The Science and Information (SAI) Organization
  • Home
  • About Us
  • Journals
  • Conferences
  • Contact Us

Publication Links

  • IJACSA
  • Author Guidelines
  • Publication Policies
  • Metadata Harvesting (OAI2)
  • Digital Archiving Policy

IJACSA

  • About the Journal
  • Call for Papers
  • Author Guidelines
  • Submit your Paper
  • Current Issue
  • Archives
  • Indexing
  • Editors
  • Reviewers
  • Apply as a Reviewer

IJARAI

  • About the Journal
  • Archives
  • Indexing & Archiving
  • Editorial Board

Special Issues

  • Home
  • Archives
  • Proposals
  • Guest Editors

Computing Conference 2021

  • Home
  • Call for Papers
  • Submit your Paper/Poster
  • Register
  • Venue
  • Contact

Intelligent Systems Conference (IntelliSys) 2021

  • Home
  • Call for Papers
  • Submit your Paper/Poster
  • Register
  • Venue
  • Contact

Future Technologies Conference (FTC) 2021

  • Home
  • Call for Papers
  • Submit your Paper/Poster
  • Register
  • Venue
  • Contact

Future of Information and Communication Conference (FICC) 2021

  • Home
  • Call for Papers
  • Submit your Paper/Poster
  • Register
  • Venue
  • Contact
  • Home
  • Call for Papers
  • Guidelines
  • Submit your Paper
  • Current Issue
  • Archives
  • Indexing
  • Editors
  • Reviewers
  • Subscribe

Article Details

Copyright Statement: This is an open access article licensed under a Creative Commons Attribution 4.0 International License, which permits unrestricted use, distribution, and reproduction in any medium, even commercially as long as the original work is properly cited.

An Iterative, Self-Assessing Entity Resolution System: First Steps toward a Data Washing Machine

Author 1: John R. Talburt
Author 2: Awaad K. Al Sarkhi
Author 3: Daniel Pullen
Author 4: Leon Claassens
Author 5: Richard Wang

Download PDF

Digital Object Identifier (DOI) : 10.14569/IJACSA.2020.0111279

Article Published in International Journal of Advanced Computer Science and Applications(IJACSA), Volume 11 Issue 12, 2020.

  • Abstract and Keywords
  • How to Cite this Article
  • {} BibTeX Source

Abstract: Data curation is the process of acquiring multiple sources of data, assessing and improving data quality, standardizing, and integrating the data into a usable information product, and eventually disposing of the data. The research describes the building of a proof-of-concept for an unsupervised data curation process addressing a basic form of data cleansing in the form of identifying redundant records through entity resolution and spelling corrections. The novelty of the approach is to use ER as the first step using an unsupervised blocking and stop word scheme based on token frequency. A scoring matrix is used for linking unstandardized references, and an unsupervised process for evaluating linking results based on cluster entropy. The ER process is iterative, and in each iteration, the match threshold is increased. The prototype was tested on 18 fully-annotated test samples of primarily synthetic person data varied in two different ways, good data quality versus poor data quality, and a single record layout versus two different record layouts. In samples with good data quality and using both single and mixed layouts, the final clusters had an average F-measure of 0.91, precision of 0.96, and recall of 0.87 outcomes comparable to results from a supervised ER process. In samples with poor data quality whether mixed or single layout, the average F-measure was 0.78, precision 0.74, and recall 0.83 showing that data quality assessment and improvement is still a critical component of successful data curation. The results demonstrate the feasibility of building an unsupervised ER engine to support data integration for good quality references while avoiding the time and effort to standardize reference sources to a common layout, design, and test matching rules, design blocking keys, or test blocking alignment. Also, the paper proposes how unsupervised data quality improvement processes could also be incorporated into the design allowing the model to address an even broader range of data curation applications.

Keywords: Unsupervised entity resolution; data curation; frequency blocking; entropy regulated; data washing machine

John R. Talburt, Awaad K. Al Sarkhi, Daniel Pullen, Leon Claassens and Richard Wang, “An Iterative, Self-Assessing Entity Resolution System: First Steps toward a Data Washing Machine” International Journal of Advanced Computer Science and Applications(IJACSA), 11(12), 2020. http://dx.doi.org/10.14569/IJACSA.2020.0111279

@article{Talburt2020,
title = {An Iterative, Self-Assessing Entity Resolution System: First Steps toward a Data Washing Machine},
journal = {International Journal of Advanced Computer Science and Applications},
doi = {10.14569/IJACSA.2020.0111279},
url = {http://dx.doi.org/10.14569/IJACSA.2020.0111279},
year = {2020},
publisher = {The Science and Information Organization},
volume = {11},
number = {12},
author = {John R. Talburt and Awaad K. Al Sarkhi and Daniel Pullen and Leon Claassens and Richard Wang}
}


IJACSA

Upcoming Conferences

Future of Information and Communication Conference (FICC) 2021

29-30 April 2021

  • Virtual

Computing Conference 2021

15-16 July 2021

  • London, United Kingdom

IntelliSys 2021

2-3 September 2021

  • Amsterdam, The Netherlands

Future Technologies Conference (FTC) 2021

28-39 October 2021

  • Vancouver, Canada
The Science and Information (SAI) Organization
BACK TO TOP

Computer Science Journal

  • About the Journal
  • Call for Papers
  • Submit Your Paper
  • Indexing

Our Conferences

  • Computing Conference
  • Intelligent Systems Conference
  • Future Technologies Conference
  • Communication Conference

Help & Support

  • Contact Us
  • FAQ's
  • Terms and Conditions
  • Privacy Policy

© 2018 The Science and Information (SAI) Organization Limited. Registered in England and Wales. Company Number 8933205. All rights reserved. thesai.org