Another great post and on point! I find it very interesting linking “point in time” occupancies to the current state location of an entity. Public records, while fruitful, are spotty in availability and lack many standard data quality measures. Name distributions per a given geography (zip or zip+4) are helping in making links between names with materially different addresses — Zawarek Timonsky 123 Main St and Zawarek Timonsky 456 Elm Dr in same zip code where only one Zawarek first name is known and 3 Timonsky surnames known … the unique combination creates a high degree of confidence we are talking same person — even with differing addresses.

As for the example of St. in the street not always meaning Street, it is clear that the software causing the incorrect classification and standardization is not looking at both the keyword AND the pattern or semantics in which the keyword or phrase is referenced. This type of semantic parsing and standardization is gaining traction in document classification and phrase searching (aka Google).

Keep up the thought provoking articles!

Steve

Great post. You may recall the slides I’ve used at IAIDQ conferences about my name and how it got me into information quality at an early age.

13+ spelling variants, can be male/female, can be miskeyed as Tara, or mangled to be Darren, Daryn, Daryl (also a male/female name), Dora (hence my love of exploring). And let’s not get started on my home address as a kid which seems to still confuse data quality tools (here’s a hint… St. in an address is not always an abbreviation of “street”). I have other examples…

I think one of the mental gear-shifts that needs to be made when looking at these issues is to remember that data is a representation of a real world thing (in this case a person). It is not the thing itself. When we are elbows deep in the data it can be all too easy to loose sight of that.

Looking forward to the follow ups to this.

Topics covered in the series:

• Why a symbiosis of technology and methodology is necessary when approaching the common data quality problem of identifying duplicate customers
• How performing a preliminary analysis on a representative sample of real project data prepares effective examples for discussion
• Why using a detailed, interrogative analysis of those examples is imperative for defining your business rules
• How both false negatives and false positives illustrate the highly subjective nature of this problem
• How to document your business rules for identifying duplicate customers
• How to set realistic expectations about application development
• How to foster a collaboration of the business and technical teams throughout the entire project
• How to consolidate identified duplicates by creating a “best of breed” representative record

Here is the link to article series on my blog:

http://www.ocdqblog.com/home/identifying-duplicate-customers.html

Best Regards…

Jim Harris

The “deliberate attempt to conceal a connection” seems to be common in fraud, law enforcement and homeland security applications.

It’s important to remind people about one of your main points (matching being a necessary but not sufficient part of entity resolution).

There’s a lot more going on in true entity resolution than simple matching!