Identity Resolution Daily

By Robert Barker, Infoglide Senior Vice President & Chief Marketing Officer

Andy Hayler recently commented on the “scale of the problem that Master Data Management (MDM) is attacking.” He specifically cited a survey of IT leaders recently conducted by Information Difference and reported on by ITPro that indicates a majority of senior IT executives believe their data quality is poor, yet many believe the sheer number of data sources they have to deal with is a big obstacle. In fact, “13 percent of companies had over 100 systems storing customer data, while 11 percent had over 100 systems storing product data.”

Imperfect data can be so daunting that it leads to paralysis. Maybe the bigger obstacle is not the number of data sources – it’s the mindset. Contrary to common belief, complete cleansing and integration of all data into a warehouse is not mandatory before you can gain value from MDM. Cleansing and combining data no longer has to be the first step in MDM. Newer technology exists that not only reaps value from un-cleansed data but has the ability to create a virtual data warehouse that allows an organization to access all its data in real time.

Identity resolution technology can search across multiple, disparate databases to find associated records without needing to merge and purge. And a sophisticated identity resolution solution can do this in real time without performance issues. Additionally, identity resolution is not “one size fits all” but supports flexible and highly configurable solutions that work with data in specific domains.

The task of cleaning and combining over 100 sources of data seems overwhelming, which is probably why those companies still have scores of data sources. Identity resolution addresses this issue directly.

Using the right technology before applying ETL processes, you can incorporate identity resolution into existing applications in order to combine similar identities, find hidden relationships, and enhance automated decision-making. Demonstrating ROI early will lead to continuing organizational support for additional MDM projects.

Share This

smart (enough) systems: How to address decision making challenges - ownership

“You want to manage the decision so that the value of the customer, their retention risk and more is considered and so that the person who REALLY made the decision was the person who works on how to handle the best customers. Only taking control of decisions, being explicit about how they are made and using automation to deliver the decision in every channel, every time is really going to work.”

Evolution of Security: ID Update and Word on the Blog

“Now that the new ID requirement is almost one week old, we wanted to provide additional stats.”

Hi-Tech-Blog: Economy Down Shoplifting Up

“The rise in retail thefts is also being blamed on the presence of organized crime, which account for up to $30 billion in losses every year, according to the FBI. These rings of professional thieves can easily move stolen items through the Internet, where consumers go to look for deals.”

Andy on Enterprise Software: MDM Fashions

“There are some good points in the DM Review article on MDM by William McKnight. In particular, I like that he clearly highlights some issues that some parts of our industry are still in denial over:”

Homeland Security Watch: Technology Task Force Presents 7 Recommendations to Chertoff

“Before he left, Chertoff charged the HSAC membership with one more task: ‘What are the ten tasks for the next Administration to take up and accomplish over its first year or two?’ It seemed odd to charge this group with something so trite. However, he explained, rightly, that it is important that efforts be made to preserve the institutional knowledge of the Department into and through its first ever Presidential transition. I’d like to know what you think should make the top ten list. Comment below.”

Hub Solution Designs: Building a Data Governance Organization

“I’m often asked how many data stewards a company needs. A rule of thumb I’ve used for quite a while is ‘one data steward per $1 billion in annual revenue’, but of course that can’t be a hard and fast rule. Depending on the company, the degree of automation, what data quality tools are available, and the major processes around the enterprise, that number is going to vary.”

The First Post: More snooping please, we love it

“Freeborn Englishmen like nothing more than the state’s jackboot on their windpipes, says David Cox”

Share This

[Post from Infoglide] The Importance of Identity Resolution to MDM

“In a recent post on the Hub Solution Designs blog, I wrote about the importance of integration to Master Data Management (MDM). Today, I’d like to dig into the importance of identity resolution to MDM.”

b-eye.com - Business Intelligence Network: Loss Prevention in the Retail Environment

“Leading retailers are using business intelligence to implement enterprise-wide loss prevention solutions that take into account the wide variety of factors that contribute to economic loss.”

DataFlux Community of Experts: Data Quality Strategy

“I was just creating a data quality strategy for a customer going through a conversion from multiple sources to one integrated database. While doing the conversion, new software will be introduced and propagated out to the organization. Not an easy task for any organization!”

Wired Danger Room: FBI Data-Mining Slashed After G-Men Dis Congress

“Earlier today, a House appropriators voted to pull $11 million to expand a controversial FBI data-mining project, after the Bureau repeatedly stiff-armed Congressmen and their gumshoes in the Government Accountability Office.”

b-eye.com - Business Intelligence Network: The Dirty Job of Data Governance

“Hence, this inaugural edition of the Jill Dyché Data Governance newsletter, courtesy of our buddies at the Business Intelligence Network (BeyeNETWORK.com). We’re encountering some new clients who’ve asked us to do cleanup duty on their failed data governance efforts, and that’s taught us a lot about how to deliver data governance, data management, data quality, master data management (MDM) and data integration initiatives the right way. Through this monthly newsletter, I’ll be sharing some of those hard-won lessons with you.”

Data Governance and Data Quality Insider: Data Quality Events – Powerful and Cozy

“For those of you who enjoy hobnobbing with the information quality community, I have a couple of recommendations for you. These events are your chance to rub elbows with different factions of the community. In the case of these events, the crowds are small but the information is powerful.”

Share This

By Dan Power, President and Founder, Hub Solution Designs

In a recent post on the Hub Solution Designs blog, I wrote about the importance of integration to Master Data Management (MDM). Today, I’d like to dig into the importance of identity resolution to MDM.

In a previous life, I worked for Dun & Bradstreet, and I had a chance to learn quite a bit about D&B’s technology for identity resolution, which it calls Entity Matching.

But the general question of identity resolution, or matching, is critical to a successful MDM initiative. There’s a good article by Roger Wolter and Kirk Haselden of Microsoft which says “the matching accuracy of MDM tools is one of the most important purchase criteria.”

I agree with that. If your goal is to build an MDM solution, with a central hub, integrated data quality, robust middleware, and useful external content, all driven by a data governance organization composed of experts from the business and from IT, then you’re going to need highly accurate matching tools.

The consequences of false matches (showing one customer record as a duplicate of another when that isn’t the case) or missed matches (not grouping two customer records together when they should be) will strongly affect the level of trust your MDM solution receives from the organization as a whole.

And depending on your industry, too many false positive or false negative matches could have serious business impacts, and in a medical or law enforcement MDM application, could even have life and death consequences.

The crucial question is “are the built-in matching tools provided by your MDM hub vendor good enough, or are you better off integrating a purpose-built identity resolution engine with your MDM hub?”

There’s one good way to answer that question - do a “bake off” or side-by-side comparison of the MDM hub with the dedicated matching tool, and see if the difference in results warrants integrating a matching tool with your MDM solution.

I know the first time that the business users of your new MDM solution find records grouped together that shouldn’t be, or find a large number of duplicates that to the human eye are obviously the same, their level of trust in your MDM hub will drop, and it may be very difficult to get it back again.

Dan Power is president of Hub Solution Designs, Inc., a consulting firm specializing in master data management and data governance. He has 20 years of experience in management consulting, enterprise applications and strategic alliances at companies such as D&B, Deloitte & Touche, Computer Sciences Corporation, eCredit and Parson Consulting. He speaks frequently at technology conferences, has written several articles for DM Review magazine, and regularly advises clients on developing and implementing high impact MDM strategies.

Share This

Evolution of Security: New ID Requirements Begin Tomorrow

“If you’re flying tomorrow, or anytime in the near future, you may want to make a note that tomorrow is the day the TSA enhances its ID requirements. There have been many misconceptions of the new requirements and I just wanted to attempt to clear things up a little bit. . . . Is this about control? No. It’s about knowing who is getting on the plane. It’s about shifting our focus towards people instead of items on a list. . . . If our goal is to keep bad people off of planes, and our law enforcement and intelligence partners have gone to the lengths of creating watch lists of known terrorists to keep them off said planes, we have to know for sure that each person who goes through matches the name on their boarding pass and is who they say they are.”

Andy on Enterprise Software: MDM Research

“13% of the (mostly larger) 112 companies surveyed had over 100 systems that hold and maintain customer data, which gives some idea of the scale of the problem that MDM is tackling. . . . The good thing was that plenty of companies seem to have begin measuring the costs of poor master data, and those costs are high, which should make it easier to justify master data management initiatives.”

NRF Loss Prevention Convention & Expo: Day One - General Session: The State of the Loss Prevention Industry

“Sheriff Judd and his team recently broke up an organized retail theft ring the stole an estimated $100 million worth of retail goods. . . . Sheriff Judd pointed out that many agencies think the crime ends with the single arrest. That’s not always the case, that simple shoplift may be organized retail theft. There needs to be a paradigm shift within law enforcement. Retail theft needs to be taken more seriously. Stopping retail theft can impact other crime. Your minor thieves are your major thieves. Thieves are good at what they do, it’s their industry.”

CQ.com: We Rage, Europeans Yawn, Over Domestic Counterterrorism Ops

“Ever since I attended a conference on homeland security in Paris four years ago, I’ve been fascinated by how little the French, Italians, Germans and other continentals worry about violations of their civil rights by their spy agencies. In fact, outside the United Kingdom, which invented civil liberties with the Magna Carta 993 years ago last Sunday, ordinary Europeans couldn’t care less about wiretapping, national ID cards, preventive detention and police spies in mosques, all of which have millions of Americans, not to mention the ACLU and libertarian Rep. Ron Paul, R-Texas, up in arms. . . . It’s likely because Germans, like the French and Italians, ran out of patience with domestic terror groups years ago. . . . ‘There is a more highly developed sense [in the United States] of what freedom means,’ he said.”

Business Insurance: Risk management not yet boardroom issue: Survey

“Risk management is not making its way into most of the boardrooms at Europe’s leading companies, a recent survey reveals. . . . Quantifying risk and demonstrating value were the biggest risk management challenges cited by three-quarters of the respondents, according to the survey.”

Share This

[Post from Infoglide] Identity Resolution Daily - One Year Later . . . or Thereabouts

“Well, here we are feeling like the stereotypical bad husband. We can’t believe we missed our first year anniversary. I guess a couple dozen long stemmed roses are in order.”

b-eye.com - Business Intelligence Network: Master Data Management Readiness

“The selection and purchase of a core master data management system should be the culmination of a series of preparatory steps that are described in this article.”

newsday.com: NYPD arrests 15 suspected in car insurance fraud

“The 13 people were arrested Tuesday on insurance fraud and grand larceny charges stemming from at least five wrecks. Two suspects are still at large. Police say the ring targeted car insurance companies, taking advantage of a state law that entitles people involved in a crash to $50,000 worth of medical benefits. . . . Police are investigating whether the clinics were in on the scam.”

DataFlux Community of Experts: CIO Forum

“This time, I am on my way to be a panel member at a local CIO Forum. . . . Data integration, and data quality issues are going to be on the top of the list for this discussion.”

supermarket.co.za: ‘E-Fencing’ Is Growing Wave In Organized Crime

“These losses are clearly affecting retailers’ bottom lines. According to the survey, organized crime accounts for approximately $30 billion in retail losses every year. As a result, retailers are also spending more to ward off these detrimental incidents. In fact, the average retailer spends approximately $230,000 per year on labor costs, the study said. This price tag can hit $1 million a year at larger chains that are constant organized retail crime (ORC) targets.”

Data Governance and Data Quality Insider: Get Smart about Your Data Quality Projects

“With all due respect to Agent Maxwell Smart, there is a mini battle between good and evil, CONTROL and KAOS, happening in many busy, fast-growing corporations. It is, of course, with information quality. Faster growing companies are more vulnerable to chaos because by opening up new national and international divisions, expanding through acquisition, manufacturing offshore, and doing all the other things that an aggressive company does, it leads to more misalignment and more chaotic data. While a company may have a strong desire to ‘own the world’ or at least their market, they may wind up owning chaos and disorder instead - in the form of disparate data.”

Share This

Homeland Security Watch: Homeland Security & Technology Panel Event

“Greg Nojeim, Director of the Project on Freedom, Security, and Technology at the Center for Democracy and Technology offered insightful warnings about the unintended consequences of technology when it is not developed or deployed with privacy protections at the initial stages.”

b-eye.com - Business Intelligence Network - Blog: Jill Dyche: At Night She Sells Oranges: Sarah Mclachlan Eats Our Lunch

“You may have seen Baseline’s latest YouTube video, one of several in our ongoing Get in the Game series. To film it we took advantage of the actor’s strike in L.A. and ended up producing several videos on the cheap. We were pretty proud of our opportunism, which let us deliver a funny and tongue-in-cheek send-up of master data management. The video illustrates how customer identity resolution —a key part of the MDM stack—is a problem that can often be found in your own backyard, as it were.”

Fraud, Phishing and Financial Misdeeds: NRF Survey shows Organized Retail Crime activity is growing!

“When I stated that this activity hurts all of us, the reason is that retailers have to make up the $30 billion they are losing to this activity somewhere. This normally equates to higher prices, or in extreme circumstances (especially in tight economic times) cutting payroll. Simply stated, people might be losing their jobs because of this activity.”

Andy on Enterprise Software: Data Quality stories

“The most entertaining thing I saw was a presentation by a consultancy firm on some real-life data quality situations they had encountered at clients, showing just how tricky data quality algorithms can be.”

National Retail Federation: The Most Famous Sheriff You Never Heard Of: Sheriff Grady Judd of Florida’s Polk County Sheriff’s Office will speak about high-profile cases of organized retail crime

“Sheriff Judd and I have something in common—we truly believe in the value of educating the public about organized retail crime. During Sheriff Judd’s keynote at the opening session of the LP show, he will address the importance of partnering with local retailers and others in the private sector. He will also share his story about uncovering the organized retail crime ring and about how his team worked with local retailers to track the criminals.”

[Note: While not exactly on topic, we felt that anyone who reads or writes blogs should be aware of the following story.]
Concurring Opinions: The Associated Press, Copyright, and the Blogosphere

“The Associated Press, one of the nation’s largest news organizations, said that it will, for the first time, attempt to define clear standards as to how much of its articles and broadcasts bloggers and Web sites can excerpt without infringing on The A.P.’s copyright.”

Share This

[Post from Infoglide] Integrating Identity Resolution into MDM Solutions: Real-Time or Batch?

“Dan Power at Hub Solution Designs recently challenged ‘MDM project team leaders, program managers, and data champions’ to think about the importance of integration. In particular, he asks about MDM integration in three contexts: real-time, business modeling, and two-way hub communication.”

Jeff Jonas: Smart Systems Flip-Flop

“Certainty often shifts with observations over time. And this is good.”

DataFlux Community of Experts: Integration in a Big Way!

“I am so excited! I get to work on an integration project that is huge. I mean really, really big — and it affects an entire organization. . . . After the project plan is solidified, roles and responsibilities are identified. Now a role is not necessarily a person! For example, there could be a data architect, business architect, technical architect, and a business intelligence architect role. Are you confused? I was, until we dug into the responsibilities of each role.”

Evolution of Security: Why is ID Important for Security?

“Last week we announced on our Web site a plan to begin REQUIRING ID from travelers on June 21st. This plan includes enabling our officers to refuse entry into the area beyond the security checkpoint anyone that does not cooperate with us to establish his or her identity. The exclusive reason to do this is to ensure people are who they say they are and are not gaming the system by using a boarding pass in a fake name; a well-known endeavor of professionals and college kids alike that could potentially circumvent the no-fly list.”

b-eye.com - Business Intelligence Network: An Update on Knowledge Management in the Federal Government

“A survey of the attendees of this year’s Knowledge Management Conference reveals that knowledge management in the federal government is alive and well.”

Share This

By Robert Barker, Infoglide Senior Vice President & Chief Marketing Officer

Dan Power at Hub Solution Designs recently challenged “MDM project team leaders, program managers, and data champions” to think about the importance of integration. In particular, he asks about MDM integration in three contexts: real-time, business modeling, and two-way hub communication.

Additionally, Jill Dyché recently pointed out that Identity Resolution supports and enhances five of eight core MDM functions outlined in her book on CDI, and we agree wholeheartedly. Given that Identity Resolution (IR) is integral to MDM, we can ask the first of Dan’s integration question: how should IR be integrated with MDM? More specifically, should this integration be batch or real-time?

The answer is, of course, it depends. Instead of asking “what will we be giving up by not providing for any real-time capabilities?” maybe we should ask “is the value of the MDM application sufficiently enhanced to justify the incremental infrastructure cost of using real-time data sources?” Sometimes the answer is yes, and sometimes it’s no.

Let’s consider two examples in the area of fraud detection. In a bank or retail setting, discovering a hidden relationship between two fraudulent people and keeping a fraudulent transaction from being consummated may save thousands of dollars in apprehension costs over catching it a day later, so a real-time approach is easily justified. On the other hand, in a case like insider securities trading, the acts are not preventable and the data connecting someone “in the know” with the person who makes a transaction based on a tip persists forever and can be caught and prosecuted months after it occurs.

Regardless of whether real-time or batch integration is selected, having multiple modes of integration available to match project requirements is important. In some cases, simple point-to-point XML data exchange may be enough, but in others it can be imperative that IR decisions be smoothly integrated into existing business processes. That may require using BPM, queuing, web services and other broadly supported methods of integration, again depending on the specific application context.

The takeaway: it’s essential to include multiple integration capabilities among the criteria you use in evaluating Identity Resolution technology.

Share This

[Post from Infoglide] That’s the Ticket: Addressing a Different Kind of Lottery Fraud

“Type ‘lottery fraud’ into Google, and you get lots of returns warning about the dangers of Nigerian lottery scams and other phishing attacks on unsuspecting, and/or perhaps overly optimistic, consumers. But there are less well known types of lottery fraud also being perpetrated, and some of them hurt consumers as well.”

b-eye.com - Business Intelligence Network - Blog: Jill Dyche: Doublin’ Down at the Data Governance Conference

“In a prior blog I bemoaned having to miss Informatica World in Vegas this week. (Regular readers of the Jillblog know that what happens in Vegas gets blogged about almost immediately.) But I feel much better now, since the Wilshire Data Governance Conference in San Francisco has already exceeded my expectations.”

Washington Business Journal: Retail survey reveals growing theft problems

“Retailers are battling an increase of organized retail crime, according to the National Retail Federation’s (NRF) 2008 Organized Retail Crime report published Wednesday.”

Informatica: Data Quality - Ensuring Trust in Your Data

“TDWI once published a study that poor data quality was costing companies an estimated $600B annually in the US alone. And yet, industry analysts estimate the data quality tools market to be between $300-400M annually. The demand for better data quality is clearly there, but the size of the data quality tools market does not appear to reflect that same demand. Could it be that traditional Data Quality tools are not delivering enough value for customers?”

PogoWasRight.org: Few Canadians are ‘very confident’ their personal data is safe with retailers, banks and governments

“Of the three types of organizations, Canadian retailers fared the worst, with less than one per cent (O.5 per cent) of consumers saying they are very confident retailers can protect their customers’ on-line personal and private information.”

Andy on Enterprise Software: The dust clears (a bit)

“I have wondered for some time about Informatica’s intentions in the MDM space. There had been some market rumours about them possibly buying Siperian, but it seems as if their not so secret meetings with Siperian were actually to form a partnership, which was just announced.”

Share This