Data Must Be Kept Fresh!

Understanding data decay

Published: Thursday, March 16, 2023 - 12:03

Data decay, or the aging of data, is data degradation over time—which results in bad data quality (Mahanti 2019).

There are some data, such as date of birth and place of birth, which are evergreen and not subject to decay; that is, if you have captured the data values correctly in the first place and the data are untouched, the data will not change. Time has no effect on such data. However, other data are subject to aging and decay, even if left untouched. The time factor and what triggers the decay are the variables. 

Photo by George Firican

The quote in the image above is from Cesar Augusto Lima in response to a LinkedIn post by George Firican in relation to natural decay of both lattes and data. Firican, the founder of LightsOnData and a recognized thought leader, made an interesting comment in his post on Oct. 31, 2022, on LinkedIn with a cup of latte. His statement was: “The design of the latte starts to be ruined after a while, even if left untouched. Same with certain data.”  

Data such as stock market data are extremely volatile and change every few seconds. 

Other data, like passport expiration dates, are comparatively less volatile, with an expiration ranging from five to 10 years. 

Still, in the case of other data such as contact data, decay is event driven, such as: change of address triggered by expiration of rent lease dates or due to movement to a new location for job purposes; change in telephone numbers due to transfer to a new country or due to operator changes; and so on. About 25 to 30 percent of an organization’s contact data can go bad each year under normal circumstances (Neubarth 2013). Hence, if an organization’s customer database has 12 million customer contact records, then approximately 3 million to 4 million customer contact records will be obsolete annually, resulting in significant dollar costs in terms of postage as well as missed opportunities (Mahanti 2019). 

How do you keep contact data relatively fresh? 

Contact data need regular maintenance. To ensure that data are up to date, it’s important to set guidelines for how often each field should be updated. For example, according to Reserve Bank of India guidelines, Indian banks require Know Your Customer (KYC) updates once every three years, and more frequently if a transaction has not occurred. Know Your Customer (KYC) standards are designed to protect financial institutions against fraud, corruption, money laundering, and counterterrorist financing, but because they establish customer identity, the contact details are also updated. 

Concluding thoughts 

Guidelines and processes for updating data should be defined for critical data elements in an organization. This will ensure that data are up to date, of high quality, and fit for usage. “Latte must be consumed fresh; data must be kept fresh!”

This article draws significantly from the research presented in the book Data Quality: Dimensions, Measurement, Strategy, Management, and Governance (ASQ Quality Press, 2019). Future research will focus on how to measure data quality and data quality strategy. 

About The Author

Rupa Mahanti

Rupa Mahanti is a business and information management consultant and has extensive and diversified consulting experience in different solution environments, industry sectors, and geographies (United States, United Kingdom, India, and Australia). With work experience that spans industry, academics, and research, Mahanti has guided a doctoral dissertation, published a large number of research articles, and is the author of the book Data Quality: Dimensions, Measurement, Strategy, Management and Governance (ASQ Quality Press, 2019). She is a reviewer for several international journals and publisher of “The Data Pub” newsletter on Substack.