Single Source of Truth in Data Management
Understand the importance of data consistency in automation projects and learn how to build a Single Source of Truth (SSoT) architecture.
Happy Automation Friday!
In today's newsletter, I want to address a problem. It's not the main focus of this newsletter, but it greatly affects automation: data consistency.
The Context
Today’s data environments are quite complex.
With all the integrations, it's easy to duplicate data everywhere. It's easier than ever to keep a record in many places.
This is a common mistake for anyone starting an automation project.
You can easily integrate many sources and duplicate parts of the data into various places without even knowing it.
I won't discuss the legal issues, including GDPR implications. This includes when users want to unsubscribe, but you keep their data.
Instead, I want to talk about the complexity of the problem that results from this action.
The Problem
A common example is when you want to sync contacts from the CRM to the email marketing list.
If the sync is one-way (for example, sending data from the CRM to the email solution), that data will only be consistent for a short time. It will change once a user tries to unsubscribe from the list.
From that moment on, your “customer” lists will have two different sources of truth, depending on whom you ask.
This situation occurs often.
Below is an example. It has a small integration scheme. It includes email marketing, transactional emails, and invoicing solutions.
Of course, in a small or medium-sized company, you might have a few dozen integrations like this one.
At first, it may seem logical to automate sending email addresses to an external email marketing tool, like Mailchimp.
But in automation, more than in other domains, things can easily go wrong.
In the first few hundred interactions, you could have different data everywhere. Some could be bounced emails from the transactional platform. Some could be updated invoices from the invoicing solution. And some could be unsubscribed emails from the email marketing solution.
If you don’t integrate the solutions both ways, that data will stay there. Forever.
The Solution
You must think about what kind of data you want to return to the main source of truth.
The goal should be to create an SSoT architecture (Single Source of Truth).
Map every aspect available on each platform that could be returned to the main database:
Invoicing solution: the final amount and due date.
Email marketing platform: unsubscribes, customer ratings, and perhaps email marketing segments.
Transactional email platform: hard bounce emails.
This way of making a data architecture is more expensive. It costs more in discovery, analysis, and implementation. But it has long-term benefits, like:
Reduced errors
Reduced client friction
Enhanced data consistency
Enhanced security
Faster development times for future integrations
The Conclusion
As a Product Manager, invest time in the discovery phase before integration and automation.
Understand how data flows in and out of the systems you want to integrate. Don’t skip this part. Integrate what you understand.
Don’t overlook inconsistency bugs—treat them as real problems and investigate where they come from.F
Focus on creating an SSoT architecture rather than integrating solutions.