Unlocking data means making data from a source system available so other systems, reports or AI applications can work with it. It is needed because data is often locked in software that was never built to share. There are four routes: through an API, directly on the database, through export files or through a middleware layer.
What exactly does unlocking data mean?
Unlocking data means making data from a source system accessible to other applications. Think of order lines from an ERP that need to appear in a customer portal, or readings from a machine that end up in a report.
Unlocking is the first step, not the whole journey. What you do with the data afterwards, such as combining, synchronising or enriching it, is integration. The distinction matters because the problems differ. Unlocking is about access and load on the source system, integration about meaning and consistency.
Why is data locked in a source system?
Data is locked because the source system was built for its own task, not for sharing. An accounting package has to do accounting well. Whether it could also pass that data on neatly was rarely a criterion when it was bought.
Practical barriers come on top of that. The system has no API, or an API that returns only part of the data. The vendor charges extra for access. The data model is so complex that nobody knows which table holds what. And with older systems, knowledge of how they were built sometimes left with the original developer.
Which methods are there to unlock data?
There are four methods, each with its own trade-off between speed, reliability and load on the source system.
Through the source system's API.
An API is the agreed entry point through which software requests and writes back data. Most modern packages offer a REST API. This is the cleanest route: the vendor decides what is available and keeps the API stable across updates. Watch out for authorisation and rate limiting, the maximum number of requests per time unit.
Directly on the database.
If the system has no usable API, you can read the underlying database directly. That gives access to everything, but you depend on a data model the vendor can change with an update. Preferably read from a copy of the database rather than from the production environment.
Through export and import files.
The system periodically prepares a file, for example a CSV file on a secure server, which another system reads in. This route is simple and works almost everywhere. The downside is that the data is only as recent as the last export, and an export that stops often goes unnoticed.
Through a middleware layer or data platform.
With several sources, a layer in between that handles unlocking centrally pays off. It retrieves data, stages it and translates it into a shared data model. This is called ETL (extract, transform, load), or ELT when the transformation happens after loading. Such middleware as an intermediate layer prevents every system from needing its own connection to every other system. In a Data Management Platform that approach comes together with storing and managing the data.
How do you choose between real-time and batch?
Choose based on how old the data may be at the moment someone makes a decision on it. A stock level in the web store has to be current, a monthly report does not.
Real-time unlocking works with webhooks, where the source system sends a notification as soon as something changes, or with polling, where you ask at short intervals whether anything is new. Batch processing retrieves data in one go, for example overnight. Batch is simpler and loads the source system at a quiet moment. Real-time is faster but asks more of the set-up and the monitoring.
What goes wrong in practice when unlocking data?
The most common problem is that the source system cannot handle the extra read load. An ERP processing orders during the day slows down if a full read runs every minute at the same time. So retrieve only the changes, or read from a copy.
The second problem is silence. An export that stops or an API key that expires produces no error in the receiving system, just data that is no longer updated. Without monitoring, you only notice when someone asks why the figures are from last week.
The third problem is permissions. In the source system an employee only sees their own department. After unlocking, everything sits in one report anyone can open. Include authorisation in the design, not as an afterthought.
Unlocking data from legacy systems.
Legacy systems are older systems that are still critical to operations but are no longer actively developed. They often have no API and run on an outdated database. Even so, there are three routes.
You can read the database directly, with the risks that come with it. You can build your own API layer on top of the system, so other applications work through a stable entry point and the old system can later be replaced without the rest noticing. And if the system itself is the problem, you end up at replatforming outdated software.
From unlocked data to usable data.
Unlocked data is not yet usable data. Data from different sources often uses different codes, different spellings and different definitions of the same concept. An active account means something different in the CRM than in the ERP.
That is why unlocking is almost always followed by a step of translating and checking. Which data appears twice and where the differences are shows how deep the data silos in your landscape run, and what state your data quality is in. Bringing sources together for good is the domain of data integration.
In short.
Unlocking data means making data from a source system available to other applications. The four routes are an API, direct database access, export files and a middleware layer. The choice between real-time and batch depends on how current the data needs to be. Watch the load on the source system, monitoring and permissions, because that is where it goes wrong most often.
Is your data locked in a source system? Schedule a no-obligation call.
Worth reading next...
What is Single Source of Truth (SSOT?
Learn what SSOT means and how to implement it for reliable data integration in your organization
What is a data silo?
How data silos form, how to recognise them and when to break them down.
What is data quality?
The six dimensions, how to measure them and how to improve data quality for good.
Questions about unlocking data? No problem.
Unlocking data makes data from a source system accessible. Data integration is what comes next: combining, translating and keeping data from several sources in sync.
Yes. You can read the database directly, work with export files or build your own API layer on top of the system. Each route has its own trade-off between simplicity and reliability.
It can, especially with full reads during office hours. Retrieve only changes, schedule heavy processing at quiet times or read from a copy of the database.
As often as the decision based on it requires. A stock level in a web store has to be current, a monthly report can make do with overnight processing.
That depends on the route and the state of the source system. A system with a good API is unlocked faster than a legacy system without documentation. An estimate takes a short exploration.