Data streams are data sent from one point in a system to another point in a system through a connection. They are crucial to ensuring that the data from the source is accessed correctly and can ultimately deliver the insights needed.
Unfortunately, it still happens regularly that problems occur with data flows. Data streams may fail unexpectedly. There is also the possibility of a situation where data streams contain only part of the data without you being aware of it. Of course, you want to proactively gain insight into data flows and not be told afterwards by customers or users that it is not good. It is therefore important to have a good understanding of the data flows present in your organization, and whether they are still “healthy.
At Hendrikx ITC we work with many (complex) data streams on a daily basis and regularly discover these types of problems with our clients. Fortunately, there is always a solution and we are happy to share with you more information about causes of data flow problems and how to prevent them.
What can cause data flow problems?
- Problems at the source
Basic data may have to come from outside, and for whatever reason, may not come in or may come in too late. It may also be that the data is coming in, but something has changed in the format of the data. For example, at Hendrikx ITC we recently had the case where data that previously came in as separate files was merged and delivered in a tar archive.
2. Internal problems
Internally, changing the format of data can also easily occur. A seemingly innocuous change, such as changing the data type of a column in a database, can interfere with further processing, and lead to empty data streams or processes that fail completely.
Monitoring data streams
In order to solve problems with data flows, it is first of all necessary to make these problems understandable. At Hendrikx ITC, we use Prometheus and Grafana for this purpose. These are open-source solutions. Prometheus monitors metrics of our system, including the presence and completeness of data flows, both inbound, internal, and outbound. Grafana makes these metrics understandable in charts in a simple yet powerful way. It also creates alerts when the metrics exceed certain thresholds, for example, minimum or maximum values.
To see if there are any problems, the data stream must be viewed at the appropriate point. It goes wrong, for example, if one looks at whether the data to be sent is present, but not whether it has actually been sent. If the problems are in the shipping process, for example, because that process does not have permissions on the database table it needs to read, then they will not be found. This can be solved by having the process that writes the metrics to Prometheus be a subprocess of the process that sends the data. Of course, it is also possible to be a little smarter about it. You can detect deviation (deviation) through the subprocess. Smart triggers can then be set here.
Want to learn more about understanding systems and networks using metrics?
On November 25th, 2021 11:00 am CET we will discuss smart applications of metrics in addition to the basics with a webinar. Registration is free and can be done by clicking here.