Future Time-series Architecture
Summary
Over the past 6 years of experimenting with different designs for time-series data, and the more recent designs for integrating time-series data into GraphQL and the Rhize Data Hub, our ability to reason about the different types of data and requirements has evolved.
This post tries to capture that understanding as the basis for discussion on what the future of time-series should look like.
Retrospective
Motivation
What use-cases are time-series use-cases? What makes time-series different?
- We are mostly taking about machine data when we talk about time-series data. The Rhize Data Hub allows you to connect to external streaming data sources like OPC-UA servers or MQTT Brokers, and allows you to ābindā topics from those external data sources to equipment properties in your equipment model.
- This is traditionally the role of the āProcess Historianā and is primarily concerned with Observed data. I.e. the value of a sensor was observed at a point in time. That observation has a time-stamp, and a set of observations across time creates a time-series.
Time Series use-cases often include the need to Infer context for the observation. for example:
- What type of equipment is the sensor attached to?
- What product is the equipment producing at the time of the observation?
- Which shift is operating the equipment at the time of the observation?
Time Series use-cases also include the need to derive information from the observations. Examples include:
- Calculating the volume of liquid in a vessel based on the observation of a level sensor
- Aggregating the total amount of time that an observed value stayed within a specified range
- Calculating the % of waste over a particular time interval by considering the total production and the total rejects
- Determining the time lost due to running at less than the desired speed by calculating the area between to observations over time
Some of these calculations only need to be determined at the time of running a query, but others may need to be used in the generation of events. (see RFC - Design Doc - Event Publishing )
Where there is a need to persist Derived or Inferred information, complications can arise when the underlying information changes. This can and does happen in scenarios like:
- A calcuation changes which invalidates the previously persisted results
- Context information changes historically: An example would be where the Production Order details for a machine where updated for a previously run order.
- Data arrives late: This often occurs with lab test results which can arrive some days or weeks after production has occurred. Can also occur if source systems are not capable of publishing real-time events, and instead publish information in batches.
- late arriving data can cause derived or inferred information to be invalidated.
Stateful calculations can also be required for some use-cases, particularly those relating to āBottom-Upā MES applications that infer the state of the process by monitoring the state of the equipment against defined rules.
A stateful calculation refers to a computational process or algorithm that maintains and updates internal state information across multiple invocations or iterations. Unlike stateless calculations, where each computation is independent and produces an output based solely on the input parameters, stateful calculations rely on the accumulation and retention of intermediate state between computations.
Examples of stateful calculations include:
- Running Totals:
- Calculating the cumulative sum of a sequence of values over time. The stateful calculation maintains the current total as it processes each input value, updating the total with each new value.
- Moving Averages:
- Computing the average of a sliding window of values from a data stream. The stateful calculation maintains a buffer of recent data points and updates the average as new values are added to the window while removing older values.
- Iterative Algorithms:
- Implementing algorithms that require multiple iterations to converge on a solution, such as optimization algorithms or machine learning algorithms like gradient descent. The stateful calculation maintains the current state of the computation between iterations, updating parameters or variables based on the results of each iteration.
- Session Management:
- Tracking the state of user sessions in web applications or distributed systems. The stateful calculation maintains session information, such as user authentication status, session duration, and activity logs, across multiple HTTP requests or interactions.
- State Machines:
- Modeling stateful systems or processes using state machines, where the system transitions between different states based on input events or conditions. The stateful calculation maintains the current state of the system and transitions between states as events occur.
Historical changes can invalidate the accumulation and retention of intermediate state in stateful calculations causing them to become invalid.
In Transactional source applications, data integrity structures and data validation techniques are used to ensure that errors are generated when invalid scenarios occur. Transactions only move forward in time, and even when historical adjustments are required, like a journal entry in a financial ledger, that adjustment is applied at the current date, and marked with an effective data in the past. History is never altered.
In a DataHub that must handle scenarios of late arriving data, and historical updates, the need for solutions that handle Back-Filling and Re-Statement arise. These solutions are inherently complex.
So what should be considered when historical changes happen?
- How can we determine the impact of a historical change to data?
Not all changes will invalidate the current state. Consider a moving average. If the change is prior to the moving average window, the current Moving Average value will not be effected. Only the values in the window of the historical value change will be impacted. - Should the historical change cause new historical āEventsā to be published?
In a real-time, event-driven architecture, it is very important to separate historical events from current events. Imagine the confusion if events occurring as the result of a back-filling operations were not handled separately to currently occurring events.
