Repository navigation
Define a module for windowing streaming data #113
Description
What's needed?
We need to access a window of data that is received from the LogicalMeter and can be used by downstream users, usually trained machine learning models.
This should use a ring buffer as an underlying data structure
The anticipated feature are
- updates the data regularly
- can access data by time windows
- can create an ordered copy of a slice of the ring buffer as numpy array
- OPTIONAL: can return a resampled ordered copy of a slice of the ring buffer
The last feature is marked as optional because it's possible to create two windows with the same data with different sampling rates without that feature.
Additional Features are
- aggregations/operations on windows like mean and standard deviation shared with the TimeseriesAvg (for further ideas see also
Proposed solution
Essentially this is the user facing interface to the ring buffer taking care of dealing with all kinds time related and providing bounded and ordered data batches to the user when requested.
Furthermore it's supposed the update the ring buffer on new incoming samples from the LogicalMeter.
Use cases
This will be used in the forecasting pipeline as the central data storage for
- storing time series data that can be piped into ml predictions,
- calculating features as
- daily averages
- energy consumption/production in a certain time window
- mean and standard deviation
- polynomial coefficients (for power curves)
Alternatives and workarounds
This could, instead of being a wrapper around, as well directly end up in the ring buffer.
In the past pandas dataframes have been used. It turned out as too inefficient due to
- memory allocation when operating (e.g. (joining, resampling,...),
- creation of new dataframes on new incoming data.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Projects
- Status
Activity
matthias-wende-frequenz commentedon Dec 19, 2022
This data structure could furthermore take care of loading historic data, as an alternative approach to the seamless transition between disk data and streaming data. This could be simplified if we would cache transformed logical meter data on disk.
[-]Define a data structure for storing data batches[/-][+]Define an actor for storing data batches[/+][-]Define an actor for storing data batches[/-][+]Define a module for storing data batches[/+]ela-kotulska-frequenz commentedon Dec 20, 2022
How frequently would the mean be updated and send?
matthias-wende-frequenz commentedon Dec 21, 2022
Does this question belong to this issue? This module is just about batching data from a stream.
matthias-wende-frequenz commentedon Dec 21, 2022
For resampling we might consider using the python wrapper around libsamplerate
[-]Define a module for storing data batches[/-][+]Define a module for buffering streaming data[/+]25 remaining items
ela-kotulska-frequenz commentedon Jan 9, 2023
I am wondering how the data would be returned to the user.
If I understand correctly it will copy the ringbuffer (to not override it by mistake).
If yes, then for example:
Consider actor that every 5 seconds needs last 1 hour data resampled every 1 sec.
Would it copy whole 1 hour array every 5 seconds?
Or would it return only the latest 5 seconds data to be appended to the previously received data?
matthias-wende-frequenz commentedon Jan 10, 2023
That's not needed as mentioned here.
matthias-wende-frequenz commentedon Jan 10, 2023
That's indeed a very good question.
The ringbuffer is designed to copy only if
As we try to avoid coping as much as possible I think we should inherit from the default ring buffer behavior.
idlir-shkurti-frequenz commentedon Jan 10, 2023
Just to make sure I understand this properly. It seems to me that the second example is a bit more general if we need the classic sliding window functionality. I understand that if we want these windows to represent things like days or weeks then the windows will not overlap but isn't it better to generalise this by applying the second example?
idlir-shkurti-frequenz commentedon Jan 10, 2023
By copy do you mean load the entire window again instead of appending the ringbuffer? If so then I agree with the bullet points.
mathias-baumann-frequenz commentedon Jan 23, 2023
There is no "appending", only overwriting already allocated memory and possibly old out-of-date data.
You can request any datetime range from the ringbuffer
buffer.window(start, end)and the returned array/list will be a view to the actual raw buffer except for the cases @matthias-wende-frequenz outlined above.In those cases it will be a newly allocated buffer.
thomas-nicolai-frequenz commentedon Jan 24, 2023
@idlir-shkurti-frequenz this is the main reason for the ring buffer. its has a fixed sizes and thus there is no such things as appending. appending always means new memory allocation and that in turn may invoke the garbage collector.
Please try to use that feature carefully. There has to be a really good reason using that.
matthias-wende-frequenz commentedon Jan 25, 2023
There is no overlap as it's just one window that constantly moves with the data flow. So it'll always have the latest data and with each new incoming data point the oldest data point is removed.
There has been a confusion between the
TimeseriesAvgand theMovingWindow, that is sketched here so please forget the two examples Tiyash gave.matthias-wende-frequenz commentedon Jan 25, 2023
The underlying ringbuffer already minimizes coping to the cases described above.
Additionally I'll add examples howto arrange the buffer when initializing the
MovingWindowto further minimize the likelihood of copying.