Add a PeriodicFeatureExtractor

What's needed?

For forecasting we have been using average load and production profiles.

Given a time series, we want a module that is able to calculate the item wise average of repeating time windows of the same size and the same sample-rate, possibly all having the same distance.

Example:

Consider the time series $T = [1,2,3,4,5,6,7,8,9,1]$ and a window a length $2$ with distance $2$ starting at the first position.

Hence we get the sub-series $t_1=[1,2], t_2=[5,6], t_3=[9,1]$ and therefore the average $$t_{avg}=[(1+5+9)/3, (2+6+1)/3] = [5, 3]$$

Note: A position could likewise be a timestamp in a usual time-series representation.

Proposed solution

We want to design such a module in a way such that it

uses batches of data (most likely cached in memory)
is designed to work together with the data structure defined here
it should be able to run calculations on demand and memoize the result

Use cases

It's been used for calculating average load and production profiles certain days.
For example it could just calculate the average load of some consume of the last seven days as well as the average PV production of the last four Tuesdays.

Alternatives and workarounds

It's possible to run the calculation and update the mean on a data stream whenever a new sample arrives. However we plan on storing batches of data in any case, as this can be used as a feature for ml models, and thus calculating the averages from the batches seems obvious.
Additionally this option hasn't been further explored so far.

Additional context

We have had a use-case where we where looking at a full days averaged data plus a 15 minute margin. Thus we will need to add a margin for a window size in order to re-engineer this feature.

We'll probably be able to use https://numpy.org/doc/stable/reference/generated/numpy.lib.stride_tricks.as_strided.html#numpy.lib.stride_tricks.as_strided for accessing arrays in strides in numpy

possilby helpful: https://numpy.org/doc/stable/reference/generated/numpy.mean.html

We need a margin around the window, such that we can say we want a specific load profile of a certain weekday +-15min. That would translate to the following parameters:

window_size = 1day
distance = 6days
margin = 15min

We could also consider introducing independent left and right margins.

Note: maybe we want padding (the one that goes to the outside)

possilby helpful: numpy.org/doc/stable/reference/generated/numpy.mean.html

Indeed useful. We can reshape the array into an 2 dimensional first and use the where array to select which values to include, i.e. leaf out all values that are not in any window or window margin. Using numpy's internal functions is likely faster then iterating over the values in python for calculating the mean.

Note: Average Time Series can be empty or a series of zeros because there is not enough data existing in the buffer or the buffer is full of NaN or in case the averaging calculation hasn't finished in time.

Should be able to calculate mean, standard deviation and polynomial coefficients.

Yes, if you don't have enough data. For instance, what is the expected behaviour in case we want a last 4 same weekdays average on a completely new site

Thats not a question about the MovingWindow but rather application level of what the fallback behaviour would/should look like. Can you put that on the DS agenda?

Right so, if you replace the MovingWindow with a numpy array you essentially just pumping all data thats coming out of the Resampler into memory. So is the use case about model training? Can you please give some more precise examples/explanation, please :-)

Yes, it's about model training. We are using data frames for this, so essentially numpy arrays. If it's possible we could apply only the TimeSeriesAverager from the SDK already now for creating features on our training data. Example could look like:

def tsavg(pd_series):
    # Code using TimeSeriesAverager with a pandas Series or numpy array
    ...
    return avg

# Calling tsavg over a 7 day rolling window in a pandas data frame
df.roll("7D")["some_column"].apply(tsavg)

Thats not a question about the MovingWindow but rather application level of what the fallback behaviour would/should look like. Can you put that on the DS agenda?

Sure, the application should always be responsible how this is handled. There should still be a clearly defined behaviour of the TimeSeriesAverager in this case. Added it to the agenda.

@matthias-wende-frequenz

In future we would also need other metrics like e.g. min, max, std.

@matthias-wende-frequenz this should be pretty straight forward to implement right? can you >> create a seperate issue for that once the PR is merged?

@cwasicki can you please define what you mean by min, max, std in this context?

@cwasicki

@cwasicki can you please define what you mean by min, max, std in this context?

For the TimeSeriesAverager you select a subset of the moving window and then take the average of these values. Similarly, you could also get the maximum, minimum or standard deviation on this subset. For instance, what was the maximum value in a specific time period of the day over the last 7 days.

@cwasicki

can you please define what you mean by min, max, std in this context

@cwasicki is that the min, max and std of the averaged timeseries or the underlying MovingWindow? @matthias-wende-frequenz that indeed raises a good question if renaming the TimeseriesAverager to MovingWindowAverager is really to way to go. At the very end the derived timeseries isn't really anymore connected the the MovingWindow itself that contains raw data whereas the TimeseriesAvergager contains derived statistical measures from the MovingWindow.

@cwasicki

@cwasicki is that the min, max and std of the averaged timeseries or the underlying MovingWindow?

I meant the underlying moving window. To clarify with an example: Assuming it's 15:00. Based on a time series window with data points every 15 min for the last 28 days, if you want to have the minimum for a time period of today 10:00 - 10:30 (i.e. two data points) and a distance of 7 days (i.e. 4 times a window with 2 data points) you would receive two values. The first value is the minimum value of the four values of 10:00-10:15 data points today, 7, 14 and 21 days ago. Equivalently you determine the second value, but for the 10:15-10:30 data points instead.

A potential use-case could be for a PV forecast: the production of a recent sunny day gives you at least some indication of the potential of your PV system for the given time period. You can get some estimate by taking the maximum of that time period over the last couple of days. In contrast to that, the production on cloudy days don't tell you much about the future unless you combine it with very good weather forecast data.

Add PeriodicFeatureExtractor (frequenz-floss#299)

Navigation Menu

Repository navigation

Add a `PeriodicFeatureExtractor` #120

What's needed?

Proposed solution

Use cases

Alternatives and workarounds

Additional context

12 remaining items

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Add a PeriodicFeatureExtractor #120

Description

What's needed?

Proposed solution

Use cases

Alternatives and workarounds

Additional context

Activity

mathias-baumann-frequenz commented on Dec 20, 2022

matthias-wende-frequenz commented on Dec 20, 2022

matthias-wende-frequenz commented on Dec 21, 2022

matthias-wende-frequenz commented on Dec 22, 2022

matthias-wende-frequenz commented on Dec 22, 2022

12 remaining items

thomas-nicolai-frequenz commented on Apr 6, 2023

cwasicki commented on Apr 6, 2023

matthias-wende-frequenz commented on Apr 6, 2023

cwasicki commented on Apr 6, 2023

thomas-nicolai-frequenz commented on Apr 8, 2023

cwasicki commented on Apr 11, 2023

cwasicki commented on Apr 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Fields

Projects

Milestone

Relationships

Development

Issue actions

Add a `PeriodicFeatureExtractor` #120