Energy Forecast Benchmark Toolkit

Energy Forecast Benchmark Toolkit is a Python project that aims to provide common tools to benchmark forecast models.

Installation

Use the package manager pip to install foobar.

pip install enfobench

Usage

Load your own data and create a dataset.

import pandas as pd

from enfobench import Dataset

# Load your datasets
data = pd.read_csv("../path/to/your/data.csv", parse_dates=['timestamp'], index_col='timestamp')

# Create a target DataFrame that has a pd.DatetimeIndex and a column named 'y'
target = data.loc[:, ['target_column']].rename(columns={'target_column': 'y'})

# Add covariates that can be used as past covariates. This also has to have a pd.DatetimeIndex
past_covariates = data.loc[:, ['covariate_1', 'covariate_2']]

# As sometimes it can be challenging to access historical forecasts to use future covariates, 
# the package also has a helper function to create perfect historical forecasts from the past covariates.
from enfobench.datasets.utils import create_perfect_forecasts_from_covariates

# The example below creates simulated perfect historical forecasts with a horizon of 24 hours and a step of 1 day.
future_covariates = create_perfect_forecasts_from_covariates(
    past_covariates,
    horizon=pd.Timedelta("24 hours"),
    step=pd.Timedelta("1 day"),
)

dataset = Dataset(
    target=target,
    past_covariates=past_covariates,
    future_covariates=future_covariates,
)

The package integrates with the HuggingFace Dataset 'EDS-lab/electricity-demand'. To use this, just download all the files from the data folder to your computer.

from enfobench import Dataset
from enfobench.datasets import ElectricityDemandDataset

# Load the dataset from the folder that you downloaded the files to.
ds = ElectricityDemandDataset("/path/to/the/dataset/folder/that/contains/all/subsets")

# List all meter ids
ds.list_unique_ids()

# Get dataset for a specific meter id
target, past_covariates, metadata = ds.get_data_by_unique_id("unique_id_of_the_meter")

# Create a dataset
dataset = Dataset(
    target=target,
    past_covariates=past_covariates,
    future_covariates=None,
    metadata=metadata
)

You can perform a cross validation on any model locally that adheres to the enfobench.Model protocol.

import MyModel
import pandas as pd
from enfobench.evaluation import cross_validate

# Import your model and instantiate it
model = MyModel()

# Run cross validation on your model
cv_results = cross_validate(
    model,
    dataset,
    start_date=pd.Timestamp("2018-01-01T00:00:00"),
    end_date=pd.Timestamp("2018-01-31T00:00:00"),
    horizon=pd.Timedelta("24 hours"),
    step=pd.Timedelta("1 day"),
)

You can use the same crossvalidation interface with your model served behind an API. To make this simple, both a client and a server are provided.

import pandas as pd
from enfobench.evaluation import cross_validate, ForecastClient

# Import your model and instantiate it
client = ForecastClient(host='localhost', port=3000)

# Run cross validation on your model
cv_results = cross_validate(
    client,
    dataset,
    start_date=pd.Timestamp("2018-01-01"),
    end_date=pd.Timestamp("2018-01-31"),
    horizon=pd.Timedelta("24 hours"),
    step=pd.Timedelta("1 day"),
)

The package also collects common metrics used in forecasting.

from enfobench.evaluation import evaluate_metrics

from enfobench.evaluation.metrics import (
    mean_bias_error,
    mean_absolute_error,
    mean_squared_error,
    root_mean_squared_error,
)

# Simply pass in the cross validation results and the metrics you want to evaluate.
metrics = evaluate_metrics(
    cv_results,
    metrics={
        "mean_bias_error": mean_bias_error,
        "mean_absolute_error": mean_absolute_error,
        "mean_squared_error": mean_squared_error,
        "root_mean_squared_error": root_mean_squared_error,
    },
)

In order to serve your model behind an API, you can use the built-in server factory.

import uvicorn
from enfobench.evaluation.server import server_factory

model = MyModel()

# Create a server that serves your model
server = server_factory(model)
uvicorn.run(server, port=3000)

Benchmarking

The package also provides a benchmarking framework that can be used to benchmark your model against other models.

The results of the benchmarking are openly accessible here.

Contributing

Contributions and feedback are welcome! For major changes, please open an issue first to discuss what you would like to change.

If you'd like to contribute to the project, please follow these steps:

Fork the repository. Create a new branch for your feature or bug fix. Make your changes and commit them. Push your changes to your forked repository. Submit a pull request describing your changes.

License

BSD 2-Clause License