Welcome to Zoltar 4!¶
Zoltar version 4 is a major upgrade to Zoltar's fundamental forecast data representation, enabling new forecast versioning features. Read on for details.
Recall that the version of Zoltar prior to this new one added support for storing forecast versions. From the new Forecast Versions page:
Forecasts in Zoltar can be versioned. A version is identified by the combination of the forecast's model, time zero, and issue date. Multiple versions for the same model and time zero will have different issue dates. These can be identified by the gray version numbering shown in the version column on the model detail page's Forecasts list (e.g., "1 of 2"), and on the Issue date row of the forecast detail page's information table at the top (e.g., "(Version 3 of 3)").
This upgrade required migrating existing data, which we've done, but which resulted in a few forecasts that needed modifying. We've notified individual modelers of any problematic ones.
Forecast version rules¶
We now enforce these version rules:
- No uploaded forecast can have no rows of data.
- No uploaded forecast can load 100% duplicate data of a previous version.
- New forecast versions cannot imply any retracted prediction elements in existing versions, i.e., you cannot load data that's a subset of the previous forecast's data.
- New forecast versions cannot be positioned before any existing versions.
- Editing a version's issue_date cannot reposition it before any existing forecasts.
- Deleted forecasts cannot be positioned before any newer versions.
Bullet three means we require forecasts to be uploaded in
issue_date order. If you need to back-fill older versions, you'll first have to delete forecasts with newer issue_dates before uploading older ones.
Skipping of duplicate forecast data¶
Zoltar now implements a space optimization where duplicate data between forecast versions is not saved on disk. The forecast query feature hides this from users by reassembling the data as needed.
Zoltar now supports the notion of "retracting" forecasts. Such retractions mark individual prediction elements so that the
as_of forecast query parameter hides those predictions. The retracted data is not deleted, and earlier
as_of values will still return that data when passing an
as_of value that is prior to the retraction.
From the Retracted predictions page:
Zoltar supports retracting individual prediction elements. A retracted element marks a particular combination of unit, target, and prediction type to be ignored when executing a forecast query if the user passes an
as_ofvalue that's on or later than the retraction
issue_date. In that case there will be no value returned for the retracted prediction element. However, an
as_ofthat's earlier than the forecast's
issue_datewill result in the element's pre-retraction value being returned. Users who want to return the forecast's original data including retractions, should use the API or web UI as documented in download a single forecast. In this case Zoltar will include retractions as they were uploaded - with
null"prediction" value in the JSON (see below).
For users of the quantile forecast CSV format:
Retractions are indicated in quantile CSV files by
NULLpoint and quantile values. The libraries take care of converting these into the JSON below. See Quantile forecast format (CSV) for quantile retraction format details.