Scoring in Zoltar

Zoltar automatically computes a fixed set of scores (more are in development) for all projects and forecasts. They are updated in the background on a per-forecast basis by a scheduler that currently runs Daily at 5:30 AM UTC. The update is performed only on models that have changed, i.e., those that have added or removed forecasts since the last update. Following are details about scoring.

Scoring requirements

For Zoltar to calculate a forecast's scores, the following must be true:

Current scores

At the time of writing Zoltar implements these scores:

  • Error: The the truth value minus the model's point estimate.
  • Absolute Error: The absolute value of the truth value minus the model's point estimate. Lower is better.
  • Log score (single bin): Natural log of probability assigned to the true bin. Higher is " "better." More detail is here.
  • Log score (multi bin): This is calculated by finding the natural log of probability " "assigned to the true and a few neighbouring bins. Higher is better. See this page for how it works.
  • Probability Integral Transform (PIT): The probability integral transform (PIT) is a metric commonly " "used to evaluate the calibration of probabilistic forecasts.

In addition an aggregate score values are calculated (see below).

Viewing scores

Scores are accessed from the project detail page of the project of interest via the links in the Analysis row in the project details table at the top of the page. There are three links:

  • Visualizations: Takes you to a page that shows a time chart of the project's predictions. See the visualization page for details.
  • Aggregate scores: Goes to the "Mean Absolute Error" page. Absolute error is the absolute value of difference between the eventually observed value and the point prediction. This page is an aggregate of scores that are currently in Zoltar, and only as up-to-date as they are. No scores are shown for a particular unit or target if the errors could not be calculated because not all truth values were available for it.
  • Score data: The linked-to page shows score counts at both the project and model levels. Only those scores that have values are shown.

Here's an example score page:

Scores page

Downloading scores

Scores are downloaded from the project detail page. To download a project's scores:

  1. Go to the model detail page of the one you want to upload into.
  2. Click the "Download CSV" button and save the file.

The scores are saved in the CSV format documented at Score download format (CSV).

Note that because creating score CSV files can be time consuming, Zoltar caches creating them to help keep the site responsive. Normally the cached version will be available, which should download as soon as you click the button. However, the download will be delayed (and likely time out) otherwise. The state of caching is indicated by the text to the right of the button. If it says something like "(Cached file updated 2020-04-20 05:00:30)" then the faster download is available. Otherwise it will say "(No cached file - download may be slow.)"