An Overview Of Forecasting Performance Metrics

https://miro.medium.com/max/1200/0*ramo1Mj_UNrRQFdL

Original Source Here

An Overview Of Forecasting Performance Metrics

Photo by Scott Graham on Unsplash

Introduction

There are many error and performance metrics out there for your machine learning or statistical model. Not to mention new ones being developed year on year.

One metric is not better than the other, after all they are just a single number. However, picking the most suitable one for your model is important so you can optimise the model correctly.

In this post I want to go over the classic metrics and some forecasting specific ones along with their pros and cons. This should help you gain a better understanding to where and when to use certain metrics in your work.

The Classics

Mean Absolute Error (MAE)

Equation by author in LaTeX.

Mean absolute error is simply the mean of the differences between the forecasts, F, and the actual values, A.

Pros:

  • Easy to interpret
  • The error is in the units of the data and forecasts

Cons:

  • Doesn’t penalise outliers (if needed for your model)
  • Scale dependent, therefore cannot compare to other time series models which use different units

Mean Squared Error (MSE)

Equation by author in LaTeX.

Mean squared error is similar to MAE, but this time we square the differences between the forecasted and actual values.

Pros:

  • Outliers are heavily punished (if needed for your model)

Cons:

  • The error will not be in the original units of the time series, therefore can be harder to interpret
  • Scale dependent, therefore cannot compare to other time series models which use different units

Root Mean Squared Error (RMSE)

Equation by author in LaTeX.

Root mean squared error is the same as MSE apart from at the end we square root the result.

Pros:

  • Still heavily punishes outliers (if needed for your model)
  • Error will be in the original units of the time series
  • Kind of a best of both worlds of MSE and MAE

Cons:

  • Less interpretable as you are still squaring the errors
  • Scale dependent, therefore cannot compare to other time series models which use different units

Forecasting Specific

Mean Absolute Percentage Error (MAPE)

Equation by author in LaTeX.

Mean absolute percentage error is the percentage difference between the actual value and forecasted value. This is often the baseline metric used to measure most forecasting models.

Pros:

  • Easy to interpret
  • Scale independent, so can compare across different time series

Cons:

  • Infinite error if the actual value is near or at zero
  • Lower forecasts are bound to 100% error but higher forecasts can have an infinite error, hence it is biased to under-forecast

Symmetric Mean Absolute Percentage Error (SMAPE)

Equation by author in LaTeX.

Symmetric mean absolute percentage error is an extension of MAPE but accounting for its limitation of penalising negative errors more than positive ones.

Pros:

  • No longer favours under forecasting as the output is now fully bounded between 0 and 200%

Cons:

  • Still a chance of infinite values as the denominator can still be near or at zero
  • A percentage error between 0 and 200% is hard to interpret
  • In reality not actually symmetric as it doesn’t treat the errors of under and over-forecasting equally (see here)

Mean Absolute Scaled Error (MASE)

No seasonality:

Equation by author in LaTeX.

With seasonality:

Equation by author in LaTeX.

This is the mean absolute scaled error for both seasonal and non-seasonal time series and is probably the best and most fair metric to use.

If you want to learn more about seasonality, checkout my previous blog on the topic:

If the error is less than one, then the forecast is better than an averaged naive forecast on the training data of length T. On the other hand, if it is greater than one, the forecast is worse than an averaged naive forecast. In the above equation m refers to the seasonal index.

A naive forecast is where you set your prediction to the latest observed actual value. For a naive seasonal forecast you set it to the previous observed value in the season you are forecasting for.

Pros:

  • Scale independent as the numerator and denominator have the same unit
  • Predictable behaviour when the actual is near or at zero
  • Under-forecasting and over-forecasting are equally penalised

Cons:

  • Can be difficult to explain to stakeholders

Mean Squared Logarithmic Error (MSLE)

Equation by author in LaTeX.

Mean squared logarithmic error measures the ratio or relative difference between the forecasted values and the actuals.

Pros:

  • Punishes under-forecasting more than over-forecasting (if needed by your model)

Cons:

  • Not easy to interpret
  • Can have an issue of dividing by values that are at or close to zero

Summary and Further Thoughts

In this post we have gone over the classical error metrics: MAE, MSE and RMSE and some forecasting specific ones: MAPE, SMAPE, MASE and MSLE. All of these metrics have different pros and cons, whether that be being scale independent, able to divide by zero or to punish under-forecasts. The key thing to remember is that not one size fits all and you are better off using a range of metrics to evaluate your forecasting model.

References and Further Reading

Connect With Me!

AI/ML

Trending AI/ML Article Identified & Digested via Granola by Ramsey Elbasheer; a Machine-Driven RSS Bot

%d bloggers like this: