# An Overview Of Forecasting Performance Metrics

https://miro.medium.com/max/1200/0*ramo1Mj_UNrRQFdL

Original Source Here

# Introduction

There are many error and performance metrics out there for your machine learning or statistical model. Not to mention new ones being developed year on year.

One metric is not better than the other, after all they are just a single number. However, picking the most suitable one for your model is important so you can optimise the model correctly.

In this post I want to go over the classic metrics and some forecasting specific ones along with their pros and cons. This should help you gain a better understanding to where and when to use certain metrics in your work.

# The Classics

## Mean Absolute Error (MAE)

Mean absolute error is simply the mean of the differences between the forecasts, F, and the actual values, A.

Pros:

• Easy to interpret
• The error is in the units of the data and forecasts

Cons:

• Doesn’t penalise outliers (if needed for your model)
• Scale dependent, therefore cannot compare to other time series models which use different units

## Mean Squared Error (MSE)

Mean squared error is similar to MAE, but this time we square the differences between the forecasted and actual values.

Pros:

• Outliers are heavily punished (if needed for your model)

Cons:

• The error will not be in the original units of the time series, therefore can be harder to interpret
• Scale dependent, therefore cannot compare to other time series models which use different units

## Root Mean Squared Error (RMSE)

Root mean squared error is the same as MSE apart from at the end we square root the result.

Pros:

• Still heavily punishes outliers (if needed for your model)
• Error will be in the original units of the time series
• Kind of a best of both worlds of MSE and MAE

Cons:

• Less interpretable as you are still squaring the errors
• Scale dependent, therefore cannot compare to other time series models which use different units

# Forecasting Specific

## Mean Absolute Percentage Error (MAPE)

Mean absolute percentage error is the percentage difference between the actual value and forecasted value. This is often the baseline metric used to measure most forecasting models.

Pros:

• Easy to interpret
• Scale independent, so can compare across different time series

Cons:

• Infinite error if the actual value is near or at zero
• Lower forecasts are bound to 100% error but higher forecasts can have an infinite error, hence it is biased to under-forecast

## Symmetric Mean Absolute Percentage Error (SMAPE)

Symmetric mean absolute percentage error is an extension of MAPE but accounting for its limitation of penalising negative errors more than positive ones.

Pros:

• No longer favours under forecasting as the output is now fully bounded between 0 and 200%

Cons:

• Still a chance of infinite values as the denominator can still be near or at zero
• A percentage error between 0 and 200% is hard to interpret
• In reality not actually symmetric as it doesn’t treat the errors of under and over-forecasting equally (see here)

## Mean Absolute Scaled Error (MASE)

No seasonality:

With seasonality:

This is the mean absolute scaled error for both seasonal and non-seasonal time series and is probably the best and most fair metric to use.

If the error is less than one, then the forecast is better than an averaged naive forecast on the training data of length T. On the other hand, if it is greater than one, the forecast is worse than an averaged naive forecast. In the above equation m refers to the seasonal index.

A naive forecast is where you set your prediction to the latest observed actual value. For a naive seasonal forecast you set it to the previous observed value in the season you are forecasting for.

Pros:

• Scale independent as the numerator and denominator have the same unit
• Predictable behaviour when the actual is near or at zero
• Under-forecasting and over-forecasting are equally penalised

Cons:

• Can be difficult to explain to stakeholders

## Mean Squared Logarithmic Error (MSLE)

Mean squared logarithmic error measures the ratio or relative difference between the forecasted values and the actuals.

Pros:

• Punishes under-forecasting more than over-forecasting (if needed by your model)

Cons:

• Not easy to interpret
• Can have an issue of dividing by values that are at or close to zero

# Summary and Further Thoughts

In this post we have gone over the classical error metrics: MAE, MSE and RMSE and some forecasting specific ones: MAPE, SMAPE, MASE and MSLE. All of these metrics have different pros and cons, whether that be being scale independent, able to divide by zero or to punish under-forecasts. The key thing to remember is that not one size fits all and you are better off using a range of metrics to evaluate your forecasting model.