Forecast Accuracy Metrics in Python: WAPE, MASE, Bias and Forecast Value Add

TL;DR & Quick Summary

Forecast accuracy comes down to three questions: how big are the misses (WAPE), which way do they lean (bias), and is the forecast worth running (forecast value add). This guide gives a short Python function for each, plus MAPE and MASE, and tests them on 60 items.

• WAPE: total absolute error ÷ total actual demand. Volume-weighted and defined when actuals are zero.

• MAPE: average percentage error per item. Undefined at zero, so it silently drops slow movers.

• MASE: your error ÷ naive forecast error. Below 1 means you beat the benchmark.

• Bias: (total forecast − total actual) ÷ total actual. Shows whether you'll overstock or stock out.

• Forecast value add (FVA): how much your forecast beats a naive one.

• Key Takeaway: In our backtest, MAPE couldn't be calculated for 20 of 60 items, all of them slow movers, and a forecast deliberately biased 10% low still looked respectable on error alone. Report WAPE, bias and FVA together, by segment.

• Get Started: For the concepts behind these metrics, read forecast accuracy metrics: MAPE, WAPE, bias and forecast value add. To have your own forecasts measured, schedule a strategy call with Cogniq AI.

The Five Functions

All five take arrays of actual and forecast values. MASE also needs the training history.

import numpy as np

def mape(actual, forecast):
    a, f = np.asarray(actual, float), np.asarray(forecast, float)
    with np.errstate(divide="ignore", invalid="ignore"):
        return np.mean(np.abs(a - f) / np.abs(a)) * 100

def wape(actual, forecast):
    a, f = np.asarray(actual, float), np.asarray(forecast, float)
    return np.sum(np.abs(a - f)) / np.sum(np.abs(a)) * 100

def bias_pct(actual, forecast):
    a, f = np.asarray(actual, float), np.asarray(forecast, float)
    return (np.sum(f) - np.sum(a)) / np.sum(a) * 100

def mase(actual, forecast, history):
    scale = np.mean(np.abs(np.diff(np.asarray(history, float))))
    return np.mean(np.abs(np.asarray(actual, float) - np.asarray(forecast, float))) / scale

def fva(metric_naive, metric_forecast):
    return metric_naive - metric_forecast   # positive = the forecast adds value

WAPE (Weighted Absolute Percentage Error) is the total absolute error divided by total actual demand. Because it divides totals, large items count in proportion to their volume, and it stays defined when individual periods are zero.

MAPE (Mean Absolute Percentage Error) averages the percentage error of each item equally. It divides by each actual value, so a zero makes it infinite.

MASE (Mean Absolute Scaled Error), from Hyndman and Koehler (2006), divides your error by the in-sample error of a one-step naive forecast. It works for any demand pattern, including intermittent items.

Bias is signed. Negative means the forecast runs low, positive means it runs high.

Forecast value add compares your forecast with a naive benchmark on the same metric.

Why MAPE and WAPE Disagree

Take two items. One sells 2 units and was forecast at 4. The other sells 1,000 and was forecast at 900.

mape([2, 1000], [4, 900])   # 55.0
wape([2, 1000], [4, 900])   # 10.2

MAPE reports 55% because the small item's 2-unit miss counts as a 100% error, weighted equally with the large item's 10%. WAPE reports 10.2% because it weights by volume: the business missed 102 units out of 1,002.

Neither is "wrong". They answer different questions. But a planning team that reports MAPE will spend its time chasing tiny items that barely affect inventory.

Add a single zero and MAPE breaks entirely:

mape([0, 1000], [1, 900])   # inf

A 60-Item Backtest

To see how these metrics behave on a realistic catalogue, we generated 60 items with a fixed random seed:

• 20 fast movers: 200–800 units a week with moderate noise
• 20 medium movers: 20–80 units a week
• 20 slow movers: selling in only 15–50% of weeks

Each had 52 weeks of history and a 12-week test period. We compared three forecasts:

1. Naive: last week's actual, repeated
2. 8-week average: the mean of the last eight weeks
3. 8-week average × 0.9: the same forecast deliberately biased 10% low

Forecast value add against the naive forecast, measured in WAPE points: +5.8 for the 8-week average and +3.0 for the biased version.

What the Results Show

1. MAPE quietly dropped a third of the catalogue. All 20 slow movers had at least one zero week in the test period, so MAPE couldn't be calculated for any of them. The MAPE column reflects only the fast and medium items. Anyone reading "13.1% MAPE" would assume it covers everything. WAPE and MASE covered all 60 items.

2. A biased forecast can look acceptable on error alone. The version deliberately 10% low scored a respectable 15.7% WAPE and a MASE below 1, beating the naive forecast. Only the bias column, at −10.5%, shows that it would consistently leave you short of stock. Bias doesn't average out; it repeats every period.

3. The naive benchmark is the right yardstick. The naive forecast's MASE of 1.046 is close to 1, as expected, since MASE is scaled by naive error. The 8-week average beat it on every metric. Without the naive comparison, "12.9% WAPE" has nothing to be judged against.

These are synthetic results. Your numbers will differ, but the behaviour of the metrics is the same on real data.

Report by Segment

A single number across the whole catalogue hides more than it shows. Group the results before reporting:

import pandas as pd

def report(df):
    # df columns: segment, actual, forecast, naive
    rows = []
    for seg, g in df.groupby("segment"):
        rows.append({
            "segment": seg,
            "WAPE": wape(g.actual, g.forecast),
            "bias_%": bias_pct(g.actual, g.forecast),
            "FVA_pts": fva(wape(g.actual, g.naive), wape(g.actual, g.forecast)),
        })
    return pd.DataFrame(rows).round(1)

Typical segments are fast, medium and slow movers, or the intermittent and lumpy categories from the ADI and CV² classification. A forecast often adds value on fast movers and destroys it on slow ones. Segmenting is how you find out, and decide where a simpler method should replace the current one.

Choosing the Naive Benchmark

Forecast value add is only as meaningful as the benchmark you compare against. Common choices:

• Last period's actual (the one used here): simple and standard, and works well for steady demand
• Same period last year (seasonal naive): the right benchmark when demand is strongly seasonal, because last week's number would be an unfairly weak comparison
• A short moving average: a tougher benchmark for noisy items, and a realistic alternative you could adopt if the forecast fails to beat it

Pick the benchmark that a reasonable planner could actually use without a forecasting system. If your process can't beat that, its cost isn't justified for those items. Use the same benchmark across reports so the numbers are comparable over time.

Measure at the Horizon That Matters

A forecast made one week ahead will always look better than one made eight weeks ahead, because less can change in between. Measure accuracy at the horizon your decisions actually use:

• If suppliers need orders six weeks in advance, the forecast that drives purchasing is the one made six weeks before the sales period. Its accuracy is what determines your stock position.
• Store each forecast with the date it was made, not just the period it predicts. Without that, you can't reconstruct what the forecast said when the decision was taken.
• Report accuracy at one or two fixed horizons, for example one week and lead time, so trends over time are comparable.

Measuring the most recent forecast against actuals is the most common way teams overstate accuracy without realising it.

Explaining the Numbers to Non-Specialists

Metrics only change decisions if the people who make them understand them. Some phrasings that work:

• WAPE: "Across all products, our forecast missed by 13 units for every 100 we sold."
• Bias: "Our forecast runs about 1% low, so we're not systematically under- or over-buying." Or, for the biased case: "It runs 10% low, which is why we keep running out."
• Forecast value add: "Our forecast beats simply reusing last week's sales by about 6 points."

Show these three numbers by segment, on one page, every month. Leave MASE and the formulas in the appendix for the analysts.

Common Mistakes

• Averaging per-item WAPE. WAPE should be computed on totals. Averaging item-level WAPE turns it back into something like MAPE.
• Letting MAPE skip zeros silently. If you must report MAPE, also report how many items it excluded.
• Measuring on the training period. Accuracy only counts on held-out data the forecast didn't see.
• Comparing forecasts of different horizons. A one-week-ahead forecast will always beat a twelve-week-ahead one. Compare at the horizon your purchasing decisions use.
• Forgetting stockouts. If actual sales were capped by stockouts, every forecast looks too high. Flag or remove those periods.
• Ignoring bias because error looks fine. As the backtest showed, a forecast can be 10% low every week and still look reasonable on WAPE and MASE.

Turning Metrics Into Decisions

Metrics are only useful if they change something:

• FVA negative on a segment: switch it to a simpler method or a naive forecast, and stop spending effort on it
• Bias consistently negative: expect stockouts; investigate the cause before increasing safety stock
• Bias consistently positive: expect excess inventory; check promotions, stockout-capped history and planner overrides
• High WAPE but no bias: noise rather than systematic error; safety stock, not the forecast, is usually the lever

For intermittent items, the forecasting methods themselves are covered in Croston, SBA and TSB in Python. For turning forecasts and their errors into stock levels, see safety stock and reorder point in Python.

Conclusion

Five short functions cover forecast accuracy properly. Use WAPE for the size of errors, bias for their direction, and forecast value add to check the forecast beats a naive benchmark, all broken down by segment, with MASE for intermittent items. Be wary of MAPE: in our backtest it silently excluded every slow mover. And remember that a biased forecast can look accurate until you check which way it leans.

Schedule a strategy call with Cogniq AI to have your forecasts measured against a naive benchmark, or explore our predictive analytics services.