TL;DR & Quick Summary
Most businesses should start with the forecasting already in their ERP or planning software and only build a custom model where that software measurably falls short. A custom model earns its cost when your demand depends on drivers the software ignores (price, promotions, events, customer orders), when much of your catalogue is intermittent or short-lived, or when forecast errors are expensive enough that small improvements matter.
Software first: if it beats a simple benchmark on your own items, it may be all you need
Custom where it pays: important drivers missing, unusual demand patterns, or high-value inventory
Decide with a bake-off: same items, same holdout, same metrics, plus a simple benchmark
Judge by stock outcomes: fill rate and inventory value, not just forecast error
Hybrid is common: the planning system stays the workflow, and a custom model feeds the segments where it wins
Key Takeaway: This isn't a one-time choice between software and a model. It's an item-by-item question of which forecast performs better, answered with a fair test.
Get Started: Want a bake-off between your current forecasts and a custom model? Schedule a strategy call with Cogniq AI or see our predictive analytics services.
What Forecasting Software Usually Gives You
Forecasting software means the demand forecasting built into an ERP or supply chain suite, or a specialist inventory planning tool. Most of these work in a similar way:
- They take sales history from your transactions
- They fit standard statistical methods and pick the best fit per item
- They let planners view, adjust and approve the forecast
- They feed the approved forecast into purchasing and production planning
- They report forecast accuracy
Microsoft's documentation for Dynamics 365 Supply Chain Management demand forecasting is a clear example. It generates a statistical baseline from historical transactions using Azure Machine Learning, chooses among several time-series methods, removes outliers, and requires manual adjustments to be authorised before planning uses them. The same documentation also notes that the feature might not be the best fit for some industries, such as commerce and wholesale. That kind of limit is exactly what a fair comparison should test.
The strength of software is the workflow: forecasts live where planners already work, with approvals and audit history. Its usual limit is the inputs and methods: it mostly sees sales history and applies general-purpose methods to every item.
What a Custom Model Adds
A custom model is a forecast built for your data and your decisions. It typically adds four things:
- More drivers. Price changes, promotions, marketing calendars, weather, school holidays, open customer orders and pipeline data. If these drive your demand and the software can't use them, its forecasts will consistently miss the moves you already know about.
- Methods suited to your demand. Intermittent spare parts, short-lifecycle products and new launches each need different approaches. See our guides to intermittent demand forecasting and the data new products need.
- Learning across products. A single model trained across many related items can forecast short-history items better than per-item methods. The top entries in the M5 forecasting competition, built on Walmart sales, used this approach.
- Evaluation tied to your outcomes. A custom model can be tuned to reduce bias on the items that drive stockouts, rather than to minimise one generic accuracy number.
The costs are real. Someone has to build it, maintain the data pipeline, monitor accuracy, and load forecasts into the planning system on schedule.
Side-by-Side Comparison
| Factor | Forecasting software | Custom model |
|---|---|---|
| Upfront cost | Low if already licensed | Higher: data work, modelling, integration |
| Time to first forecast | Days to weeks | Weeks to a few months |
| Inputs used | Mostly sales history | Sales plus any drivers you can supply |
| Methods | General-purpose, per item | Chosen and tested for your demand patterns |
| Intermittent and new items | Often weak | Can use specialised or cross-item methods |
| Planner workflow | Built in | Needs integration into the planning system |
| Ownership | Vendor maintains the product | You or a partner maintain the model |
| Best for | Regular demand, smaller catalogues | Driver-heavy demand, large or complex catalogues |
Signs the Software Is Enough
- Your demand is fairly regular and driven mainly by trend and season
- The software's forecasts beat a simple benchmark, such as last year's same period or a moving average, on your important items
- Planners trust the forecasts and adjust only occasionally
- Forecast bias is small and not consistently in one direction
- Stockouts and excess stock come from supply problems, not forecast misses
If these hold, a custom model is unlikely to repay its cost. Better use of the software, such as cleaner history, stockout flags and regular accuracy reviews, will usually do more.
Signs a Custom Model Will Pay Off
- Planners routinely override the forecast because they know something it doesn't, like a promotion, a price change or a large order
- Forecasts are consistently biased on an important segment
- A large share of items are intermittent, seasonal-short or new, and the software handles them poorly
- Inventory value is high or stockouts are expensive, so a few points of improvement are worth real money
- You have useful driver data that the software can't take in
Frequent manual overrides are the clearest signal. Each override is a planner adding information the model lacks. A custom model can often learn that information systematically.
How to Run a Fair Bake-Off
Don't decide on a vendor demo or a data scientist's enthusiasm. Test it:
- Choose the items: a representative sample across categories, including important intermittent and seasonal items.
- Hold out recent history: for example, the last 3–6 months, with no peeking.
- Produce forecasts three ways: from the software, from the custom model, and from a simple benchmark.
- Score them the same way: a volume-weighted error such as WAPE, MASE for intermittent items, and bias. See forecast accuracy metrics.
- Convert to stock levels and compare fill rate and inventory value. Our safety stock formulas cover the conversion.
- Decide by segment: the winner may differ between, say, fast movers and spare parts.
If the simple benchmark beats both, fix data and process before buying or building anything.
Common Mistakes When Comparing
- Comparing against the planners' adjusted forecast for one option and the raw baseline for the other. Compare like with like: raw versus raw, or adjusted versus adjusted.
- Testing on the period the model was tuned on. A custom model will always look better on data it has seen. Only the held-out period counts.
- Reporting one accuracy number across the catalogue. A blended figure hides the segments where each option wins. Score by segment.
- Ignoring bias. Two forecasts with the same error can produce very different stock positions if one consistently runs low.
- Leaving stockout periods in the history. Both options will learn that demand drops whenever you run out, and both will look worse than they are.
- Judging on a demo dataset. Vendor demos use clean, regular data. Your intermittent spare parts and promotional items are where the difference shows.
A fair comparison takes a few weeks of analysis. That's small compared with the cost of committing to the wrong option for years.
The Hybrid Setup
Most mid-sized businesses that invest in custom forecasting end up with a hybrid:
- The planning system remains where forecasts are reviewed, adjusted, approved and used for purchasing
- A custom model produces the baseline forecast for the segments where it won the bake-off
- Custom forecasts are loaded into the planning system on a schedule, for example weekly
- The software's own forecast remains the fallback for everything else
Planners keep their workflow and audit trail. The business pays for custom modelling only where it measurably helps. The integration that loads forecasts into the ERP is usually the main engineering work, and it's covered in our guide to connecting inventory systems to predictive reordering.
What a Custom Model Costs
Most of the cost isn't the model. It's the data and the integration:
- Data preparation: reconstructing stockout periods, aligning promotions and prices, cleaning units of measure
- Modelling and backtesting: building, comparing and validating models by segment
- Integration: loading forecasts into the planning system and pulling fresh data automatically
- Monitoring: tracking accuracy and bias after go-live, and retraining as patterns change
For build-versus-buy economics in general, our custom AI agent vs SaaS guide shows a three-year break-even calculation. The same approach works for forecasting if you value the stock and service improvements the bake-off measures.
Conclusion
Use the forecasting software you already have until a fair test shows it falling short on items that matter. When it does, usually because important drivers are missing, demand is intermittent or short-lived, or errors are expensive, add a custom model for those segments and feed it into the same planning workflow. Decide by bake-off and by stock outcomes, not by demos.
Schedule a strategy call with Cogniq AI to run a forecasting bake-off on your own data, or explore our predictive analytics services.
