Forecasting, a Wicked Problem?

Malcolm Gladwell wrote an excellent article in the New Yorker about predicting box office hits. I could use a similar machine right about now to predict what people are going to buy. I've found forecasting to be fascinating and insanely difficult (hard to predict?) at the same time. It's about the only field that I can think of where there are few benchmarks of "goodness", but no dearth of people with opinions on how to make it better! This is a field that aspires one to do better even though there is no clear end goal in sight - a truly wicked problem.

In 1973, Horst W.J. Rittel and Melvin M. Webber, two Berkeley professors, published an article in Policy Sciences introducing the notion of “wicked” social problems. The article, “Dilemmas in a General Theory of Planning,” named 10 properties that distinguished wicked problems from hard but ordinary problems. I believe Forecasting displays many of these properties and can be aptly classified as a wicked problem.

1. There is no definitive formulation of a wicked problem. It’s not possible to write a well-defined statement of the problem, as can be done with an ordinary problem.

Forecasting is usually a means to an end - for example, better forecasting improves supply chain efficiency, or forecasting demand for consumer items is critical to determine broader macro-economic trends or commodity/currency forecasting to hedge risk. These are just some examples of the wide applicability of forecasting. While it is easy to formulate the forecasting problem and what one has to do to improve upon it, it is much harder to see how improvements in forecasting affect the end goals, which could be several stages removed. For instance, What is the direct attributable effect of a 100 basis point improvement in forecast accuracy on a company's inventory turns?

2. Wicked problems have no stopping rule. You can tell when you’ve reached a solution with an ordinary problem. With a wicked problem, the search for solutions never stops.

The forecasting problem definitely follows this property- after all, what is considered a "stopping point" for forecast accuracy? Is an 85% forecast accuracy good enough? also, good enough compared to what?

3. Solutions to wicked problems are not true or false, but good or bad. Ordinary problems have solutions that can be objectively evaluated as right or wrong. Choosing a solution to a wicked problem is largely a matter of judgment.

This is the epitome of forecasting. You're never right (duh!), but you can be better or worse compared to how you've been doing in the past. Most solutions to forecasting involves improving accuracy for some items while worsening it for others. What matters is that overall (over a large set of items), the solution is better.

4. There is no immediate and no ultimate test of a solution to a wicked problem. It’s possible to determine right away if a solution to an ordinary problem is working. But solutions to wicked problems generate unexpected consequences over time, making it difficult to measure their effectiveness.

This again is a classic problem in forecasting. You can do all the backtests or simulations you want to test a hypothesis, but as underlying things change, so will the need to re-learn your previous solutions. Furthermore, as your solutions are predicated on 'what happened before will repeat itself" to some degree, you're never going to be spot-on.

5. Every solution to a wicked problem is a “one-shot” operation; because there is no opportunity to learn by trial and error, every attempt counts significantly. Solutions to ordinary problems can be easily tried and abandoned. With wicked problems, every implemented solution has consequences that cannot be undone.

Yes and No. Forecasting usually drives critical business processes in a company, so the cost of error is only known once the approach is implemented. After all relying on either Monte-Carlo simulations or backtesting assumes that the past will repeat itself, which is seldom doesn't. Now, auditing and putting appropriate monitoring can help, but there are usually too many edge cases to catch every permutation of undesirable outcomes. On the other hand, you can minimize the impact by making small changes and learning as you go.

6. Wicked problems do not have an exhaustively describable set of potential solutions, nor is there a well-described set of permissible operations that may be incorporated into the plan. Ordinary problems do have a set of limited solutions.

This again is a classic forecasting problem. The limit of solutions is extensive with minute variations in explanatory variables (or new variables) changing the ultimate answer.

7. Every wicked problem is essentially unique. An ordinary problem belongs to a class of similar problems that are all solved in the same way. A wicked problem is substantially without precedent; experience does not help you address it.

There is very little one can draw from experience when it comes to solving forecasting problems. The uniqueness of the data in a given environment usually dictates what is and what isn't possible. The techniques to solve problems are transferable across industries and problem domains, but the patterns in the data is what makes one forecasting problem completely different from another problem.

8. Every wicked problem can be considered to be a symptom of another problem. While an ordinary problem is self-contained, a wicked problem is entwined with other problems. However, those problems don’t have one root cause.

Price changes affect forecasting and lack of product availability affects demand and lack of demand affects forecasts. This is just a simple way in which forecasts get intertwined with supply issues and demand shocks.

9. The existence of a discrepancy representing a wicked problem can be explained in numerous ways. A wicked problem involves many stakeholders, who all will have different ideas about what the problem really is and what its causes are.

Touche! As I've always maintained, "The forecast is right, the demand is wrong".

10. The planner has no right to be wrong. Problem solvers dealing with a wicked issue are held liable for the consequences of any actions they take, because those actions will have such a large impact and are hard to justify.

Touche-squared!

While it's impossible to find perfect solutions to wicked problems, companies learn to cope with them. When all else fails, the simplest possible solution is usually the right way to proceed. Here are some ways to deal with them:

- Social Brainstorming: Getting the stakeholders involved and making them understand the trade offs is the best way for everyone to understand the limitations of any proposed approach. Seems simple enough, but don't underestimate the havoc a lack of understanding can wreak on a team. It's far better to get people to understand what they're trying to solve together, formulate the problem and solution in terms that your clients can understand and be honest about the options and techniques that you think are feasible.

- Stick to your knitting: Pressure to succeed and to quell the wicked problem sometimes tends to have folks move away from their basic tenets and optimize for short-term. Keep true to what you're good at and how your core competency can help with this specific problem. Another way of saying this is "focus on your customers and not on your competition or on short-term pressures". That is one way to break ties when you have multiple options to solve a particular problem.

- Experiment incessantly: Wicked problems rarely have linear approaches/solutions. Try a million things, test changes in small increments and keep what works. That's the only way to deal with both the ambiguity of the problem as well as the unintended consequences of your solutions. Add to that a regular habit of auditing both your inputs and outputs as that will keep you honest.

Forecasting is truly both an art and a science and the wickedness property of the problem usually attracts masochists to the problem space. No pain, No gain as they say. Now, where is that prediction machine again?