Modeling to Learn

You’re on an adventure. The quest is challenging, you are tired, and you are completely uncertain about what the path ahead looks like. Who would you want assistance from? A robot from the future who can accurately tell you the direction and distance to travel? Or a wise mountain man who cannot remember exactly how far to walk, but says you will want to stay close to the creek until you reach the edge of the forest and then be on the look out for a cave that looks like a quarter moon? And what does this have to do with finance?

I have fun building models in my job. They usually are not that sophisticated, but they force me to consider what I know about things today and what things will be like in the future. Lately I have also learned more about “real” statistical modeling and forecasting methods. Auto regression, exponential smoothing, and even deep learning forecasts all seem much more advanced than my usual year-over-year-drag-to-the-right model. The accuracy could be better and they are fairly easy to create and update. All the same, I have struggled to put my hope into them and rely on them for planning. Why would I not use a model that is less time intensive and more actuate?

Unfortunately they do not tell a story about the future that I can understand. They can and do provide a distance and direction towards the future, but not much color about the space in between. That is a big problem when businesses or products experience hiccups, diverge from the model’s forecast and you have to figure out why. A detailed model incorporating business drivers gives you an anchor upon which you can try and figure out where you went wrong and how to get back on track.

The other big advantage of a detailed model grounded in reality is that when it is wrong it can still be useful. With a model that describes the future world as you know it, any discrepancies are a chance to get smarter about the prediction surface. In the quest we imagined earlier, if the creek turns out to be dry and the cave actually looks like a crescent moon, we now know to expect a creek with or without water and to check out any roughly moon-shaped caves. The robot’s vector cannot teach us much of anything if we get off track, or if it turns out to be wrong.

My boring vanilla year over year growth models do not seem very descriptive. The reason growth is such a useful model is that it maps the business today +/- changes that you can understand. Then the main thing you have to gain comfort around is just the magnitude of changes between now and then. If you can decompose the business into the key drivers and predict how those will change, even better.

To conclude, your main models for a business should focus on the actual future reality you can imagine, with as much color and detail as you can reason about in your head. Use statistically grounded or AI enabled outputs as sanity checks or guides when you do not know what the future will look like at all. That way when you’re wrong you will still learn the right things to get closer. And people will be grateful for the wise man in the mountains who showed them the (approximate) way.