4 minutes
Five ways to overcome your inertia on using this great tool, plus four questions to answer to ensure you’re making the most of your backtest
“What good is the warmth of summer without the cold of winter to give it sweetness.” - John Steinbeck
We are all too familiar with the winter blues, the dread that accompanies those long, cold winter days that seem to drag on and zap our energy and motivation. We know it is coming, but somehow it sneaks up on us. It’s unavoidable but, as Steinbeck points out, it’s necessary to appreciate the warmth of summer.
When I recall conversations I have had with credit unions about backtesting, this same “blues” mindset can set in. For anyone new to the concept, backtesting evaluates the accuracy of a forecasting model by using the projected data and comparing the results to the actual performance.
When I walk through the process of backtesting with CFOs, they are often eager to start the process. However, that enthusiasm is completely depleted once it’s time to roll up sleeves and evaluate the test results. The backtest blues set in, and the benefit of the results seem like a distant summer day.
To combat the typical winter blues, we may head on a tropical vacation or tackle a New Year’s resolution with refreshed vigor. Similarly, it’s natural to search for ways to fight the backtest blues. And while I can’t write a surefire prescription, I do have five suggestions to overcome them.
1. Correct Blurred Vision
Want to sharpen the blurred vision your asset/liability committee has of your model results? Show them that it works. Sharing results with the ALCO will add validity to the model. Gaining their confidence opens new avenues to strategic success. The model has huge benefits to assist with the planning. Turning a fuzzy view into a measurable idea is a step in the right direction.
2. Bask in Your Success
When your examiner asks for backtest results, be prepared to bask in your accomplishment. It might seem like a minor success, but you’ll be thankful you took the time to have this ready—and not have to pull an all-nighter to complete it.
3. Strengthen the Model
Dramatic changes to the balance sheet, driven by pandemic liquidity, often lead to sweeping changes to the balance sheet as funds are reinvested. Many times, I see this resulting in the addition of new types of assets. Modeling errors can occur, particularly when it’s a new product type. For example, the premium paid for a loan participation should be amortized in the model, reducing the yield. The backtest process can reveal model errors, making it a seriously effective tool for your financial arsenal. In this case, backtesting will prevent the model from overstating the projected income. At a time when margins are tight, this process can bring to light these oversights and help you make your model better.
4. Consult Your Vendor
Just like it can be hard to talk about your winter blues, it can be difficult to seek with backtesting. Still, I encourage you to benefit from others’ experiences and the way to do that is by reaching out for support. Contact your vendor and let them help you interpret the model’s results. As a credit union vendor and partner, I can confidently say that we want your model to be as accurate as possible, and we’re eager to guide you through this process.
5. Trust Your Method
Knowing that your model is running at optimum levels builds confidence in the results. Plain and simple, the backtesting process works. Over the years, we have found enhancements originating from both sides of the balance sheet that resulted in overstatements as well as understatements to the projected income.
I have to ask … After viewing these backtesting tips, are you ready to beat the backtesting blues? If so, here are recommendations for setting up an effective backtest based on the most common questions we receive about the process.
- Where do I start? Start with a rate-volume analysis. This simple tool identifies if a variance exists and reveals where it originates. Differences can be due to balance, yields or a combination of both.
- What time frame should I use? Best practice where time frames are concerned is to compare a quarter. If you prepare December ALM reports, you will use the projected income from the January, February and March model projections and compare it to the actual first-quarter financial statements. Remember, the purpose of backtesting is to gauge the model’s accuracy and, since the model is a projection, you have to wait for the actual data to occur before you can evaluate a model’s accuracy.
- How much variance is too much? There is not a definitive answer to the variance percentage. To make sure you aren’t chasing immaterial variances, try taking a deeper look into variances that exceed 3% to 5%.
- What if I can’t find the issue? Reach out for help. Your model vendor wants you to have confidence in the model. Use this to your benefit.
With these suggestions and recommendations at your disposal, it’s my hope that backtest blues become a thing of the past for you and your credit union. Don’t get down on the process. Instead, move forward with confidence. Trust in the great Steinbeck’s words: Better things lie ahead if you can just get past the difficult hurdles.
Melissa Scott is VP/ALM services, Vizo Financial.
