The World is Non-Stationary
The world, well, just about everything in the universe, is non-stationary. Non-stationary means it is constantly changing, what is, in one moment in time, isn’t quite the same a bit later. This is a problem opportunity.
This is an opportunity especially in data analytics. You spend a lot of time getting data together, synchronized, cleaned, converted into something useful, you build a model and put that model to use. Then, non-stationarity starts eating away at your model like langoliers (if you’ve seen the movie). Bit by bit, the system you modeled changes. It happens everywhere: Demographics shift. An industrial process corrodes or fouls. Resources and energies deplete. Instruments drift and fall out of calibration. Consumer trends change. Data interrelationships change… you get the idea.
This happens always everywhere with rare exception. You MUST deal with it otherwise the performance of your solution degrades, becoming less and less useful over time until it becomes useless.
How to Deal With It
Stage I: Adapt Your Models
Monitor the performance of your solution and calculate its “bias”, how far off it is, on average, using a walking window of time. Use that bias to adjust up or down your model’s output to keep it aligned. This will work fairly well for quite a while, until the inter-relationships, the functional relations, between the inputs to output(s) change enough to be notable. A that time your models are not just “off” but going wrong.
Stage II: Rebuild Your Models
After adaptation has run its course, when the inter-relationships in the data have notably changed, then rebuild your models with more recent information, dropping older data if necessary. Since rebuilding a model is actually an optimization process, this sometimes can cause a discontinuity in your results.
Stage III: Run an Ensemble
One way around having rebuilding discontinuities is to run an ensemble of models, dropping older lower performing ones while adding newer top performers, thus getting a smoother blend of results as
Do It Autonomously!
If you need a human to maintain your models, re-calibrating them by hand, it’s not going to be done, or there will be resistance… who like to do work?! People might actually not want to do performance testing because then they might have to recalibrate models. In that case, things go bad fast. Instead, do model maintenance automatically by the system. We can in our Intellect server. If your technology does not support it, it’s time for some new technology! Set it up, let it run, just check in on it every now and then.
Just about everything in the universe is non-stationary, constantly shifting through time. This is a problem opportunity to be a self-maintaining top performer, using autonomous self-maintenance.
What is YOUR experience and how did you deal with this analytical challenge? Comment below.
If you liked this article, please share it.
President / CTO
BioComp Systems, Inc. / IntelliDynamics
Call me at 1-281-760-4007