CASE STUDY - GLOBAL BUSINESS TRAVEL PLATFORM
A churn model that held up outside the lab.
A global business-travel platform flags accounts whose spend is about to collapse, three months ahead. NEXUS beat a heavily-tuned model on new data, and held that edge far better.
The Task
Flag accounts likely to churn, 3 months out
Churn Caught
40% more, same false-alarm rate
Generalization
Holds up on new data: 5% drop vs. 29%
A well-tuned model that looked better on paper than in production
The platform's team had built a gradient-boosted model to catch accounts about to churn - usage patterns, billing history, support tickets and admin activity all folded into a carefully tuned classifier. It scored well on the data it was trained on, but that score didn't fully carry over once the model saw new data it hadn't trained on: a classic sign of overfitting, and a risk in any model asked to catch problems before they happen.
Same features, no hyperparameter tuning
NEXUS was given the same engineered account features the existing model used - usage, billing, support, and account-management signals - and trained once with no hyperparameter tuning. Both models were trained on one stretch of history, then tested on later new data neither had seen.
More accurate on new data, not just the past it learned from
Measured Outcomes
+40%
more churners caught than the existing model on new data, at the decision threshold where four out of five flagged accounts really churn
+10%
higher PR-AUC on new data (0.509 vs. 0.463)
5% vs. 29%
drop in PR-AUC from training data to new data. NEXUS held its performance. The existing model lost almost a third of its score
On new data, NEXUS caught 40% more churners than the existing model, at the decision threshold where four out of five flagged accounts really churn. It also scored 10% higher on PR-AUC. The existing model looked strong on its training data, but its PR-AUC fell 29% on new data. NEXUS fell just 5%.
Precision is the share of flagged accounts that really churn. PR-AUC scores how well a model catches churners while keeping false alarms low. It runs from 0 to 1, and higher is better.
A model that trusts new data, not just familiar patterns.
The team's original build was thorough - dozens of engineered features, careful class-imbalance handling, an extensive tuning pass. NEXUS reached a more reliable result on the same features, without the tuning work. That matters for a churn model specifically: it has to be right about what happens next, months before obvious warning signs show, and a model that just memorizes the past is the one most likely to miss them.
