Evaluation methodology
The fraud detector and the forecaster: metrics, overfitting, validation, tuning
In this chapter, you will learn to
- Measure honestly whether a model works (the right metrics)
- Spot overfitting and guard against it (train, validation, test)
- Tune a model without cheating on the test data
Tools: scikit-learn
12 exercises
Each exercise is graded automatically, in your browser. The full statements are available once you have an account.
What this is for in real life
A model that looks excellent can be useless, or dangerous. It all depends on how you grade it. - Medicine. A screening test can be wrong in two ways: miss a sick person, or alarm a healthy one. These two…
Beat the champion
Warm-up. sklearn forbidden. You work in the AI team of a bank. After months of development, your fraud detector is finally running in production: every day, it sifts through thousands of transactions and flags the…
The intern's triumph
sklearn forbidden. This morning seemed completely normal. You arrived at your desk at the Super AI Agentic Bank at 8:30, as usual. You had your coffee, as usual. But a strange feeling crosses your mind. At…
Neither too much nor too little
sklearn forbidden. The director has recovered from the trouble with the wrongly accused customers: a few refunds and negotiations later, the relationship is restored. But there is no question of it happening again.…
Comparing without a threshold
sklearn forbidden. Your risk-score detector has proven itself. The director is considering replacing it and is comparing several competing models, but he wants to decide before setting the triggering threshold…
One hundred and twenty days of predictions
sklearn allowed. The bank sponsors a sports-prediction platform that publishes a prediction every day for 12 matches from several leagues. It wants to display on its home page the success rate the in-house model…
The honest verdict
sklearn forbidden. The director wants a number to rely on: “what accuracy will your model really have on future matches?” The problem: you have few matches, and each one counts. You are provided with…
Lesson: train, validation, test, the golden rule
As soon as a number is used to make a decision, it stops being a neutral measurement. That is why a serious ML project gives its data three distinct roles: | Set | Used for | How many times | |---|---|---| |…
Too Good to Be True
An analyst colleague proudly shows you his credit-scoring model. On the bank's thousands of already-closed loan files, it guesses who defaulted with near-perfect success. He wants to plug it into new applications…
Promise to the partner
sklearn allowed. The bank is about to sell its loan-approval model to a partner network. The partner will send the applications of its own customers, people the bank has never served, and it demands in the…
Lesson: learning curves
A learning curve plots two scores as a function of the number of training examples: the score on the data the model learned from (train curve) and the score on held-out data (validation curve). It is the…
Fine-Tuning
sklearn allowed. You are given a default
DecisionTreeClassifier. It fits the training data perfectly... but on test points it plateaus (~74%). Write the functiontrain_and_predict(X_train, y_train, X_test): it…