Data preparation & feature engineering

Clean without breaking, then build the feature that makes the problem separable

In this chapter, you will learn to

• Clean messy data without skewing the results
• Build new features that make the problem easier
• Handle imbalanced classes

Tools: pandas, scikit-learn

9 exercises

Each exercise is graded automatically, in your browser. The full statements are available once you have an account.

1. What this is for in real life

   Data never arrives clean, and a model only sees what you give it. What you do before training often matters more than which model you pick. - Real records. In a hospital, a test was never ordered, a field was…

2. Cleaner-in-chief

   You are given a very dirty real-estate dataset: df_raw, a little over 1000 rows and 12 columns. Some are useful, others are noise. There are NaNs, outliers, inconsistent types, duplicates. Your mission: clean the…

3. Dirty Data

   sklearn allowed. It's your first day at a startup. You are handed a customer dataset to predict whether they will cancel their subscription (churn). Problem: the data is raw. - Some columns have missing values…

4. Mirage

   You are given points with two features (x, y) split into two classes (0 and 1). A linear model trained directly on x and y plateaus around 50%: it cannot find any straight line that separates the…

5. Epiphany

   Two classes again (X_train of shape (N, 2), y_train), and again a linear model that plateaus (around 50%): no straight line separates the classes. Explore the data: plot the class against each feature…

6. Hidden key

   Two classes (X_train (N, 2) = (x, y) points, y_train). Plotting the scatter colored by class, the boundary between the two classes is not a straight line: it is a curve. A linear model cannot follow it. Find…

7. Blind spot

   Two strictly positive features (X_train (N, 2), values > 0), two classes. A linear model fails: the boundary between the classes is not a straight line. Explore the data (plot it, color by class, try different…

8. The Feature Engineer

   sklearn allowed. A real-estate agency wants to spot, right when a property goes on sale, the ones that will sell above the market's median price. Each property is described by six columns: surface…

9. Imbalanced Classes

   sklearn allowed. Your startup pivots to fraud detection. Transactions arrive in a table data: columns f0 to f14 are the measurements of a transaction, and the column label is 0 (legitimate) or 1…