Pandas & visualization
The data investigation: filtering, aggregation, histograms, correlation
In this chapter, you will learn to
- Load a data table and ask it the right questions
- Group, aggregate and reshape a table
- Plot a chart to spot an anomaly or a correlation
Tools: pandas, Matplotlib
16 exercises
Each exercise is graded automatically, in your browser. The full statements are available once you have an account.
What this is for in real life
A model is only as good as the data you feed it, and you only really know your data once you have looked at it from every angle. - Epidemiology. In 1854, in London, the physician John Snow plotted cholera deaths…
Lesson: what pandas is for
What pandas is for. Before any model, you need to load, explore and prepare data. pandas is the standard tool: it manipulates DataFrames, tables made of rows and named columns, like a spreadsheet, but…
Mystery file
You are a data journalist. An anonymous source sends you a file containing information about the cities of a country. The data arrives as a Python list of dictionaries (each dictionary = one row of the table). An…
Suspicious cities
Your investigation moves forward. You are now looking for patterns in the city data (variable
data, same shape as in the previous exercise). Write the functionanalyze_network(data)that returns, in this order…Regions under the lens
Your editor-in-chief wants a summary by region: how many cities, what total population, what average population? The problem: several cities belong to the same region. You need to group the data by region, then…
Cross-checking sources
A second source sends you economic data about the regions: GDP per capita and the unemployment rate. This data is in
eco_records. The problem: your city data is organized by city, while the economic data is…Rebel bolt
Back to the factory! You hold the quality-control post on a bolt-making line. All day long, the machines send you batches of freshly machined bolts, each measured automatically to the hundredth of a millimeter. Most are…
Dirty data
You receive a survey dataset with 500 responses. The problem: the file is a disaster. The DataFrame
dfhas 6 columns:nom,age,ville,salaire,satisfaction,signup_date. Known problems: - Missing values…Lesson: your lists of dictionaries, upgraded
In the Exploration chapter, you handled data as lists of dictionaries, with plain Python loops: counting rows, taking a mean, filtering. A pandas DataFrame stores exactly this same data as a table (rows ×…
Transform
You have a dataset of 300 products with:
prix,categorie(text),poids_kg,customer_rating(1-5),en_stock("oui"/"non"). For an ML algorithm, every column must be numeric. Transform the DataFrame and print…Lesson: making the columns numeric
An ML model only eats numbers. Two cases, two tools: 1. Ordered or binary category →
.map. When the values have an obvious numeric correspondence, you replace them one by one:Reshaping
You have a table of monthly sales in "long" format: each row = one sale with
magasin,months,montant. Your boss wants a cross table: stores as rows, months as columns, and the total sales in the cells. From the…Lesson: long format ↔ cross table
Your data is in long format: one row per sale (
magasin,mois,montant). Convenient to store, but unreadable for comparing. You want a cross table: stores as rows, months as columns, the total in each cell.…Log detective
You receive the connection logs of a website:
logsis a DataFrame withuser_id,page,timestamp(sorted chronologically). The product manager wants to know: 1. How many unique users visited the site? 2. What…Lesson: group then aggregate
Many questions about data have the same shape: "for each X, compute Y". For each user, how many pages? For each city, the average salary? This is the split-apply-combine pattern, and pandas sums it up in…
Lesson: the data investigation
In this chapter, you did by hand the work that comes before any model: exploring and preparing data. In practice we use pandas, which expresses each of these moves in a single line. What matters is remembering the…