Part II: Data Wrangling

The 80% of the Job That Nobody Warns You About

Here is a number that shocks every aspiring data scientist: professional data scientists report spending roughly 80% of their time not on modeling, not on visualization, not on the exciting stuff they imagined when they chose this career — but on getting data into a shape where analysis is even possible. Cleaning. Reshaping. Merging. Fixing. Transforming. The unsexy, unglamorous, absolutely essential work of data wrangling.

If you completed Part I, you have already tasted this reality. In Chapter 6, you loaded a real dataset and immediately encountered missing values, inconsistent formats, and columns that were not quite what you expected. You wrote loops and conditionals to count things, filter things, and compute things, and it worked — but it was slow, verbose, and fragile. Every operation felt like it required too many lines of code.

Part II changes everything. Over the next seven chapters, you will learn pandas, the Python library that turns data wrangling from a chore into a craft. You will also develop something more valuable than any library: the mindset of a skilled data wrangler, someone who looks at a messy dataset and sees not a problem but a series of specific, solvable challenges.

Why Wrangling Deserves Your Respect

It is tempting to view data wrangling as a necessary evil — the vegetables you have to eat before dessert. Resist that temptation. Wrangling is where you develop your deepest understanding of the data. When you clean a column of vaccination dates and discover that three countries use a different date format, you are learning something real about the data collection process. When you merge vaccination rates with GDP data and discover that the country names do not match — "United States" in one dataset, "USA" in another, "United States of America" in a third — you are confronting the fundamental truth that data is created by humans, with all the inconsistency that implies.

The decisions you make during wrangling are analytical decisions. Choosing to fill missing vaccination rates with the regional average rather than dropping those rows entirely is not a mechanical step — it is an assumption about the data that will ripple through every downstream analysis. Part II will teach you not just how to make these decisions, but how to think about them carefully and document them honestly.

What You Will Find in These Chapters

Chapter 7: Introduction to pandas is the great liberation. Remember those fifteen-line loops from Chapter 6? pandas will replace them with single, expressive lines. You will learn the two core objects — Series (a single column) and DataFrame (a table) — and the fundamental verbs of data manipulation: select, filter, sort, and create. We will rebuild your Chapter 6 analysis in pandas, and the contrast will be vivid. But this chapter is not just about speed. It introduces a way of thinking — vectorized operations, working with entire columns at once rather than row by row — that will reshape how you approach data problems.

Chapter 8: Cleaning Messy Data is arguably the single most important chapter in this book. Real data is never clean. It arrives with missing values, duplicate rows, inconsistent types, and contradictory entries. You will learn to diagnose these problems systematically and apply appropriate fixes: detection with isnull, deletion with dropna, imputation with fillna, deduplication with drop_duplicates, and type correction with astype. More importantly, you will learn that every cleaning decision is an analytical decision that deserves documentation and justification. We work through a deliberately messy version of the vaccination dataset, and by the time you finish, you will have a clean, trustworthy dataset saved to disk — along with a cleaning log that explains every choice you made.

Chapter 9: Reshaping and Transforming Data teaches you to change the shape of your data to match the shape of your question. You need vaccination rates next to GDP figures? That is a merge. You need one row per country-year instead of one row per country? That is a melt. You need average vaccination rates by region? That is a groupby. These operations — merge, join, pivot, melt, groupby — are the structural transformations that turn raw data into analytical data. The chapter uses visual diagrams to show what each operation does to the shape of a DataFrame before you write a single line of code, because understanding the geometry of your data is just as important as knowing the syntax.

Chapter 10: Working with Text Data confronts the reality that much of the world's data arrives as text rather than numbers. Survey responses, product descriptions, medical notes, messy categorical labels — they all need processing before they become useful. You will learn pandas string methods for vectorized text operations and regular expressions for pattern matching and extraction. Regular expressions have a reputation for being cryptic, and that reputation is not entirely undeserved, but we introduce them gently, starting with simple literal matches and building toward the kind of pattern that can extract vaccine manufacturer names from a messy free-text field.

Chapter 11: Working with Dates, Times, and Time Series tackles one of the trickiest dimensions in data. Time seems simple — it is just dates, right? — but the details are surprisingly treacherous. Date formats differ across countries and datasets. Time zones create subtle bugs. Parsing a date string into an object that Python actually understands as a date, rather than just a piece of text, unlocks powerful operations like computing rolling averages, resampling to different frequencies, and slicing data by time ranges. You will parse the vaccination dataset's date columns and compute rolling 7-day averages, transforming noisy daily data into smooth trend lines.

Chapter 12: Getting Data from Files expands your ability to ingest data from multiple formats. CSV files are the bread and butter, but real projects involve Excel workbooks with multiple sheets, JSON files with nested structures, and databases that speak SQL. You will learn to handle all of these with pandas, along with the common gotchas that trip up beginners: encoding errors in CSVs, merged cells in Excel, deeply nested JSON that needs flattening. This chapter also introduces basic SQL — enough to pull data from a database, not enough to become a database administrator.

Chapter 13: Getting Data from the Web takes you beyond downloadable files to the living, breathing internet. You will learn to retrieve data from web APIs using the requests library and to scrape information from web pages using BeautifulSoup. But this chapter is as much about ethics as it is about technology. Just because you can scrape data does not mean you should. We discuss robots.txt, terms of service, rate limiting as courtesy, and the fundamental question of data ownership. You will pull live vaccination data from a public health API and discuss the responsibilities that come with automated data collection.

The Progressive Project Advances

Your vaccination rate investigation grows substantially in Part II. You will convert raw data into clean DataFrames, handle missing records and fix date formats, pivot data for cross-region comparison, merge with GDP figures, extract manufacturer names from messy text, compute rolling averages over time, pull in supplementary data from Excel and JSON files, and even retrieve live data from an API. By the end of Part II, you will have a rich, clean, multi-source analytical dataset — the foundation on which Parts III through V will build.

A Shift in Difficulty

Part II marks a step up from Part I. The concepts are more nuanced, the code is more complex, and the judgment calls are harder. Cleaning decisions require thinking about what the data means, not just what the syntax does. Reshaping operations require visualizing how the structure of your DataFrame changes. Regular expressions require a new kind of pattern-thinking.

This is normal. This is growth. And the skills you build here are among the most transferable in all of data science. Visualization libraries will change. Modeling frameworks will evolve. But the need to get data into shape — to clean, merge, reshape, and transform — will be with you for your entire career. Invest in these skills now, and they will pay dividends forever.

Do not be discouraged if a particular operation does not click immediately. Reshaping, in particular, often takes several encounters before it feels natural. The exercises at the end of each chapter are there to give you that practice. Do them. Then do them again with different data. The fluency will come.

What Comes After

Once you have a clean, well-structured dataset, two questions naturally arise: "What does this data look like?" and "What does this data mean?" Part III answers the first question through visualization. Part IV answers the second through statistical thinking. But neither is possible without the solid data wrangling foundation you are about to build.

Roll up your sleeves. The data is messy, and that is exactly how it should be.