Data Cleaning and Missing Values – B.Pharm Notes (Ch 20)

These free B.Pharm notes on data cleaning and missing values in pandas cover chapter 20 of Basics of Python Programming for Pharmaceutical Sciences (BP101T), a first-semester subject. Read the key points below, then study the complete chapter PDF. You can browse every chapter in the Basics of Python Programming for Pharmaceutical Sciences category.

Read the following document for full information

Overview of data cleaning and missing values in pandas

These notes explain why pharmaceutical datasets need cleaning and how Pandas handles missing data shown as NaN. They cover detecting and counting missing values, converting “BLQ” text with pd.to_numeric(), dropping rows with dropna(), imputing with fillna(), forward and backward fill, removing duplicate records and renaming columns, using PK and ADR example datasets.

Key points on data cleaning and missing values in pandas

Pandas represents a missing value with a special floating-point marker called NaN, short for “Not a Number”, which propagates through calculations.
The expression df.isnull().sum() counts missing values in each column, while df.isnull().mean() * 100 gives the percentage missing per column.
Using pd.to_numeric() with errors=”coerce” converts text such as “BLQ” into NaN, so the concentration column becomes float64.
The dropna() method removes rows with missing values but should be used carefully in clinical datasets because removing observations may introduce bias.
The fillna() method replaces missing values with a statistic such as the median or a category like “Not Recorded”, while ffill() and bfill() suit time-series data.

Relevance to pharmacy practice

Pharmaceutical data can contain manual entry errors, instrument issues, inconsistencies between study centres and values below a quantification limit. Systematic cleaning maintains data quality, supports regulatory requirements and makes conclusions from clinical, pharmacokinetic and pharmacovigilance data more reliable.

Every chapter in this subject builds on the one before it, so do not skip topics that look simple. Definitions, principles, procedures and examples are the parts that examiners return to most often, and a clear understanding here will also make later subjects easier to follow in the coming semesters.

Frequently asked questions on data cleaning and missing values in pandas

How do you count missing values in each column in Pandas?

Use isnull() together with sum(), as in df.isnull().sum(). This gives the number of missing values in every column of the DataFrame.

Why is using dropna() risky in pharmaceutical data?

Deleting records may introduce bias. For example, a late BLQ concentration may be a real value below the assay limit, so removing it can affect the concentration-time profile.

What is the difference between forward fill and backward fill?

Forward fill (ffill) carries the most recent valid observation forward, while backward fill (bfill) uses the next valid observation to fill a preceding gap. Use them only when this is scientifically reasonable.

Quick revision checklist

Use this self-test after reading the chapter on data cleaning and missing values in pandas. If you can do all of the following without looking at your notes, you are ready for the exam.

I can explain: what NaN means in pandas
I can explain: how to count and percentage missing values
I can explain: how to convert BLQ to NaN with to_numeric()
I can explain: when to use dropna() or fillna()
I can explain: how to find and remove duplicate rows

Study and exam tips

Start by reading the chapter once for the overall idea, then revise data cleaning and missing values in pandas using headings, definitions, principles, reactions and examples. Short-answer questions in the university exam usually ask for definitions, principles, procedures, reactions and uses, so write these in your own words and practise the equations, tables and flow charts that the chapter has.

For wider reading, standard books and open resources such as the NCBI Bookshelf help you confirm terms and see more examples. Revise data cleaning and missing values in pandas again a day later and again a week later, because spaced revision improves long-term memory.

This page is part of the Basics of Python Programming for Pharmaceutical Sciences series for first-semester B.Pharm students on B.Pharm Guru, where every chapter has a short summary and a full PDF. Bookmark this page and share it with classmates who are preparing for the same exam.

Next: 21. FILTERING AND SELECTING DATA BASED ON CONDITIONS. Back to the Basics of Python Programming for Pharmaceutical Sciences category.