Filtering and Selecting Data with loc and iloc in Pandas

Unit IV · Basics of Python Programming for Pharmaceutical Sciences (BP101T) · As per PCI B.Pharmacy Syllabus, NEP 2020

Once a pharmaceutical dataset has been cleaned, the next task is almost always to extract only the records and variables that matter for a specific question. A pharmacovigilance officer may need only the adverse events still “Ongoing”; a pharmacokinetic analyst may need only the concentration readings taken after the one-hour mark. Pandas gives you a complete toolkit for this — selecting columns, selecting rows by label or position, and filtering rows using logical conditions — and this article works through each of these with pharmacy-specific examples.

Selecting Columns

A single column can be pulled out of a DataFrame by placing its name inside square brackets. This returns a Pandas Series — a one-dimensional labelled array. Supplying a list of column names instead returns a smaller DataFrame containing just those columns.

pk_data = pd.read_csv("pk_study_data.csv")
ADR_data = pd.read_csv("ADR.csv")

# Select one column as a Series
subject_ids = ADR_data["SubjectID"]
print(subject_ids)

# Select multiple columns as a DataFrame
pk_subset = pk_data[["SubjectID", "Time_h", "Concentration_ugmL"]]
print(pk_subset.head())

Notice the syntactic difference: single brackets with one string (df["col"]) give a Series; double brackets with a list (df[["col1", "col2"]]) give a DataFrame, even if the list contains only one name. This distinction matters because Series and DataFrames support slightly different sets of methods.

Selecting Rows by Label and Position: loc[] and iloc[]

Pandas provides two indexers for pulling out rows (and optionally columns) from a DataFrame, and the difference between them is one of the most commonly confused topics for beginners:

• loc[] selects using index or column labels.
• iloc[] selects using integer positions, regardless of what the labels actually are.

# Select the row with index label 2
print(pk_data.loc[2])

# Select rows 0 to 3 (inclusive) and only two named columns
print(pk_data.loc[0:3, ["SubjectID", "Time_h"]])

# Select the first 3 rows and first 3 columns by position
print(pk_data.iloc[0:3, 0:3])

The Critical Slicing Difference

The single most important distinction to remember for exams and for writing correct code: with loc[0:3] the ending label 3 is included in the result (four rows: 0, 1, 2, 3), whereas with iloc[0:3] the ending position 3 is excluded (three rows: 0, 1, 2), following ordinary Python slicing rules. Mixing this up is one of the most frequent sources of off-by-one bugs when working with Pandas.

Boolean Indexing: Filtering by Condition

Boolean indexing is the core technique for filtering rows in Pandas. Applying a comparison operator to a column produces a Series of True/False values (a Boolean mask); passing that mask back into the DataFrame’s square brackets retains only the rows where the mask is True.

# Keep only adverse-event records whose Outcome is "Ongoing"
ongoing_ADR = ADR_data[ADR_data["Outcome"] == "Ongoing"]
print(ongoing_ADR)

# Keep only pharmacokinetic measurements at or after the 1-hour mark
late_pk = pk_data[pk_data["Time_h"] >= 1.0]
print(late_pk)

In the ADR example, only the record for subject S005 (Abdominal Pain, Severe, still Ongoing) satisfies the condition — every other adverse event in the dataset has already resolved.

Combining Multiple Conditions with & and |

Real filtering questions are rarely single-condition. Pandas lets you combine conditions using & (logical AND — every condition must hold) and | (logical OR — at least one condition must hold). Each individual condition must be wrapped in its own parentheses, because Python’s operator precedence would otherwise evaluate the comparison and the bitwise operator in the wrong order.

# AND: relation to drug is "Probable" AND outcome is "Resolved"
probable_resolved = ADR_data[
    (ADR_data["Relation_to_Drug"] == "Probable") &
    (ADR_data["Outcome"] == "Resolved")
]
print(probable_resolved)

# OR: severity is "Severe" OR onset time is under 1 hour
urgent_ADR = ADR_data[
    (ADR_data["Severity"] == "Severe") |
    (ADR_data["Onset_Time_h"] < 1.0)
]
print(urgent_ADR)

The AND query returns three records (S002, S004, S009) — all three are both drug-related and resolved. The OR query, in this dataset, happens to isolate a single record (S005) because both conditions independently point to the same severe, early-onset event.

Filtering with .isin()

When you need to keep rows matching any one of several specific categories — several drug names, several subject IDs, several causality gradings — writing a chain of OR conditions quickly becomes unwieldy. The .isin() method offers a compact alternative: it tests whether each value in a column is a member of a supplied list.

# Keep every ADR record classed as Probable or Possible
drug_related = ADR_data[
    ADR_data["Relation_to_Drug"].isin(["Probable", "Possible"])
]
print(drug_related)

This single line replaces what would otherwise require (col == "Probable") | (col == "Possible"), and scales cleanly to any number of categories without the expression growing longer for each one.

Filtering Rows and Selecting Columns Together

A very common real-world task is to filter rows by a condition and restrict the output to only the columns needed for the analysis, in one step. The .loc[] indexer accepts a row condition and a column list simultaneously.

pk_data_numeric = pk_data.copy()
pk_data_numeric["Concentration_ugmL"] = pd.to_numeric(
    pk_data_numeric["Concentration_ugmL"], errors="coerce"
)

mean_conc = pk_data_numeric["Concentration_ugmL"].mean()
print(f"Mean concentration: {mean_conc:.2f} µg/mL")

# Filter rows above the mean, keep only Time and Concentration columns
above_mean = pk_data_numeric.loc[
    pk_data_numeric["Concentration_ugmL"] > mean_conc,
    ["Time_h", "Concentration_ugmL"]
]
print(above_mean)

Because the "BLQ" text value was first converted to NaN, it is automatically excluded from the mean calculation (which comes out to 3.26 µg/mL in the sample dataset). The final result contains only the time points whose concentration exceeded that mean, with the extraneous Sex, Weight, and Age columns dropped from the view — exactly the kind of tight, analysis-ready subset a report or graph typically needs.

Frequently Asked Questions

What happens if I forget to put parentheses around each condition when combining with & or |?

Python will try to evaluate the bitwise operator before the comparison operator, which usually raises a TypeError or produces an incorrect result. Always write each condition as (df["col"] == value) before joining them with & or |.

Can I use the Python keywords "and" / "or" instead of & and | for filtering?

No. The keywords and/or expect a single True/False value, but a Pandas condition produces a whole Series of True/False values — Python cannot resolve the ambiguity and raises a ValueError: The truth value of a Series is ambiguous. You must use the vectorised operators & and | instead.

Why does .loc[0:3] give me 4 rows when I expected 3?

Because loc treats the numbers as labels, not positions, and label-based slicing in Pandas is inclusive of the endpoint by design (unlike ordinary Python list slicing). If you want the exclusive-endpoint behaviour you are used to from lists, use iloc instead.

Summary

Selecting and filtering data is the bridge between a cleaned dataset and a focused analysis. Single and multiple columns are pulled out using square-bracket notation; rows are selected by label with loc[] or by position with iloc[], remembering that loc slicing is inclusive of the endpoint while iloc slicing is not. Boolean indexing, combined with & and | for multi-condition queries and .isin() for multi-category membership tests, lets you extract exactly the records relevant to a pharmacokinetic or pharmacovigilance question. Combining a row condition with a column list inside .loc[] lets both operations happen in a single, efficient line of code.

References

• Pharmacy Council of India (PCI) — B.Pharm Regulations, NEP 2020, BP101T Syllabus
• Python Software Foundation — Official Python Documentation, docs.python.org
• Pandas Development Team — Pandas Official Documentation, pandas.pydata.org, especially the "Indexing and selecting data" user guide