Missing Data and How to Spot It

In a perfect world, every row would have every column filled in. In reality, datasets are full of gaps. A survey respondent skipped a question. A sensor cut out for three seconds. A database migration dropped a field. All of this creates missing data.

Three flavors of missingness

• MCAR — Missing Completely At Random: a sensor glitched. The gap has nothing to do with the value.
• MAR — Missing At Random: men are less likely to answer a health survey question. The missingness depends on another column (gender) but not on the answer itself.
• MNAR — Missing Not At Random: people with very high incomes refuse to report their income. The value itself causes the missingness. This is the dangerous one.

Common ways to handle missing data

1. 1Drop rows with missing values (simple but throws away data)
2. 2Fill with the mean or median of the column (imputation)
3. 3Fill with a predicted value from other columns
4. 4Flag missingness as its own feature (is_missing = true)
5. 5Leave it for the model to handle (some models tolerate NaN)

import pandas as pd
import numpy as np

df = pd.read_csv('survey.csv', na_values=['-999', 'N/A', 'unknown'])

# How much is missing in each column?
print(df.isna().sum())
print(df.isna().mean())  # as a fraction

# Fill age with the median
df['age'] = df['age'].fillna(df['age'].median())

# Flag missingness before filling
df['income_was_missing'] = df['income'].isna()
df['income'] = df['income'].fillna(df['income'].median())

The big idea: missing data is not just absence, it is often information. Treat every gap as a question, not an error.