Pandas - Cleaning Empty Cells

Empty Cells

Empty cells can potentially give you a wrong result when you analyze data.

Cleaning empty cells, also known as missing values or NaNs (Not a Number), is a crucial step in data preprocessing. Pandas provides several methods to handle empty cells effectively. Here's a guide on how to clean empty cells in Pandas:

Identifying Empty Cells

• isna() and isnull(): These methods return a DataFrame of the same shape as the original, where each cell contains True if it's empty and False otherwise.

          print(df.isna())   # Boolean DataFrame indicating missing values
         




Removing Empty Cells

• dropna(): This method drops rows or columns containing empty cells from the DataFrame.

          df.dropna()  # Drops rows with any missing values
         




• Threshold: You can specify a threshold to drop rows or columns based on the number of non-null values.

          df.dropna(thresh=2)  # Drops rows with less than 2 non-null values
         

Filling Empty Cells

• fillna(): This method fills empty cells with a specified value or method.

          df.fillna(0)  # Fill missing values with 0
        




• Forward Fill (ffill) and Backward Fill (bfill): These methods propagate non-null values forward or backward along a Series or DataFrame.

          df.fillna(method='ffill')  # Forward fill
          df.fillna(method='bfill')  # Backward fill
        




Imputation

• Mean, Median, or Mode Imputation: You can fill missing values with the mean, median, or mode of the respective column.

          mean_value = df['Column_Name'].mean()
          df['Column_Name'].fillna(mean_value, inplace=True)
        




Dropping Rows or Columns

• drop(): You can drop rows or columns containing empty cells based on your requirements.

          
          df.dropna(axis=0)  # Drop rows with any missing values
          df.dropna(axis=1)  # Drop columns with any missing values