Significance of Data Cleaning
What Is Data Cleaning?
Data cleaning is the process of identifying and correcting errors in a dataset to improve its quality and ensure the reliability of machine learning models.
Poor-quality data can lead to inaccurate predictions, inefficient models, and unreliable decision-making.
Why Is Data Cleaning Important?
Data quality significantly impacts the performance of machine learning models. Clean data ensures:
- Higher Model Accuracy: Reduces noise and irrelevant patterns.
- Efficient Training: Removes redundancy and inconsistencies.
- Better Generalisation: Ensures models work well on new data.
- Reduced Bias: Prevents incorrect data from skewing results.
Data Cleaning Techniques
Common Problems with Data
Handling Outliers
Outliers are data points that differ significantly from others in the dataset. They can skew analysis and impact model performance.
Purpose: To prevent rare but extreme values from distorting the model’s learning process.
Example: In a house price dataset, a property listed at £10 million among average homes priced at £300,000 could be flagged as an outlier.
Sample Data (Before)
| PropertyID | Bedrooms | Price (£) |
|---|---|---|
| 101 | 3 | 295000 |
| 102 | 4 | 310000 |
| 103 | 3 | 10000000 |
| 104 | 2 | 280000 |
Sample Data (After: outlier removed)
| PropertyID | Bedrooms | Price (£) |
|---|---|---|
| 101 | 3 | 295000 |
| 102 | 4 | 310000 |
| 104 | 2 | 280000 |
Duplicate Data Removal
Duplicate records occur when the same observation appears more than once in a dataset.
Purpose: To avoid bias from repeated entries that might overrepresent certain outcomes.
Example: A customer appearing twice in a purchase history dataset could falsely influence purchase frequency metrics.
Sample Data (Before)
| OrderID | CustomerID | Date | Amount (£) |
|---|---|---|---|
| 5001 | C12 | 2025-02-11 | 48.00 |
| 5002 | C18 | 2025-02-11 | 19.99 |
| 5002 | C18 | 2025-02-11 | 19.99 |
| 5003 | C05 | 2025-02-12 | 12.50 |
Sample Data (After: duplicates removed)
| OrderID | CustomerID | Date | Amount (£) |
|---|---|---|---|
| 5001 | C12 | 2025-02-11 | 48.00 |
| 5002 | C18 | 2025-02-11 | 19.99 |
| 5003 | C05 | 2025-02-12 | 12.50 |
Correcting Incorrect Data
This involves fixing values that are invalid, inconsistent, or out of expected range.
Purpose: To improve accuracy by ensuring all data conforms to expected formats or categories.
Example: A user’s age recorded as 350 is likely an error and needs correction.
Sample Data (Before)
| UserID | Age | Country |
|---|---|---|
| U01 | 16 | UK |
| U02 | 350 | uk |
| U03 | 22 | USA |
| U04 | -5 | France |
Sample Data (After: corrected/flagged)
| UserID | Age | Country (Std) | Issue |
|---|---|---|---|
| U01 | 16 | UK | |
| U02 | UK | Age out of range → set blank | |
| U03 | 22 | USA | |
| U04 | France | Age negative → set blank |
Filtering Irrelevant Data
Not all available features or entries contribute value to a machine learning model.
Purpose: To reduce model complexity and training time by removing unrelated or unnecessary information.
Example: A dataset predicting loan approval may include a customer’s favourite colour, which should be filtered out as irrelevant.
Sample Data (Before)
| ApplicantID | Income (£) | CreditScore | FavouriteColour |
|---|---|---|---|
| A001 | 32000 | 690 | Blue |
| A002 | 54000 | 720 | Green |
| A003 | 41000 | 650 | Purple |
Sample Data (After: irrelevant field dropped)
| ApplicantID | Income (£) | CreditScore |
|---|---|---|
| A001 | 32000 | 690 |
| A002 | 54000 | 720 |
| A003 | 41000 | 650 |
Data Transformation
Transformation reshapes data into a format suitable for analysis (e.g. encoding, scaling, or restructuring).
Purpose: To ensure consistency and compatibility with machine learning algorithms.
Example: Converting "Yes"/"No" answers to 1s and 0s for use in a classification model.
Sample Data (Before)
| CustomerID | Subscribed | Plan |
|---|---|---|
| C01 | Yes | Basic |
| C02 | No | Plus |
| C03 | Yes | Pro |
| C04 | No | Basic |
Sample Data (After: encoded for ML)
| CustomerID | Subscribed (1/0) | Plan |
|---|---|---|
| C01 | 1 | Basic |
| C02 | 0 | Plus |
| C03 | 1 | Pro |
| C04 | 0 | Basic |
Handling Missing Data
Missing values are common in real-world datasets and must be addressed before training models.
Purpose: To maintain data integrity and prevent errors or bias in model training.
Example: A survey dataset missing age values might use the average age of respondents to fill gaps.
Sample Data (Before)
| RespondentID | Age | City |
|---|---|---|
| R01 | 18 | Leeds |
| R02 | Manchester | |
| R03 | 20 | York |
| R04 | 22 | Sheffield |
Sample Data (After: mean imputation)
| RespondentID | Age (Imputed) | City |
|---|---|---|
| R01 | 18 | Leeds |
| R02 | 20 | Manchester |
| R03 | 20 | York |
| R04 | 22 | Sheffield |
Techniques to handle missing values
- Imputation: Fill the gap with a sensible value (e.g. mean/median for numbers, most common category for text, or previous value for time series). Keeps all rows but adds some uncertainty.
- Deletion: Remove rows with too many gaps or drop a column that is mostly missing. Simple and safe when only a small fraction is affected, but reduces your dataset.
- Predictive: Estimate the missing value using other fields (e.g. a small regression or classifier). Can be more accurate when features relate well, but requires care to avoid using information that wouldn’t be known in practice.
Tip: Add a flag column (e.g. Age_was_missing) so you can audit or reverse fills later.
Normalisation and Standardisation
These preprocessing techniques scale values consistently across features to improve model performance and convergence speed.
Normalisation
Normalisation: A technique that scales each feature to a fixed range - typically between 0 and 1.
Why it matters: Normalisation is useful when features have different scales or units (e.g. height in cm vs income in £), ensuring that no single feature dominates the model.
Example: In a dataset predicting car prices, one feature might be engine size (ranging from 1.0 to 5.0 litres), while another is mileage (ranging from 0 to 200,000 km). Normalising both ensures that engine size doesn't get overwhelmed by the much larger scale of mileage values.
from sklearn.preprocessing import MinMaxScalerscaler =
MinMaxScaler()normalized_data = scaler.
fit_transform(data)
Standardisation
Standardisation: A scaling method that adjusts data to have a mean of 0 and a standard deviation of 1.
Why it matters: Many machine learning algorithms (e.g. logistic regression, SVMs) assume normally distributed data - standardisation helps meet that assumption.
Example: In a dataset of student test scores, the maths scores may range from 0–100 while the reading scores might be out of 40. Standardising the scores ensures both features are treated equally by the model, even though they originally had different ranges.
from sklearn.preprocessing import StandardScalerscaler =
StandardScaler()standardized_data = scaler.
fit_transform(data)
Example: Data Cleaning in Python
Scenario: Handling missing values and duplicates in a dataset.
import pandas as pd# Load dataset
df = pd.
read_csv("data.csv")# Remove duplicate rows
df = df.
drop_duplicates()# Handle missing values using imputation
df.
fillna(df.mean(), inplace=True)# Filter irrelevant data
df = df[df["age"] > 18]
print(df.head())
Impact of Poor Data Quality
Unclean data can lead to:
- Biased Results: Misleading insights affecting decision-making.
- Model Overfitting: Learning from noise instead of meaningful patterns.
- Inefficient Training: Longer training times due to redundant or irrelevant data.
- Inconsistent Predictions: Unreliable outputs affecting real-world applications.
Key Takeaways
- Data cleaning is essential for ensuring accurate and reliable ML models.
- Handling outliers, duplicates, missing values, and irrelevant data improves model performance.
- Normalisation and standardisation are crucial preprocessing steps.
- Unclean data leads to bias, inefficiency, and unreliable results.