Data Cleaning Fundamentals
Master essential skills for cleaning and preparing data for analysis
The 80/20 Rule of Data Analytics
Data scientists spend approximately 80% of their time cleaning and preparing data, and only 20% actually analyzing it. Data cleaning isn't glamorous, but it's absolutely critical for accurate results.
Why Clean Data?
"Garbage in, garbage out" - if your data is messy, your analysis will be unreliable. Clean data ensures:
- ✓Accurate insights - Correct conclusions from your analysis
- ✓Reliable predictions - Models that actually work
- ✓Consistent results - Reproducible findings
- ✓Better decisions - Trust in your data-driven choices
Common Data Quality Issues
1. Missing Values
Empty cells, null values, or placeholders like "N/A", "Unknown"
John | 25 | NYC
Jane | null | Boston
Bob | 30 | ""
Solutions:
- • Delete rows with missing values (if few)
- • Fill with mean/median/mode (for numerical data)
- • Fill with most frequent value (for categorical)
- • Use forward/backward fill for time series
- • Flag as "missing" category
2. Duplicate Records
Same entry appears multiple times, inflating counts
1 | John | john@email.com
2 | Jane | jane@email.com
3 | John | john@email.com
Solutions:
- • Remove exact duplicates
- • Check for fuzzy duplicates (slight variations)
- • Keep first/last occurrence
- • Aggregate if needed (sum, count)
3. Inconsistent Formatting
Same data represented differently
Phone: (555) 123-4567 | 555-123-4567 | 5551234567
Case: USA | usa | U.S.A.
Solutions:
- • Standardize formats (dates, phone numbers)
- • Convert to consistent case (upper/lower)
- • Remove extra spaces and special characters
- • Use consistent units (kg vs lbs)
4. Outliers
Values that are suspiciously high or low
Prices: $19.99, $24.99, $999999, $29.99
Solutions:
- • Investigate - is it an error or real?
- • Remove if clearly wrong
- • Cap at reasonable limits
- • Transform data (log scale)
- • Keep but flag for special treatment
5. Incorrect Data Types
Numbers stored as text, dates as strings, etc.
Date: "2024-01-15" (string) → should be datetime
ZIP: 12345 (number) → should be "12345" (string)
Solutions:
- • Convert to appropriate data type
- • Parse dates correctly
- • Remove currency symbols before converting
- • Preserve leading zeros (ZIP codes)
Data Cleaning Workflow
Inspect the Data
Look at first/last rows, check data types, understand structure, get summary statistics
Identify Issues
Check for nulls, duplicates, outliers, format inconsistencies, wrong data types
Clean and Transform
Handle missing values, remove duplicates, standardize formats, fix types, deal with outliers
Validate Results
Check row counts, verify transformations, ensure logic, spot check random samples
Document Changes
Keep record of what was changed and why - crucial for reproducibility
Best Practices
Always keep original data
Work on copies, never modify the source
Document everything
Write down what you changed and why
Use scripts, not manual edits
Automate for reproducibility
Validate at each step
Check results after every transformation
Understand context first
Know what the data represents before cleaning
Collaborate with domain experts
They know what's normal vs suspicious
💡 Key Takeaways
- ✓Data cleaning takes most of your time but is essential for accurate analysis
- ✓Common issues: missing values, duplicates, formatting, outliers, wrong types
- ✓Follow a systematic workflow: inspect, identify, clean, validate, document
- ✓Always preserve original data and document your changes
What's Next?
Now that your data is clean, you're ready to analyze it! Learn about descriptive statistics to summarize and understand your data.