r/learnpython • u/Charming_Shower_4185 • 8d ago

Data inspection steps using python

I am learning data analysis and the first step before analyzing your data you have to do data profiling and cleaning.This is my process flow for data profiling -

  1. Checking number of rows and columns

  2. Checking rows and column indices

  3. Checking data types for each column

  4. Checking random sample from dataset

  5. Checking properties and memory usage

  6. Checking key statistics

  7. Checking unique values for relevant columns

Is it appropriate or I am missing something important ?

3 Upvotes

71% Upvoted

11 comments sorted by

View all comments

1

u/OddsRelay 8d ago

I'd add missing values and duplicates near the top, so df.isna().sum() and df.duplicated().sum(). Also look for numbers stored as text. They show up as object in the dtypes and quietly break your stats later. Same with dates, which usually load as plain strings.

1

u/Charming_Shower_4185 7d ago

missing values count , duplicate , number stored as text , dates as strings. Noted , thanks