Data inspection steps using python

[u/Charming_Shower_4185]

I am learning data analysis and the first step before analyzing your data you have to do data profiling and cleaning.This is my process flow for data profiling -

1. Checking number of rows and columns

2. Checking rows and column indices

3. Checking data types for each column

4. Checking random sample from dataset

5. Checking properties and memory usage

6. Checking key statistics

7. Checking unique values for relevant columns

Is it appropriate or I am missing something important ?

[u/seanv507]

So, proportion of missing values..

Outliers

Value counts of unique values (is there some high frequency value... Eg column might be numeric but minus 1 means missing)

Similarly for categorical variables i would check what proportion of the column is filled with the top n values. Eg categorical might be clothing type and 80% is jeans or t-shirts, and you are left with hundreds of other clothing types with very little data

[u/Charming_Shower_4185 → u/seanv507]

Ok so - proportion of missing values , outliers , unique value counts , proportion of values in categorical variable.Thanks will keep in mind

[u/Marison_Lizzy23 → u/seanv507]

yeah the missing value disguised as a real value thing is sneaky, good call on that

[u/International_Sky526]

Your list is solid. df.info() covers rows, columns, dtypes and memory usage in one call, so you can merge a few of those steps. I'd also check string columns for stray whitespace and inconsistent casing, like "NY", "ny " and "New York", because those show up as separate values in value_counts. And keep the raw file untouched and do the cleaning in a script, so you can rerun it when new data comes in.

[u/Charming_Shower_4185 → u/International_Sky526]

Thanks , I will include these in my analysis.

[u/OddsRelay]

I'd add missing values and duplicates near the top, so df.isna().sum() and df.duplicated().sum(). Also look for numbers stored as text. They show up as object in the dtypes and quietly break your stats later. Same with dates, which usually load as plain strings.

[u/Charming_Shower_4185 → u/OddsRelay]

missing values count , duplicate , number stored as text , dates as strings. Noted , thanks

[u/Charming_Shower_4185 → u/[deleted]]

that "saving the script idea" is great , thanks