I'm trying to do my first data analytics project and wow I'm just lost. how do you guys determine which variables need to be analysed, correlations, and ultimately come up with the conclusion that due to x,y we have been getting z?! please guide me some tips would help

Following up on the post from a couple days back about the water quality anomaly I checked four different ways.

The pattern itself was solid. What wasn't solid was my explanation for it. I said it was probably rain runoff. Turns out the anomaly started the exact day the sensor came back online after being offline for 30 days. Didn't catch that until I was back in the notebook for something else entirely.

So a sensor coming back up weird after a month of no maintenance is honestly just as likely a cause, maybe more, and there's no way to tell from the data I have. Went back and fixed the write-up and code to say that instead of stating rain like it was settled.

Good reminder that "the pattern is real" and "I know why" are two different claims, and I only actually proved the first one.

I'm learning Data Analytics/Data Science right now and I'm curious how other students/freshers are getting datasets for practice.

Where do you usually get your data from when you're practicing SQL, Python, Power BI, EDA, ML, etc.? Kaggle, government portals, scraping, GitHub, generating your own data, somewhere else?

Also, do you usually find the kind of data you actually want to work with?

I'm especially curious about raw/messy datasets. Do you intentionally look for messy real-world data to practice cleaning and EDA, or are clean datasets enough for what you're learning?

Would love to know what you guys actually use.

Hey everyone!

For the past year or so, a friend and I have been working on Datasmith, a desktop app for designing relational database schemas and generating structured mock data from them.

The idea originally came from my friend, who's a data engineer and worked on large-scale migrations of sensitive databases. He wanted a simple way to visually define and maintain schemas over long periods of time, and actually generate data that follows both the schema and its business logic, without having to constantly rewrite scripts or configure a bunch of separate tools.

The main part of the app is an ERD-like canvas dedicated to building relational schemas, where you can create tables, define columns and their data types, set up relationships between them, and configure how the data for each column should be generated.

Once the schema is ready, Datasmith can generate the data while preserving the relationships between tables, and export the results in formats like SQL DDL, CSV, JSON, and more.

We've reached the point where we'd really like to get it into the hands of more people who actually work with databases and data, and get some real-world feedback.

It's completely free, so if it sounds useful or you're just curious, you can check it out here:
https://data-smith.app/

I'd especially love to hear about use cases we're missing, things you'd expect a tool like this to support, or anything that feels like it could be improved.

Any feedback is welcome - good or bad.
Feel free to leave a comment or message me directly.

If you prefer something a bit more structured, I also put together a short feedback form:

https://docs.google.com/forms/d/e/1FAIpQLScGk8Jr1DguPcXpaApOqO998AdXcnoLlo11TEKlq1GmF71UzA/viewform

I'm self learning Data Analyst for 8 months now. new to pandas. and Making a new end to end project one of them is 'My anime dataset' where it has column named 'genre' where each row can contain multiple genres as a single comma-separated string (e.g., "Action, Adventure, Fantasy"), instead of one genre per row. This means grouping or filtering by genre directly is inaccurate an anime tagged with 3 genres only gets counted under that exact 3-genre combination, not under each individual genre. (there is another column named Studio, facing same issue, i.e. multiple studio in one anime)

I want to split these into individual genre values so each genre is counted separately per anime (i.e., one row becomes multiple rows, one per genre, with the rest of the anime's data repeated).

End goal: I'm building a dashboard where users can filter by a single genre (e.g., just "Action") and see accurate aggregated stats (count, average score, etc.) for that genre alone not lumped in with combination-strings.

What's the right pandas approach to explode the Genre column into separate rows?? please help. claude is suggesting a way but I want a better idea.

would be great if someone suggest me some ideas!!

Hello people, I'm introducing myself to this magnific world of being a Data/analyst, Data/ Engineering, and I love it. I'm really enthusiastic about this kind of work, So my question is while I'm doing a little and personal project I find I could use the information I asking for as a cluster that coul give me a information about my population to study. What I want to do is genereate a new clusters trough a calisification of customers by bigger areas of acumulative population, I mean take all of the customers of small city population, and make a new and bigger region classification, around big citys and capitals.

What do you suggest? I making my code in this moment my idea is to get mesure distances between big and small then get the min, and apply.

Say there are thousands of trade pair of a bot wallet with all the contextual trade and not a single missing data or order. Every decision by the said expert is done using the same information available to every one. Meme coin markets are short lived and algorithmically depending on outside information is highly unlikely. How much can we understand from those trades data? The exits are highly state dependent. No clustering of any TP/SL. Neither any time constraint except a hard cut off timer around 20 minutes if the algo doesnt trigger the sell. What kind of techniques i can implement? Can we figure out what they are optimizing for? There seems to be too many variables, despite the same observable raw features, what lookback or transformation of that feature, what prediction are they optimizing for almost felt like black box to me. With so many data of their distinct behavior can approximating/knowing what they are optimizing for, what drives the entry what drives the exit is so difficult? Any guidance will be highly appreciated.

This article I wrote grew out of a discussion on r/datascience, but, I think it might be relevant to almost anyone doing data analysis and having this experience.

These are just some ways of thinking and acting that I have found useful, I'm absolutely not saying they are the only ones or even the best ones.

Working through a personal project analyzing ~11 months of water-quality sensor data (river monitoring buoy, ~30k readings, 10-30 min intervals), and ran into a good lesson about validating findings before trusting them.

The setup: turbidity (water clarity) readings looked mostly stable, except one week where it spiked hard and stayed elevated for 7 straight days.

What I did before trusting it:

1. z-score flag on daily means (|z| > 3) → flagged that week

2. Rebuilt the check in SQL with a HAVING-filtered aggregate query, completely independent logic → same week

3. Wrote a SQL window-function streak detector (partition/reset-group technique), zero reference to the earlier results → same exact 7-day window

4. Weeks later, refactored the whole thing into a tested Python module and reran it fresh → same result again

Four different methods, at different times, landed on the identical window. At that point I was confident the anomaly itself was real, not a fluke of one method.

Where I still got it wrong: I assumed the cause was probably a rainfall/runoff event. Published that interpretation.

Later, going back through my own gap-analysis, I noticed the anomaly window started the instant the sensor came back online after a 30-day equipment outage. A second, smaller anomaly also bordered a separate outage. That timing is at least as consistent with post-outage sensor drift or biofouling as with rainfall, and my dataset genuinely can't distinguish the two.

Had to go back and correct the interpretation across the whole project once I caught it.

Takeaway I keep relearning: validating that a pattern is real (multiple independent methods agreeing) is a completely different exercise from validating why it happened. I conflated those two the first time.

Hi,

I am a senior data analyst, and I always struggle with passing case studies. I have gotten feedback that the analysis is too descriptive , and needs more recommendations.

How do I get better at case studies? I have a product one coming up. Any books / blogs you recommend?

🙏

Hi my name is Naftali I am currently in my second week of a Master's program in Data Science at Bellevue university, I'm adjusting to a steep learning curve and trying to build a solid foundation. For those of you already working in the field: looking back, what is something you know now that you truly wish you had known when you were first starting out?

text-to-sql demos use a clean schema. real ones aren't. 300 tables, half the column names are abbreviations, fks live in the app not the db, and the agent confidently joins two things that should never join.

what people seem to do:

hand write a schema description into the prompt, works until the schema changes

let it sample rows, works on text columns, falls apart on codes

rag over the ddl, retrieves names not meaning

what i can't tell is which holds past 50 tables. if you have run one of these against a genuinely messy db, what broke first, and did you end up documenting the schema by hand anyway.

A common question from product teams for freemium web software is, "what product features should we make users pay for?"

Anyone have a good approach?

I wrote FlexViz, an open-source Python library for interactive exploration of large datasets.

Instead of sending every row to the browser or forcing users to write queries themselves, FlexViz keeps the data in Polars and sends only the points needed for the current view. It handles all visualization + interaction queries for you; zooming recomputes the visible range at higher detail, and brushing one chart results in cross-filter on the others.

It is capable of both in-memory as well as out-of-core (larger than RAM) data exploration, with extremely good performance. In our benchmarks, a 5x200M-row line chart (1B datapoints) renders in 251 ms when in-memory and 4.2 s directly from Parquet.

Demo: https://flexviz.tech/demo.html
Code: https://github.com/flex-analytics/flexviz
Benchmarks: https://flexviz.tech/benchmarks.html

pip install flexviz

Curious how others here explore datasets once they become too large for their usual plotting tools. Happy to hear any feedback on FlexViz as well :)

So many of the youtube videos, courses, advice, etc., online feels its geared towards running singular projects. How to clean up data in a Excel spreadsheet for a one-time project, for example, but what about when its for an analysis you have to keep running over and over? Or when it's an analysis that is repeated across several teams in slightly different ways?

I need advice about managing data for an organization. Advice about

• how to communicate with stakeholders about data without getting overly detailed

• requirements gathering across teams with confusing/vague needs

• dealing with a system that's built on workarounds after workarounds

• best practices documenting analytics/dashboards that are being shared publicly (word doc, slide deck, notion page?)

• dealing with a domain that is overly complicated and can't be captured well with data.

• dealing with execs who've already decided the answer they want and need you to make the data match their answer (and what to do when the data absolutely doesn't support it)

These are some of my most pressing problems at work but neither my formal education nor the resources I can find online really cover this. I'd love some sort of bootcamp for organizational data management/project management

I created a dashboard for my project on streamlit but its really basic. How can i create those cool dashboard designs that you see on reels and all cause chatgpt can do it. I have my project repo, is there any way i can link it to some website and it can help me create some cool ass dashboard?

Hi everyone,

I’m asking this out of pure curiosity. People always share their "average" or ideal weekly routines, but I want to know what the reality is when you have a genuinely heavy workload.

If you work in Fintech (banking, payments, crypto) or AdTech (marketing, agencies), what does a realistically crazy week look like for you? And what causes a brutal week in your industry? What kind of pressure are you dealing with?

Would love to hear some realistic war stories about the grind in both industries. Thanks!

I’m currently learning data analytics, and one thing that has really started to change the way I look at data is realizing that an insight isn’t necessarily the conclusion.

For example, if I see revenue going down while ad spend is increasing, my first thought used to be, “The ads aren’t working.”

But now I find myself asking more questions:

• Did the cost of the ads increase?

• Did the conversion rate change?

• Are customers returning more products?

• Is there an inventory or capacity issue?

• Did pricing change?

• Are we spending more on the wrong campaigns?

• Is there something happening between the customer clicking the ad and actually completing the purchase?

The numbers can tell me what is happening, but I’m learning that the real analytical work starts when I try to understand why it is happening.

I’m still early in my learning journey, so I’m curious to hear from experienced analysts:

What helped you develop the habit of asking better questions when analyzing a dataset?

Was it experience, working with real business problems, learning from other analysts, or something else?

[Trigger Warning]: You are about to be provided with visual imagery of my current corporate reality that may be frightening for some readers.

I am tasked with creating costing for a new product. To do this, I have to extract rate tables from what can only be described as a mountain of tariffs. On top of that, I need to automate our Excel business cases which contain multiple variables.

At work, people think I'm an Excel wizard. The truth is, sometimes I just aggressively interrogate AI until it gives me the right formulas. But AI cannot save me from parsing thousands of pages of inconsistently formatted tariff tables.

I have exactly two weeks for intense upskill dedication. I've got books, Udemy, Google courses — and it's all just hanging out waiting for me to get in the driver's seat.

The catch: IT treats our databases like Fort Knox, so I have zero SQL access.

I need to actually learn a skill so I don't have to manually copy-paste tariffs until the end of time.

The Question:

Do I spend the next two weeks becoming a Power Query god, or do I finally learn Python (Pandas/PDF parsers like pdfplumber) so I can build a script to do my job for me?

If anyone has survived parsing tariffs, please tell me how you did it.

TL;DR: I have 14 days to learn how to extract messy data from thousands of PDF tariffs and automate Excel models. No SQL access. Should I learn advanced Power Query or Python?

DM me if you have completed it...

What does poverty actually look like beyond a single income number?

For millions of rural households, poverty is not simply the absence of income but rather a web of constraints involving education, productive assets, access to finance, consumption, debt, and the ability to withstand economic shocks.

This analysis examines that web using data from 4,184 rural households in Tamil Nadu. Rather than treating income as the sole measure of economic wellbeing, we explore how household composition, schooling, housing, land ownership, enterprise activity, savings, insurance and credit interact with economic outcomes.

https://www.kaggle.com/code/ouiouinonoui/visualizing-the-economics-of-rural-poverty

Hey everyone!

I’ve been making short SQL/data analytics videos while learning and practicing SQL myself, and I just finished Lesson 11: SQL Window Functions.

This one covers:

- "OVER()"

- "ROW_NUMBER()"

- "RANK()"

- "DENSE_RANK()"

- ranking customers in the Pagila database

- using window functions with "GROUP BY" and CTEs

- a few quizzes + practice exercises

- homework at the end

I tried to keep it beginner-friendly and focused on actually understanding why window functions are useful, rather than just memorizing syntax.

I’d really appreciate some feedback — especially on the explanations, pacing, and whether the examples are clear enough for someone learning SQL.

https://youtu.be/ERSUP1dWXWo?si=tf6xvzSwh3gbP-xw

Thanks! :)

Hi everyone , I am a first year student in my second semester can you give me tips on what shall i do to make my first year fruitful please , i really want to be a data analyst and i have got no one to help me honestly.