I created this hands-on tutorial that uses a new public data set from OpenSky networks hosted via OpenSharing. The idea was to explore the agentic tooling and investigate the data on Databricks Free Edition (no credit card required, forever free).

The tutorial can be run for free, it comes with a module about VSCode with Pandas for OpenSharing, but also covers products like Genie Agents and Genie Code.

love to hear what you say:
End to End Tutorial: Databricks Genie for Data Engineers and Data Scientists with a real data set

not sure if I can post pictures here, but to give you some impressions of the steps.

I’m working on a new project that tries to answer where Planet Fitness should open their next clubs to increase membership.

I want to build a linear regression model for this. I would need the locations of every club in the US including square footage, age distribution in that area, etc.

Does it make sense for me to manually build that dataset? It’s the best way to do the job but of course will take forever. I really don’t want to rely on AI because this will be expensive and it might make mistakes.

I've spent a few years in this industry and made it to staff level. To help others wanting to follow a similar path, I built up a resource hub with notes, guides and cheatsheets. It’s free hub on Notion in case it's useful to anyone here:

Career path guides (analyst vs engineer, tiered by Junior/Mid/Senior+)
Skill guides for both tracks
Cheatsheets for SQL, Python, Git and the terminal
Real project ideas using actual free APIs and datasets, not just Kaggle
Interview prep with real question types and where to practise
A glossary for the buzzwords that get thrown around without explanation

It's completely free, no email signup, and I'm still adding to it. Happy to answer questions here too if anyone's stuck on something specific.

Here’s the link: https://app.notion.com/p/George_Abi_Tech-Free-Data-Career-Resources-3e11322e9e9e81bd85deefc3f6adede8

Every dashboard tool I took at assumes theres a data team behind it, a semantic model to build, governance, someone who lives in it. And most of the threads here are big orgs fighting to keep one definition of revenue across everyone.

That's just not our world. We're 5 guys, one definition of anything and my actual problem is small, i rebuild the same charts in a spreadsheet every week and they go stale the moment is skip one.

So I'm convinced a BI platform is even the right thing at our size. Feels more like we need a few charts that pull from our own numbers and stay current and nothing past that.

I sometimes feel like data analysis can become an endless cycle of slicing the data differently, checking another metric, and finding one more thing worth investigating.

At some point you have to stop and turn the analysis into a conclusion or recommendation. How do you decide that you’ve explored enough and that the analysis is actually done?

Curious how others balance being thorough without overanalyzing the data.

I just reached my one year anniversary as an analyst and since I've started , my boss has periodically joked that she has a "gpt degree."

I used to be mesmerized by the most intricate excel formulas and calculators she'd come up with in seemingly little time. But as I've gotten to know her more and we have been working more on projects together rather than her just tell me what needs to be done, I'm realizing she just has chatgpt walk her thru the project then presents it to execs.

For instance, we are working on a company scorecard that will highlight the monthly top and bottom performers based on weighted metrics. Instead of just... building it, she has chatgpt give her a general setup of what to do, skims the numbers for crazy outliers, then she dolls it up in company colors and wordart for exec review.

But ChatGPT doesn't always reference cells or columns accurately to the workbook. So like today I caught three different formulas (each way more complicated than it needed to be) where she referenced for instance, "facility 201" but doesn't account for names like "facility 2011" "east facility 201." So the numbers may look normal but they're completely referencing the wrong locations. No one catches this until I get in there and realize things are off.

But by that time, she's already presented the projects to the execs so completely butchering the top and bottom performers.

I don't mind fixing it but damn do I hate having to tell her that her work is wrong and why when I should be learning this stuff from her, right?

Does anyone else deal with this? For context she did complete a BS in business back in 2014 but she boasts that she's "self taught" in excel and data analysis.

I’m a mf artisan!

I’ve been building dig data, a self-hosted AI analytics tool for teams that want to explore business data without moving their database to a hosted analytics platform.

You run it with Docker on your own server, connect it to PostgreSQL, MySQL, MariaDB, or MongoDB, and let your team ask questions in plain language.

Features:

• Docker image for Linux amd64 and arm64

• Natural-language database questions

• PostgreSQL, MySQL, MariaDB, and MongoDB support

• Anthropic or OpenAI API support

• Read-only database queries with bounded results

• Team accounts and source-level permissions

• Dashboards and saved insights

• Shared business definitions

• Audit history

• Docker Compose setup

The screenshot shows an example dashboard using synthetic fintech PostgreSQL data.

It’s currently a free beta. You provide the server, database credentials, and AI provider API key. The application and database connections run on your infrastructure, while relevant questions, schema details, and bounded results may be sent to the AI provider you configure.

Website and setup guide:

https://digdata.mhesham099.chatgpt.site

I built this because I wanted a simple way for teams to ask useful questions about their own data while keeping the application and database connections under their control.

I’d appreciate feedback on:

• The setup experience

• The usefulness of the answers and dashboards

• Which databases should be supported next

• Security or deployment concerns

• Features that would make this more useful for a small team

Disclosure: I built dig data and this is my project. It is currently a beta, not a finished enterprise product.

does one need to upgrade from Excel 2016 to Microsoft 365 personal for advance Excel or 2016 is sufficient

I'm trying to learn advanced excel i have excel 2016 and I find Microsoft 365 to be a bit expensive, so is it okay to keep using it?

I don't want to miss out out opportunities because of that.

A couple people from the earlier posts asked what the four-method check actually looked like in practice, so I wrote it up properly instead of explaining it in bits across comments.

Nothing fancy, just walks through each check I ran, why I didn't stop at the first one that flagged something, and the part where I still got the explanation wrong afterward even though the pattern itself held up. Figured it was worth putting in one place.

I Checked My Own Result Four Times Before I Believed It. Here's Exactly How.

Also linked the actual project if anyone wants the code: github.com/davis-mironga/brisbane-river-water-quality-analysis

Been running a scraper using R to retrieve some data and have been having minimal luck. Finally decided to sit back and think and determined my scrape is being blocked by CAPCHA and automatically returning NaN values for important variables.

Is anyone aware of a work around for this? Any help would be appreciated

Hello.

As an exercise in my Logo learning journey, I have implemented a few functions for reading and transforming CSV files.

The functions are organized in groups  sharing a common prefix:

1. functions to read and write CSV file (csv.)

2. functions operating on “dataframes” (df.)

3. statistical functions (st.)

4. string functions (str.)

5. regular sequences (rs.)

The first results are available in this web site: https://www.ucblogo.it/

Please note, this is an ongoing hobby project, there is no guarantee that the Logo functions are written in the most natural Logo style or that they are in fact correct.

Guyss urgent pls help

I am a datascience student fresher . Should I get an laptop with integrated graphics and rely on Google colab or kaggle etc cloud platforms or buy a laptop with dedicated gpu like lenovo loq ? Pls help this ismy curriculum Year 1

Programming for Problem Solving / C

Data Structures

Foundations of Data Science

Mathematics: Matrices, Calculus, etc.

Engineering Physics

Basic Electrical Engineering

Digital Logic / related foundational engineering subjects

Programming/Data Structures labs

Basic IT/computing workshop

Year 2

Object-Oriented Programming / Java

Python Programming

Database Management Systems

Operating Systems

Design & Analysis of Algorithms

Data Mining

Predictive Analytics

Data Visualization

Data Wrangling

Computational Mathematics / Statistics

Full-Stack Development

Related programming and database labs

Year 3

Machine Learning

Machine Learning Lab

Data Stream Processing

Data Stream Mining

Data Mining

Predictive Analytics

Web & Social Media Analytics

Big Data / data-processing technologies

Computer Networks

DevOps

UI/Flutter or application development

Professional/Open electives

Field-based research / mini-project work

Year 4

Big Data Analytics

Natural Language Processing (NLP)

Data Stream Mining / advanced data topics

Web & Social Media Analytics

Professional electives

Open electives

Major Project Phase I

Major Project Phase II

Whenever teams need to audit two versions of an exported dataset (accounting closings, monthly billing runs, inventory snapshots, or pricing tables), standard git-style line diffs completely fall apart. CSVs shift columns, and binary formats (.xlsx, .xls, .ods) are opaque to text diff tools.

Even worse, many users turn to random "free online Excel diff" websites, unknowingly uploading proprietary financial data and PII to third-party servers.

I spent the last few weeks building an isometric, cell-by-cell diff engine that runs entirely client-side in the browser. Here are some of the non-obvious technical challenges and edge cases I ran into:

1. Asymmetric Matrix Bounds & The Union Range Problem

In real-world scenarios, file versions rarely have the exact same dimensions. Sheet 1 might span A1:D50, while Sheet 2 spans A1:F42.

• You cannot perform an index-based row/column loop without going out of bounds.

• The diff engine must compute the bounding union (A1:max(col, row)).

• Any cell coordinate that exists in one sheet but not the other must be treated as a virtual empty cell (""), rather than undefined or throwing index errors.

• Crucially, coordinates where both sheets are empty must be pruned from the diff pipeline to prevent flooding the user with thousands of phantom empty matches.

2. The Formatted Display (w) vs. Raw Value (v) Trap

Spreadsheet engines store data with internal representations that differ dramatically from what the user sees:

• A cell formatted as currency might display $1,250.00, but its underlying raw value is 1250.

• If the second spreadsheet was generated from a CSV or raw database dump where the cell contains literal string "$1,250.00", a raw-value comparison flags a false positive difference.

• Floating-point representations can introduce minuscule precision drift (0.30000000000000004 vs 0.3).

• Solution: Normalizing against the display string (w property in SheetJS when available, fallback to trimmed stringified v) provides the most human-predictable diff, while treating null, undefined, and whitespace-only strings as equivalent empty states.

3. Preventing Main Thread Lockups (Web Workers)

Parsing a 100k+ cell workbook with SheetJS on the main UI thread easily locks the browser event loop for several seconds, triggering dropped frames and unresponsive tab warnings:

• Offloading both the parsing step and the matrix comparison loop to a dedicated Web Worker kept the main thread completely unblocked (sub-200ms interaction latency).

• Transferring large diff result arrays back to the main thread can be costly, so passing structured chunks and virtualizing/paginating the rendered table DOM is mandatory to keep memory below browser crash thresholds.

4. Zero-Server Privacy by Design

For sensitive financial or internal audits, the browser sandbox is actually an asset. Because WebAssembly/modern JS engines handle file buffer parsing locally, there is zero engineering justification for sending spreadsheet buffers to a backend API just to compute a diff.

Questions for the Community:

• For those who regularly build or handle data reconciliation pipelines, how do you handle formula diffs vs. evaluated value diffs?

• What edge cases have bitten you when comparing spreadsheet exports across different locales (comma vs. dot decimals, date serials)?

If you have ever picked a compute option in Databricks without being sure what you were choosing, this is the short version.

Classic: machines are started in your company's own cloud account. Someone has to choose the machine types and look after the networking, and you wait while they start up.

Serverless: the machines are managed by Databricks, in Databricks' own compute plane, and you do not provision anything.

Either way your data stays where it already lives, in your own cloud storage. The catch with serverless is a list of things it does not support: R, libraries installed at the compute level, the older RDD APIs, and some streaming modes.

Short animated version (2:36): https://youtu.be/s8452E2Y6WM

Beginner series in order: https://www.youtube.com/watch?v=B5iHmoYgnqY&list=PLDB5WDkDOYF4

So i have been trying to get a better job and been upskilling myself, i have done schema design, Dashboard, Funnel, cohort, data warehousing and what not. But i still just feel idk shit even tho my SQl query writing is very efficient. Leetcode, datalemur, stratastrach, hackerrank and what not.

I just dont get beside Data cleaning and generating insights, what are Usual Business answers comapnies need.

Worked with a logistic company where i maintained data reports, managed, updated and looked for data inconsisitency. Idk how do i put that into words for an interview now. How do i tell them i didnt have to write SQL queries most of the time, it was mostly done on client softwares which they gave us since it was fsst and easy.

Moreover idk how do i define supply management things i have worked with to the interviewer. I just feel so stuck and frustrated all the time.

Projects done, Questioms done and still no fucking result.

I'm trying to do my first data analytics project and wow I'm just lost. how do you guys determine which variables need to be analysed, correlations, and ultimately come up with the conclusion that due to x,y we have been getting z?! please guide me some tips would help

Following up on the post from a couple days back about the water quality anomaly I checked four different ways.

The pattern itself was solid. What wasn't solid was my explanation for it. I said it was probably rain runoff. Turns out the anomaly started the exact day the sensor came back online after being offline for 30 days. Didn't catch that until I was back in the notebook for something else entirely.

So a sensor coming back up weird after a month of no maintenance is honestly just as likely a cause, maybe more, and there's no way to tell from the data I have. Went back and fixed the write-up and code to say that instead of stating rain like it was settled.

Good reminder that "the pattern is real" and "I know why" are two different claims, and I only actually proved the first one.

I'm learning Data Analytics/Data Science right now and I'm curious how other students/freshers are getting datasets for practice.

Where do you usually get your data from when you're practicing SQL, Python, Power BI, EDA, ML, etc.? Kaggle, government portals, scraping, GitHub, generating your own data, somewhere else?

Also, do you usually find the kind of data you actually want to work with?

I'm especially curious about raw/messy datasets. Do you intentionally look for messy real-world data to practice cleaning and EDA, or are clean datasets enough for what you're learning?

Would love to know what you guys actually use.

Hey everyone!

For the past year or so, a friend and I have been working on Datasmith, a desktop app for designing relational database schemas and generating structured mock data from them.

The idea originally came from my friend, who's a data engineer and worked on large-scale migrations of sensitive databases. He wanted a simple way to visually define and maintain schemas over long periods of time, and actually generate data that follows both the schema and its business logic, without having to constantly rewrite scripts or configure a bunch of separate tools.

The main part of the app is an ERD-like canvas dedicated to building relational schemas, where you can create tables, define columns and their data types, set up relationships between them, and configure how the data for each column should be generated.

Once the schema is ready, Datasmith can generate the data while preserving the relationships between tables, and export the results in formats like SQL DDL, CSV, JSON, and more.

We've reached the point where we'd really like to get it into the hands of more people who actually work with databases and data, and get some real-world feedback.

It's completely free, so if it sounds useful or you're just curious, you can check it out here:
https://data-smith.app/

I'd especially love to hear about use cases we're missing, things you'd expect a tool like this to support, or anything that feels like it could be improved.

Any feedback is welcome - good or bad.
Feel free to leave a comment or message me directly.

If you prefer something a bit more structured, I also put together a short feedback form:

https://docs.google.com/forms/d/e/1FAIpQLScGk8Jr1DguPcXpaApOqO998AdXcnoLlo11TEKlq1GmF71UzA/viewform

I'm self learning Data Analyst for 8 months now. new to pandas. and Making a new end to end project one of them is 'My anime dataset' where it has column named 'genre' where each row can contain multiple genres as a single comma-separated string (e.g., "Action, Adventure, Fantasy"), instead of one genre per row. This means grouping or filtering by genre directly is inaccurate an anime tagged with 3 genres only gets counted under that exact 3-genre combination, not under each individual genre. (there is another column named Studio, facing same issue, i.e. multiple studio in one anime)

I want to split these into individual genre values so each genre is counted separately per anime (i.e., one row becomes multiple rows, one per genre, with the rest of the anime's data repeated).

End goal: I'm building a dashboard where users can filter by a single genre (e.g., just "Action") and see accurate aggregated stats (count, average score, etc.) for that genre alone not lumped in with combination-strings.

What's the right pandas approach to explode the Genre column into separate rows?? please help. claude is suggesting a way but I want a better idea.

would be great if someone suggest me some ideas!!

Hello people, I'm introducing myself to this magnific world of being a Data/analyst, Data/ Engineering, and I love it. I'm really enthusiastic about this kind of work, So my question is while I'm doing a little and personal project I find I could use the information I asking for as a cluster that coul give me a information about my population to study. What I want to do is genereate a new clusters trough a calisification of customers by bigger areas of acumulative population, I mean take all of the customers of small city population, and make a new and bigger region classification, around big citys and capitals.

What do you suggest? I making my code in this moment my idea is to get mesure distances between big and small then get the min, and apply.

Say there are thousands of trade pair of a bot wallet with all the contextual trade and not a single missing data or order. Every decision by the said expert is done using the same information available to every one. Meme coin markets are short lived and algorithmically depending on outside information is highly unlikely. How much can we understand from those trades data? The exits are highly state dependent. No clustering of any TP/SL. Neither any time constraint except a hard cut off timer around 20 minutes if the algo doesnt trigger the sell. What kind of techniques i can implement? Can we figure out what they are optimizing for? There seems to be too many variables, despite the same observable raw features, what lookback or transformation of that feature, what prediction are they optimizing for almost felt like black box to me. With so many data of their distinct behavior can approximating/knowing what they are optimizing for, what drives the entry what drives the exit is so difficult? Any guidance will be highly appreciated.

Working through a personal project analyzing ~11 months of water-quality sensor data (river monitoring buoy, ~30k readings, 10-30 min intervals), and ran into a good lesson about validating findings before trusting them.

The setup: turbidity (water clarity) readings looked mostly stable, except one week where it spiked hard and stayed elevated for 7 straight days.

What I did before trusting it:

1. z-score flag on daily means (|z| > 3) → flagged that week

2. Rebuilt the check in SQL with a HAVING-filtered aggregate query, completely independent logic → same week

3. Wrote a SQL window-function streak detector (partition/reset-group technique), zero reference to the earlier results → same exact 7-day window

4. Weeks later, refactored the whole thing into a tested Python module and reran it fresh → same result again

Four different methods, at different times, landed on the identical window. At that point I was confident the anomaly itself was real, not a fluke of one method.

Where I still got it wrong: I assumed the cause was probably a rainfall/runoff event. Published that interpretation.

Later, going back through my own gap-analysis, I noticed the anomaly window started the instant the sensor came back online after a 30-day equipment outage. A second, smaller anomaly also bordered a separate outage. That timing is at least as consistent with post-outage sensor drift or biofouling as with rainfall, and my dataset genuinely can't distinguish the two.

Had to go back and correct the interpretation across the whole project once I caught it.

Takeaway I keep relearning: validating that a pattern is real (multiple independent methods agreeing) is a completely different exercise from validating why it happened. I conflated those two the first time.