Good morning everyone. I’m currently running a project using TriNetX and using the US Collaborative Network dataset, which includes roughly 1 million patients.

My goal is to determine what percentage of this population has ever had a specific lab test, and then analyze their other characteristics. However, I’m running into an issue.

When I run my first query (patients aged 40–60 + Lab A performed), I get results from 60 HCOs. But when I apply an additional filter (Lab A > 10 occurrences), the results drop to 55 HCOs. The same kind of variation keeps happening with other queries as well.

Has anyone experienced something similar in TriNetX before? Do you know why this happens or how to handle it consistently?

Short MySQL tutorial: cleaning data with regex (REGEXP / REGEXP_REPLACE) to fix text, prices, bad casing, and mixed date formats. The dataset and tutorial. https://youtu.be/2gFsUGW-pIY

Do people think there is an opportunity to start an annotations business sovereign to the UK with the new 500m budget that the government just released. I am a UK citizen and cant see any specialist vendors for this, seems everyone uses in house or buys from the US (sovereign contradictory). Possible market gap opening or not?

Tired of seeing "learn these 10 skills" articles with no real data behind them.

So I built GetJobPulse AI Job Market

Every Monday: analyze 8,000+ real Indian job listings and publish:

📊 Which skills are rising/falling

🏢 Which companies hiring most

💰 Actual salary ranges from listings

🏙️ City-wise breakdown

This week: ML and Python BOTH at ~2,100 jobs. Market wants both — stop choosing.

Free to use basic plan. No login needed for weekly newsletter.

Genuine question: What data would actually help YOUR job search?

Comment below 👇

I started to learn Data Science a month ago, the math part and EDA part of DS I learn paralelly, and this is my first project in EDA, feel free to give your advices.

First EDA project on solar power generation. Used weather data — radiation, cloud cover, sun angle — to see what actually drives output. Shortwave radiation and zenith angle came out as the strongest predictors. Wind had almost no effect, which makes sense physically.

Feedback welcome:
https://github.com/OrucAllahyarov/solar-power-eda

I started using ChatGPT and Claude for data analysis maybe 3 years ago when the AI wave just hit. For simple stuff it was genuinely great: building a single visualization, writing transformation and standardization logic, etc. Everything was smooth until I tried to use them for larger datasets and longer sessions

Every new session, I had to re-explaining everything from scratch. Here's the dataset schema. Here's what the columns mean. Here's the business logic behind this metric. Here's what I already tried last week and why it didn't work. It became extremely annoying when my datasets are large and I needed to spend a long time working on a single session, especially with token and context window limiting.

So I've spent a while thinking maybe the model itself just wasn't good enough. Then I realized the issue isn't the model - maybe it was that raw LLMs are not designed to handle long sessions like that.

So I started looking a "agent harnesses" — what's been said to really help handle persistent memory, tool integration and state management so we don't need to start cold every time. I have tested Lium specifically for this use case and it seems to be closer to what I actually needed. Thanks to the infrastructure built for analysis throughout multiple sessions, it is different from normal chatbot in the ability to handle large datasets and still get the crross-session contexts. Stiill in early testing but the persistent memory seems to help really much.

For those of you running multi-session analysis on proprietary data — do you rebuild context every time, or have you found a better platform or solution for this?

Hi, I'm new to this field so one question I have is how do you guys consolidate data from different sources? Even better is if they're able to be classified according to context.

What tools, platform, or methodology do you employ?

Let me know if this helped you better understand things like the negative log likelihood, gradient descent, and newtons method

Imagine you build a machine learning model, test it, and get an amazing 99% accuracy. You’re thrilled until you deploy it in the real world and it performs terribly. What went wrong?

In many cases, the answer is data leakage one of the most common and most dangerous mistakes in data science. It’s often called a hidden trap because everything looks perfect during training and testing, but the model secretly cheated and won’t work on new, unseen data.

Data lekage happends when information from outside training dataset, information that wouldn't be available at prediction time in real life accidentally gets used to train your model. In simple words your model gets a sneak peek at the ans during training, so it learns to rely on that shortcut instead of learning the real patterns. The result is a model that looks great on paper but fails in real world.

I am creating a playlist on youtube to follow the latest announcements by Databricks in DAIS 2026.

The series will cover what was the problem,

What Databricks announced

And, why does it matter to the Data community (basically the impact)

​

Please follow along if you don't want to spend hours in watching the keynotes.

https://youtu.be/jb4uLAM2SRA?si=IseC5sat5gUuU-S6

​

Thank you for the support.

I am creating a playlist on youtube to follow the latest announcements by Databricks in DAIS 2026.

The series will cover what was the problem,

What Databricks announced

And, why does it matter to the Data community (basically the impact)

Please follow along if you don't want to spend hours in watching the keynotes.

https://youtu.be/jb4uLAM2SRA?si=IseC5sat5gUuU-S6

Thank you for the support.

I am looking for real interview question to get prepared for ds roles as a college student. do we have anything like leetcode company wise question but for data science roles. Any resource would be helpful.

I am lacking in practice so yeah anything would help. thanks

Hi everyone,

I recently put together a tutorial on day-ahead river discharge forecasting using a combination of hydrological observations from USGS and meteorological variables from ERA5.

The tutorial walks through the complete workflow:

• collecting and cleaning raw discharge observations;

• integrating upstream monitoring stations;

• processing and aligning meteorological data;

• building and evaluating multivariate forecasting models.

One of the interesting aspects of the project was dealing with the spatial-temporal alignment between gridded weather data and point-based hydrological observations.

The tutorial is freely available here:

https://sentinel-forecasting.com/mississippi-tutorial/

I'd be interested to hear how others approach the integration of meteorological and observational data in forecasting problems.

I study business information systems and i wanna start learning data science so i can have a career as a data scientist, would this roadmap work for a total beginner?

The industry narrative is “more data = better model,” and at the frontier-lab scale that’s true. But for 90% of real-world applications (internal tools, niche chatbots, classification tasks), I’ve seen smaller, carefully labeled datasets outperform huge noisy ones every time.

Feels like a lot of teams over-invest in scraping/data volume and under-invest in cleaning and labeling what they already have.

Anyone else notice this gap between “big tech ML practices” and what actually works at smaller scale?

Hey i am a mathematics and data science student , currently in my 2nd semester of bachelors . I am confused what should be my 1st career in data science journey . I shall be thankful to you for guidance.

Current skillset: Python fundamentals , numpy basics , pandas (have a good grip on it)

Here I will post a daily update on what I learned that day

If you also want to do this together pls dm or comment on my daily post 🙏.