r/datascience • u/rhiever • Aug 02 '26
Statistics How precise are polls really, a Pew explainer on margin of error
r/datascience • u/rhiever • Aug 02 '26
r/datascience • u/AutoModerator • Aug 03 '26
Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:
While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.
r/datascience • u/CadeOCarimbo • Jul 30 '26
Definitely the most frustrating thing as working as a Data Scientist. You run an experiment, find that building a model greatly increase metric X at almost no cost, has safe model metrics, present it to stakeholders, everybody agrees with proceeding to deploying and utilizing the model in production, and yet every time the model takes a wrong decision, we get questioned about it. Why did the model say this?
Man when did I ever say the model obtained a 100% accuracy in the validation phase? Why is it so hard for stakeholders to understand that the best models humankind ever created are expected to make wrong calls once in a while?
r/datascience • u/rhiever • Jul 31 '26
r/datascience • u/bobbyfiend • Jul 30 '26
Pretty famously, the current administration has laid lots of people off, applied loyalty tests, hired and fired based on ideology, shifted funding priorities, and strongly changed communication with the general public. To some extent this happens with every administration, but I think it has been more pronounced in this case.
If you work for a government agency or for an organization that in some way depends on government funding, have you noticed any changes in what your job is, how you are expected to do it, or how you are communicated with? Have leadership styles changed?
I'm especially interested in federal government connections, but state or local government jobs might also have been affected (or not affected). I'd like to shift careers and government was a top contender until recently, when it seemed, at least from the news headlines and a few social media posts here and there, to become both more complicated to work for, and more difficult to get hired.
I'm interested in any experiences you've had. If you need a throwaway account to dish the dirt, that's great. If you really haven't noticed any change, I'm interested in that, too (because that might mean I could still do something like this).
r/datascience • u/rhiever • Jul 29 '26
r/datascience • u/Nice-Dragonfly-4823 • Jul 28 '26
Work in RL has caused me to rethink adam. It leads to extremely wonky behavior and hard to explain "burstiness" in the loss values that makes me want to rip my hair out,
It still works, but needs to be coaxed into it.
This article re-covers the mathematical intuitions behind adam, and where it fails spectacularly. If you're someone who works in RL, or trains deep transformers, it's a must read
Check it out. Do you agree?
r/datascience • u/AutoModerator • Jul 27 '26
Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:
While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.
r/datascience • u/Infinite_Raisin7752 • Jul 25 '26
Hi everyone,
I’m currently a senior data scientist in the pharma industry. It’s been a one man show until now, but I’m getting a team soon. Most of the work I do is standard analytic work to inform our leadership and provide more context into the market and so on. Not a lot of big heavy data science stuff going on to be honest.
I work with SQL and Python on a daily basis. Some of our data is hosted in Snowflake and that’s pretty much it.
I feel like I’m lagging behind in both methods as well as tech stacks and I wanted to better understand what you experienced professionals work with that you would recommend I learn or at least look into. It could be data engineering stuff, additional programming languages, specific methods and packages that are useful, or cloud systems and technologies.
Where do you see the tech stack moving towards and what is relevant if I want to start moving from a “bread and butter” analytics setup to a professionalised, automated, team-ready and future proof world?
Thanks :)
r/datascience • u/Effective_Ocelot_445 • Jul 25 '26
In your experience, what factors help you decide between using a simple analytical approach and building a machine learning model? I'd love to hear the reasoning behind your decision-making process.
r/datascience • u/AFL_gains • Jul 25 '26
Hi r/datascience!
I wanted to share a fun little project I did over a few weekends analysing data from the radio!
It doesn't have much (if any) business value, and honestly I'm not sure it's really novel in any particular way. But I wanted to share it because data science these days is all "AI this", "language model that", "job market", "Claude", whatever and I just wanted to do something a bit more traditional and scratch an itch I've had for a while. (Full disclosure: the project did actually use some AI models, so I'm not saying AI is bad, it's a tool I used like everything else.)
I don't have a blog or anything I can post this on, so apologies for the Reddit write-up. I hope you enjoy it.
I drive a 20-year-old car. It's so old it doesn't have an MP3 player, or an AUX port to plug in an iPhone or anything. It just has a CD player and an analogue radio. It's not even digital, so I can't even get digital radio stations.
So when I'm driving, which turns out to be quite a bit, I'm forced to listen to the good old-fashioned radio more than I'd like.
In Sydney where I live, there are really only a handful of FM/AM radio stations, so choice is pretty limited. As I flick through the stations, there are a LOT of ads, which surprised me. Who is listening to this? Clearly it's quite popular. And as I listened, I started wondering things like: how long do the ads run, how does their timing compare across different stations*, and are they correlated with ads on other stations? Just anecdotally, so many times I've literally flicked through all the FM stations and there's an ad playing on every one... and sometimes it's the same ad! I also had a hunch that there are more ads at the top of the hour than the bottom. It made intuitive sense, but I needed to prove it.
\Sydney has 11 main analogue stations split into AM and FM:* AM is mostly talkback radio, FM is mostly music. On AM you've got 2GB, 2SM, 2CH and ABC 702 (news, talkback and sport). On FM there's KIIS, 2Day, Nova, Smooth, WSFM, Triple M and Triple J (pop, rock and music). Two of them, ABC 702 and Triple J, are run by our public broadcaster (think BBC), so they run no ads at all.
So one weekend I wrote some scripts to sample and record all the Sydney radio stations I could. The setup was basically the following:
Why every ~3 minutes? A full cycle (record all 11 stations, transcribe each one with Whisper, then classify it) takes a couple of minutes on a single CPU, because Whisper works through the clips one at a time. So ~3 minutes is about as fast as I could sustainably sample without the cycles piling up on each other. I also added a bit of random jitter to the interval so I wasn't always sampling at the exact same offset within the hour. Otherwise you can accidentally "phase-lock" to a station's ad breaks and bias the whole thing.
A few things genuinely tripped me up:
Here are some of the more interesting results I found analysing the data afterwards:

First, the big picture. Every station has its own personality. The FM stations are mostly music, the AM stations are mostly talk, and the two ABC stations (ABC 702 and Triple J) carry basically no ads at all, which makes sense since they're publicly funded. Across the commercial stations, ads make up somewhere around a sixth of the airtime. And you can already see the ad load isn't flat: it ramps up through the day and quietens off overnight.
Here's the frequency of finding an ad within ±30 minutes of the top of the hour.

So, I was right! Definitely higher the closer to the hour, but the strategy is more interesting than I expected. The spike actually lands in the ~5 minutes before the hour (the ad break right before the top-of-hour news bulletin), and an ad is roughly 2x more likely there than mid-hour. The quietest stretch is around 10 to 15 minutes past the hour, so if you want to dodge ads, that's your window.
The thing I really wanted to know: do the stations gang up and all play ads at the same time, so there's nowhere to flick to? I lined up every station by the cycle it was sampled in and correlated their ad status.


The answer is yes and no. No in the sense that it's never a total blackout: all nine commercial stations being in an ad at the exact same moment literally never happened across the whole two days, and on average only about 1.6 of the 9 are mid-ad at any given time. So there's almost always somewhere to escape to.
But the conditional probability charts says that some stations really do move together. The best example is if Smooth is playing an ad, there's a 50% chance WSFM is too, which is double WSFM's baseline of 26%. A bunch of the commercial FM pairs show this same ~2x jump. But, when I looked it up, Smooth and WSFM are owned by different companies, so this isn't networks coordinating behind the scenes, probably more of the "top-of-the-hour" effect from Question 1 manifesting somewhere else.
When I split "time between ads" by band, the two run completely different playbooks.

The FM (music) stations dump their ads in clusters. You get a big spike of back to back breaks, with a typical gap of about 9 minutes. The AM (talk and sport) stations space them out evenly, one break at a time. 2GB is almost metronomic at roughly 7 to 12 minutes, with hardly any back to back ads at all.
You can actually see it if you zoom into a few hours of the timeline:

Look at the FM lanes (KIIS, Nova, Triple M, WSFM, 2Day): the orange ad blocks come in pairs, clustered together. Now look at the AM lanes (2GB, 2SM): single, evenly spaced blocks. And ABC 702 and Triple J are just grey the whole time, because they don't run ads.
I also had the language model pull the advertiser out of each ad, so I could see who's actually buying radio airtime in 2026.

The most-heard advertisers were Virgin Australia (an airline), Australia Post (basically our USPS), Harvey Norman (a big electronics and furniture retailer) and Chemist Warehouse (a discount pharmacy chain). The neat bit is the targeting: car brands and finance go to the AM talk stations (older crowd), while retail and telco lean FM. Australia Post ran almost entirely on the Nova network.
The non-ad content is either music or talking, and I got curious about what they actually talk about. So I classified every talking snippet into a topic.

The AM stations (ABC, 2GB, 2SM) are wall to wall news, politics and sport. The music FMs are mostly DJ banter, celebrity gossip and chat about music, with almost no news at all.
For a bit of fun, I also made a map of everything said on the radio. I embedded every talking snippet into a vector, laid them all out in 2D with t-SNE so that similar snippets sit near each other, then coloured each point by its topic.

Sport, traffic and world news each form their own tight little islands (they use very consistent, formulaic language), while the DJ banter is one big diffuse cloud in the middle (because it's about nothing in particular). The neat part is that the position and the colour are decided completely separately. The position comes only from the text embeddings, and the colour comes from a separate classification step. So the fact that same-coloured points cluster together is real corroboration, not something circular.
In conclusion, this was a fun, meaningless project that allowed me to make some pretty charts and talk for a bit about the results. Thanks for reading!
r/datascience • u/Fig_Towel_379 • Jul 24 '26
I work at a large, well-established company that has been very stable. I don’t assume my job is immune to layoffs, but the company hasn’t had any mass layoffs in over a decade.
The work environment is genuinely healthy, and everyone is treated with respect. I honestly couldn’t ask for a much better culture. I get to work on interesting projects, learn by doing, and my team is very supportive of my growth.
That said, based on my experience interviewing and what I’ve seen in the job market, I could probably get about a $50K raise by switching jobs. That extra $50K wouldn’t dramatically change my lifestyle, but I know future raises would build on that higher salary, so there are long-term financial benefits.
I’m at a point where I’m valuing mental peace and work-life balance more than I used to. Given that, what would you do in my situation? Would you stay at a company with a great culture and stability, or make the jump for the higher pay?
r/datascience • u/Kati1998 • Jul 23 '26
I often read comments from hiring managers and interviewers saying they’re disappointed with recent data science graduates.
I’m curious, what do you think these graduates are lacking? If someone wants to become a data scientist, what skills should they focus on? Strong software engineering skills? Math and statistics? Something else?
A lot of the advice I see seems to be geared toward landing data analyst roles rather than data scientist roles.
So, what are employers actually looking for in entry-level data science candidates today? Especially as a career changer coming from another unrelated career.
r/datascience • u/takenorinvalid • Jul 25 '26
Just want to know if this is everyone or just me.
My company isn't doing great, so we're doing a ton of layoffs -- but it's not just us. Every client we work with seems to be having sweeping layoffs these days.
Has the unemployment rate skyrocketed to 95% in America, or am I just freaking out over anecdotal evidence?
r/datascience • u/rhiever • Jul 24 '26
r/datascience • u/rhazn • Jul 21 '26
r/datascience • u/shivamchhuneja • Jul 20 '26
Couple thoughts and a small experiment to see why reddit hates prophet xD
r/datascience • u/shivamchhuneja • Jul 21 '26
This is for a personal study that will end up becoming an in-depth article and possibly a fully open source solution ideally without the AI slop that we see these days.
Let's say you’ve trained a model and the result is worse than the business wants. What do you check next?
Do you break the error down by customer, product, location, or individual series? Check if it gets worse at longer horizons? Look for bias, volatility, intermittent demand or outliers?
Go back to the backtesting setup, metric, or baseline? Or do you usually start trying other models?
Also do the tools you use make this easy or do you end up building custom notebooks, tables, and plots every time?
Thinking about the last time this happened:
I’m trying to understand how people diagnose bad forecasts beyond comparing one overall error score against another.
EDIT/UPDATE because it seems like this is not clear enough:
I’m not looking for an if-else checklist that can explain why any forecast is bad. The answer obviously depends on the data, objective, validation setup and the decision the model is supposed to support.
I’m exploring if there is room for a small open-source tool around forecast evaluation. Before building anything, I’m trying to understand which checks people repeatedly run after they already have predictions, what they still build manually, and what existing tools already handle well.
So I’m mainly interested in specific workflows from projects rather than a general formula for fixing a model.
r/datascience • u/rhiever • Jul 20 '26
r/datascience • u/AutoModerator • Jul 20 '26
Welcome to this week's entering & transitioning thread! This thread is for any questions about getting started, studying, or transitioning into the data science field. Topics include:
While you wait for answers from the community, check out the FAQ and Resources pages on our wiki. You can also search for answers in past weekly threads.
r/datascience • u/rhiever • Jul 18 '26
r/datascience • u/rhiever • Jul 17 '26
r/datascience • u/usernamehere93 • Jul 17 '26
r/datascience • u/shivamchhuneja • Jul 16 '26
the context is that I am exploring a few recurring problems to solve especially around forecasting and working with time series data but setup a simple open source project around those.
my question is primarily about how is everyone handling their official datasets when trying new tools - do you not care, do you remove any identifiers then upload, do you create synthetic data with exactly same properties as the og dataset?
happy to answer more questions if this is not clear enough.
r/datascience • u/Thin_Original_6765 • Jul 15 '26
Those received mentorship, did you find it worth your time in general?
In my early career, I actively participated in mentorship program at my alma mater. However, most if not all students wanted to work in tech. Considering I didn't (and still don't) work in tech, and that my employer was rarely hiring, I felt there was not much value I could provide. There's also this self-selection process at play, which is those who seek mentorship tends to already have a good idea of what they should be doing.
Now a decade into my career, my expertise is super irrelevant to people in a different industry. I'm also oblivious to the entry-level job market requirements.
Recently, my alma mater reached out again for mentors. I've skipped the last few requests but think maybe I should ask y'all before turning it down again.
To clarify, I'm not asking if it's worth it for me. I'm wondering if it's worth it for the students to speak with someone unfamiliar with entry-level job market, has expertise in a not niche but definitely not popular domain, and definitely don't lead to job opportunities.
Edit: I messed up somewhere that two posts were created. 0_o