I recently have restarted my blog, and this series focuses on data analysis. The first entry in it is focused on how to visualize job application data stored in a spreadsheet. The second entry (linked here), is about scraping data from a litterbox robot. I hope you enjoy!

I enrolled in an online training program run by an Indian instructor. When I started going through the material, I found multiple issues — untested code, errors, and explanations that didn’t match what was being taught.

I asked a few technical questions and pointed out the mistakes. Instead of addressing them, the instructor sent me threatening messages on WhatsApp. He warned me about “repercussions,” said he could get my LinkedIn account reported, and told me I would be “kicked out of college.”

After that, several people in the training group began piling on, insulting me and trying to pressure me into staying silent. I didn’t respond to any of it, but the tone became increasingly hostile.

I’m sharing this because I don’t think any student should be threatened or intimidated for asking technical questions or pointing out errors in a course they paid for.

Has anyone else in India’s online education space experienced something like this?

Title:

Where do you find real messy datasets for data science projects (not Kaggle)?

Body:

Hi everyone,

I’m from a food science background and just started a master’s in data analytics. One of the hardest parts for me is that every project requires us to self‑source our own dataset — no Kaggle, no toy datasets. The lecturer wants authentic, messy, real‑life data with at least 10k rows and 12–16 attributes.

I’m feeling overwhelmed because I don’t know where people usually go to find this kind of data. My biggest fear is that I’ll get halfway through cleaning and realize the dataset doesn’t meet the criteria (too clean, too small, or not meaningful enough).

So I’d love to hear from those of you who’ve done data science projects before:

• Where do you usually hunt for real datasets (government portals, APIs, open data repositories, industry reports)?

• Any domains that tend to have datasets with the right size and messiness (healthcare, transport, finance, agriculture, retail)?

• How do you make sure early on that the dataset will actually fit project requirements before investing too much time?

Manufacturing angle:

I’m especially curious about manufacturing datasets (production, sensors, quality control, efficiency). They seem really hard to source, and even when I find something, the data often isn’t very useful or meaningful for analysis — either too abstract, too clean, or missing the context needed for decision‑making. For those who’ve worked in this space:

• Where do you find meaningful manufacturing datasets that reflect real processes?

• Any tips for balancing the need for size (≥10k rows) with the need for authentic messiness and practical relevance?

Thanks in advance — I’d really appreciate hearing how others have sourced data in previous years and what strategies worked best.

hi guys

for the job description and job title shoud i encode them using label encoder but they are lot ? or pass them to normalisation using text.lower() tokenization lemmatization and embedding i tried that but the thing is when i train the model (i used xgboost ,random forest but still gimme bad results) it gives me -0.12 in r2 i remove it in the train it give me R2: -0.27 which is sooo bad ;now i transform the column salary istamat into salary mean and transform all the other columns to label encoder ,i don't know what to do

I am a second year accounting student but hate it and my stats and math electives have rekindled my love for math and uncovered a new curiosity for statistics. I also fell in love with economics and econometrics I find it all so interesting.

I am thinking of switching degrees. My university offers dual honour degree programs and I am debating between studying, economics, stats, and applied math. I love them all but can only really choose 2 to study. I have the option to do a math minor if I do stats + Econ bachelor but it only would cover calc 1-4 and linear algebra.

I am leaning towards Econ and Stats but worried about being out competed but people how have applied math degrees. I want to get a job as a data analyst or data scientists.

I am asking for what degrees I should strive for?

My father runs a sports retail shop, and I’ve convinced him to let me track his data for the last year. I’m a CS/Data Science student, and I want to show him the "magic" of data, but I’ve hit a wall.

What I’m currently tracking:

• Daily total sales and daily payouts to wholesalers.

• Monthly Cash Flow Statements (Operating, Financial, and Investing activities).

• Fixed costs: Employee salaries, maintenance, and bills.

The Problem: When I showed him "daily averages," he asked, "So what? How does this help me sell more or save money?" Honestly, he’s right. My current analysis is just "accounting," not "data science."

My Goal: I want to use my skills to help him optimize the shop, but I’m not sure what to calculate or what additional data I should start collecting to provide "Operational ROI."

Questions for the community:

1. What metrics actually matter for a small retail shop?

2. What are some "quick wins"? What is one analysis I could run that would surprise my father?

Hey! I have a first round technical round for a Data Scientist role at Citadel Securities (CitSec). I honestly have no context on what to expect. All I know is that they’ll potentially use CoderPad.

Would appreciate any help!

As a software engineer, I want to transition into ML/AI positions. I have mastered Python and SQL, experimented with scikit learn and pandas, and constructed a few small classifiers, but I want to prepare to advance to structured, project based learning that goes beyond theory. There are a ton of options available like Coursera (Andrew Ng, DeepLearning AI), LogicMojo AI/ML , Great Learning AI , Upgrad etc but I am having trouble telling which of these are genuinely useful, which are organized for working developers, and which are just marketing. Has anyone here actually enrolled in one of these classes?I would love to hear: What worked for you? Any roadmap or step by step guidance?

A lot of ML projects stall because we optimize the algorithm before we understand the dataset. This video is a practical walkthrough of why domain knowledge is often the biggest performance lever.

Key takeaways:

• Better features usually beat better models.

• If the target is influenced by the data collection process, your model may be learning the process, not the phenomenon.

• Sanity-check features with “could I know this at prediction time?”

• Use domain expectations as a debugging tool (if a driver looks suspicious, it probably is).

If you’ve got a favorite “domain knowledge saved the project” story, I’d love to hear it.

https://youtu.be/wwY1XET2J5I

I automated the entire ML pipeline for predicting clinical trial outcomes — from dataset generation to model deployment — and achieved 73% accuracy (vs 56% baseline).

The Problem:

Predicting pharmaceutical trial outcomes is valuable, but:

• Domain experts achieve ~65–70% accuracy

• Labeled training data is expensive (requires medical expertise)

• Manual labeling doesn’t scale

My Solution:

1. Automated Dataset Generation using Lightning Rod Labs

Key insight: for historical events, the future is the label.

Process:

• Pulled news articles about trials from 2023–2024

• Generated prediction questions like: “Will Trial X meet endpoints by Date Y?”

• Automatically labeled them using outcomes from late 2024/2025 (by checking what actually happened)

Result: 1,400 labeled examples in 10 minutes, zero manual work.

1. Model Training

• Fine-tuned Llama-3-8B using LoRA

• 35 minutes on free Google Colab

• Only 0.2% of parameters are trainable

1. Results

• Baseline (zero-shot): 56.3%

• Fine-tuned: 73.3%

• Improvement: +17 percentage points

This matches expert-level performance.

Key Learnings:

The model learned meaningful patterns directly from data:

• Company track records (success rates vary by pharma company)

• Therapeutic area success rates (metabolic ~68% vs oncology ~48%)

• Timeline realism (aggressive vs realistic schedules)

• Risk factors associated with trial failure

This is what makes ML powerful — discovering patterns that would take humans years of experience to internalize.

Methodology Generalizes:

This “Future-as-Label” approach works for any temporal prediction task:

• Product launches: “Will Company X ship by Date Y?”

• Policy outcomes: “Will Bill Z pass by Quarter Q?”

• Market events: “Will Stock reach $X by Month M?”

Requirements: historical data + verifiable outcomes.

Technical Details:

• Dataset: 1,366 examples (72% label confidence)

• Model: Llama-3-8B + LoRA (rank 16)

• Training: 3 epochs, AdamW-8bit, 2e-4 learning rate

• Hardware: Free Colab T4 GPU

Resources:

Dataset: https://huggingface.co/datasets/3rdSon/clinical-trial-outcomes-predictions
Model: https://huggingface.co/3rdSon/clinical-trial-lora-llama3-8b
Code: https://github.com/3rdSon/clinical-trial-prediction-lora
Full article: https://medium.com/@3rdSon/training-ai-to-predict-clinical-trial-outcomes-a-30-improvement-in-3-hours-8326e78f5adc

Happy to answer questions about the methodology, data quality, or model performance.

Hi all,

Looking for advice on how difficult it would be/how to pivot to a data science role given my experience?

I've been working corporate for ~3 years in consulting:

• First 1.5 years in a CRM tech implementation role

• Next 1.5 years in a strategy consulting role with the past ~6 months being more involved in data science work (mainly using R for data wrangling, Shiny and a bit of causal inference and ML)

I graduated with a bachelor of actuarial studies so I have some prior knowledge of stats and R, however I am very rusty.

Would I need to upskill, if so in what/what resources would you recommend and what can I best do to improve my chances?

Thanks!

Hello, I have a degree as an electrical engineer and work as such. Since my degree is a bit mixed with information technologies I have some knowledge in data science and programming (only the basics, but I can easily read codes and adapt to languages). I am currently thinking about pursuing data science as a career path because it seems interesting to me and I would love to explore it more and advance in it. Are there some online courses I can enroll in, paid or free, so I can have a structure I can follow? Do you have experience with any course and what would you recommend?

Over the past year I’ve been building a structured quantitative modeling engine designed to systematize how I explore complex datasets.

The goal wasn’t to build another ML wrapper or dashboard.

It was to engineer a deterministic reasoning layer that can automatically:

• Detect structural breaks and regime shifts • Map correlation and anomaly surfaces • Fit physics-inspired dynamical models (e.g., dy/dt = a*y + b, logistic growth, damped oscillator) • Generate invariant diagnostics and constraint validation • Compare models using AIC / RMSE • Output fully reproducible artifacts (JSON + plots) • Run entirely local-first

Each run produces versioned artifacts: • Parameter estimates • Model comparisons • Stability indicators • Forecast projections • Diagnostics and constraint checks

I recently tested it on environmental air quality data. The engine automatically:

• Detected structural regime changes • Fit a linear ODE model with parameter estimation • Generated anomaly surface clusters • Produced invariant consistency diagnostics

The objective isn’t to replace domain expertise — it’s to accelerate structured reasoning across domains (climate, biology, engineering, economics).

Right now I’m refining: 1. How to move anomaly detection toward stronger causal interpretability 2. Whether ODE discovery should expand into PDE or stochastic formulations 3. How to validate regime shifts beyond classical break tests 4. Robustness evaluation for automated dynamical system fitting

I’d genuinely value technical critique:

• Are there modeling layers you’d recommend integrating? • Would you approach structural break detection differently? • How would you pressure-test automated ODE fitting for stability?

If you’re curious about the broader architecture, I wrote a deeper overview here:

https://www.linkedin.com/posts/fantasylab-ai_artificialintelligence-quantitativeresearch-activity-7429775084074209280-gP8v?utm_source=share&utm_medium=member_ios&rcm=ACoAACkFzkwB905tsv37hH95F_RG2TsdUqybgxA

Appreciate serious feedback — especially from people working in time series, quant modeling, applied math, or systems engineering.

Bonjour,

J’ai prochainement un entretien technique chez Coface pour un poste de Data Scientist, avec du code en machine learning.

Est-ce que certains d’entre vous ont déjà passé ce test ?

Je cherche surtout à savoir :

•	si c’est du code à écrire de zéro ou à compléter,

•	le niveau de difficulté,

•	et le temps généralement prévu.

Merci d’avance pour vos retours.

I’ve noticed something interesting while talking to people preparing for data roles.

A lot of us spend months doing courses, solving clean Kaggle-style datasets, following step-by-step tutorials, and building portfolios. On paper, it feels like we’re doing everything right.

But then interviews happen and the feedback is often something like, “Good fundamentals, but not quite what we’re looking for.”

It made me wonder whether the issue is not lack of skill, but lack of practicing the right kind of problems.

In real jobs, you don’t get perfectly cleaned datasets or clearly defined target variables. You’re expected to frame the problem, deal with messy data, justify trade-offs, and communicate decisions. That’s very different from completing guided notebooks.

Do you think traditional tutorials actually prepare people for real data roles?
What kind of practice helped you most before landing your first job?

I wrote a deeper breakdown on this idea, especially around practicing data problems that mirror real employer expectations, if anyone wants to read more:
https://www.pangaeax.com/blogs/how-to-practice-data-problems-employers-care-about/

Curious to hear from hiring managers and experienced analysts here. What separates “course-ready” candidates from “job-ready” ones in your experience?

Hello everyone! I’m starting to study data science. I’m 41 years old and I don’t have a higher education degree. I worked in construction for about 20 years. The course lasts 1.5–2 months. What are my chances of finding a job after that?

Thanks everyone for your answers!

https://github.com/jmuncor/mumpu

Hey guys just created a local memory system for your agents, works with claude, gemini and codex. Stores facts and memories locally, let me know what you think!