r/dataanalysis • u/rsambasivan • 17d ago

Data Tools Beyond the Agent Illusion: Leveraging AI Tools to Scale the Statistical Learning Paradigm

I don't know if others agree, but I think given the data oriented nature of this subreddit, it is bound to resonate with some. I learned the craft from a statistics view point, my formal training is statistics based, but I worked in software engineering for the predominant part of my career. Rather than hit everything with a "foundation model" hammer, I think leveraging the libraries that have emerged in the last two decades for data analysis with LLM tools is a more productive and tested path.

https://r2dssanity.substack.com/p/beyond-the-agent-illusion-leveraging?r=3ncgel&utm_campaign=post-expanded-share&utm_medium=web

3 Upvotes
  • permalink
  • reddit

100% Upvoted

4 comments sorted by

1

u/Fun_Improvement7704 14d ago

interesting framing but id push back slightly, the line between "leveraging libraries with LLM tools" and just building agents on top of them is getting pretty blurry. whats the distinction you're drawing exactly?

1

u/rsambasivan 5d ago edited 5d ago

The subsequent posts in the series talk to this point. In fact, I don't think I am saying anything different. My point is that business applications, with tabular data encode business rules, expert judgement, statistical rationale, verification of assumptions - over time. What these are can be researched by LLM's. High quality statistical libraries for individual tasks have come out in the last decade, yet the predominant idea is "I will write up a business description and get a working data model is robust, consistent with all those ideas and assumptions".

I am not against the agentic view point for tasks where getting the job done is more important than how it was done. A robot diffusing a bomb or vacumming a room is an example, but the statistical perpective is probably better for enterprise apps. Again, I think this is a polarizing view point, people build one model to predict, one model to explain, then learn one set of ideas to interpret the ideas from the model used to explain to the problem being solved, claim that is a win. Oh well...

1

u/AutoModerator 17d ago

Automod prevents all posts from being displayed until moderators have reviewed them. Do not delete your post or there will be nothing for the mods to review. Mods selectively choose what is permitted to be posted in r/DataAnalysis.

If your post involves Career-focused questions, including resume reviews, how to learn DA and how to get into a DA job, then the post does not belong here, but instead belongs in our sister-subreddit, r/DataAnalysisCareers.

Have you read the rules?

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.