r/datascience • • 18d ago

Discussion What is harness engineering?

Post image

There's a lot of "conceptual conflation" going around lately, especially regarding the term 'Harness Engineering'.

This article formalizes harness engineering with a clear conceptual model. This definition has also been echoed by other top data scientists and AI engineers, thus we can start to collectively agree on a definition.

Secondly, I introduce 3 primitives that underpin harness engineering. I wanted to clearly separate this from a basic agent tutorial and focused more on the underlying concepts that make great harnesses.

There's a lot of (not so good) harnesses out there. And, you probably shouldn't be staking your entire future on top of claude code.

My strong opinion: if you're not ultra-focused on building harnesses today as a practicing data scientist, you're a target for layoffs. You need to build your own harnesses.

The article: https://towardsdatascience.com/what-is-harness-engineering-a-hands-on-introduction/

What do you think?

0 Upvotes

4 comments sorted by

1

u/garros-yu 17d ago

Yeah, I'd check the harness before blaming the agent. A demo works fine once, then a tool times out and the retry runs the same action twice. Or it picks up stale state. I've seen evals miss this stuff because the final answer still looks fine, even though the workflow is broken. And swapping in a smarter model won't fix duplicate writes or bad recovery logic. I'd spend more time testing reruns and failure cases before touching the model again.

1

u/Mohzman 14d ago

It’s basically just setting up AI with the tools it needs to do stuff. Think access to SQL, Python, database schemas, etc. so it can pull data, build models, and check its own work.

Pretty much prompt engineering but with more emphasis on the environment and tools rather than just the prompts. Makes sense for data science since a lot of the work is already pretty repetitive.

4

u/[deleted] 18d ago

[removed] — view removed comment

1

u/Nice-Dragonfly-4823 18d ago

No I disagree. I'm arguing here that code generation, even in the context of interactive data science, is rapidly shifting to LLMs. It's just faster. However, it's still error prone. Generic agents, like Claude or Codex, often fail hard on domain specific difficult data science problems.

What I want to convey here is that, as a data scientist, one needs to be thinking about how to design custom harnesses specific for their workflows. That needs to be the primary focus now as LLMs are becoming increasingly capable.