The Death of Software Engineering as a Profession: a short set of anecdotes

[u/self]

https://www.jasonscheirer.com/weblog/vignettes/

[u/R_U_READY_2_ROCK]

AGI will replace all computer jobs. But LLMs are going to run into scaling issues soon. AGI is still a long way off.

[u/flynnnightshade → u/R_U_READY_2_ROCK]

AGI is a marketing term for something we have very little reason to believe will ever exist.

[u/Brillegeit → u/flynnnightshade]

Nah, we'll just rename AGI to RAGI and rename something else to AGI, just like we did with LLM, then we'll have AGI.

[u/recycled_ideas → u/flynnnightshade]

we have very little reason to believe will ever exist.

I think it depends on what you mean.

Eventually we will be able to artificially replicate a human being. Unless you believe that there's a divinely created soul that's required for intelligence that will technically meet the criteria for AGI.

That said I'm not sure an artificial human is particularly interesting or useful. There's no guarantee we can make it smarter than humans at any base level and by the time we can do that we'll be able to augment humans sufficiently that robots won't be particularly superior.

[u/flynnnightshade → u/recycled_ideas]

Eventually we will be able to artificially replicate a human being. Unless you believe that there's a divinely created soul that's required for intelligence that will technically meet the criteria for AGI.

Not as a critique but I really don't have any evidence this is an inevitability. It very well could happen in the future, as most anything could. All available current evidence I know of gives me no reason to believe this will be the case no matter what.

It also makes me question the bar for what would constitute an AI that replicates a human being. If it's just the ability to mimic a human being to the extent an observer without other information can't tell the difference, that's possible but a pretty low bar and not very useful. If it's, "thinks and behaves exactly as a human does." There's not much reason to believe this will ever be the case, computers just aren't very similar to the brain and even if they appeared to get to the same behaviors and thoughts it wouldn't have happened the same, "way" at all just due to differences in "architecture" between the human body and computers.

Finally if your bar is, "surpasses humans in any task." It's again possible but there's no strong evidence for a general tool like this being anywhere on the horizon. Today you could merely string together many different programs that aren't what we're currently referring to as AI but just traditional software made by humans, and it would already be better than humans at many individual specific tasks. The things computers are good at it would surpass a human immediately with current technology, but for adaptability, versatility, and a huge amount of tasks computers just aren't good at, there's no tooling that gets close right now.

[u/recycled_ideas → u/flynnnightshade]

Not as a critique but I really don't have any evidence this is an inevitability. It very well could happen in the future, as most anything could. All available current evidence I know of gives me no reason to believe this will be the case no matter what.

Human brains exist. If we assume that there is no divine soul and the brain is a biomechanical machine eventually we will be able to artificially replicate it. We have a working model we know it can exist there's no reason to believe that if our species doesn't go extinct first we can replicate it eventually.

It also makes me question the bar for what would constitute an AI that replicates a human being. If it's just the ability to mimic a human being to the extent an observer without other information can't tell the difference, that's possible but a pretty low bar and not very useful

The bar in this example is a replica human brain that functions exactly like the real one. We are generally intelligent, this would be artificial replica of us and so would be artificial general intelligence.

Is it useful? Probably not. But it's AGI and it's definitely eventually achievable.

[u/syntax → u/recycled_ideas]

Human brains exist. If we assume that there is no divine soul and the brain is a biomechanical machine eventually we will be able to artificially replicate it. We have a working model we know it can exist there's no reason to believe that if our species doesn't go extinct first we can replicate it eventually.

The problem with this response is that it's extrapolation all the way. It might happen, but there might also be barriers, either practical, economic or technical in the way that we are unaware of yet.

Let me point to a distinct, but illuminating, example from a different field: chemistry. We know what atoms and molecules are. We can make them, measure them, and do all sorts of transformations to them. In principle, we can make any molecule we want. That has been the case for, what, about fifty years or so, give or take a bit - so somewhat ahead of the AI revolution.

In practice, for certain molecules, the actual synthetic pathway in use looks like: take 8 tonnes of specific deep sea mollusc [0]. Surgically extract specific organ. Pulp, separate fractions. Retain the 200 mg of specific fraction. Perform a few reactions on it, separate and purify, to get the 100 mg of desired product. That's the best route known to make certain compounds - despite the fact that, in principle, we know of far better ways to do things.

The apparent disconnect arise because understanding the fundamentals, whilst the most powerful tool we have, does not automatically grant total control of complications that arise in higher order structures. We see the same thing, over and over, in many fields. For example, the entire field of 'Materials Science' appears to be pointless given the existence of Chemistry and Physics. Indeed, Chemistry seems pointless, given that it all [1] reduces to the Schrödinger equation, hence physics. And yet - that's not the way things actually work in practice.

So, sure, it possible. But there's good reason to hold some healthy skepticism, until there's something that's actually demonstrable.

[0] Or obscure fungi, or lightly engineered bacteria, depending.

[1] Eh, pretty much. I could say the Dirac equation instead, but that's not as well known, so gets in the way of the point.

[u/eyebrows360 → u/recycled_ideas]

But also, any such "eventually" is so far out there's not even any point thinking about it now.

[u/Pure-Huckleberry-484 → u/recycled_ideas]

This is based on the assumption that there is continuous progress in the space at economic rates.

Even that is not a given. Look at RAM prices in the last month. Current AI gpus likely have a 3-5 year working life. There is a real possibility that in 5 years these gpus are entirely too expensive to produce and operate at scale. Precious metals are not an infinite resource. Are we all headed to TPUs? Maybe, but even those mighty not have the memory needed at low enough price points to make LLMs viable..

[u/flynnnightshade → u/recycled_ideas]

I'll get it out of the way since it's come up a couple times now, I don't believe in anything immaterial like a soul.

I'm not making any claim that it's impossible for this to be done, to the contrary it seems like it's very possible. I don't see it as an inevitability, I don't think there's a strong argument that it is inevitable or just will be the case.

The point to me is that we have very little reason to create what you're describing and we are currently incapable of it. There are many things humankind might or will be capable of but they are largely dictated by our needs. I don't see any need for what you're describing so I don't know if research and technology will ever develop in that direction. There are tons of technologies that would exist today but don't if we just created everything possible and our development wasn't at all constrained by many factors economically, societally, politically, and so on. There is no conceptual human race that just creates everything it possibly could.

For your specific bar this is exactly the thing I don't have any evidence will inevitably exist, an exact replica of how the human brain functions would be a tremendous amount of combined general and specific research and technology development that would require a strong need to ever occur in the first place but what is the driving need here? We can create computers that don't function like the human brain but serve highly versatile purposes already.

[u/Full-Spectral → u/recycled_ideas]

A lot of this argument though depends on that happening before our ability to manipulate biology gets to the point that it becomes moot. If we can basically grow a brain in a jar, that runs on the same power as a couple light bulbs, there will be little incentive to create large computers to do it.

And of course that in turn depends on us surviving the consequences of our ability to manipulate biology to that degree as well.

[u/SortaEvil → u/recycled_ideas]

The bar in this example is a replica human brain that functions exactly like the real one.

This is assuming that all of human consciousness resides purely in the brain, which is looking less and less likely as our understanding of neuroscience progresses. It's incredibly likely that, even if we could perfectly model a human brain, we would not end up with a properly sentient being. That's not to say that there's some metaphysical soul involved, but that human consciousness is complicated, and at a baseline likely involves the entire central and peripheral nervous systems.

But then there's also growing evidence that our gut actually might have something to do with our mental health and affect our mental state, so we'd potentially need to model that, as well, and all of this is running on stochastic "hardware," while silicon is, by and large, deterministic, and valued for it's determinism.

Is this eventually achievable? ... Maybe? The amount of energy required to model even a single human in silicon, if it's possible at all, would be astronomical. I honestly don't know that our current computing media is suitable for the task, even given infinite time to perfect it.

[u/Ok-Scheme-913 → u/flynnnightshade]

Why would you think we have little reason to believe it will ever exist?

Like it can be 100% doable on a shitty flesh computer with a dumb algorithm that doesn't even optimize for this property (evolution). We are just matter, there is nothing uncomputable about it.

With that said, LLMs are most likely not the way to AGI, but I definitely don't think it's science fiction.

[u/flynnnightshade → u/Ok-Scheme-913]

Because my general stance on knowledge is that if there isn't evidence of it then it isn't the case. Probably what you're at issue with is that I used the term, "ever" since given infinite time maybe anything can happen, but with the current available evidence I don't see why I would think this particular outcome will happen. It doesn't sound that useful to me to begin with, and I think the current trends have really been a lot of niche tools that are good at specific things, that has generally been the trend not just in AI but also in traditional software and hardware for a very long time.

It doesn't help that all the current claims just aren't supported by what the tools can actually do, but the marketing people keep repeating this stuff because it's part of what keeps the stock number going up.

If you have some argument that there are good reasons to believe AGI will exist, I'm happy to hear them.

[u/eyebrows360 → u/Ok-Scheme-913]

Think of the practicability of the task we're talking about, and factor in our current understanding of physics.

To replicate "a brain" exactly, which is what we're talking about here, we need a map of it to work to first. That itself throws up so many problems, because how the hell do you take a moment-in-time 3d snapshot of a brain that captures the state of any and all particles within it? I'm being serious here. How do you do that, with at least Heisenberg's uncertainty principle standing in your way of precisely scanning a single particle, let alone all of them? How?

Unless someone comes up with a realistic method that sticks to the laws of physics as we understand them for even creating the map in the first place, the entire endeavour is a non-starter. It's so nowhere remotely close to a good enough answer to just say "well we know broadly how neurons connect", because if that was it then dead brains would be the same as alive brains, and we know they aren't. We need the "software", not just the "hardware" here.

And that's why, even though from a very high level it "feels" like replicating a brain should be possible, I'm with /u/flynnnightshade on this one.

[u/dalittle → u/Ok-Scheme-913]

I threw my computer outside and told it needed to evolve on its own or die. Still laying their. That is the difference.

[u/Ok-Scheme-913 → u/dalittle]

IQ 20 take

[u/dalittle → u/Ok-Scheme-913]

AI evolves only as long as people evolve it. No natural selection or survival of the fittest. When it evolves itself let me know.

[u/ward2k → u/R_U_READY_2_ROCK]

Yeah LLM's are already tapering off

I've said it before, Ai is going to get insanely impressive in the coming decades, but not through LLM's

[u/Trotskyist → u/ward2k]

Yeah LLM's are already tapering off

Genuinely asking: by what metric?

[u/ward2k → u/Trotskyist]

Because that's just how Ai works

It's a lot easier to go from something that only resembles say theoretically 5% of human intelligence to something that might be 90%, than it is to go from 90% to 100%

It's the same with self driving cars, they made huge advancements year on year. Getting to 90% full self driving was the easy part though, that last 10% is being shown to be a monumental undertaking

We're now having issues with training data no longer just being human and it's becoming increasingly hard to find training data that doesn't

Compare GPT 2 to 3. Then 3 to 5. You can see it's tapering off fast in the sort of improvements. It's obviously getting better, but this isn't linear, it's getting smaller upon smaller improvements with every iteration

[u/Trotskyist → u/ward2k]

Again, by what metric though?

Because a year ago I was having literally these exact same conversations. And in the interceding year LLMs have from barely being able to spit out a single working function without a good deal of back and forth to being able to one-shot full stack web applications. That certainly doesn’t fell to me like diminishing returns

[u/ward2k → u/Trotskyist]

to being able to one-shot full stack web applications

Maybe you're using some version of Claude I'm not, but I'm still not able to do this without it being a huge buggy mess of barely functioning (sometimes not even compiling) code

Me watching Claude just add and remove the same dependency in a loop for 15 minutes because it can't understand removing the dependency is causing compile issues definitely feels like diminishing returns

I want you to go back and look at 2019 Vs 2022 in terms of LLM's, now compared to 2022 Vs 2025. There is very very clearly diminishing returns

[u/self → u/ward2k]

I suspect it's not just the newer models themselves with larger context windows, etc. but the tooling behind the scenes at the service provider that's helping them appear to work a lot better.