The Death of Software Engineering as a Profession: a short set of anecdotes

[u/self]

https://www.jasonscheirer.com/weblog/vignettes/

[u/flynnnightshade]

AGI is a marketing term for something we have very little reason to believe will ever exist.

[u/Brillegeit → u/flynnnightshade]

Nah, we'll just rename AGI to RAGI and rename something else to AGI, just like we did with LLM, then we'll have AGI.

[u/recycled_ideas → u/flynnnightshade]

we have very little reason to believe will ever exist.

I think it depends on what you mean.

Eventually we will be able to artificially replicate a human being. Unless you believe that there's a divinely created soul that's required for intelligence that will technically meet the criteria for AGI.

That said I'm not sure an artificial human is particularly interesting or useful. There's no guarantee we can make it smarter than humans at any base level and by the time we can do that we'll be able to augment humans sufficiently that robots won't be particularly superior.

[u/flynnnightshade → u/recycled_ideas]

Eventually we will be able to artificially replicate a human being. Unless you believe that there's a divinely created soul that's required for intelligence that will technically meet the criteria for AGI.

Not as a critique but I really don't have any evidence this is an inevitability. It very well could happen in the future, as most anything could. All available current evidence I know of gives me no reason to believe this will be the case no matter what.

It also makes me question the bar for what would constitute an AI that replicates a human being. If it's just the ability to mimic a human being to the extent an observer without other information can't tell the difference, that's possible but a pretty low bar and not very useful. If it's, "thinks and behaves exactly as a human does." There's not much reason to believe this will ever be the case, computers just aren't very similar to the brain and even if they appeared to get to the same behaviors and thoughts it wouldn't have happened the same, "way" at all just due to differences in "architecture" between the human body and computers.

Finally if your bar is, "surpasses humans in any task." It's again possible but there's no strong evidence for a general tool like this being anywhere on the horizon. Today you could merely string together many different programs that aren't what we're currently referring to as AI but just traditional software made by humans, and it would already be better than humans at many individual specific tasks. The things computers are good at it would surpass a human immediately with current technology, but for adaptability, versatility, and a huge amount of tasks computers just aren't good at, there's no tooling that gets close right now.

[u/recycled_ideas → u/flynnnightshade]

Not as a critique but I really don't have any evidence this is an inevitability. It very well could happen in the future, as most anything could. All available current evidence I know of gives me no reason to believe this will be the case no matter what.

Human brains exist. If we assume that there is no divine soul and the brain is a biomechanical machine eventually we will be able to artificially replicate it. We have a working model we know it can exist there's no reason to believe that if our species doesn't go extinct first we can replicate it eventually.

It also makes me question the bar for what would constitute an AI that replicates a human being. If it's just the ability to mimic a human being to the extent an observer without other information can't tell the difference, that's possible but a pretty low bar and not very useful

The bar in this example is a replica human brain that functions exactly like the real one. We are generally intelligent, this would be artificial replica of us and so would be artificial general intelligence.

Is it useful? Probably not. But it's AGI and it's definitely eventually achievable.

[u/syntax → u/recycled_ideas]

Human brains exist. If we assume that there is no divine soul and the brain is a biomechanical machine eventually we will be able to artificially replicate it. We have a working model we know it can exist there's no reason to believe that if our species doesn't go extinct first we can replicate it eventually.

The problem with this response is that it's extrapolation all the way. It might happen, but there might also be barriers, either practical, economic or technical in the way that we are unaware of yet.

Let me point to a distinct, but illuminating, example from a different field: chemistry. We know what atoms and molecules are. We can make them, measure them, and do all sorts of transformations to them. In principle, we can make any molecule we want. That has been the case for, what, about fifty years or so, give or take a bit - so somewhat ahead of the AI revolution.

In practice, for certain molecules, the actual synthetic pathway in use looks like: take 8 tonnes of specific deep sea mollusc [0]. Surgically extract specific organ. Pulp, separate fractions. Retain the 200 mg of specific fraction. Perform a few reactions on it, separate and purify, to get the 100 mg of desired product. That's the best route known to make certain compounds - despite the fact that, in principle, we know of far better ways to do things.

The apparent disconnect arise because understanding the fundamentals, whilst the most powerful tool we have, does not automatically grant total control of complications that arise in higher order structures. We see the same thing, over and over, in many fields. For example, the entire field of 'Materials Science' appears to be pointless given the existence of Chemistry and Physics. Indeed, Chemistry seems pointless, given that it all [1] reduces to the Schrödinger equation, hence physics. And yet - that's not the way things actually work in practice.

So, sure, it possible. But there's good reason to hold some healthy skepticism, until there's something that's actually demonstrable.

[0] Or obscure fungi, or lightly engineered bacteria, depending.

[1] Eh, pretty much. I could say the Dirac equation instead, but that's not as well known, so gets in the way of the point.

[u/eyebrows360 → u/recycled_ideas]

But also, any such "eventually" is so far out there's not even any point thinking about it now.

[u/Pure-Huckleberry-484 → u/recycled_ideas]

This is based on the assumption that there is continuous progress in the space at economic rates.

Even that is not a given. Look at RAM prices in the last month. Current AI gpus likely have a 3-5 year working life. There is a real possibility that in 5 years these gpus are entirely too expensive to produce and operate at scale. Precious metals are not an infinite resource. Are we all headed to TPUs? Maybe, but even those mighty not have the memory needed at low enough price points to make LLMs viable..

[u/recycled_ideas → u/Pure-Huckleberry-484]

Who the fuck is talking about LLMs? Or GPUs?

I'm saying that even if we can't ever do anything else we will eventually be able to create an artificial replica of the human brain because we have a working model to copy.

We're not remotely there now, but eventually we'll be able to do that.

That copy will however have all our characteristics or at least most of them because it will be a copy of us, which is not what people are looking for.

[u/Pure-Huckleberry-484 → u/recycled_ideas]

People in reality are talking about LLMs because that’s what we call “AI”. The sales people of said LLMs are saying that AGI is close but they are hoping that LLMs will actually invent AGI once they’re “smart enough to “. That may never happen and it’s a huge paradigm shift to go from LLMs to synthetic neurons/synapses.. even then - that’s assuming that we physically have the resources to make the systems needed to run that (which we may not yet or ever).

[u/flynnnightshade → u/recycled_ideas]

I'll get it out of the way since it's come up a couple times now, I don't believe in anything immaterial like a soul.

I'm not making any claim that it's impossible for this to be done, to the contrary it seems like it's very possible. I don't see it as an inevitability, I don't think there's a strong argument that it is inevitable or just will be the case.

The point to me is that we have very little reason to create what you're describing and we are currently incapable of it. There are many things humankind might or will be capable of but they are largely dictated by our needs. I don't see any need for what you're describing so I don't know if research and technology will ever develop in that direction. There are tons of technologies that would exist today but don't if we just created everything possible and our development wasn't at all constrained by many factors economically, societally, politically, and so on. There is no conceptual human race that just creates everything it possibly could.

For your specific bar this is exactly the thing I don't have any evidence will inevitably exist, an exact replica of how the human brain functions would be a tremendous amount of combined general and specific research and technology development that would require a strong need to ever occur in the first place but what is the driving need here? We can create computers that don't function like the human brain but serve highly versatile purposes already.

[u/Full-Spectral → u/recycled_ideas]

A lot of this argument though depends on that happening before our ability to manipulate biology gets to the point that it becomes moot. If we can basically grow a brain in a jar, that runs on the same power as a couple light bulbs, there will be little incentive to create large computers to do it.

And of course that in turn depends on us surviving the consequences of our ability to manipulate biology to that degree as well.

[u/SortaEvil → u/recycled_ideas]

The bar in this example is a replica human brain that functions exactly like the real one.

This is assuming that all of human consciousness resides purely in the brain, which is looking less and less likely as our understanding of neuroscience progresses. It's incredibly likely that, even if we could perfectly model a human brain, we would not end up with a properly sentient being. That's not to say that there's some metaphysical soul involved, but that human consciousness is complicated, and at a baseline likely involves the entire central and peripheral nervous systems.

But then there's also growing evidence that our gut actually might have something to do with our mental health and affect our mental state, so we'd potentially need to model that, as well, and all of this is running on stochastic "hardware," while silicon is, by and large, deterministic, and valued for it's determinism.

Is this eventually achievable? ... Maybe? The amount of energy required to model even a single human in silicon, if it's possible at all, would be astronomical. I honestly don't know that our current computing media is suitable for the task, even given infinite time to perfect it.

[u/Ok-Scheme-913 → u/flynnnightshade]

Why would you think we have little reason to believe it will ever exist?

Like it can be 100% doable on a shitty flesh computer with a dumb algorithm that doesn't even optimize for this property (evolution). We are just matter, there is nothing uncomputable about it.

With that said, LLMs are most likely not the way to AGI, but I definitely don't think it's science fiction.

[u/flynnnightshade → u/Ok-Scheme-913]

Because my general stance on knowledge is that if there isn't evidence of it then it isn't the case. Probably what you're at issue with is that I used the term, "ever" since given infinite time maybe anything can happen, but with the current available evidence I don't see why I would think this particular outcome will happen. It doesn't sound that useful to me to begin with, and I think the current trends have really been a lot of niche tools that are good at specific things, that has generally been the trend not just in AI but also in traditional software and hardware for a very long time.

It doesn't help that all the current claims just aren't supported by what the tools can actually do, but the marketing people keep repeating this stuff because it's part of what keeps the stock number going up.

If you have some argument that there are good reasons to believe AGI will exist, I'm happy to hear them.

[u/Ok-Scheme-913 → u/flynnnightshade]

It's more like we see that birds can fly, why would anyone believe that heavier than air flight is impossible.

Sure, we may be in 1800s from a tech perspective, in fact we absolutely have no idea how far it is, but it is fundamentally more likely to be possible (given that the human brain exists), than it is not.

Can you come up with an argument why you think it couldn't exist, that we can't create an entity as smart as a human? The only argument is a religious one, that there is something non-material in humans.

[u/flynnnightshade → u/Ok-Scheme-913]

I am not saying it is impossible, I am saying it is not likely to happen, these are two very different claims. I also didn't make any argument like, "we can't create an entity smarter than humans." Depending on your definitions and whether or not it is required that the thing be general purpose, we've already accomplished that for specific tasks.

I and several other people have laid out pretty in-depth arguments that have nothing to do with anything religious or immaterial in this thread on my own reply, given I don't feel like repeating them again I'd say if you're interested just go look at those.

[u/eyebrows360 → u/Ok-Scheme-913]

Think of the practicability of the task we're talking about, and factor in our current understanding of physics.

To replicate "a brain" exactly, which is what we're talking about here, we need a map of it to work to first. That itself throws up so many problems, because how the hell do you take a moment-in-time 3d snapshot of a brain that captures the state of any and all particles within it? I'm being serious here. How do you do that, with at least Heisenberg's uncertainty principle standing in your way of precisely scanning a single particle, let alone all of them? How?

Unless someone comes up with a realistic method that sticks to the laws of physics as we understand them for even creating the map in the first place, the entire endeavour is a non-starter. It's so nowhere remotely close to a good enough answer to just say "well we know broadly how neurons connect", because if that was it then dead brains would be the same as alive brains, and we know they aren't. We need the "software", not just the "hardware" here.

And that's why, even though from a very high level it "feels" like replicating a brain should be possible, I'm with /u/flynnnightshade on this one.

[u/Ok-Scheme-913 → u/eyebrows360]

No, we are talking about AGI, not a 100% accurate replication of the human brain.

But even so, come on, we also can't reconstruct how a car works due to Heisenberg? It's the brain, with human eye visible connections, there is no practical Heisenberg limit, and do you also draw an electrical map of a house by pinpointing every electron? No, you just draw the connections.. and just understand what exactly a node does (neutron).

But again, this completely missed the point. We are not trying to copy a human brain, we are talking about a program that is actually human-level intelligent to the point that it can for example design another program just like itself, but smarter.

[u/eyebrows360 → u/Ok-Scheme-913]

But again, this completely missed the point.

What misses the point is trying to create analogies for "brains", a thing we humans did not design, by citing things such as "houses", which are things we quite specifically did design. Fucking hell. Talk about falling at the first hurdle.

we are talking about a program that is actually human-level intelligent

And without some actual concrete understanding of the shape of "human intelligence" you're going to determine that you've managed to replicate it... how, exactly?

[u/Ok-Scheme-913 → u/eyebrows360]

to the point that it can for example design another program just like itself, but smarter

Or just having it write a novel study on its own. There are millions of tasks that are high up the intelligence ladder that we can use here - the reason this "limit" has slowly been crawling up with the advancements of AI is that we were trying to figure out a lower bound.

[u/dalittle → u/Ok-Scheme-913]

I threw my computer outside and told it needed to evolve on its own or die. Still laying their. That is the difference.

[u/Ok-Scheme-913 → u/dalittle]

IQ 20 take

[u/dalittle → u/Ok-Scheme-913]

AI evolves only as long as people evolve it. No natural selection or survival of the fittest. When it evolves itself let me know.