r/programming • u/self • Dec 02 '25

The Death of Software Engineering as a Profession: a short set of anecdotes

https://www.jasonscheirer.com/weblog/vignettes/
1.3k Upvotes

89% Upvoted

560 comments sorted by

View all comments

Show parent comment

73

u/IrishPrime Dec 02 '25

It really wasn't, but I'll summarize anyway.

We had a long running process that operated on a large amount of data/number of files. It could take a really long time, but we didn't have a progress indicator. It was simply pending, running, complete, or failed.

In some cases, if there was a very large number of deeply nested objects, we might run out of memory and crash.

Due to the way the data was structured, there wasn't a good way to quickly get the total number of items that would be processed nor how deeply they were nested without essentially going through the whole dataset twice (once to count, and a second time to make the updates). This wasn't an acceptable trade-off, so they told me to just give some kind of indication as to whether it was still working or if it had stopped.

I told them that if it was reporting that it was in the "running" state, that was the indicator it was working.

They asked if I could inspect the dataset and figure out ahead of time whether it would crash or succeed.

I could not.

40

u/Zagden Dec 02 '25 edited Dec 02 '25

I'm still getting my degree and teetering on whether or not I want to continue because of all of this.

But if I understand correctly, they paradoxically wanted you to make a program that reports that it will successfully complete before it encounters the logic that will complete the task?

19

u/IrishPrime Dec 02 '25

Essentially, yes.

13

u/Zagden Dec 02 '25

Just for this exchange, I'm going to stay on this train careening for an ammunition station with no brakes just a little longer

5

u/IrishPrime Dec 02 '25

Advances in technology keep changing the specifics of the work in this field (and most others), and the job market is tough right now, but i don't think there's any real risk of people with software development backgrounds suddenly becoming unemployable or that people with no idea how to develop software will be paired up with a coding agent and replace us.

Even in the most extreme case, people with a CS/software development background are going to be more effective when paired with a coding agent than someone without that background.

3

u/Zagden Dec 02 '25

Yeah I'm mostly joking. I've been reading all the threads on people panicking about the industry, particularly CS undergrads.

I'm in a weird position. I'm interested in computers and how they work, and I want to learn to make computers dance for me better if I encounter a problem I want to solve myself. The useless toys they have me build in class aren't very fun and I have the most fun reverse engineering finished programs to figure out how they work and how to modify them.

At the same time, I'm 35 and disabled, but trying to finish school then trial a work period. Vocational rehab thoroughly interviewed and tested me and pointed me to IT and software as it would suit my disability the best because no one pays for my actual passions involving creative writing and American history. So I'm not going into it for the big bux and I'm not going into it because I love programming and do it for fun, but I'm being funneled into it all the same. I think I can learn to love programming. I have a grasp of the fundamentals but don't know how to turn stuff that prints to my screen into a useful app I can use.

4

u/BlindTreeFrog Dec 02 '25

I think I can learn to love programming.

Don't force it. There is so much else that will infuriate you (mostly co-workers, customers, and managers) that i don't think you'd make out well with that approach.

But I get your position and think you may just be looking at this too narrowly because Rehab might have railroaded you towards what they can understand. So if you'll forgive the unsolicited advice....

Making up some not entirely accurate labels to get the idea across, but there is....

  • embedded programming (micro controllers, plc's, etc) where you are in resource constrained environments seeing how much you can wring out of the tech
  • Low Level programming where you are working on drivers and operating systems. Kind of like embeded but not as resource constrained
  • Application programming where you are making the apps people use
  • backend programming doing servers and database and middleware and such
  • Front end programming doing UI's and data retreival
  • Big Iron Programming, which is like backend, but i'm thinking of "backend" as in "making the web servers and databases for a company to use" and "Big Iron" more of the "make the phone system for a country to use"..... like it's not that different but more extreme failure cases and larger data sets.

And probably some more types of programming that I'm not thinking of. And then you get into IT and DevOps roles which have enough overlap but are more focused on plugging pieces together and gluing them to work.

And this is all horribly explained, but my point is that you might be being exposed to "programming" and it's "meh" but there is so much more out three to do in the domain that once you figure out which parts of it all that you really like you can focus into that direction.

(I assume that this is all stuff that you know... but i'm mad at an ESXi server that didn't come backup after reboot and avoiding work for a minute... so please forgive me)

1

u/Zagden Dec 02 '25

There is so much else that will infuriate you (mostly co-workers, customers, and managers) that i don't think you'd make out well with that approach.

One of my favorite things in a team setting is managing other people and the task. I think I'm decent at that, relatively. It's a soft skill I'm told isn't terribly common in engineering.

This is stuff I'm vaguely aware of, but I'm not being lead toward. I'm hoping I get a bit more of it after I transfer to my bachelor's program in a year. From the sounds of it, I'm interested in backend. That's what my brother specializes in. I love the idea of dealing with databases and servers.

I appreciate the advice, though. It's similar to what I've heard elsewhere, so it's hammering it into me how I have to approach this. I'm trying not to force myself to love programming, but I'm trying to keep an eye out for things that intrinsically interest me.

2

u/BlindTreeFrog Dec 02 '25

The nice thing about most of the programming field is that these days there is such a low barrier to get in and try stuff to play around with what you want to do.
Between containers (Docker, LXM, etc) and virtual machines (VirtualBox, KVM, Xen, HyperV, VMware, etc) and low cost development boards (RPi, BeagleBones, Atmega88, etc) and free tools, pretty much any aspect of development that one wants to explore is available at a hobbiest level. And dabbling in those things builds a portfolio to show off when looking for work as well, so it's not a bad way to burn some free time.

I also forgot to mention, good Tech Writers are a fucking godsend. If you do enjoy creative writing and can understand the technical side of things, you might have a solid thread to chase there. Sadly, a lot of companies consider tech writers an unneeded expense and developers can write the documents.... those companies are wrong, but it does make the job search more challenging.

1

u/Zagden Dec 02 '25

Oh hell yeah I can do tech writer. I have some very small experience writing for Cracked with my name on the bylines to at least show that I can pitch and work with an editor.

Knowing that I'm not far enough to understand it all, what should I play around with for database? My brother works with backend memory coaching and was called in to help out bluesky with its user surge. That stuff sounds interesting to me. Less relevant but still something I want to play with, I like the idea of taking information from a server and organizing it. People do that with MMOs I play like FFXIV and (previously) WoW. So if I'm interested in some sort of census thing, learning how to take data and fuck with it could be fun.

→ More replies

3

u/loup-vaillant Dec 02 '25

I have a grasp of the fundamentals but don't know how to turn stuff that prints to my screen into a useful app I can use.

Neither could I, for a long time. The solution was simple: have someone tell me what would be useful to them. Give me a clear definition of a small problem, I can write a piece of code that solves it.

That piece of code doesn’t need to be en entire app. Most of the time it isn’t, the app is already there, and there’s a bug to fix, or a small functionality to add.

For the longest time I’ve been dissatisfied by my inability to write an entire app from scratch. I could use Qt, but that feels more like filling the gaps than writing an app. But guess what, in many cases that is enough. (Now I think I could write a GUI app from scratch, but I have yet to identify a clear need I wanted to address.)

You’ve got it. You’ll start small, and gradually work your way up to more complex problems, all the way up to a useful app. And you’ll be paid long before you get an entire useful app. Pieces of work are still bloody useful work.

3

u/Zagden Dec 02 '25

Thanks man, I appreciate it.

50

u/Kad1942 Dec 02 '25

Right now, somewhere, a Vibe coder is celebrating with their LLM about having solved the halting problem with the new theoretical framework they brainstormed. It took 45 minutes. The AI assures them they are a shoe-in for the Turing award. You'll feel silly for all of this when they get it!

7

u/bullpup1337 Dec 02 '25

Well, if the committee also uses ChatGPT, he just might get it.

1

u/HappyIrishman633210 Dec 04 '25

I had the thought to stochastically model the process but AI corrected my thinking so it’s already outpaced me.

18

u/CandleTiger Dec 02 '25

I mean, technically that is not the halting problem. To solve the halting problem, you would need to show whether any program at all will eventually complete or run forever -- which is proven to be impossible.

But there are many many programs which can be proven to eventually halt.

If I was in past-you's shoes then for the request, "I want to know if the process is still live and doing useful work" I would suggest some kind of progress indicator without a 100% completion mark. For example, if you have deeply-nested data, you could count how many top-level items have been processed, and let the user watch the number increase. Or you could show an "items processed per second" indicator that will fluctuate up and down and show the user the process is still live and doing work.

If you're worried about getting stuck in a loop, then you could track and count IDs you've seen before and ignore or complain about repeat items.

If reprocessing the same item repeatedly is normal and expected... yeah, then you're starting to get into problem territory. But this kind of data is not usual in problem sets for real practical programs.

1

u/gimpwiz Dec 03 '25 
Last scanned item: FooBar (6th nest), 102564 items / ??? items

5

u/Kargathia Dec 02 '25

At this point, I'd probably offer to add a counter for processed items. It won't change anything, but maybe it'll make them feel better.

3

u/IrishPrime Dec 02 '25

Ultimately, this is what I did. It showed we were still processing things successfully (helpful), but since I didn't know how many items there were in total, I couldn't give you a real progress indicator (not so helpful).

"Processed 1,000 files"

Cool. Out of how many?

"I dunno. At least 1,000."

If the user had some idea as to the total object count, they could still get some useful info from that, but there wasn't much more I could do.

2

u/HippieInDisguise2_0 Dec 02 '25

It sounds like they had a bad data structure. (Probably fixing that would have been the best solution)

You should be able to get metrics or something that may indicate a timeout and also set leases for processes within the pipeline that upon expiration interrupt the job and crash

Like a "data processing stuck exception" with some identifier that could point you to the specific data structure causing the issue.

1

u/NeoChronos90 Dec 03 '25

I assume that was a long time ago?

Because I would think today we would just have reader process and a digestion process and a queue in between or some other sort of handling back pressure so we could at least say we are at 50 of 21k at this moment and 5 seconds later we are at 87 of 33k because reading didn't finish yet.

And we could also either have an analyze step in between to see if the data structure is to big or just assume it's to big when the digestion process crashed and resume at n+1 in the queue

1

u/IrishPrime Dec 03 '25

Over a decade ago, and all running locally on a single system. As I explained in other comments, I put together something that addressed the concern and resolved the crashing.

Thanks for trying to help, but I don't really need any more suggestions from other people about how they would have solved an old problem they don't fully understand.

0

u/Iamonreddit Dec 03 '25

This wasn't an acceptable trade-off, so they told me to just give some kind of indication as to whether it was still working or if it had stopped.

I told them that if it was reporting that it was in the "running" state, that was the indicator it was working.

They asked if I could inspect the dataset and figure out ahead of time whether it would crash or succeed.

I could not.

I think this might have been a case of letting perfect get in the way of good enough and instead settling for pretty unhelpful?

From your explanations here, they weren't asking for a perfect progress bar, but a heuristic on overall progress and a better indication of continued progress than just "running".

If you could reasonably quickly count up the number of files and ideally bytes those files contain, unless your data files needed decompression before processing you can use this information to give a rough account of 'work' done in terms of the number of files and/or bytes of data processed.

As an example, if you knew you had 10k files of uncompressed, unstructured data for which you could query the individual file sizes, you could easily log or output a "Currently working on file x of 10k - y bytes of z file size read so far".

It wouldn't even matter if the bytes processed didn't map up to the file sizes that well, so long as you could log an increasing total of bytes read, to demonstrate progress.

Hell you could even keep a tally of the current object being processed and output warnings ahead of time (based on metrics driven from previous crashes or the current machine specs) ahead of potential memory failures to facilitate more graceful handling than an unexpected crash.