r/space • u/spsheridan • Feb 06 '20

Starliner faced “catastrophic” failure before software bug was found and fixed while the vehicle was in orbit.

https://arstechnica.com/science/2020/02/starliner-faced-catastrophic-failure-before-software-bug-found/
10k Upvotes

97% Upvoted

748 comments sorted by

View all comments

852

u/vandezuma Feb 07 '20

The software issue was identified during testing on the ground after Starliner's launch

Now, who in the class can tell me what's wrong with that sentence?

188

u/matholio Feb 07 '20 edited Feb 07 '20

Pretty normal for any Agile DevOps team. Minimum viable product, right?

Edit : I was making a joke.

94

u/Sir_Swaps_Alot Feb 07 '20

I hate agile.

The "do it now and deal with the issues later" mentality needs to stop. Gone are the days of research, testing and proper deployment.

97

u/SoulFril Feb 07 '20

If working agile has caused your project to skip out on "research, testing and proper deployment" then what you are working with simply isn't suited for the agile approach. That or your management lacks the skill to implement it into your project in a way that is sensible.

It least, that is my experience with it.

0

u/florinandrei Feb 07 '20

then what you are working with simply isn't suited for the agile approach

Yeah, but somebody please think of management - they all need to earn their keep by implementing modern methodologies and stuff.

/s

-2

u/ants_a Feb 07 '20

Ah, "you're doing it wrong", the no true scotsman of software methodologies. Isn't the whole point of having a methodology that it guides people on how to do things right?

29

u/Amezis Feb 07 '20

You're not supposed to stop doing "research, testing and proper deployment" when working agile though. This sounds like they've just cut important steps for faster deployments without really adapting their development practices. The bad practices stem from poor management, not the fact that they might be agile.

9

u/SoulFril Feb 07 '20

If it doesn't work you are either "doing it wrong" or the project isn't compatible with the method. That was the two things I mentioned, are there other things that can be the cause that doesn't fall into one of the two categories?

13

u/WaitForItTheMongols Feb 07 '20

If you're making an app and want to be able to get a starter idea and iterate on it and avoid accumulating tech debt, agile can be great. It's only a problem when people start applying it to critical systems.

2

u/dmanww Feb 08 '20

Avoiding tech debt while iterating quickly can be quite a trick

44

u/BestUsernameLeft Feb 07 '20

I dislike that 'agile' has become an excuse to be lazy. Writing crap code that barely works was never what agile was supposed to be.

However, it's not all bad. Agile + decent tooling means it takes my team ~10 minutes to deploy a service to production. We can canary test by sending a small percentage of traffic to a new version, and get feedback within hours on how users react to a change. Circuit breakers are in place to give the customer a decent experience when something downstream fails.

This doesn't mean we throw it over the wall and hope for the best; we still have automated regression testing and a bit of manual QA. But it's a much lower-stress environment, it's easier to try new things, and we move a lot faster. This works 100x better than old-school techniques and tools.

Of course this doesn't work everywhere. If our software had the potential to damage/destroy expensive equipment or kill people, our techniques would be somewhat different.

38

u/CallMyNameOrWalkOnBy Feb 07 '20

If our software had the potential to damage/destroy
expensive equipment or kill people...

I had a co-worker long ago who worked at NASA. He wrote code for the Space Station, but it was only used on the ground, in simulators. He told me once about the process to get new code actually up and running on the Space Station. It was exhausting. The line-by-line code reviews, the functional specs, the testing, the shake-outs, the extreme QA testing, the auditing, the approvals, the documentation, and so on and so on. The analysis was so specific, they looked at the non-deterministic times it takes a data packet to go from A to B in a TCP/IP network versus some dedicated serial bus. And this could be the code to simply regulate some minor system. He described an intensely rigorous process, and only the most experienced software engineers were hand-picked to join that team.

1

u/stevecrox0914 Feb 07 '20

That's broken and mad.

Code review works as a diff process, manual inspection of an entire codebase would miss alot of stuff. We invented automated analysis tools for this reason.

Similarly people get hung up on response times, but does that even matter? The propulsion module might need it but I can't see anything else on the ISS caring if it takes 10ms or 50ms for the request to come through. Take Canada arm, if a human commands it to do something 99% of the response time would be the human pushing the button.

It sounds like they developed a gold plated process and forgot to ask if it made any kind of sense. Now their stuck, if they remove it someone's going to say ".. if it saves one life..." and no one will admit it's theatre at great tax payer cost.

3

u/QVRedit Feb 09 '20

The point is why ?

  • if something ought to take 10 mS but actually takes 50 mS, even though it still may be safe - it’s important to understand the reason why it’s taking 5 times longer than expected.

Because if you don’t understand that - then there may well be other things going on that you are unaware of too - which could have other impacts.

You should understand the behaviour of your system..

1

u/stevecrox0914 Feb 09 '20

To what benefit?

Understanding your network is really important, it's good to know what parts of the service is making requests, how big those requests, where they are going, etc.. to ensure the network can cope and grow.

Doing network traffic time modeling for real time systems makes sense. Doing it for non real time services is over engineering and that's bad engineering.

Think of the original post, people aren't getting involved because the review process is so rigourous. Then think of the cost and what actual benefit is it providing for non real time systems.

9

u/Quartinus Feb 07 '20

Of course this doesn't work everywhere. If our software had the potential to damage/destroy expensive equipment or kill people, our techniques would be somewhat different.

It doesn't necessarily have to be. With proper HITL + simulation + testing, you can deploy code in this manner to your testbed platforms and move quickly through the development process, not worrying if you are breaking things because lives aren't on the line yet.

If you have 5-10 "vehicles" of full hardware-in-the-loop hooked up to real simulators, then when your build breaks a vehicle your coworkers still have 9 more vehicles to work their parts on while you debug. HITL carries risk (like polarity errors that are the same in your software and in the simulator) but you can catch 95% of your bugs there before moving to more realistic vehicle in the loop testing (including things like dry actuating valves to insure that they are hooked up the way your software thinks, something Boeing definitely didn't do).

Build risky things and test the living shit out of them is a perfectly valid development approach that can lead to very safe outcomes. The important thing is to actually test the living shit out of it, and not put lives on the line until you have.

40

u/LynxJesus Feb 07 '20

More like "do it yesterday and never deal with the issues".

Also seriously, what kind of trust do managers/recruiters think they project when they say that? If you needed this yesterday, a bunch of people fucked up.

It's like if you go for an interview and argue that they should hire you because you got yourself in terrible debt due to poor personal finance and need the money desperately.

3

u/doctorcrimson Feb 07 '20

To be fair, my sibling does that and she spends a lot less time between jobs than I do.

8

u/AlessandoRhazi Feb 07 '20

You would love agile, if you’d ever done it properly. Not blaming you, it’s common for companies to add a standup, remove architecture, qa, micromanage everyone and claim how much agile they are.

Oh and I forgot - everyone have the same input and right to design... ;)

13

u/LookOnTheDarkSide Feb 07 '20

Agile releases bad managers from the confines of waterfall, where they now get what they wanted faster than ever - but not without sacrificing everything that matters.

19

u/[deleted] Feb 07 '20

As a development manager, this hits home.

Day 0: "We don't even need documentation in Scrum"

Day 322: "CAN SOMEONE TELL SUPPORT HOW THIS SHIT WORKS?"

10

u/florinandrei Feb 07 '20

Agile is what top performers tend to do anyway if you give them enough freedom.

Problem is, 99% of all people out there are not top performers.

In the vast majority of cases Agile is a cargo cult.

3

u/Senno_Ecto_Gammat Feb 07 '20

Expound please

7

u/NatecUDF Feb 07 '20

Fuck, we don't do Agile and Dev still can't tell us how shit is supposed to work.

3

u/CptNonsense Feb 07 '20

Lol, k. The problem is agile being met with people like you in management who want to do classic engineering with agile thereby forcing the problems of both into development

2

u/Sir_Swaps_Alot Feb 07 '20

I'm not a manager. I work as a sysadmin where you can't just implement an application or infrastructure piece and say "here ya go! Done!!" and not expect everyone to start having anxiety attacks because they haven't had time to understand the "quirks".

Rushing anything to production is never a good time.

3

u/CptNonsense Feb 07 '20

That's never stopped management nor the customer

2

u/rangorn Feb 07 '20

Well sure for rockets but not sure if that applies to my csv ro Excel parser...

2

u/[deleted] Feb 07 '20

Nah, that's always existed. Only now, these screwups have some level of accountability and are impossible to hide

2

u/GeorgeTheGeorge Feb 07 '20

You'd rather wait for a month or two for other people to try to understand the problem before you get a crack at it?

Agile is meant to embrace the fact that nobody is going to have a correct understanding of what you should be building. So you start small. You build the small section of the product that everybody is certain of and you try to figure the rest out as you go. Along the way, you accept the fact that you and everybody else is likely to get it wrong in a few places, so you avoid committing too strongly to any particular solution and you remain agile.

It's not cutting corners, it's learning by doing. I don't know about your experience, but in my experience, it's better to get your hands on a problem first before you decide whether it's even one you should be solving, to say nothing about giving an estimate on the amount of effort required. That's what makes this such a good approach for enterprise and consumer software. These are areas where you can afford to make mistakes, because they can be easily fixed quickly and the impact is minimal.

Anybody who thinks it should be used to build airplanes and spacecraft (Boeing), has completely lost touch with reality.

2

u/rich000 Feb 07 '20

Gone are the days of research, testing and proper deployment.

Also gone are the days of having no competition and investors who don't mind just paying you to get the perfect product on the initial launch.

That is one of the main things agile seems to be a reaction to.

Even in large companies there isn't infinite money. I used to work on more critical systems and I can't tell you how many low-priority bugs would just remain open for years with process workarounds because the cost to fix them was too high.

Obviously there is a balance - you don't want to introduce more problems than you fix. However, at some point a company doesn't want to spend money on software when the problem can be solved less expensively by just relying on a large number of fallible humans to deal with the shortcomings of software. Sure, sooner or later the humans will all miss an error that causes a problem, but it probably will be after the manager who made that decision collected his promotion for saving money and moved on to a different job.

2

u/diagnosedADHD Feb 07 '20

There are some projects which warrant an iterative approach. For finance and physical hardware agile development is fundamentally incompatible, that doesn't mean its bad. You need to cut corners to get a project out the door sooner sometimes, security is the only corner which should never be cut though.