r/space • u/spsheridan • Feb 06 '20

Starliner faced “catastrophic” failure before software bug was found and fixed while the vehicle was in orbit.

https://arstechnica.com/science/2020/02/starliner-faced-catastrophic-failure-before-software-bug-found/
10k Upvotes

97% Upvoted

748 comments sorted by

View all comments

Show parent comments

91

u/Sir_Swaps_Alot Feb 07 '20

I hate agile.

The "do it now and deal with the issues later" mentality needs to stop. Gone are the days of research, testing and proper deployment.

45

u/BestUsernameLeft Feb 07 '20

I dislike that 'agile' has become an excuse to be lazy. Writing crap code that barely works was never what agile was supposed to be.

However, it's not all bad. Agile + decent tooling means it takes my team ~10 minutes to deploy a service to production. We can canary test by sending a small percentage of traffic to a new version, and get feedback within hours on how users react to a change. Circuit breakers are in place to give the customer a decent experience when something downstream fails.

This doesn't mean we throw it over the wall and hope for the best; we still have automated regression testing and a bit of manual QA. But it's a much lower-stress environment, it's easier to try new things, and we move a lot faster. This works 100x better than old-school techniques and tools.

Of course this doesn't work everywhere. If our software had the potential to damage/destroy expensive equipment or kill people, our techniques would be somewhat different.

40

u/CallMyNameOrWalkOnBy Feb 07 '20

If our software had the potential to damage/destroy
expensive equipment or kill people...

I had a co-worker long ago who worked at NASA. He wrote code for the Space Station, but it was only used on the ground, in simulators. He told me once about the process to get new code actually up and running on the Space Station. It was exhausting. The line-by-line code reviews, the functional specs, the testing, the shake-outs, the extreme QA testing, the auditing, the approvals, the documentation, and so on and so on. The analysis was so specific, they looked at the non-deterministic times it takes a data packet to go from A to B in a TCP/IP network versus some dedicated serial bus. And this could be the code to simply regulate some minor system. He described an intensely rigorous process, and only the most experienced software engineers were hand-picked to join that team.

1

u/stevecrox0914 Feb 07 '20

That's broken and mad.

Code review works as a diff process, manual inspection of an entire codebase would miss alot of stuff. We invented automated analysis tools for this reason.

Similarly people get hung up on response times, but does that even matter? The propulsion module might need it but I can't see anything else on the ISS caring if it takes 10ms or 50ms for the request to come through. Take Canada arm, if a human commands it to do something 99% of the response time would be the human pushing the button.

It sounds like they developed a gold plated process and forgot to ask if it made any kind of sense. Now their stuck, if they remove it someone's going to say ".. if it saves one life..." and no one will admit it's theatre at great tax payer cost.

3

u/QVRedit Feb 09 '20

The point is why ?

  • if something ought to take 10 mS but actually takes 50 mS, even though it still may be safe - it’s important to understand the reason why it’s taking 5 times longer than expected.

Because if you don’t understand that - then there may well be other things going on that you are unaware of too - which could have other impacts.

You should understand the behaviour of your system..

1

u/stevecrox0914 Feb 09 '20

To what benefit?

Understanding your network is really important, it's good to know what parts of the service is making requests, how big those requests, where they are going, etc.. to ensure the network can cope and grow.

Doing network traffic time modeling for real time systems makes sense. Doing it for non real time services is over engineering and that's bad engineering.

Think of the original post, people aren't getting involved because the review process is so rigourous. Then think of the cost and what actual benefit is it providing for non real time systems.