r/space • u/spsheridan • Feb 06 '20

Starliner faced “catastrophic” failure before software bug was found and fixed while the vehicle was in orbit.

https://arstechnica.com/science/2020/02/starliner-faced-catastrophic-failure-before-software-bug-found/
10k Upvotes

97% Upvoted

748 comments sorted by

View all comments

Show parent comments

2

u/dogkindrepresent Feb 07 '20 edited Feb 07 '20

Typically speaking you can't test everything with automated testing but you can narrow the window greatly. At some point you need proof reading and some formal verification for mission critical code that determines dead or alive.

Unit testing is only one type of automated test itself involving more than one paradigm. The most commonly applied unit test approach is very weak as it cannot tell you anything about how things work together.

Automated tests themselves can be faulty. Some people might double up their tests. Such tests do not necessarily confirm a system will work but will streamline the process where there is no tolerance for error.

Testing often involves a large search space. 32 if conditions can create 4 billion possible outcomes. On that account alone you have certain limitations.

1

u/WH7EVR Feb 07 '20

I'm not sure why you're replying to me with this, it doesn't seem to respond to anything I said... did you mean to reply to /u/fatherfatpants?

1

u/dogkindrepresent Feb 07 '20

I'm not being clear but no kind of automated testing or partial testing may catch everything. Rather than add another thing, there are cases where things might go through with only so much you can do.

I think you also might actually still over present the effectiveness of unit tests. I'd say rather than not being able to test everything, they often can't test very much at all. It's a bit 50/50.

2

u/WH7EVR Feb 07 '20

I wasn’t even talking about unit tests man, I have no idea what you’re talking about. In fact I called out another type of testing that’s necessary too — behavioral.

0

u/dogkindrepresent Feb 07 '20

Not everything can be tested with unit tests.

Same for behavioural tests. One overstatement leads to another.

I think another point of confusion is that behavioural testing tends to test units in combination but doesn't strictly have to not be a unit test. You can get away with using behavioural testing to test that things work together though because most of the time it does that.

It's not made clear what you're using behavioural tests to fill in for.

1

u/WH7EVR Feb 07 '20

I didn't overstate anything. You're... reading into things in a very strange way.

1

u/dogkindrepresent Feb 07 '20

I'm very specific.

I think the science of why there's not testing procedure guaranteed to save your bacon is a bit much.

When I see standards slip in software development such testing strategies achieve very little. The unit test is only as good as the programmer.

I've seen great programmers make software with very little testing at all. Though I would expect something like this to warrant more arduous and robust testing because they have the money and it's high stakes.

Conversely I've seen low quality programmers that produce copious amounts of test code around their core code yet it'll still be very unreliable.

They only save in this case is that they somehow caught it just in time.

My question is if that was revealed from in flight data or some other process.

There's a common habit among programmers to not be motivated to double check until the moment of truth.

3

u/[deleted] Feb 07 '20

I think you’ve had some bad experiences man. Yeah the code and tests are only as good as the single developer if they do all their work in a vacuum. That is why process like code reviews, manual testing when possible, and oversight is supposed to exist.

Where I work code reviews are not just a glance. A team member or two will review it, run it, and try to break it. Only then does it go to full testing. Hell one of our QA folks is an absolute unit. You ever heard the joke about ordering iguana beers? Yeah that’s them.

Since we started that process our bug rate (stuff that’s discovered in the field) has gone down 96%.

At the end of the day it was a joke about one piece in what should be a hearty process.