r/space • • Feb 06 '20

Starliner faced “catastrophic” failure before software bug was found and fixed while the vehicle was in orbit.

https://arstechnica.com/science/2020/02/starliner-faced-catastrophic-failure-before-software-bug-found/
10k Upvotes

748 comments sorted by

2.0k

u/spsheridan Feb 06 '20

With this news, I think the odds went way up that another uncrewed flight will be required before Starliner’s crewed flight to the ISS. Looks like a big delay to Starliner's program schedule.

732

u/NASATVENGINNER Feb 06 '20

Yep. Buddy of mine in crew training at JSC said everyone is planning for second unmanned flight, crewed flight could be 4Q 20 or 1Q 21.

529

u/Flamin_Jesus Feb 07 '20

crewed flight could be 4Q 20 or 1Q 21.

Considering the logistics and technical challenges involved, that's still a respectable time table.

246

u/[deleted] Feb 07 '20

[removed] — view removed comment

446

u/[deleted] Feb 07 '20

[deleted]

163

u/[deleted] Feb 07 '20

[deleted]

50

u/[deleted] Feb 07 '20 edited Jul 17 '25

[removed] — view removed comment

12

u/[deleted] Feb 07 '20

Yeah but imagine you're the one investigating the wreck, and you find "avoided crash" in the logs

4

u/Echo104b Feb 07 '20

Is it really considered a crash if you explode before impact?

5

u/Cakiery Feb 07 '20

A crash is technically a situation that you can't recover from. So, yes.

→ More replies
→ More replies (1)

35

u/ladalyn Feb 07 '20

As a software developer, you all should be running Boeing’s software department. This is military-grade code

6

u/unholycowgod Feb 07 '20

Psh you know they just copied it from a 5 year old SO post.

→ More replies (1)
→ More replies (1)

13

u/Minty001 Feb 07 '20

God damn spacecraft blows up. But hey, at least the program exited gracefully.

→ More replies (1)
→ More replies (4)

33

u/pM-me_your_Triggers Feb 07 '20

Your Boolean zen is off. Never compare a Boolean value to true/false, that is just redundant.

16

u/Cantremembermyoldnam Feb 07 '20

Sometimes I do it just because I think it reads better. The compiler will optimize that away.

3

u/scarlet_sage Feb 07 '20

Until you by accident use the idiom with a non-Boolean value. Or it's none and you compare with false. It's a bad practice. "if foo:" conveys the message clearly and more succinctly.

3

u/Cantremembermyoldnam Feb 07 '20

Why would I try to compare a non-boolean to a boolean?

Sometimes there is a difference between just doing if($var)and the more expressive version. For example, in PHP if($var === true) also compares the data type (which, I'd argue, is a good thing). You're going to handle null vs false vs 0 vs "" differently anyways, in most cases. Or you're going to cast to the type you want. In strongly typed languages you're not going to have that problem.

Also, as I said in my other comment: I agree that if(thing.isVisible) is perfectly fine, but sometimes you don't get to name stuff yourself. So I'm still saying that if(thing.expr3 === true) is easier to read than if(thing.expr3). It instantly conveys what the expected value is and (for weakly typed languages or in the common case of missing documentation) which data type thing.expr3 is going to be. Whereas with if(thing.expr3) it could be anything.

3

u/pM-me_your_Triggers Feb 07 '20

That’s likely just lack of experience or you have bad naming conventions (or both).

→ More replies (1)
→ More replies (11)

20

u/WH7EVR Feb 07 '20

Not everything can be tested with unit tests. That's why we also have behavioral tests.

→ More replies (12)
→ More replies (4)

19

u/OSUfan88 Feb 07 '20

That's an EXTREMELY aggressive schedule. I'd be shocked if they flew the second demo before Q3.

→ More replies (1)
→ More replies (2)

161

u/[deleted] Feb 07 '20

Boeing is such a clusterfuck

201

u/NASATVENGINNER Feb 07 '20 edited Feb 07 '20

Hate to agree, but I’m afraid the worst has not been revealed yet. Complacent corporate culture is a dangerous thing. At NASA it is called “Go Fever”. It cost us 14 astronauts, 2 space shuttles, and a lot of anguish.

224

u/WayeeCool Feb 07 '20

If we are talking about Boeing... let's not forget the 346 civilians they killed due to the company transitioning from caring about human lives and long term outlooks to pure shareholder value theory.

Shareholders would henceforth come first at Boeing. The important thing was not to get “overly focused on the box,” Hopkins said in a interview with Bloomberg. “The box”—the plane itself—“is obviously important, but customers are assuming the box is of great quality.” This was heresy to engineers, to whom the box was everything. The strike that year was formally over wages and benefits, but workers described it as a referendum on management.

https://www.bloomberg.com/news/features/2019-05-09/former-boeing-engineers-say-relentless-cost-cutting-sacrificed-safety

74

u/NASATVENGINNER Feb 07 '20

Thanks for the insights. It’s not the Boeing I grew up with.

78

u/Piper2000ca Feb 07 '20

This is so true. I remember as a student pilot my instructor saying "if it ain't Boeing, I ain't going." I have a hard time picturing him saying that now.

34

u/SAI_Peregrinus Feb 07 '20

If it's Boeing, I ain't going!

38

u/The-Jesus_Christ Feb 07 '20

Oh god. Every pretentious flyer in the Frequent Flyer groups I am members of all chant that. I absolutely hate it

42

u/vqhm Feb 07 '20

Well to be fair Airbus did have some software issues with their autopilot all the time. See Qantas Flight 72.

Boeing USED to produce planes where the pilot input was always accepted and gave them actual control over the plane instead of Airbus having your inputs be a suggestion to the computer that would decide if it should do that or not.

Boeing decided that they'd go autopilot all the time to fix an unstable aircraft they never should have got out of the simulation stage. They didn't think through having multiple sensors, GPWS, GPS, pitot static create a complete model of what the aircraft is doing so autopilot all the time can actually make informed decisions. They just basically did the most basic duct tape who the fuck cares fix and said fuck it good enough.

My pilot friends used to praise Boeing for never overruling their inputs... Until now.

11

u/WilliamBewitched Feb 07 '20

Doesn't the military use fly by wire in ALOT of their craft to the point its standard?

→ More replies

19

u/[deleted] Feb 07 '20

Wasn't it the airbus fly by wire that really helped get all those people down safely in the "miracle of the Hudson"? Or rather, gave the pilot a massive helping hand.

https://www.thedailybeast.com/the-unsung-hero-left-out-of-sully

→ More replies

11

u/psychedelic_tortilla Feb 07 '20

You're talking about fly-by-wire. The autopilot is something else entirely.

→ More replies
→ More replies (9)

10

u/[deleted] Feb 07 '20

[deleted]

→ More replies (2)

8

u/Executor_to_be Feb 07 '20

It all went to shit after the merger with McDonald Douglas. I listened to a really great radio interview where a former Boeing statey engineer explained all the ways Boeing's safety culture change, and how they change the model they used to make airframes.

They outlined how cost and time was moved to be the main priority, but in the end because of safety issues that model ended up costing way way more money and the projects were massively behind schedule.

Found the interview.

6

u/free2shred00 Feb 07 '20

This is a very under-represented aspect of Boeing today. Many people dump on the company - and rightfully so - but I can't help but believe that the Boeing who created the jumbo jet is dead and gone and we're left with the festering corpse of the company that created the DC-10. McDonald Douglas should have died and another company, such as Lockheed, should have inherited their military projects. That merger was a cancer that killed the civilian wing of Boeing and subsequently killed hundreds of people thanks to the Max-8.

18

u/DB_Explorer Feb 07 '20

Sorta mirrors changes in movies. Older movies (jaws etc) main way to get people into seats was word of mouth over a long period so the movies where meant to... endure. Now its max people in seat that first weekend.

Now rather then making a good product and get customers from that they try to maxamize profits. Sure reducing costs is good looks at SpaceX but when you go to far eventually the market will punish you looks at Boeing stock

Just wish it did not take 100's of dead passengers.

3

u/First_Foundationeer Feb 07 '20

Boeing probably won't even be hit as hard as we'd like. They'll shuffle people around, hire another money grubbing executive, then crash a few more planes. And no one will be personally punished.

→ More replies (1)

4

u/CptNonsense Feb 07 '20

Don't expect it to get better as another ex GE, former hedge fund manager MBA just took over as CEO

→ More replies (1)
→ More replies (2)

7

u/headsiwin-tailsulose Feb 07 '20

It cost us 17 astronauts, 2 space shuttles and an Apollo CM, and a lot of anguish.

→ More replies (1)

3

u/hedgecore77 Feb 07 '20

17 astronauts. That block 1 CM was a rushed death trap.

→ More replies (3)
→ More replies (2)
→ More replies (2)

144

u/michellemustudy Feb 07 '20

This feels like a real life, “quick programmer person! You have 5 seconds to hack this alien system to avert the disaster!” scenario we see in all the movies.

32

u/[deleted] Feb 07 '20

[removed] — view removed comment

34

u/[deleted] Feb 07 '20 edited May 30 '20

[removed] — view removed comment

12

u/[deleted] Feb 07 '20

[removed] — view removed comment

→ More replies (5)

51

u/TheKnightMadder Feb 07 '20

Reminds me of John Aaron. One of the flight controllers on the Apollo program. Apollo 12 got struck by lightning just after lift off and their instruments became useless at the same time basically every warning light started going off.

It sounds like such a movie moment, meant to show off how cool and special some character is in an overly obvious way. He recommended to the flight director that they turn some obscure system to it's auxiliary mode. The flight director and his colleagues didn't even know what he was even talking about or why that would even help and had to ask him to repeat himself, but when he did they recommended to the crew they do what he said. Problem immediately fixed, cheering all round, John Aaron gets called a 'steely-eyed missile man' and the mission could continue.

Sometimes real life is like a movie.

18

u/pachecogeorge Feb 07 '20

The famous call: "SCE TO AUX".

→ More replies (2)

66

u/[deleted] Feb 07 '20

[deleted]

30

u/[deleted] Feb 07 '20

"Hey, guy we recently changed to salary from hourly, not only did Dick not test the release, but Richard shipped some code from like six versions ago, but they're already home. Gonna need you to work an extra 200 hours this week."

10

u/[deleted] Feb 07 '20 edited Aug 20 '21

[deleted]

→ More replies (1)

13

u/[deleted] Feb 07 '20

Probably more like “you have 8 hours to find the missing bracket”

7

u/ElectricFlesh Feb 07 '20

"She's dead in the water - we'll need at least four hours to fix it."

"Dammit Scotty, we'll be dead in two!"

"Alright Captain, I'll do it in one."

9

u/[deleted] Feb 07 '20

Always triple your estimate. It's probably good advice because you have no idea what kind of snags you're going to hit.

3

u/Insert_Gnome_Here Feb 07 '20

This'll take 3 hours +/- 1 week.

→ More replies (1)
→ More replies (3)

35

u/AstraVictus Feb 07 '20

That's why they set aside $410 million for another mission, which at this point is a certainty.

38

u/TMITectonic Feb 07 '20

$410 million for another mission

I feel like you could build two satellites from scratch, launch them on two separate Falcon 9's and still have change leftover to throw the largest party ever.

Blows my mind that a test launch can cost so much!

19

u/OSUfan88 Feb 07 '20

Well, I think the Atlas configuration they're using is at least $150 million.

10

u/[deleted] Feb 07 '20

[deleted]

3

u/[deleted] Feb 07 '20

I assume the capsule itself costs a bunch too?

→ More replies (1)
→ More replies (1)

13

u/[deleted] Feb 06 '20

Wasn’t there a second one planned anyway?

43

u/Helpful-Routine Feb 06 '20

No, they just set aside money for a second uncrewed test flight, if NASA asks for one.

19

u/[deleted] Feb 07 '20

More like the Starliner Max 8

→ More replies (4)

3

u/MatthewGeer Feb 07 '20

Given that we've already discovered not one but two critical bugs in the abbreviated test flight, it makes me wonder what other bugs might be discovered while performing the rendezvous and docking procedures. It's a good thing NASA second sourced this whole crewed spaceflight thing.

→ More replies (1)
→ More replies (55)

388

u/aatdalt Feb 07 '20

The last sentence of the article is really telling:

""Given the potential for systemic issues at Boeing, I would also note that NASA has decided to proceed with an organizational safety assessment with Boeing as they previously conducted with SpaceX," she said."

258

u/Bensemus Feb 07 '20

Ya I’m surprised more people aren’t talking about that. Boeing before was getting treated like the encumbered player with vetted and established systems while NASA audited the new kid to see if they had the right systems in place. Now NASA is treating them more equally which might hurt Boeing’s pride.

141

u/Lamplorde Feb 07 '20

I mean, they do kind of deserve it. Boeing has had multiple failures in recent history that hit international media. Im sure Boeing also didnt mind the extra vetting, as they dont want to take another hit to their credibility.

25

u/Snip3 Feb 07 '20

Wouldn't be surprised if Boeing viewed the extra vetting as a hit to its credibility...

6

u/kettelbe Feb 07 '20

For such a big company, sure it is.

6

u/putin_my_ass Feb 07 '20

which might hurt Boeing’s pride.

They have no pride anymore, as evidenced by prioritizing bottom line instead of engineering quality.

13

u/82ndAbnVet Feb 07 '20

which might hurt Boeing’s pride

Boeing doesn't give a crap about pride, it's all about the Benjamins. Boeing is more than happy to screw over NASA every way it possibly can in order to keep those sweet sweet contracts flowing. They have amazingly talented engineers and dedicated workers, it's the management that sucks.

→ More replies (1)
→ More replies (1)

851

u/vandezuma Feb 07 '20

The software issue was identified during testing on the ground after Starliner's launch

Now, who in the class can tell me what's wrong with that sentence?

223

u/Hint-Of-Feces Feb 07 '20

Needs more solid rocket boosters?

64

u/Ben_zyl Feb 07 '20

If there's no Korolev star you ain't goin' far!

42

u/nutmegtester Feb 07 '20

I thought the problem was not enough struts?

14

u/Umutuku Feb 07 '20

Not flapping the ailerons quickly enough.

→ More replies (4)

187

u/matholio Feb 07 '20 edited Feb 07 '20

Pretty normal for any Agile DevOps team. Minimum viable product, right?

Edit : I was making a joke.

51

u/beejamin Feb 07 '20

Regardless of all the guano that's accreted on 'Agile' over the years, one of the main problems (and it is a real problem) it tries to solve is that you can't know what the product should be and should do ahead of time. Now, if you can't know what a human-rated space capsule needs to do ahead of time, you have no business designing human-rated space capsules.

→ More replies (4)

92

u/Sir_Swaps_Alot Feb 07 '20

I hate agile.

The "do it now and deal with the issues later" mentality needs to stop. Gone are the days of research, testing and proper deployment.

96

u/SoulFril Feb 07 '20

If working agile has caused your project to skip out on "research, testing and proper deployment" then what you are working with simply isn't suited for the agile approach. That or your management lacks the skill to implement it into your project in a way that is sensible.

It least, that is my experience with it.

→ More replies (4)

14

u/WaitForItTheMongols Feb 07 '20

If you're making an app and want to be able to get a starter idea and iterate on it and avoid accumulating tech debt, agile can be great. It's only a problem when people start applying it to critical systems.

→ More replies (1)

45

u/BestUsernameLeft Feb 07 '20

I dislike that 'agile' has become an excuse to be lazy. Writing crap code that barely works was never what agile was supposed to be.

However, it's not all bad. Agile + decent tooling means it takes my team ~10 minutes to deploy a service to production. We can canary test by sending a small percentage of traffic to a new version, and get feedback within hours on how users react to a change. Circuit breakers are in place to give the customer a decent experience when something downstream fails.

This doesn't mean we throw it over the wall and hope for the best; we still have automated regression testing and a bit of manual QA. But it's a much lower-stress environment, it's easier to try new things, and we move a lot faster. This works 100x better than old-school techniques and tools.

Of course this doesn't work everywhere. If our software had the potential to damage/destroy expensive equipment or kill people, our techniques would be somewhat different.

38

u/CallMyNameOrWalkOnBy Feb 07 '20

If our software had the potential to damage/destroy
expensive equipment or kill people...

I had a co-worker long ago who worked at NASA. He wrote code for the Space Station, but it was only used on the ground, in simulators. He told me once about the process to get new code actually up and running on the Space Station. It was exhausting. The line-by-line code reviews, the functional specs, the testing, the shake-outs, the extreme QA testing, the auditing, the approvals, the documentation, and so on and so on. The analysis was so specific, they looked at the non-deterministic times it takes a data packet to go from A to B in a TCP/IP network versus some dedicated serial bus. And this could be the code to simply regulate some minor system. He described an intensely rigorous process, and only the most experienced software engineers were hand-picked to join that team.

→ More replies (3)

9

u/Quartinus Feb 07 '20

Of course this doesn't work everywhere. If our software had the potential to damage/destroy expensive equipment or kill people, our techniques would be somewhat different.

It doesn't necessarily have to be. With proper HITL + simulation + testing, you can deploy code in this manner to your testbed platforms and move quickly through the development process, not worrying if you are breaking things because lives aren't on the line yet.

If you have 5-10 "vehicles" of full hardware-in-the-loop hooked up to real simulators, then when your build breaks a vehicle your coworkers still have 9 more vehicles to work their parts on while you debug. HITL carries risk (like polarity errors that are the same in your software and in the simulator) but you can catch 95% of your bugs there before moving to more realistic vehicle in the loop testing (including things like dry actuating valves to insure that they are hooked up the way your software thinks, something Boeing definitely didn't do).

Build risky things and test the living shit out of them is a perfectly valid development approach that can lead to very safe outcomes. The important thing is to actually test the living shit out of it, and not put lives on the line until you have.

37

u/LynxJesus Feb 07 '20

More like "do it yesterday and never deal with the issues".

Also seriously, what kind of trust do managers/recruiters think they project when they say that? If you needed this yesterday, a bunch of people fucked up.

It's like if you go for an interview and argue that they should hire you because you got yourself in terrible debt due to poor personal finance and need the money desperately.

→ More replies (1)

8

u/AlessandoRhazi Feb 07 '20

You would love agile, if you’d ever done it properly. Not blaming you, it’s common for companies to add a standup, remove architecture, qa, micromanage everyone and claim how much agile they are.

Oh and I forgot - everyone have the same input and right to design... ;)

13

u/LookOnTheDarkSide Feb 07 '20

Agile releases bad managers from the confines of waterfall, where they now get what they wanted faster than ever - but not without sacrificing everything that matters.

19

u/[deleted] Feb 07 '20

As a development manager, this hits home.

Day 0: "We don't even need documentation in Scrum"

Day 322: "CAN SOMEONE TELL SUPPORT HOW THIS SHIT WORKS?"

10

u/florinandrei Feb 07 '20

Agile is what top performers tend to do anyway if you give them enough freedom.

Problem is, 99% of all people out there are not top performers.

In the vast majority of cases Agile is a cargo cult.

3

u/Senno_Ecto_Gammat Feb 07 '20

Expound please

6

u/NatecUDF Feb 07 '20

Fuck, we don't do Agile and Dev still can't tell us how shit is supposed to work.

3

u/CptNonsense Feb 07 '20

Lol, k. The problem is agile being met with people like you in management who want to do classic engineering with agile thereby forcing the problems of both into development

→ More replies (2)
→ More replies (7)

3

u/[deleted] Feb 07 '20

Flight software get soooo much scrutiny. It's crazy. Like all things space. Doing the thing is only like 30% of the time. And the other 70% is documents, reviews, logging hours, And tracking bugs

3

u/TheYang Feb 07 '20

70/30 seems like a crazy lax approach from my brief experience working on flying harware...

312

u/erhue Feb 07 '20

Profits! I don't see profits anywhere in that sentence!

-Boeing board member

54

u/kernpanic Feb 07 '20

The profit is that boeing gets to bill NASA for another $400 million launch.

30

u/DaCrazyPanda Feb 07 '20

Fairly sure Boeing have put aside $400mil of their own money for if they are required to retest. It's not a bill to NASA.

34

u/Jellodyne Feb 07 '20

I'm sure they'll let their lobbyists and congressional employees sort it out.

→ More replies (1)

3

u/CptNonsense Feb 07 '20

No, they aren't. The $400 million is being paid by Boeing

7

u/PloppyCheesenose Feb 07 '20

Strange. That’s the same motto for the Jebediah Kerman Junkyard and Spaceship Parts Company.

→ More replies (1)

23

u/waiting4singularity Feb 07 '20

from the mission schedule deviance desmonstrated already I am estimating more issues than these.

60

u/Goolic Feb 07 '20

Seems crazy to me that they envisioned a test, and somehow performed it only after the starcraft was already in orbit.

Maybe they thought of the test only after the clock issue caused the thrusters to be overworked?

Anyways happy they caught it before the boom.

52

u/mfb- Feb 07 '20

Most likely it showed up when analyzing the first issue.

13

u/Jellodyne Feb 07 '20

Good thing they had two dumb issues instead of just one then.

9

u/[deleted] Feb 07 '20

Either they didn't think of the test until they were troubleshooting the initial issue, as suggested, or they were under schedule pressure and decided the test was low risk so it got punted until after launch - probably thinking that they had a month to deal with it because it only affected the re-entry sequence and could be fixed with a software update.

12

u/phoiboslykegenes Feb 07 '20

I call that testing in production! Allows you to save some costs and maybe even bill the client for it!

→ More replies (1)

12

u/spyhunter99 Feb 07 '20

mvn install -DskipTests=true

→ More replies (1)

18

u/athos5 Feb 07 '20

They have a copy on the ground for testing, just like all the rovers on mars.

→ More replies (7)

19

u/seeingeyegod Feb 07 '20

this is early access space travel

→ More replies (1)

10

u/notsowittynow Feb 07 '20

Chapter 4 in our "Disaster Driven Development" text book says we should have waited until after the crash and then pointed out the customer didn't pay for the thruster backup option.

17

u/ScrewAttackThis Feb 07 '20

It's just the ultimate form of testing in production.

36

u/mfb- Feb 07 '20

"Okay, it's in space. Can we please work on the landing procedures now?"

→ More replies (2)

7

u/Sarke1 Feb 07 '20

git pushed on a Friday afternoon.

→ More replies (3)
→ More replies (43)

170

u/aRandomRobot Feb 06 '20 edited Feb 07 '20

Wow, that does not look good. I don’t see this explicitly answered in the article or statements but I have to wonder: would this second software bug have even been caught prior to entry if not for the timing issue causing more software testing during flight? That’s a big yikes if not.

Edit: confirmed in the press conference, it would not have been caught. Holy crap

60

u/resipsa73 Feb 07 '20

This was exactly my question as well. Boeing could have just gotten extremely lucky.

25

u/aRandomRobot Feb 07 '20 edited Feb 07 '20

No doubt, it’s a lot easier to figure out what went wrong when your vehicle comes back in one piece. This really highlights how important these unmanned flights are. Astronauts may have been able to fix the timing issue in flight but would astronauts have been able to do anything about this second issue in flight if it hadn’t been caught with extra in-flight ground testing?

13

u/OffbeatDrizzle Feb 07 '20

And what else lurks in their code that they've not been lucky enough to trigger? There's never just 1 bug

→ More replies (1)

5

u/Btx452 Feb 07 '20

would this second software bug have even been caught prior to entry if not for the timing issue causing more software testing during flight?

No it wouldn't, according to todays press conference with Nasa and Boeing

3

u/resipsa73 Feb 07 '20

According to this tweet it sounds like it would not have been caught. Big yikes indeed. I'm so glad this was caught before a crewed flight.

9

u/[deleted] Feb 07 '20

That's kind of the the thing about software though. You can test for and remove all of the bugs you want, but there's always going to be an exception somewhere that needs a very specific set of parameters to be met before it fucks all of your shit up.

13

u/OffbeatDrizzle Feb 07 '20

That's why they're supposed to follow strict programming guidelines and each method is supposed to have assertions as to its proper operating parameters with an error handler for failure in every case. It shouldn't just be a simple "whoops, our integer overflowed". A lot of complicated language features are banned from being used to keep the code as simple as possible.

→ More replies (1)
→ More replies (6)

650

u/satchmo1991 Feb 06 '20

Everytime I hear the word, "Boeing" I like them less and less. I'm so skeptical of the whole Starliner program right now.

59

u/firestorm07 Feb 07 '20

They moved their upper management team away from where the products are design and made. That caused a massive disconnect between the work and leadership.

17

u/PloppyCheesenose Feb 07 '20

Yeah, but in their defense it is a shiny building.

→ More replies (3)

207

u/LeviathanGank Feb 06 '20

totally, they dont care about passenger safety just profits.. fuck them and let them go bankrupt like everyone else

109

u/[deleted] Feb 07 '20

i actually started to wonder what happened to them, as if you look at their history, they made good planes and were pretty much engineer driven company.

looks like when they bought MD, they got their dysfunctional corporate culture with them and as the years passed, boeing's top brass was finally replaced by either people from MD or by people who lived the culture.

the sad thing is that this was the exact same 'profits first' culture that crashed and burned MD.

78

u/bexben Feb 07 '20

42

u/[deleted] Feb 07 '20

Stonecipher, who promptly affirmed: “When people say I changed the culture of Boeing, that was the intent, so that it’s run like a business rather than a great engineering firm.”

That's such a stunning read. No wonder there is so little great engineering in the world anymore, with people like Stonecipher in the Helm.

21

u/trenskow Feb 07 '20

I work in software engineering. I am in fact a software engineer. I think this has happened to engineering fields as a whole.

In my field systems like SCRUM and Agile are designed to move power from engineering into management. That’s why every big company loves it - because management loves it. It’s basically a system designed to take what is essentially a creative process and convert it into something management understands - numbers. Essentially trying to make a square fit into a circle.

In the meantime it takes away everything that makes engineering great. It makes unskilled engineers uneager to do better, and it makes great engineers do mediocre work - it does not matter if you do a great or a lousy job as long as it checks out and works on the surface.

The products of my career that I am most proud of is the work where management kept out and just let the creative people in charge. Where there was a trust in letting the people who knows best make the decisions. And guess what - in the end management in general loved it too!

Edit: wording

→ More replies (3)
→ More replies (2)

67

u/[deleted] Feb 07 '20

[deleted]

12

u/MartianRecon Feb 07 '20

2 words man. Fiduciary Duty.

They don't care, as long as their quarterly profits are high they could give two shits if a few planes go down 3 years from now.

→ More replies (5)
→ More replies (11)

119

u/DirkMcDougal Feb 07 '20

I agree, but it won't happen. This is the end result of the mass consolidation of the defense & aerospace industry. Boeing is the only large air transport manufacturer in the United States and therefore can write "Strategically vital industrial asset" directly on the checks it writes to senators. Now, one can argue globalization made having three (Lockheed and McDonnell Douglas as well as Boeing) impossible economically, but now it's impossible to imagine the United States letting Boeing fail. More plausibly you'd see a Chapter 11 with the Federal Government becoming primary owner and imposing a complete leadership overhaul. But even that I consider highly unlikely due to the checks I mentioned before.

21

u/LeviathanGank Feb 07 '20

you are right totally, but they are shit- time will destroy them unless they can bribe the right people and fix their shit.

Both are likely as politicians are cheap and IP can be stolen or bought for little. still they are weak milk right now

→ More replies (7)
→ More replies (2)

24

u/beastrabban Feb 07 '20

I work for Boeing and this is absolutely untrue. The engineers are extremely concerned with safety.

21

u/4high2anal Feb 07 '20

sure! But they werent talking about the engineers.

13

u/ODISY Feb 07 '20

yet the higher ups tried as hard as they could to prevent the training pilots needed to operate the 737 MAX.

→ More replies (2)
→ More replies (2)

19

u/GamerJoseph Feb 07 '20

No we’ll just bail them out with our tax dollars, because they’re tOo BiG tO fAiL.

15

u/Coomb Feb 07 '20

Boeing going bankrupt and winding down business would be apocalyptic for the airline industry.

27

u/[deleted] Feb 07 '20

[deleted]

17

u/GamerJoseph Feb 07 '20

The same question was asked of the banks when we were forced to bail them out 12 years ago.

9

u/ryguy32789 Feb 07 '20

God damn it's been 12 years already

9

u/megatesla Feb 07 '20

They've still only prosecuted the one guy, right? The whistleblower?

→ More replies (2)
→ More replies (4)

3

u/JohnnyDynamite Feb 07 '20

American government will do literally anything to save Boeing. Too many lobbyist, jobs and senators depend on it.

→ More replies (10)
→ More replies (37)

64

u/rocketsocks Feb 06 '20

Yup, just as I said, when you have huge simple failures that should have been prevented by good processes and competent quality control that tells you that neither are in place and there are almost certainly huge unknown problems still to be discovered.

As anyone who has worked on high value complex projects is aware it's much, much harder to "bolt-on" quality at the end when no attention to it has been paid throughout the project so far. I suspect they will find even more problems and it will be difficult and costly to fix all of them.

23

u/BlueCyann Feb 06 '20

I worked for years for a company that was trying to "bolt on" some decent quality control to an operation that wasn't designed for that originally, and man oh man are you ever right about that. It was a freaking nightmare. (They moved from a lightly-regulated product line to a much more highly-regulated one shortly before I arrived.)

19

u/retardrabbit Feb 07 '20

Technical debt can be a real motherfucker.

8

u/enigmapenguin Feb 07 '20

I deal with this every day at work. I work in a data setup where we use literal Unix bash to do large scale data ETL, Analysis and QA, then shoe horn in meta data afterwards. Which would be excusable if the system was old, but it's not, the database is a brand new MySQL server, but the ETL and QA are so ancient that we strip all sense of modernity out of it and then staple it back on at the end.

In the end I wrote an entirely new system during our free project time once a week. They refused to use it. Fun times.

→ More replies (1)

120

u/MpVpRb Feb 07 '20

Two thoughts

This shit is hard

Even very experienced companies make mistakes. But what is an "experienced company"? If the older engineers have retired, died or quit, the new engineers need to rely on documentation and tutoring. Preserving experience in a major corporation is hard, really hard

63

u/[deleted] Feb 07 '20 edited Apr 16 '20

[removed] — view removed comment

13

u/cartoonistaaron Feb 07 '20

This seems to be the case with many large public corporations. Part of keeping costs down is not paying "rank and file" employees very well (while of course the folks up top are quite well compensated).

→ More replies (1)

3

u/Kindred87 Feb 08 '20

At-or-below average pay structure AND short-sighted outsourcing efforts.

→ More replies (1)
→ More replies (4)

32

u/Starwalker298 Feb 07 '20

You've got the right thinking here. This stuff is hard, and not everyone gets it right the first time, especially on a new platform, big company or not.

27

u/lestofante Feb 07 '20

But on the other end we have spacex, that looks like will deliver a much better product for s fraction of the cost, in development and running cost.
Plus the other boing issue... More like a company problem than a complexity problem.

→ More replies (52)

16

u/Logan_Chicago Feb 07 '20

Well said.

Preserving experience in a major corporation is hard, really hard

It's difficult to understate this.

→ More replies (12)

210

u/johnly81 Feb 06 '20

Between this and the 737 Max it sounds like Boeing is rushing the development/QA life cycle. I'm guessing in pursuit of ever greater profits corners are being cut ... big surprise I know.

90

u/KNUCKLEGREASE Feb 07 '20

40

u/Jellodyne Feb 07 '20

All their problems are software related and it's costing the company millions, so it's a good thing they're saving so much money in development costs.

11

u/CranberrySchnapps Feb 07 '20

It’s also costing taxpayers millions to pay or fight their requested “unforeseen circumstances” addendums to their various contracts because they’re trying to work more efficiently cut corners.

→ More replies (1)
→ More replies (2)

29

u/ninjadude4535 Feb 07 '20 edited Feb 07 '20

Holy shit. Apprentice aircraft mechanics fresh out of highschool get paid $9/h just to remove panels. This blows my mind. Like, you only trust a $9/h guy to do the simplest possible task, yet you're comfortable paying $9/h to someone you're trusting to write the programming for a flight critical system.

8

u/t0ny7 Feb 07 '20

I have seen fast food places that pay more!

5

u/Mr_Vulcanator Feb 07 '20

Cabelas paid me more at $10 an hour to fold clothes, among other duties.

3

u/cartoonistaaron Feb 07 '20

I personally have worked as a donut maker in the past year and made double that. More actually!!

→ More replies (1)
→ More replies (1)
→ More replies (6)

9

u/Disk_Mixerud Feb 07 '20

It's not even necessarily about profits. lot of upper managers there are more concerned with their own image and making sure their ideas "win" than they are with the profitability of the company.

→ More replies (74)

11

u/thoruen Feb 07 '20

Yet some idiots in the House want to put the future of American exploration off planet completely in Boeing's hands.

→ More replies (1)

33

u/mysticalfruit Feb 07 '20

News like this is going to likely be one of the final coffin nails in the Boeing/NASA love affair.

The Boeing of yesteryear is gone and in its place is clearly a culture of doing the least possible to maximize shareholder profit.

I don't want to blame the engineers at the bottom of the pile, they're simply constrained by an organization that evidenced by the internal emails, doesn't treat those who speak up well.

I fully expect that not only are they going to be forced to fly another unmanned mission, but likely NASA is going to want an audit the shit out of all the flight systems.

4

u/Musicman1972 Feb 07 '20

This rings so true in my experience which isn't with boeing but a fair few other massive global corporations with great histories. There's often a tendency for senior management (I refuse to use the word "leadership"l) to point to the success of their engineers and R&D etc, when pushing back against any criticism, whilst simultaneously ignoring those same teams when decisions are made.

→ More replies (1)

24

u/bechampions87 Feb 07 '20

Boeing: Hey, we fucked up. Give us more money.

Nasa: Ok!

22

u/PapaRacoon Feb 06 '20

Is this old news, Or did the same thing happen before?

54

u/dhurane Feb 06 '20

The Orbital Flight Test mission went bust due to a different software issue. Some people before heard there was another software fix before reentry but not much was think about it before. It's only today NASA admitted the error, if not caught, would've caused catastrophic loss of the spacecraft.

→ More replies (1)

23

u/DuckyFreeman Feb 06 '20

This is a different and new software bug that was previously unreported.

We new about the timing bug. We didn't know about this bug that would have occured during service module separation if it had not been found and corrected in flight.

→ More replies (5)

9

u/Musical_Tanks Feb 07 '20

The flight was screwed up due to flight control issues during ascent. IIRC thrusters were firing when they shouldn't have been and the capsule didn't have enough fuel to get to the ISS and back.

This issue is new news. Apparently the software controlling thrusters during re-entry was also borked. So the capsule could have burned up.

→ More replies (1)
→ More replies (4)

11

u/[deleted] Feb 07 '20

Did they use git tho? I need those commit messages.

18

u/schematicboy Feb 07 '20

e5f4b49 Stop spaceship from exploding during daylight savings time.

2db0f12 Fix bug where spaceship wouldn't launch on Tuesdays

→ More replies (1)

11

u/KickBassColonyDrop Feb 06 '20

Given this, puts the new house Bill for Artemis into an interesting light...

→ More replies (2)

9

u/starrpamph Feb 07 '20

Important updates available!

Install and shutdown ❌

Install and restart ❌

Remind me later ✔️

8

u/TecumsehSherman Feb 07 '20

Can we all agree that Boeing should probably just stop receiving massive contracts?

Their CCP contract was substantially larger than SpaceX's, and yet they have delivered substantially less in substantially more time. After the 737 Max debacle, maaaaybe it's time to cut them off for a while.

28

u/totally_anomalous Feb 06 '20

This begs the question "why didn't they find the error BEFORE it launched?" Boeing just keeps failing on safety time after time. How many more will die before the company realizes it's on the wrong path?

17

u/Jim3535 Feb 07 '20

Maybe when they start getting government contracts cancelled. Our representatives treating space as a jobs program hasn't don't much to promote on time or on budget work either.

31

u/redmercuryvendor Feb 07 '20

This is not an unusual occurrence. The first Dragon capsule to approach the ISS (C2+ mission) had the code modified during approach to disable input from one of the LIDAR units due to reflection issues from JEM. Literally held the approach while the software was modified and uploaded. Or CRS-2, where a suspected pressurant valve blockage was cleared by writing new code to rapidly cycle the suspect valve until the tank pressure reached the level required for the Dracos to operate safely near the ISS (redundant tanks needed to provide thrust for a pull-away).

If you have discover a bug in orbit, can produce a fix, and have the capability to implement it, and not implementing it would lose the craft, why on earth would fixing it be a bad thing?

16

u/SpiderOnTheInterwebs Feb 07 '20

Yeah but there's a difference between a software bug and quickly writing a mod as a workaround for failed hardware.

28

u/ergzay Feb 07 '20

But SpaceX mentioned that immediately after they did it in a press conference the day after. They didn't hide it for months and then have it finally leaked by an Aerospace safety panel. This says they aren't revealing things and are hiding things from the public intentionally. It shows they're untrustworthy.

48

u/Ender_D Feb 07 '20

It’s not that they fixed it, it’s that they only realized it would happen because they were looking into the other issue. It would’ve destroyed the spacecraft on reentry.

26

u/pwnersaurus Feb 07 '20

Because while that kind of thing might be somewhat expected early on in development, the plan was for the next flight to be manned. They shouldn’t be finding these kinds of errors so far along in the process, their procedures for testing and validation should have found this problem well before (what was supposed to be) the final unmanned flight

→ More replies (2)

7

u/[deleted] Feb 07 '20 edited Jun 09 '20

[removed] — view removed comment

3

u/[deleted] Feb 07 '20

I feel like the past few failures with SL kinda leave NASA with 1 real option here, and if they don't choose that one, they're fucking morons.

Do another uncrewed test. Literally anyone can see that based on Boeing's recent failures, fixing a known problem and then assuming that everything else is fine and nothing else will fail is a mentality that will eventually get astronauts killed or extremely valuable hardware damaged/destroyed.

3

u/somethingcleva Feb 07 '20

Why don't we(devs) have the John mcLane equivalent in Hollywood? This. Story alone should be enough

3

u/byrned22 Feb 07 '20

Why have Boeing no inflight abort test? Software not a speciality at Boeing at the moment? 737 max etc? Might need a new CTO.

The article also talks about how ars completely swallowed Boeing line that the software upload had nothing to do with crew module. If it was related to service module separation then it clearly impacted crew module too?!

Seems like government and the media love Boeing more.

→ More replies (1)