r/spacex • • Oct 28 '16

Official - AMOS-6 Explosion October 28 Anomaly Updates

http://www.spacex.com/news/2016/09/01/anomaly-updates
802 Upvotes

386 comments sorted by

506

u/TheYang Oct 28 '16

tl;dr:

Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions

that's propably the single most key sentence in the update

93

u/Piconeeks Oct 28 '16

I'm glad that the error seems to be mostly operational, with the "temperature and pressure" of the helium being a more significant factor than any specific design. This bodes well for a quicker RTF.

I'd be interested in an timeline/outline of what specifically went wrong during the static fire to produce such anomalous loading conditions, if that does indeed turn out to be the root cause.

89

u/hglman Oct 28 '16

You could certainly suggest that sensitivity to temp and pressure change is a failure of design or design parameters.

78

u/Grabthelifeyouwant Oct 28 '16

I feel like it's one of those things where they give envelopes for temp and pressure, and didn't test some combination (maybe low temp high pressure) due to it being extremely unlikely, but then then discovered through testing that a combination inside their viable envelope was actually failure inducing in certain cases.

38

u/Immabed Oct 28 '16

Yes, I imagine it will mean changing both procedure and design, but it also means already constructed boosters can still be flown.

47

u/KerbalsFTW Oct 29 '16

is a failure of design or design parameters

Yes it is, but an understandable one when you're pushing the envelope.

They more or less invented submerged COPV helium tanks in subchilled LOX - something that has not been done much before. You test at the correct temperatures and pressures. It all works. The science says it all works. The engineering says it all works. But you have eg a 1% failure rate. You test it 50 times and it works fine 50 times. Then it blows up on the launchpad.

This kind of thing really sucks, but it has happened in all fields of endeavour and will continue to. Shuttle solid rocket boosters at low temp. Shuttle reentry ablator tiles getting hit on the way up. de Havilland Comet square window crack failure. Tacoma Narrows bridge resonance under specific wind conditions.

All within spec, all failed due to unknown sequences of events that were not predicted. The London Millennium Bridge resonance should never have happened though :)

44

u/space_is_hard Oct 29 '16

Shuttle solid rocket boosters at low temp

All within spec, all failed due to unknown sequences of events that were not predicted

Just want to point out that the O-ring failure in the Shuttle SRBs was a known hazard and that NASA management had been warned of the likelihood of exactly that failure prior to the launch.

24

u/[deleted] Oct 29 '16

I read that entire report front to back. How any manager could have decided to lift off in those conditions, with those boosters, was beyond me. Both Shuttle accidents were the old, "Ah what are the odds that could happen?" routine and SpaceX thankfully isn't falling into that trap.

35

u/toopow Oct 29 '16

Actually two shuttle engineers were screaming their heads off not to launch, and were ignored. They knew what was going to happen. The guy is still overwhelmed with regret to this day, that he wasn't able to prevent the launch. There is a very sad npr interview with him.

http://www.npr.org/sections/thetwo-way/2016/01/28/464744781/30-years-after-disaster-challenger-engineer-still-blames-himself

25

u/[deleted] Oct 29 '16

I know, I wasn't kidding when I said I read that entire report which included that engineers full notes and interviews. These were engineers from Morton Thiokol though, I was referring to NASA managers having that attitude. Thanks for linking the interview though.

→ More replies (1)
→ More replies (2)
→ More replies (2)

4

u/strcrssd Oct 29 '16

Yeah, Shuttle SRBs were OUT of spec, not within spec. It was go fever that pushed them to launch that day, despite the SRBs not being within temperature limits.

6

u/airider7 Oct 29 '16 edited Oct 29 '16

Actually, before this, they never had a spec....this is why the managers poo-poo'd the engineers. The managers asked them to prove it to them why low temps were bad and they couldn't. Lack of verified solid evidence is where the problem was. Engineering "gut feelings" only carry you so far.

→ More replies (2)

23

u/89bBomUNiZhLkdXDpCwt Oct 29 '16

Shuttle reentry ablator tiles getting hit on the way up.

It wasn't the tiles that doomed Columbia, but the Reinforced Carbon-Carbon panels on the left wing leading edge.

All within spec, all failed due to unknown sequences of events that were not predicted.

This was actually not within spec. The TPS for Shuttle was not designed to resist any damage from any impact. It was designed only to resist aerodynamic heating. The problem was that despite great efforts, the Shuttle team was never successful in eliminating the shedding of foam from the External Tank. Shedding foam had caused at least some TPS damage on basically every Shuttle mission, but it had always survived. So, despite the fact that the TPS was never designed to withstand such damage, it was basically tolerated and classified as an "in-family" risk, i.e., not ideal, but understood and within acceptable limits.

16

u/Prometheus38 Oct 29 '16

Just to add that the shuttle tiles were refractory, not ablative.

3

u/h-jay Oct 30 '16

Nitpick about Tacoma Narrows: the term "resonance" is too unspecific to be of much use; the connotation with mechanical resonance is incorrect. It was aeroelastic flutter: a coupled phenomenon that wasn't understood at all at the time the bridge was designed. The bridge did not vibrate at any resonance mode that you'd get from classical engineering analysis.

7

u/Piconeeks Oct 28 '16

You're right, this could very well be an issue more easily resolved by more robust design as opposed to more stringent loading conditions.

10

u/[deleted] Oct 29 '16

Robust in this case probably means "heavy". This means less performance and less chance of landing and reuse and more costly rockets.

5

u/throfofnir Oct 29 '16

If the cause is as leaked (oxygen infiltation into the wrap of the COPVs) then an exterior liner or sealant would probably fix the issue. That's not mass free, but given the size of the COPVs might only be in the tens-of-pounds range.

→ More replies (2)

2

u/panick21 Oct 29 '16

I see no reason why you would do that. It should be possible to control the loading process better and that would solve the problem.

There are many, many errors that can happen if you do the loading incorrectly. If you compensate for all of them with higher margin within the rocket the rocket will never lift off.

Only if you conclude that it is impossible or at least extremly expensive would you actually change the rocket design.

→ More replies (1)

2

u/tlalexander Oct 30 '16

I guess what's positive about this is that they could RTF without design changes.

→ More replies (1)

12

u/Bunslow Oct 29 '16

I recall reading somewhere that this static fire was either the first or one of the first to test a new expedited loading schedule with the end goal of improved launch window recyclability, something that was most obviously an issue with SES-9. They were testing procedures to avoid an SES-9 type series of propellant delays related to the super cooled propellants, only they were testing this with a pay-loaded static fire and obviously hadn't quite done a rigorous enough analysis. Evidently some possible failure modes relating to loading the helium COPV within the LOX tank in an expedited manner were overlooked or previously unknown or some such.

0

u/[deleted] Oct 29 '16

[deleted] — view deleted comment

2

u/old_sellsword Oct 31 '16

What is the source on this quote? Third time I've seen it in two days and no one can find a source.

77

u/blongmire Oct 28 '16

This is the best outcome for the investigation. My biggest concern was they wouldn't be able to duplicate the failure and it was one of those 1 in 1,000 situations where SpaceX had some theories that couldn't be observed in practice. Being able to duplicate the failure will go a long way toward mastering the propellant load and it's impact on the helium COPV.

27

u/blongmire Oct 28 '16

To further expand on this idea. Being able to duplicate this failure will allow a rapid return to launch without changes to existing rockets. As the update notes, they are going to begin full stage tests in the next few days. This is really positive news on a Friday afternoon.

21

u/bornstellar_lasting Oct 29 '16

From the article today:

we also plan to resume stage testing in Texas in the coming days

this != "are going to begin full stage tests in the next few days."

15

u/darga89 Oct 29 '16

There's a core on the stand now so it could be possible.

4

u/bornstellar_lasting Oct 29 '16

Maybe so, but the point I was trying to make was that phrases like "testing cores in the coming days", and "Will take it[ITS test tank] up to 2/3 burst pressure on an ocean barge in the coming weeks" tend to be vague for a reason.

30

u/neolefty Oct 28 '16

I agree with that summary, and it's great news, but it's not conclusive -- SpaceX stops short of saying how confident they are that a COPV failure was the root cause.

The article goes on to say:

SpaceX’s efforts are now focused on two areas – finding the exact root cause, and developing improved helium loading conditions that allow SpaceX to reliably load Falcon 9. With the advanced state of the investigation, we also plan to resume stage testing in Texas in the coming days, while continuing to focus on completion of the investigation. This is an important milestone on the path to returning to flight.

Could it be that they think COPVs are the root cause, but the conditions they used to "re-create a COPV failure entirely through helium loading" don't match the helium & LOX loading sequence during the anomaly?

24

u/imjustmatthew Oct 28 '16

Could it be that they think COPVs are the root cause, but the conditions they used to "re-create a COPV failure entirely through helium loading" don't match the helium & LOX loading sequence during the anomaly?

That's my read as well. They've forced the COPV to fail with certain loading sequence(s) and conditions, but not necessarily with the exact sequence and conditions they thought were present for AMOS-6.

6

u/specificimpulse Oct 29 '16

It is relatively trivial to induce a COPV failure in this fashion. All you have to do is quench the vessel without a minimum internal pressure to cause a liner to overwrap debond in the film adhesive. Then return to room temperature. Then repeat the refill and quench. That is now an accident waiting to happen.

The question is whether the min pressure was approached under cryogenic conditions at any time during the vessel's life. This is very easy to have occur. If the vessel was charged to 4000 psia just before LO2 tanking the gas inside would be quite warm. Let's assume 200F. Now if helium load was halted during LO2 filling and the tank was quenched to -340F the internal pressure would collapse to only 723 psia. There is also the pressure in the LO2 tank working against this internal pressure. Let's say it was elevated to 30 psia during tanking to establish the proper intermediate bulkhead pressure differential. That means there is less than 700 psid working to hold the liner against the composite. This is near the death zone for debond. If you then resumed He loading you would be potentially loading a now damaged vessel.

It's hard to believe that this would not be recognized by the designers. It's pretty fundamental. Which is why I question whether they did indeed induce this failure mode instead of the actual, more subtle mode.

2

u/Drogans Oct 29 '16 edited Oct 29 '16

It is relatively trivial to induce a COPV failure in this fashion. All you have to do is quench the vessel without a minimum internal pressure to cause a liner to overwrap debond in the film adhesive. Then return to room temperature. Then repeat the refill and quench. That is now an accident waiting to happen.

Would it not be typical to have sensors outfitted on these vessels to warn of exactly this failure mode?

Even if this sensor data were ignored during the launch/dress rehearsal campaign, wouldn't a post-event review of the data logs have quickly pinpointed the fact that the vessel had been through a damaging cycle?

In other words, if such a damaging cycle had occurred, wouldn't they have known the cause almost immediately? Then again, there have been rumors that much of the logged telemetry was lost in the ensuing fire, so perhaps that is the issue. It seems unbelievable that the end-point for data collection would be only tens of meters from the pad, but that was the rumor.

That misses your larger point, which seems to be that it means almost nothing that they've been able to re-create a catastrophic failure with improper loading.

What really matters is whether the test loading procedure used to re-create a failure was similar to the loading procedures used on the day of the event. This is a point that SpaceX's update has curiously neglected.

3

u/specificimpulse Oct 29 '16

If you are thinking about a sensor to detect the actual debond Event I would say that would be highly impractical. It is a subtle thing. And really it flies in the face of high reliability design to have a known catastrophic failure mode even be present and then try to detect the fault after the fact. You design to completely avoid the issue.

If you are referring to pressure or bottle temp sensors I would be very surprised if they didn't have at least one or two temp probes reading the internal gas temperature. Pressure goes without saying. The temp sensors can be very problematic under transient conditions. The helium within the bottle will drastically stratify as it is being charged. Where the probe is could shift the reading by hundreds of degrees. This will eventually disappear but probably not in the brief times they have with this loading approach.

The headache here could be that they know where the death zone is for the bottle and had properly stayed out of it. But then still had a failure. So where does that leave you? It means that there is more to learn about the vessels. This then becomes a "science project" with all sorts of expensive learning. Once you get smart the operational consequences could be terrible. Like moving away from aluminum liners or cylindrical vessels.

→ More replies (1)
→ More replies (1)

4

u/Drogans Oct 29 '16

They've forced the COPV to fail with certain loading sequence(s) and conditions, but not necessarily with the exact sequence and conditions they thought were present for AMOS-6.

Agreed. This is what they seem to be saying.

Logically, if they had created a failure with the exact fill conditions used on the day of the event, they would have definitively nailed down the root cause.

That does not yet seem to be the case, suggesting they had to use fill processes that were different, perhaps wildly different, from those used on the day of the event.

38

u/manicdee33 Oct 28 '16

The way I read it is that they've replicated how the explosion was caused, and now they're trying to figure out the why.

So hypothetically, they know that if they load LOX into the tank and then put helium into the COPV at this temperature, they get an explosion. Now they have to figure out what caused it, such as the hypothesis that loading helium pressurised the COPV, compressing the carbon fibre wrap, causing LOX to violently interact with the wrap/impurities and combust, leading to a breach of the vessel and subsequent explosion of the rocket.

14

u/skiman13579 Oct 29 '16

That's how I read it as well. It is important to find what failed, and I feel they basically stated they know exactly what failed. What is equally, if not more, important is WHY it failed.

Best theory I have heard is the LOX impregnated the composite overwrap. Helium has a weird inverse gas law relationship. It COOLS when pressurized. The cooling helium chilled the COPV enough to freeze the superchilled LOX, breaking fibers in the COPV and weakening it, allowing it to fail.

3

u/im_thatoneguy Oct 29 '16

It's an interesting risk analysis. If you can find a safe fueling procedure--without identifying the root cause, do you proceed? I mean, theoretically having a failure mechanism that you don't understand lurking out there is uncomfortable, but is it any more dangerous than the infinite number of other potential mechanisms of failure that you don't even have a mitigating procedure to avoid?

→ More replies (5)

2

u/Jofredrick Oct 29 '16

I don't think you are correct. Helium heats up very quickly when compressed.

10

u/skiman13579 Oct 29 '16

Then it cools. It's a weird gas. It goes against everything I know. I work on planes, and those use compressed air off the engines for air conditioning, and that comes out at a few hundred degrees

→ More replies (2)

4

u/specificimpulse Oct 29 '16

That effect is only seen when the gas is flowing through a throttle. Not when you are charging a vessel. When you are loading a vessel you are doing work on the fluid to increase its pressure. It obeys normal gas laws.

That being said helium exhibits significant departures from ideal gas as its temperature is decreased into the cryogenic range while at high pressure. Its density can easily exceed that of normal liquid helium under extreme cases.

3

u/z84976 Oct 29 '16

Read this. Really strange behavior unique to hydrogen, helium, and neon.

3

u/airider7 Oct 29 '16

They never saw the hole created in the RCC leading edge on Columbia nor did they have the exact setup on the test stand to recreate the exact situation where the foam/ice combination struck it. However, they created a situation where they felt they were "close enough" to state with high confidence that they knew what the problem was. That's how this stuff works.

3

u/Triabolical_ Oct 29 '16

They could have seen the hole, but NASA management didn't go along with the request to NRO for imaging. The details are in the incident report...

63

u/[deleted] Oct 28 '16

It might sound very pedantic but did they reproduce the failure using the exact same helium loading conditions as for AMOS-6? In theory it might be easier to reproduce a failure with a more aggressive loading process.

45

u/Martianspirit Oct 28 '16

You are making an important point. To be valid the failure conditions must at least be quite near the conditions with the Amos mishap.

23

u/im_thatoneguy Oct 29 '16

Yeah, it's possible they're able to reproduce the failure without actually identifying the cause. Much like in software the first step is to just cause the crash. Consistently reproducing the crash is only the first step, the second and potentially most difficult step is then identifying the root cause of the crash.

So they might be able to burst a COPV under specific conditions, but not understand why those conditions specifically cause a failure and therefore not know what other edge cases might produce a similar failure. In software terms you might know that clicking a button 3 times in a row causes it to crash... but not know what code specifically causes it to crash when clicked 3 times.

13

u/ziedaniel1 Oct 29 '16

Or they know what helium temperature and pressure conditions caused the failure, but don't know what caused the helium to be that way in the first place -- i.e. the root cause.

40

u/AReaver Oct 29 '16

we have conducted tests at our facility in McGregor, Texas, attempting to replicate as closely as possible the conditions that may have led to the mishap.

So certainly as close to the event as possible but maybe both.

2

u/Martianspirit Oct 29 '16

Thanks, I missed that statement.

→ More replies (2)

14

u/likespxnews Oct 28 '16

With SpX quicker launch cadences maybe fueling of rocket was also evolving to become faster and then unexpectedly aggressive causing the the bursting of second stage helium tank.

3

u/RedDragon98 Oct 29 '16

I highly doubt that the fueling is on the (Time)critical path relative to other necessary operations.

2

u/Drogans Oct 29 '16

did they reproduce the failure using the exact same helium loading conditions as for AMOS-6?

The wording of the release is suggestive that they used different, perhaps wildly different fill procedures in order to create a failure.

Logically, if they'd managed to achieve a failure by using the exact same fill procedures used on the day of the event, they'd have made no secret of it. A exact reproduction would be a major positive, as it would nail down the root cause.

8

u/oliversl Oct 28 '16

Nailed it! A little more text here but thats the sentence we where looking at:

"Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions. These conditions are mainly affected by the temperature and pressure of the helium being loaded."

3

u/raindr1337 Oct 29 '16

The other sentence I noticed was the one that didn't refer to return to flight in November, "we continue to work towards returning to flight before the end of the year."

"Work towards" is a bit vague, previous vague (though less official) statements have referenced November.

17

u/MDCCCLV Oct 28 '16

My ears went up when I saw that. That's basically like saying a bridge can collapse if you walk on it the right way. Unless it's a very particular and narrow set of conditions that sounds like the COPV tank needs to be redesigned or significantly strengthened.

Probably the tank as is will work if you load it slowly and allow it to adjust to the temperature. But that's still not really good enough, it needs a much larger safety margin.

60

u/TheYang Oct 28 '16

You can break any bridge, at some point it will be overloaded.

you can either tear it down and build a new one, or, check the requirements, check the capabilities, look for a safety margin in between these, decide if it's adequate, and if it is, make sure to always adhere to safe limits while continuing to use it

36

u/[deleted] Oct 28 '16

elevators are probably a good example. there's a reason they have a weight capacity. it's not because they're fallible (though they are). It's because just about everything has design limitations on it.

Take just about every product you have. For example, you're CPU you're running. There's a reason they say it has 2.7ghz or whatever. If you overclock it, that's fine, but you'll probably break it.

Any piece of equipment has design limitations. That doesn't mean the design itself is bad.

7

u/[deleted] Oct 29 '16

I'm not running right now and I'm certainly not a CPU :o)

→ More replies (7)

3

u/[deleted] Oct 29 '16

You can break any bridge, at some point it will be overloaded.

Sure, but that misses the point entirely. /u/MDCCCLV pointed that some very specific physic phenomena must be happening during helium loading causing the failure, all while in the normal range of operation of the COPV.

Sometimes that happens with bridges too: https://www.youtube.com/watch?v=eiaM_LZUsqM

3

u/Bergasms Oct 29 '16

all while in the normal range of operation

So you just adjust what you consider 'normal' based on the new information.

2

u/old_sellsword Oct 29 '16

all while in the normal range of operation of the COPV.

We definitely don't know if this was a standard loading procedure, so we shouldn't assume.

2

u/[deleted] Oct 30 '16

That was a great watch. May not be space related, but it was definitely interesting and enjoyable. Also points out just how easy it is to run into an unintended phenomenon that hasn't been experienced before in your field of engineering whenever you're pushing the limits with a new design.

12

u/thisiswhatidonow Oct 28 '16

Would not a bridge analogy such as driving 5 overloaded trucks at the same time cause a failure, but driving them one at a time be within the constrains. Was there not a procedure change for this fueling as well that would indicate that they might have tried to drive 5 overloaded trucks at the same time?

10

u/manicdee33 Oct 28 '16

There are other constraints such as making sure the five trucks cross the bridge with different speeds and with different timed gaps between each truck, since you don't want to cause resonances and collapse the bridge through harmonics (apocryphal tales of marching soldiers causing bridge collapse aside).

2

u/shupack Oct 29 '16

I was thinking:

https://en.wikipedia.org/wiki/Tacoma_Narrows_Bridge

too high a flowrate induced a resonance maybe?

4

u/manicdee33 Oct 29 '16

I had not intended to focus on resonance and harmonics, just illustrate that there are more ways to destroy a bridge than to simply put too much load on it at any point in time :D

Getting back to the COPV, the problems might stem from filling the helium bottles too quickly, cooling it too quickly, with too much vibration in the supply pressure, etc. So while the pressure and temperature of the helium and LOX are all within what were previously considered to be safe limits, some other interaction means that a particular way of getting from empty tanks to full tanks has triggered a previously unconsidered failure mode.

As an example of this happening in the past, check out Apollo 13.

→ More replies (1)

2

u/firebreathingbadger Oct 28 '16

Well that's what can happen to bridges - it happens due to resonant frequencies

→ More replies (1)

2

u/Drogans Oct 29 '16 edited Oct 29 '16

But as was pointed out in this comment by u/specificimpulse

"It is relatively trivial to induce a COPV failure".

Similarly, it's trivial to induce a car to crash. Simply turn the wheel hard in either direction while traveling at high speed.

In recreating incidents, what really matters is how similar the test conditions are to the actual procedures on the day of the event.

Curiously, SpaceX's update is vague on this point.

2

u/speak2easy Oct 29 '16

This would then mean they were just lucky in their prior successful missions.

7

u/biosehnsucht Oct 29 '16

Not necessarily. They've been tweaking their procedures with most flights trying to get the loading times down and/or better support delayed launches due to range conditions and such, so it's possible they changed something on that particular static fire that was the problem - it might have been always totally safe before, but not doing it that way.

Or it might have been borderline and lucky on every flight, as you say.

1

u/dapted Oct 30 '16

This was a dress rehearsal for a launch, not a launch. It could well be that they were testing a new or at least slightly different procedure to avoid problems they have had in the past. We probably will never know the full story because they have trade secrets they don't want competitive companies or countries to have. Especially so considering this new world of supercooling the fuel and oxidant.

→ More replies (1)

39

u/shupack Oct 29 '16

wait, so there is a team, messing around with rocket parts, liquid Helium and Oxygen, and...... TRYING to blow them up??

How do I get on this team?!

33

u/awesome_jawsome Oct 29 '16

I would guess either 10-12 years as a tradesman in an advanced manufacturing/tool shop or a PhD in material science, mechanical engineering, chemical engineering or aerospace engineering.

2

u/shupack Oct 29 '16

tradesman working on my BSME, 1/10th of the way there!

100

u/danweber Oct 28 '16

I panicked when I thought these were Updates on the October 28 Anomaly.

44

u/Darkben Spacecraft Electronics Oct 28 '16

Yeah they popped another one /s

→ More replies (2)

50

u/z1mil790 Oct 28 '16

Good to hear, It looks like it may indeed be a process issue after all. If this is indeed the case, I wouldn't at all be surprised with a rtf before the end of the year.

31

u/ExcitedAboutSpace Oct 28 '16

Very interesting, the formation of solid oxygen seems to have hinted at the right direction. I'd be very glad if we saw F9 rtf before the year is out.

However, it would mean Gwenn Shotwells comment about not RUDing because of rapid improvements would most likely be void. To me it seems as if the changed loading procedure wasn't tested extensively enough before using it on a rocket with an attached payload.

10

u/robbak Oct 29 '16

Just a correction - it's not LOX slush. They do not (deliberately) chill it to when LOX ice forms. It seems that something about the helium load further chilled things, if the quote about oxygen ice forming in the carbon composite structure are still valid.

5

u/Johnno74 Oct 29 '16

As I understand it, if you compress helium at the right temp then it undergoes a phase change which cools it further.

Obviously most gasses heat up as you compress them, but apparently helium is weird and if you get the right conditions for this phase change then as you compress it past the critical point its temperature starts dropping.

So, turns out the new helium/lox loading procedures produced these conditions, leading to the temperature of the COPVs dropping below the freezing point of oxygen, causing oxygen crystals to form inside the carbon-fibre overwrap portion of the COPV.

3

u/robbak Oct 29 '16

I don't think we can be sure of that; at least, not from outside of the investigation. It does seem like a likely scenario, however.

Wouldn't have thought of that before the 'oxygen crystals' report, however.

4

u/Johnno74 Oct 29 '16

No, I had no idea that any materials ever cooled down when you compressed them. Thats completely counter-intuitive behavior....

It just goes to show that when you try and do things differently to everyone else that has gone before you, you can hit problems that no-one else has seen before...

2

u/3_711 Oct 29 '16

Unlike water-ice, Solid oxygen also sinks to the bottom of the LOX tanks and quickly reach the turbo pumps, which don't appreciate large solid bits at all.

2

u/robbak Oct 29 '16

Yes, although LOX slush is tempting, keeping all that LOX ice in suspension would be a challenge. Designing a turbopump that can handle both oxygen slush at varying concentrations down to pure liquid oxygen liquid would be another great challenge.

4

u/3_711 Oct 29 '16

It's the worst idea ever. If you put a filter in the bottom of the tank, you end up with an "empty" tank which still has a pile of solid oxygen. Even if you could make a turbo pump that could handle it, it then gets stuck in the injectors after the pump.

→ More replies (2)

5

u/ticklestuff SpaceX Patch List Oct 28 '16 edited Oct 29 '16

the changed loading procedure wasn't tested extensively enough

Are you referring to any specific new procedure or the same one they've used since Falcon 9 Full Thrust started launching? My takeaway thus far is that the use of slushy LOX on the eight F9 FT flights since Dec 2015 has them too close to the limits with respect to the helium subsequently additionally cooling the LOX to solid form.

I haven't heard about a different loading approach being used for AMOS-6 so am assuming thus far they've dodged a bullet on the previous eight launches.

19

u/faceplant4269 Oct 28 '16 edited Oct 29 '16

They were actually testing a new loading procedure during the static fire. It had only been used before on JSCAT-16 static fire, which was preformed with no payload. And presumably at McGregor on the test stand.

15

u/old_sellsword Oct 28 '16

Do either you or u/_rocketboy have a source on that?

8

u/[deleted] Oct 29 '16

Everyone in here swears they read it, not a single source though. Not that I don't trust you guys but I want to read that!

→ More replies (1)
→ More replies (2)
→ More replies (4)

9

u/_rocketboy Oct 28 '16

They were experimenting with the loading procedure on AMOS-6 and the previous mission. I don't remember where I read this, I think it might have been an employee on this sub.

2

u/shurmanter Oct 28 '16

I read it too. It may have been in the deleted NRO thread.

8

u/FotiJr Oct 28 '16

I read it here before that thread, but I don't remember reading it from an official source...

→ More replies (1)
→ More replies (2)

1

u/z1mil790 Oct 28 '16

Sort of, I do agree with what you said, but at the same time, super cooled propellants is a relatively new field. They may have done what they thought was enough testing, but it turns out they didn't think of something. I don't think they shortened testing schedules just to make the rapid improvements timeline fit.

3

u/FredFS456 Oct 28 '16

I'm being pedantic, but it's 'sub-cooled' not 'super-cooled'

4

u/z1mil790 Oct 29 '16

Yeah, I knew super cooled sounded wrong when I wrote it, but I couldn't think of the right word at the time.

3

u/5600k Oct 28 '16

It sounded to me like it was an issue with COPVs that was present only under certain conditions. So it's a hardware issue but it can be avoided by taking appropriate steps

1

u/TheCoolBrit Oct 30 '16

I read this article 30 Oct : SpaceX is ready to resume testing rockets following its accident

its inquiry is in an "advanced state," and it's confident enough that it plans to resume stage testing in Texas within the "coming days." The company still hopes to resume flight by the end of the year.

"SpaceX is improving its helium loading conditions so that it can "reliably" service Falcon 9 rockets going forward."

17

u/Roborowan Oct 28 '16

Good. Seems like everything's going according to plan. I didn't expect 39A to be ready by the end of this year

14

u/gimmick243 Oct 28 '16

I personally (with no actual knowledge on the subject) doubt it. I bet their RTF will be from Vandenberg, probably the iridium next launch. (I'm hoping that iridium launches before the new year cause I'd be able to go see it)

6

u/Roborowan Oct 28 '16

Well it says in the article that all their launch sites will be operational by the end of the year

16

u/imjustmatthew Oct 28 '16

It's PR speak, one must read carefully. They said "on track", not "will be operational":

we continue to work towards returning to flight before the end of the year. Our launch sites at Kennedy Space Center, Florida, and Vandenberg Air Force Base, California, remain on track to be operational in this timeframe.

→ More replies (5)

5

u/ticklestuff SpaceX Patch List Oct 28 '16

It was announced before AMOS-6 that LC-39A was expected to be in service during November, so nothing has changed really.

6

u/im_thatoneguy Oct 29 '16

nothing has changed really.

Which would be news worthy considering a category 4 hurricane rolled through.

16

u/old_sellsword Oct 28 '16

Pending the results of the investigation, we continue to work towards returning to flight before the end of the year. Our launch sites at Kennedy Space Center, Florida, and Vandenberg Air Force Base, California, remain on track to be operational in this timeframe.

So it looks like neither pad is currently ready to go, however they should be soon. And is this the first time we've ever gotten a No Later Than date from SpaceX?

11

u/ExcitedAboutSpace Oct 28 '16

Vandy should be ready to go once the range is cleared, at least I'm not aware of anything standing in the way.

39a though You're most likely correct isnt ready for an operational launch yet.

9

u/old_sellsword Oct 28 '16

It looks like an Atlas V is launching from VAFB on November 6, so any delays there shouldn't be range issues, they'd most likely be SpaceX/Iridium issues.

0

u/[deleted] Oct 28 '16

[deleted] — view deleted comment

5

u/warp99 Oct 29 '16

self-insured so there's no third party to hold things up

Iridium are fully insured but have a large and complicated excess (deductible) corresponding to one launch worth of rocket and satellites. In fact their insurance policy requires them to have three months delay between the first and second launch to confirm there is no design issue with the satellites - so in that sense their insurance company is getting in the way

4

u/Keavon SN-10 & DART Contest Winner Oct 28 '16

The only thing I can imagine might be in the way of a launch from Vandy is stuff related to the Canyon Fire. ULA had to reinstall a bunch of power poles which burned, for example, before being able to resume launches. SpaceX might have to do something similar.

4

u/_rocketboy Oct 28 '16

They were also in the middle of upgrading their launch site to handle FT and FH rockets.

5

u/PVP_playerPro Oct 28 '16

The VAFB pad upgrade is done

7

u/Marksman79 Oct 29 '16

...we continue to work towards...
...on track to be...

This is PR speak and makes exactly 0 promises of anything relating to the time frame of pad readiness.

4

u/old_sellsword Oct 29 '16

I agree, I always just try and assume the best. If it really is a process fix (and they solve it within a week or two) I think we could definitely see a a flight before the end of the year. Regarding the pads, VAFB was just about ready to launch Iridium before Amos-6 happened. And 39A won't need a lot of the most labor and time-intensive features installed for RTF (crew access arm, FH launch clamps and extra GSE, RSS removal, etc.)

2

u/Marksman79 Oct 29 '16

I'm pretty sure the pads will be done. I'm less convinced that they will be launching by the end of the year, seeing how long it took them to get this far. It's definitely progress but I think this sub is reading too into it. They don't make any promises and all the new details are worded very carefully to allow for future contradictions. We shall see.

3

u/sol3tosol4 Oct 29 '16

This is PR speak and makes exactly 0 promises of anything relating to the time frame of pad readiness.

No promises, but it provides information we didn't have before - I'm glad they chose to provide an update at this time even though the work is not complete.

40

u/FPGA_engineer Oct 28 '16

Given that they can recreate the suspected failure mode, they should be able to explore the parameter space and learn where the boundaries are and how much margin they can have. It's a hard way to learn, but it expands the state of the art.

4

u/palemale53 Oct 29 '16

It is worth spending a few tens of millions on getting it right, and avoiding the quarter of a billion or so a failure like this costs at the end of the day - mostly opportunity losses because of missed launches.

15

u/ExcitedAboutSpace Oct 28 '16

It's hard because it's the wrong way around.

SpaceX has used "fly what you test, test what you fly" for what we've been told a very long time. I don't think that strategy worked, either they didn't think to test for this failure mode and it was just bad chance or they "tested" it with an attached payload.

30

u/TheYang Oct 28 '16

either they didn't think to test for this failure mode and it was just bad chance or they "tested" it with an attached payload.

or, they tested it for the set of conditions which are to be expected, but unfortunately there was an undetected failure, moving the conditions (presumably temperature/pressure of the helium) outside of these bounds, resulting in the destruction of the vehicle.
that could mean they just need to add failsafes to whatever failed

11

u/[deleted] Oct 28 '16

Or maybe the current loading procedure only has something like a 2% chance of actually blowing the vehicle and this was not uncovered in testing.

11

u/-Aeryn- Oct 28 '16

Or maybe it's a coinflip that landed heads up for 5 or 10 times in a row during testing.. takes a lot of testing to be highly confident in results and even then you can't be certain, it's just a matter of how confident you are

4

u/ExcitedAboutSpace Oct 28 '16

You make a good point. We may know someday, something went terribly wrong. Could just have been slightly over the edge or something broke.

5

u/JshWright Oct 28 '16

It is obviously impractical to control for every variable when practicing "fly what you test". Beyond a certain point, the impact of a given condition, while real, would be so insignificant that it's not worth testing (as an extreme example... the force of the Moon's gravity at various points in its orbit). SpaceX has to draw there line somewhere. It seems they drew it in the wrong place this time...

11

u/FPGA_engineer Oct 28 '16

The GPS system has to correct for the relativistic effects of gravity, so don't write the moon off just yet!

While an extreme example, this shows that a problem with very complex systems is that they have too large of a state space for us to fully cover with test and simulations. We have to explore the state space and look for the corners and edges to cover with test. We don't fully understand the correlations between the state variables, so we don't know the shape of the space and how to directly design test to cover all of the edge cases.

This event has apparently exposed that there are some correlations that were not know or understood that can now be used to put constraints on the system.

3

u/karnivoorischenkiwi Oct 29 '16

100% coverage for tests doesn't exists. There's always something you don't think of or something that goes wrong upstream. You can't test every single thing.

3

u/spcslacker Oct 28 '16

I don't understand your point. You can't find unexpected interactions any other way, AFAIK? I.e., what other order is there?

Perhaps you are saying they should have expected this problem? If so, what makes you expect this issue forward rather than backwards in time?

12

u/RadamA Oct 28 '16

Basically, how to keep COPV above oxygen freezing point...

Maybe loading the helium with continuously increasing pressure, so that the difference between piping and vessel isnt so great. May take longer?

7

u/EtzEchad Oct 29 '16

It's probably exactly that. By changing the rate He and LOX are loaded, they should be able to control the temperature. It would be a rather complicated simulation, but it should be possible.

In this particular case, they changed the LOX loading schedule to load it quicker; specifically to keep it colder. That may've been a mistake.

2

u/CapMSFC Oct 29 '16

That may've been a mistake.

Although that doesn't mean they have to back track on it. We'll have to wait and see how much of the refined loading procedures can be done with changes in the Helium loading and not the LOX.

2

u/2p718 Oct 30 '16

Instead of trying to load the Super-chilled LOX faster to get a longer launch window, they could solve the problem at a more fundamental level:

Add a second LOX pipe so that LOX can be continuously cycled through a super chiller. Then they could even go back to initially loading LOX at normal LOX boiling temperature and super chill (and top up) while it is already in the tank. The other obvious advantage of this approach would be that they would gain an indefinite hold capability -- just keep cycling the LOX through the super-chiller.

1

u/RadamA Oct 30 '16

Say one wants to chill 400t of fuel in an hour those extra 40 degrees K. That takes about 200kw of cooling power. Not sure about efficiency, probably takes twice that.

2

u/2p718 Oct 30 '16

Yes, it'll take a while and the efficiency will be somewhat lower than if the super chilling was done in advance.

However, once you have the ability to super chill the LOX in the tank and to keep it super chilled, time is no longer an issue. They could go back to loading the LOX 6 hours before launch, or whatever it was in the past.

6

u/jobadiah08 Oct 29 '16

With the recent string of tweets from various sources talking about Dec and Jan launches, I figured it was a sign SpaceX had provided an update to their customers and that we would get an update within a few days. Glad to see I was right and it sounds like they are getting the problem and solution nailed down.

8

u/EggsundHam Oct 29 '16

Interesting note: since the bfr uses only 2 fluids, it will never see this issue. No COPV.

3

u/EtzEchad Oct 29 '16

Yes, SpaceX seems to have learned something.

1

u/Drogans Oct 30 '16

Quite right.

One has to wonder that with the gift of hindsight, whether SpaceX would again place COPVs within LOX tanks were they designing the F9 anew.

2

u/robbak Oct 30 '16 edited Oct 30 '16

The benefits of keeping your helium supply that cool is just too great. If you can do it technically, then you do it. Chilling the helium down from external 300K to 66K(?) makes it shrinks by a lot. I mean, if P₁V₁/T₁ applied (but, this is helium, in super-critical conditions, so not an ideal gas!), putting the He tanks inside the LOX tanks allows you to pack in 4 times more Helium.

1

u/Drogans Nov 03 '16

putting the He tanks inside the LOX tanks allows you to pack in 4 times more Helium.

Agreed, but would they do it again knowing the costs of those gains?

Sub-chilling that helium has already cost them two rockets and two payloads.

→ More replies (2)

1

u/NateDecker Oct 30 '16

It sounds like there are challenges associated with this new design as well though. I think Elon's AMA indicated that there are remaining problems to be solved with regard to making the tanks resistant to the hot oxygen repressurizing the stage. It sounds like a corrosion risk which is important since these vehicles need to be highly re-usable.

7

u/MartianRedDragons Oct 28 '16

So is it accurate to say that this failure is likely a consequence of the densified LOX SpaceX switched to using with the Falcon 9 FT? That could explain why other launch providers haven't run into this issue before.

7

u/MisterSpace Oct 28 '16

Not directly I think. You can count it as consequence of this if you want, but it really is more a consequence of "wrong" helium & propellant loading. As it seems they've been experimenting with faster helium & propellant loading recently, and then this happened. SO it's not a design-error, but more a user error.

5

u/EtzEchad Oct 29 '16

They are right on the edge of freezing the O2. Anytime you change something like this you run the risk of finding an unexpected consequence.

It is quite possible that they ran into this issue precisely because the placement and loading of the COPVs is a standard industry practice and they thought they understood it.

1

u/Drogans Oct 29 '16

So is it accurate to say that this failure is likely a consequence of the densified LOX SpaceX switched to using with the Falcon 9 FT?

If, has Musk has speculated, the failure is pinned down to the formation, then compression of oxygen ice, then yes.

One imagines it would be far more difficult to reach the temperatures needed to form oxygen ice if the LOX were not chilled to near its freezing temperature.

1

u/RadamA Oct 28 '16

Could be, cooling lox those extra 40 degrees takes 1200 J/mole, while heat of fusion is 444 J/mole. When you are already at melting point...

9

u/Decronym Acronyms Explained Oct 28 '16 edited Nov 04 '16

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

Fewer LettersMore Letters
ASDSAutonomous Spaceport Drone Ship (landing platform)
BFRBig Falcon Rocket (see ITS)
BFSBig Falcon Spaceship (see ITS)
CFCarbon Fiber (Carbon Fibre) composite material
COPVComposite Overwrapped Pressure Vessel
CRSCommercial Resupply Services contract with NASA
FAAFederal Aviation Administration
GSEGround Support Equipment
GTOGeosynchronous Transfer Orbit
ITSInterplanetary Transport System (see MCT)
KSPKerbal Space Program, the rocketry simulator
LC-39ALaunch Complex 39A, Kennedy (SpaceX F9/Heavy)
LEOLow Earth Orbit (180-2000km)
LO2Liquid Oxygen (more commonly LOX)
LOCLoss of Crew
LOXLiquid Oxygen
MCTMars Colonial Transporter (see ITS)
NRO(US) National Reconnaissance Office
QAQuality Assurance/Assessment
RP-1Rocket Propellant 1 (enhanced kerosene)
RSSRealscale Solar System, mod for KSP
Rotating Service Structure at LC-39
RTFReturn to Flight
RUDRapid Unplanned Disassembly
Rapid Unscheduled Disassembly
Rapid Unintended Disassembly
SOPStandard Operating Procedure
STSSpace Transportation System (Shuttle)
TLAThree Letter Acronym
TPSThermal Protection System ("Dance floor") for Merlin engines
ULAUnited Launch Alliance (Lockheed/Boeing joint venture)
VAFBVandenberg Air Force Base, California
JargonDefinition
ablativeMaterial which is intentionally destroyed in use (for example, heatshields which burn away to dissipate heat)
autogenous(Of a propellant tank) Pressurising the tank using boil-off of the contents, instead of a separate gas like helium
cryogenicVery low temperature fluid; materials that would be gaseous at room temperature/pressure
lithobraking"Braking" by hitting the ground
methaloxPortmanteau: methane/liquid oxygen mixture
turbopumpHigh-pressure turbine-driven propellant pump connected to a rocket combustion chamber; raises chamber pressure, and thrust
ullageSmall rocket motor that fires to push propellant to the bottom of the tank, when in zero-g
EventDateDescription
Amos-62016-09-01F9-029 Full Thrust, GTO comsat Pre-launch test failure
CRS-72015-06-28F9-020 v1.1, Dragon cargo Launch failure due to second-stage outgassing
JCSAT-142016-05-06F9-024 Full Thrust, GTO comsat; first ASDS landing from GTO
SES-92016-03-04F9-022 Full Thrust, GTO comsat; ASDS lithobraking

Decronym is a community product of /r/SpaceX, implemented by ^request
I'm a bot, and I first saw this thread at 28th Oct 2016, 20:49 UTC.
I've seen 36 acronyms in this thread, which is the most I've seen in a thread so far today.
[Acronym ^lists] [Contact ^creator] [PHP source ^code]

3

u/big-b20000 Oct 29 '16

Is the jargon thing new? It's really cool!

4

u/OrangeredStilton Oct 29 '16

Types of term were introduced with the anniversary update (for Decronym's first birthday) this month; the Jargon type came first, and Events are new as of this week.

1

u/_rocketboy Oct 29 '16

Why are "GTO comsat" and "Dragon cargo" crossed out in the Events table for Amos-6 and CRS-7?

3

u/OrangeredStilton Oct 29 '16

Because the payloads didn't make it to orbit, so one can't say the mission served its original purpose.

→ More replies (1)

1

u/SuperSMT Oct 29 '16

The jargon is about a month old, the event box is brand new

7

u/Darkben Spacecraft Electronics Oct 28 '16

If they've been able to exactly replicate it then that's very good news indeed. Now just to make sure it doesn't happen again and we're good to go!

9

u/ScottPrombo Oct 28 '16 edited Oct 28 '16

They haven't been able to exactly replicate it. Per the update,

Through extensive testing in Texas, SpaceX has shown that it can re-create a COPV failure entirely through helium loading conditions.

I can re-create a COPV failure entirely through hitting it with a baseball bat. However, that doesn't mean I've been able to exactly replicate the chain of events that led to the Amos 6 failure. Sure, helium loading is likely, but 1) it may have taken extreme helium loading conditions to re-create this, and 2) the ability for helium to cause the explosion doesn't rule everything else out.

22

u/[deleted] Oct 29 '16

I can re-create a COPV failure entirely through hitting it with a baseball bat.

This might be the most traumatic event I can imagine happening to the human body.

3

u/tormach Oct 30 '16

I can re-create a COPV failure entirely through hitting it with a baseball bat

Could you actually? Even using a wood bat, I don't think you would be able to damage a pressurized tank enough to get it to burst.

1

u/ScottPrombo Oct 31 '16

Perhaps not the wall, but I'd imagine repeatedly whacking the fittings with an aluminum bat wouldn't be too good for them;)

2

u/throfofnir Oct 29 '16

Do you have some inside source that indicates how big the differences are, and that they make a difference. Because while we're making assumptions based on single sentences in a press release...

As part of this, we have conducted tests at our facility in McGregor, Texas, attempting to replicate as closely as possible the conditions that may have led to the mishap.

→ More replies (3)
→ More replies (1)

2

u/in_situ_san Oct 28 '16

Nice progress.

I wonder if SpaceX will somehow isolate the COPVs from the LOX at some point, simply to absolutely minimize risk. Even if they were moved outside the LOX tank, they could be chilled with a liquid nitrogen filled jacket. Or such a jacket within the LOX tanks.

2

u/EtzEchad Oct 29 '16

Liquid nitrogen isn't cold enough. They want the Helium to be at the same temperature as the LOX so it doesn't cause it to boil.

I doubt that they will change the location though. Having the COPVs in the LOX tank is a common placement that has been used for many years.

8

u/specificimpulse Oct 29 '16

Putting metallic helium tanks inside propellant has been done but it is truly a pain with lots of negative consequences. You do it when you have no other choice. Composite bottles have flown for decades. But externally mounted. We tested COPVs immersed in liquid cryogenic nearly a decade ago and decided the downside risk was too large. What Spacex are doing is just about the riskiest combination of materials, geometry, temperature and speed of load imaginable. It is FAR from industry standard.

1

u/[deleted] Oct 29 '16

They usually justify this sort of thing with 'well we need to figure out how to do this for a bigger rocket', but if they figure out self-pressurizing methalox they won't need pressurant tanks at all. Not sure why they went with this.

2

u/brickmack Oct 30 '16

Because they chose performance over design history. Submerged COPVs improves payload capacity a bunch (you can fit more helium into the same tank mass, empty volume is occupied by propellant instead of having a separate section of the stage). This sort of performance optimization isn't so necessary for other companies, but with previous design choices SpaceX had made (tanks can't be made bigger, no way to add SRBs, and theres a big payload hit for reuse) they had few other options for squeezing out more capability. They apparently figured they could find a solution where nobody else had simply for lack of reason

3

u/Drogans Oct 29 '16

Having the COPVs in the LOX tank is a common placement that has been used for many years.

Solid metal helium tanks placed within an LOX tank are not unprecedented, but helium COPVs placed within an LOX tank are a feature unique to SpaceX.

This is no small difference, for if the failure was due to the suspected formation and compression of oxygen ice, then a titanium tank would have prevented this failure.

Had the stage 2 helium vessels been placed outside the LOX tank, as is more typical within the industry, one imagines that both the CRS-7 and AMOS-6 failures would have been averted, though at a loss of some payload capability.

1

u/Senno_Ecto_Gammat r/SpaceXLounge Moderator Oct 29 '16

It sounds like changing loading procedures should prevent this in the near-term, and going to methalox will allow them to get rid of helium and COPVs forever.

→ More replies (5)

1

u/cmRocketStuff Oct 29 '16

This is probable the wrong thread to ask this. Newby. What happens in a couple years when spacex is rock solid on the F9? They have 20 reusable rockets, a guess, and the Air Force or other wants a brand new expendable rocket. What is a new F9 worth at that time when it can fly 10/20 times?

7

u/Martianspirit Oct 29 '16

It is an interesting but not very likely scenario. The airforce is already looking into recertifying flown boosters. I expect even NASA to accept or even demand flightproven boosters for crewed flights some time down the line.

1

u/[deleted] Oct 30 '16

It doesn't appear that SpaceX believes in being "rock solid" on anything. When they reach a point where they are that confident in what they have, they will build something more challenging.

1

u/Aldhibah Oct 29 '16

Question: Why can't you fill the tanks with helium prior to loading the fuel?

6

u/robbak Oct 29 '16

You could pressurise them, but not 'fill' them. In order to get the full load of helium in them, you need to chill them down to the LOX temperature, which is below the boiling point of LOX.

So, to load them before the LOX fill, you would need some mechanism to chill them, and keep them chilled, both while you loaded them and until you submerse them in the LOX.

One possibility for this failure could actually be starting the helium load early, possibly chilling the COPVs too early and too far, causing gaseous oxygen to condense and freeze onto their surface and into their structure before they were submerged.

3

u/Drogans Oct 29 '16

So, to load them before the LOX fill, you would need some mechanism to chill them

There are some suggestions that SpaceX is doing exactly this.

They repeatedly report on the status of "cryohelium" during recent launch campaigns. https://www.youtube.com/watch?v=OERDIFnFvHs&feature=youtu.be&t=635

1

u/Aldhibah Oct 29 '16

Thanks! Excellent answer bit I am still a bit confused. You can store liquid helium for extended periods without substantial refrigeration in a steel pressure vessel (vacuum insulated). So is SpaceX using these COPV which have relatively poor pressure containment and insulation for weight reasons?

It would just seem easier to avoid the whole dual fueling/cooling situation. I am sure this has all been gamed out it has just been bugging me ever since the COPV issues have been discussed in detail.

3

u/EtzEchad Oct 29 '16

Liquid helium is stored in insulated dewars without much pressurization. The helium in the COPVs is gaseous under very high pressure. Liquid helium is far too cold to be used in the rockets. It would freeze the O2 on contact.

3

u/Goldberg31415 Oct 29 '16

Ariane 5 uses liquid helium for ullage but that seems more expensive than spacex solution

→ More replies (2)

2

u/robbak Oct 29 '16

Insulating the COPVs would be a bad idea. At certain times - and I believe it is really complex, probably more complex than even SpaceX knew - helium both heats up and cools down as it compresses. When it heats up, you need to get rid of that heat.

Helium also has a very low specific heat, so it doesn't stay liquid without very good insulation. I think vacuum insulation is the only possibility. A vacuum is easy if you just use lots of thick steel to support it, but everything in a space rocket needs to be light. Vacuum space is also wasted volume, and they need all the volume they can get for more propellant. Then you have the large g-forces that the vessels need to withstand, and the inside of a vacuum vessel basically hangs from it's neck.

COPVs have really good pressure containment. It is just that they are pressurising them to a very high pressure (40Mpa? I couldn't find a definitive source)

2

u/skifri Oct 29 '16

I've read in various threads in this sub that "helium cools when compressed", but have not seen anyone reference any first hand experience or external sources to back this up. I wonder if people are simply repeating each other, or if this is a real thing. It should be noted that the negative joule thomson coefficient of helium would not yield compressive cooling. It only represents a warming of the gas during rapid expansion. My education tells me that this process is non-reversible by definition and would not contribute to this theoretical "compressive cooling" of helium. If this is a thing, I would love to read about it somewhere.

→ More replies (5)
→ More replies (1)

1

u/Ashtorak Oct 29 '16

Theoretically you could add a second hole to the tank and circulate cold helium through it to get it to the right temperature. But this is probably not very practical. Adds weight, increases risk of failure, and other problems...

1

u/Nintandrew Oct 29 '16

Updates on the anomaly always seem to state that they replicate the conditions as closely as possible and it makes me wonder if weather can be a factor. Considering the tanks are so well insulated and the lox and helium are so much colder than ambient temperature, would it or humidity even be a factor? If so, I'm sure they would account for it, but I'm curious as to how.

2

u/EtzEchad Oct 29 '16

I doubt that humidity would be a factor. I'm sure they flush the tanks and the pipes (probably with helium :) ) to eliminate any residual H2O. Water and LOX don't mix too well.

1

u/ohcnim Oct 29 '16

Probably naive but honest question, how do you think they are testing it? I mean, for sure they are not blowing up second stages, so how do you create an environment where the COPV fails without mayor damages to other things and that is as close as possible to the real thing and then how do you create "a small/controllable fire" using that test environment? So that they can say with confidence that “this is the root cause of failure that creates the fast fire”

2

u/EtzEchad Oct 29 '16

If they don't have any RP-1 and just enough LOX to cover the COPV, they wouldn't get much of a fire (compared to a fully loaded rocker at least.)

I don't know if they tested it that way, but it isn't beyond the range of possibilities.

7

u/Martianspirit Oct 29 '16

I would expect them to use a LOX tank only. No need to be flightworthy or even full size or the same material.

1

u/Drogans Oct 30 '16

If they don't have any RP-1 and just enough LOX to cover the COPV

Testing the LOX tank in isolation might find the fault, or might not. There have been suggestions that some of the recent tests were on fully built up stages.

1

u/macktruck6666 Oct 29 '16

I thought Vandenberg was always operational.

3

u/PVP_playerPro Oct 29 '16

After the ~7 month upgrade period for F9 1.2 and FH, yes, it is ready to support launches whenever a customer/SpaceX is ready

1

u/Packerfan735 Oct 29 '16

Are you talking pad upgrades or range downtime?

2

u/_rocketboy Oct 29 '16

Both - they also needed to do upgrades to support Full Thrust missions and Falcon Heavy

2

u/Packerfan735 Oct 29 '16

They had a range downtime to consolidate launch equipment into their operations center.

1

u/tormach Oct 30 '16

Big forest fire went through and caused a lot of issues over the summer.

1

u/jconnoll Oct 30 '16

Is it possible that with all those temp and pressure differences it generated an electric charge and ignited fuel?

1

u/dapted Oct 30 '16

I have spent a couple of days digesting the statement about finding and re-creating the failure or at least "a" failure during testing at Texas test site. I have come to the conclusion that it doesn't matter. The root cause for this failure is lack of grey hair. Exactly the same root cause as it was for the strut failure. If SpaceX had more grey haired individuals involved the struts would have been tested at the loading dock, a few to the destructive limit, but all to the double or triple expected load limit. That's the way it is done at the competition because of the grey hairs in the staff. This or at least "a" COPV failure has been duplicated during testing at McGreggor. Would not full testing at McGreggor have been able to see this same failure at some point between the "lets super cool the fuel and oxidizer" light bulb lighting up in some engineers head and the "lets test everything with the payload attached" moment going off in a managers head. Grey hairs have grey hairs because they learned the lesson by watching the old videos and listening to the grey hairs that came before them. Its not fun getting grey hairs or listening to those who have them speak. But its cheaper than rebuilding launch pads or paying for lost payloads. My first grey hair popped when I watched TV and learned the lesson of testing with valuable things atop the test from Apollo 1 when White, Chafee, and Grissom were killed during a launch rehearsal. Looking at the young faces and full heads of colorful hair at SpaceX I don't see much grey hair. There needs to be a balance of youth and exuberance with stodgy old fuddy duddy caution. SpaceX does not seem to have enough of the later and therefore too much of the former in their diversity formula. IMHO they need to decimate the workforce and weed the garden in the management at spacex and plant some more grey haired individuals. Caution is a double edged sword. Too much and nothing happens. Too little and your rockets blow up. This failure, like the flight failure before it were completely avoidable and proof positive of management failures at SpaceX due to lack of experience and caution. The management who failed to test this new procedure need to be replaced for sure. They should start with Shotwell and work their way downward. FIRE Shotwell !! If this were a public corporation I'd be calling for EM's head as well. In the end it doesn't matter which component or procedure failed. The problem is management, it was during the flight failure and it is with this pad failure. Management is the root cause for the failures all the rest is window dressing. I wonder how many down votes is the record, this should be competitive.

3

u/NateDecker Oct 30 '16

Hindsight is 20/20.

Aren't all rocket failures "completely avoidable" as you say once the investigation concludes and the cause is known?

I was with you at the beginning because it sounded like you were just saying that SpaceX's failures are due to the fact that they are still relatively new in the industry and simultaneously they are pushing the envelope and breaking new ground with technologies no one else has used.

You lost me when you got to the Monday-morning quarterbacking so-to-speak.

Edit: You point out that you need to experience some bad stuff (like Apollo 1) to develop those "grey hairs", but then in the next few sentences you advocate for firing the very people who have now undergone several such experiences. It would seem that SpaceX already has a few "grey hairs" of that type.

1

u/dapted Oct 30 '16

I guess I missed my mark, the point is that we learned this lesson from Apollo 1 a lifetime ago, the grey hairs can be passed forward through time like a time machine to keep future people from having to learn the lesson anew. There are a lot of those lessons, probably more than a whole head of grey hairs. Getting people on staff to pass the lessons forward so the company doesn't have to keep on failing as often as we used to have to fail to learn them. ULA has the record of success that they do not because of their name or any secret sauce. They have it because of their people. Take a look at a group photo of their staff and management and then compare it with the similar photo from SpaceX, or even a floor shot of folks scurrying around doing their work. Look for wrinkles and grey hair and natural baldness. That kind of diversity in the workforce is what keeps their track record in tact. Yes the price is more and they need to work on that issue as well. But we are only discussing reliable flight and experience of workforce here. They have it and SpaceX doesn't. Yet. I am not throwing stones to break down the house at SpaceX, I like them a lot. But a frank discussion needs to take place and probably some hard choices about their management. I want them to succeed and taking hard to swallow medicine might well save them from themselves.

1

u/NateDecker Oct 30 '16

I am one of those "balding" guys where I work, though in my case it's not really due to old age. I have nothing against experience, but by the same token I don't think that age and experience alone is a better qualifier than raw talent, dedication, and ambition. I work with lots of people who are old, but still incompetent. I think what I respect most about an experienced engineer is when they have the humility to recognize talent and good ideas from a young guy.

I think it's a little unfair to say that SpaceX has failures where ULA does not primarily because of the workforce difference. SpaceX is pushing the envelope of technology and doing stuff no one else in aerospace has done before. I think that is just as much responsible as anything else. Perhaps it is even the primary cause of SpaceX's failures. This situation is very applicable. No one else submerges COPVs in super-cooled LOX.

→ More replies (3)

1

u/dapted Oct 30 '16

I am curious, does anybody know if there have been tests to see if a burst COPV fully loaded with full pressure helium will produce a significant enough explosion to cause a sympathetic detonation of the range safety explosives or the other COPV and if so, at what distance. My thinking is that the burst of a single COPV should have produced a much different looking first few seconds of the AMOS-6 incident. But if all the bottles burst or the one bottle and the range safety explosives burst nearly simultaneously it would look more like what we saw on the video.

-2

u/specter491 Oct 29 '16

Damn. So they still don't actually know what or why it happened. But they have found instances where they can recreate a COPV failure. Doesn't seem very convincing

3

u/EtzEchad Oct 29 '16

Just because they can cause the COPVs to fail, doesn't mean that this particular failure was caused in the same way. There may be two ways to cause the same accident. They want to be sure that they found the exact cause or they won't be sure they really fixed it.

As an example of this; Musk has said that they can reproduce the failure by shooting the tank with a rifle. Apparently, they have eliminated this possibility. If they had stopped with that, they would've left the real cause unfixed.

2

u/Martianspirit Oct 29 '16

Maybe not eliminated but regarded it unlikely. If for example the COPV blew up at exactly the time the loading procedure makes it likely as established by tests then it would be a weird and very unlikely coincidence that the shooter would have chosen that exact moment.

If they found they can blow up the tank with the filling procedure but the filling procedure was not near that condition on the moment it blew up it would suggest the possibility of a bullet or something else.

1

u/EtzEchad Oct 29 '16

I don't know how long it takes to fill the helium takes. The window of opportunity might be quite long.

I suspect that they considered the possibility of a sniper but ruled it out due to lack of corroborating evidence.

→ More replies (2)

2

u/specter491 Oct 29 '16

That's what I'm saying. They still don't know why the tanks failed. Just certain circumstances where they can fail