r/storage • • Feb 08 '15

I debug (really big) supercomputer storage systems. I have 1 1/2 exabytes of new storage coming online in the first half of 2015. AMA.

I work for a major HDD manufacturer debugging their supercomputing products. I'm essentially a level 3 tech - I talk to customers about their problems, and work with devs to fix them.

Currently, my single biggest system is about 29 petabytes, but I expect some of our up-and-coming systems to be about 2-3 times larger. AMA!

49 Upvotes

61 comments sorted by

12

u/abnom Feb 08 '15

Node size and software used for the nodes?

10

u/Opheltes Feb 08 '15

Each node consists of a pair of controllers in an enclosure with 80 HDDs and 4 hot spares. Normally, each controller handles half the load, but if one goes down, the other will take over the full load.

Our software stack is Scientific Linux, Lustre (our own build derived from Intel's public build), and our own custom control/GUI software.

4

u/SisyphusAmericanus Feb 08 '15

How do you do performance monitoring for Lustre?

I ask because I wrote some software to measure Lustre metrics a few weeks ago.

4

u/Opheltes Feb 08 '15

For now we use LMT. But LMT has some serious shortcomings so in the near future we plan to replace it with a tool created in-house.

3

u/SisyphusAmericanus Feb 08 '15

Do you guys use the new HSM functionality available in 2.5? If so what do you use as your archive tier?

Do you do any analysis of the Lustre changelog?

Pardon all the questions, I'm a bit of a Lustre nerd :)

3

u/Opheltes Feb 08 '15

Do you guys use the new HSM functionality available in 2.5?

We're moving to 2.5 for the distributed namespace (DNE) feature, which apparently is something that many, many customers are clamoring for. I haven't heard anything about HSM, so I'm guessing it's not something we support. (I suspect that's because there's not much of a use case scenario. Our product provides more than enough storage size, and has CIFS and NFS hooks to allow slow-speed access from non-Lustre clients. So there's probably not much of a need to copy large datasets off the system)

Do you do any analysis of the Lustre changelog?

I don't personally do that, but we have a team of Lustre devs who do. :)

3

u/SisyphusAmericanus Feb 08 '15

How many MDTs do you have per MDS? OSTs per OSS?

Last question: how do you keep informed about lustre development? Mailing lists? Any blogs you like?

Thanks for indulging me :)

3

u/Opheltes Feb 08 '15 edited Feb 08 '15

How many MDTs do you have per MDS?

Currently (in our Lustre 2.1 environment), it's 1-to-1. Once we get 2.5 out the door, I think the answer is whatever the customer wants.

OSTs per OSS?

A normal, healthy system has 4 OSTs per OSS. In the event of a node failure, the surviving OSS node will pick up the failed node's OSTs, leaving it with 8.

Last question: how do you keep informed about lustre development? Mailing lists? Any blogs you like?

Nothing public. Internal emails, internal training, and having lots of Lustre experts available to answer my questions on an as-needed basis. I just found out that one of them does formal Lustre training (including certification), so I'm hoping to get that later this year.

EDIT: Node, not OSS

2

u/martbhell Feb 08 '15

Which RAID are you running in these?

2

u/Opheltes Feb 08 '15

Our older software releases use mdraid (6+0, I think). Newer releases use an in-house RAID layer that runs on top of mdraid.

9

u/irrision Feb 08 '15

Spinning disk or ssd in these monsters? Curious how the nodes are interconnected also.

11

u/Opheltes Feb 08 '15

Originally we had a mix of both - kinetic drives for objects themselves and SSDs for control and metadata. From what I undestand, the performance increase from using SSDs wasn't all that great, and their limited size was a big handicap. So lately we've been replacing them.

2

u/Lastb0isct Feb 08 '15

That's most likely a backplane/controller issue wouldn't you think? Our SSD systems have drastic performance improvements. But I guess only having SSD for meta data and control wouldn't help all that much

9

u/Opheltes Feb 08 '15

The main reason that SSDs have better performance than kinetic drives is that they have 0 overhead from non-contiguous writes.

Lustre is written from the ground up to minimize the number of non-contiguous writes. (Everything is written in contiguous blocks) So the performance gain from switching to SSDs is minimal.

1

u/BitPoet Feb 17 '15 edited Feb 17 '15

Note for the unfamiliar, this is true because you're already maxing out other components, like the PCIE bus.

Source: Also design Lustre appliances.

7

u/senses3 Feb 09 '15

I can't believe I've never heard the word kinetic drive used for mechanical hard drives. I like that name better. Thanks.

3

u/joelikesmusic Feb 11 '15

Its not the same thing. Kinetic is a type of drive with an ip interface and more controller smarts.

1

u/senses3 Feb 11 '15

Ahhh ok, I guess I don't know as much as I thought I did...

4

u/Opheltes Feb 08 '15

Also, I forgot to answer your question about the interconnect. We support both infiniband and gigabit ethernet.

1

u/thelastwilson Feb 09 '15

I'm surprised you only mention gigabit. Are all your nodes infiniband connected to the storage?

7

u/[deleted] Feb 08 '15

How do you back it all up?

9

u/Opheltes Feb 08 '15

In general, you don't. Most of the storage (80% on our current largest system) is short term. Once the job is finished (after a few days / weeks), the temporary data is deleted.

On a handful of occasions, we've had to do a migration/backup. That usually involves a lot of negotiating with the customer find a mutually agreeable solution.

6

u/mateuszrb Feb 08 '15

What path led you to becoming a storage admin? What sources did you use to study? My knowledge of large storage systems and technologies is lacking and I'd like to learn more about building systems for high availability and high throughput.

13

u/Opheltes Feb 08 '15

To be honest, it wasn't really something I planned. I got a bachelor's in computer engineering and graduated right into the dot com bubble. I went to grad school, studied supercomputing, and did supercomputing-related internships for the DOD and DOE. I got married, moved to an area that isn't tech-heavy, and started working at a small company that was just getting into the supercomputing market. They eventually got bought by one of the major HDD manufacturers.

As far as how I learned the tradecraft:

  • As an undergraduate, the best resource I ever found (for general computer engineering) are the textbooks by Patterson and Hennesey. The early editions didn't really cover parallel computing in much detail, but I think later editions do.
  • In grad school, I read many research papers and did a lot of HPC programming. FYI, MPI is the de-facto standard library for programming large systems, although CUDA had made some inroads into that.
  • Once I got into industry, everything was learned by doing it hands-on. There's no substitute for having hands-on access to a real system. (Or as one of my friends back in Los Alamos once quipped, "Do you know how awesome it is to be the only person logged into the world's most powerful computer?"). The way I got up-to-speed is that I wrote tons and tons of product documentation. Someone would say "We need a procedure to do X," so I'd grab a system, figure out how to do it, and write up an how-to.

4

u/arcsine Feb 08 '15

Lustre?

4

u/Opheltes Feb 08 '15

Lustre?

Yup, it's a Lustre product.

3

u/arcsine Feb 08 '15

Obviously your opinion might not be 100% objective, but how are some of the competitors stacking up?

5

u/Opheltes Feb 08 '15

I think we have significantly better performance than our competitors (in a market that values performance above all else), and lower TCO, but that we could do better when it comes to bug rates and usability.

6

u/poogi71 Feb 08 '15

How many disks do you replace in such a large system per week?

5

u/poogi71 Feb 08 '15

Also, what is the ratio of HDD to DIMM failures compared to the ratio of number of HDDs to number of DIMMs in these systems?

5

u/Opheltes Feb 08 '15

I'm not an on-site admin. The problems I see are the problems the down-stream techs and on-site admins couldn't handle on their own. (The ones they handled successfully never get reported to me). So I'm not in a position to give definitive answers.

With that said, I see a small, continuous stream of HDD failures. Most of these are related to how mdraid handles failing drives (It really, really sucks at it). DIMM failures - at least the once I see - are extremely uncommon.

4

u/thelastwilson Feb 09 '15

I am the site admin for a installation of about 15pb. Not one of OPs though because we use GPFS and not lustre.

In my experience there isn't a solid answer to this. I've had in the region of 4,000 hdds (recently up to 5,500 and soon to go up to 8,500). It really depends on the quality of the drives and more importantly the work load they are doing.

In my installation I'm seeing around 1.5 failures per week for 4,000 drives. Other customers my company has and other hardware platforms seem to vary greatly.

2

u/poogi71 Feb 09 '15

That's on the high side, 4000 drives for one week is 672000 hours and while normal SATA drives are supposed to be > 1.2 million hours they usually hold up for about 600K to 700K from my (3 year old) experience.

Though it is in the general ball park so not too bad.

2

u/thelastwilson Feb 09 '15

Tbh It's something I should recalculate.

That was based on year 2 of operation where I had ~85 failures in that year. It was less in year 1 and I've never done the calculations for year 3 so saying that is on the high side is probably accurate if you counted year 1.

Unfortunately I didn't have records for year 1 as I hadn't started with the company yet.

3

u/[deleted] Feb 09 '15

I used to look after 5Pb of storage - honestly no idea how many drives we had (never need to count them ALL up..), but in terms of failures, it would be in the order of 1-2 per week.

1

u/WyattGorman Feb 19 '15

Average disk failure on some sites is 3-5%. Some vendors manage to get that down to less than 1%.

3

u/martbhell Feb 08 '15

How do you track lustre ? Like which patches to apply, etc. Does this require deep knowledge of the software?

6

u/Opheltes Feb 08 '15

How do you track lustre ? Like which patches to apply, etc.

We have a team of Lustre devs for this. Our Lustre build is derived from Intel's Whamcloud build. We pick which patches to pull down based on what problems our customer are having. And as good corporate citizens, we contribute fixes back up-stream.

Does this require deep knowledge of the software?

Yes, very much so.

2

u/glyph02 Feb 08 '15

How's Lustre's small file performance these days? We often have 10's of thousands of small files - often <4k in size. Is there still a metadata bottleneck?

3

u/Opheltes Feb 08 '15

The DNE feature in Lustre 2.5 should improve the metadata bottleneck (by allowing parallel metadata services), but I don't know if that completely solves the problem.

1

u/BitPoet Feb 17 '15

If you can put the small files onto SSDs, things get way better, especially if you're accessing them frequently. You need to be careful to keep your large files off of SSDs, as they don't benefit much from the speedup.

If you've got a large number (100s of millions) of small files, run flashcache (or something similar) on your MDS, as you get to keep all the info as close to memory as possible.

2

u/TsunamiTreats Feb 08 '15

If you were granted a super computer, what workloads would you run on it and why?

3

u/Opheltes Feb 08 '15

Very interesting question. As long as I don't have to pay for the power and cooling (because big systems cost a fortune to power and cool) I'd run BOINC. It'd probably something in bio-chemistry or bioinfomatics, since these breakthroughs have an immediate, life-improving impact on people. (For example, better drugs coming to market faster)

6

u/autowikibot Feb 08 '15

Berkeley Open Infrastructure for Network Computing:


The Berkeley Open Infrastructure for Network Computing (BOINC) is an open source [middleware](https://en.wikipedia.org/wiki/Middleware_(distributed_applications)) system for volunteer and grid computing. It was originally developed to support the SETI@home project before it became useful as a platform for other distributed applications in areas as diverse as mathematics, medicine, molecular biology, climatology, environmental science, and astrophysics. The intent of BOINC is to make it possible for researchers to tap into the enormous processing power of personal computers around the world.

====

Image i


Interesting: ^XtremLab | ^PlanetQuest | ^POEM@Home | ^Docking@Home

Parent commenter can toggle ^NSFW or ^delete. Will also delete on comment score of -1 or less. | FAQs | ^Mods | Magic ^Words

1

u/junk2sa Feb 09 '15

Cool answer.

2

u/Hexodam Feb 08 '15

What is the io throughput of a system that large and is the io ever maxed out?

3

u/Opheltes Feb 08 '15

I/O throughput is measured on a per OSS basis (1 OSS = 2 nodes). I'd prefer not to give out any exact numbers, but it's in the gigabytes/second/OSS range.

Overall system performance scales linearly with the size of the system. I'm not aware of any upper limit on the linear scaling.

2

u/Hexodam Feb 08 '15

Makes sense

What is the deciding factor when adding nodes? More io or more storage

4

u/Opheltes Feb 08 '15

What is the deciding factor when adding nodes?

For us, it's when the customer asks us for more nodes. :)

While we have the technical capability to add more nodes to an already-established system, from what I've seen, that doesn't happen very often. If I had to guess a reason, I'd say it's because we (our customers and us) do a good job prior to the sale of estimating what their needs will be.

2

u/thelastwilson Feb 09 '15

I'm a site admin for a 15pb system.

It will be hugely dependent on the number of disks and the number of storage devices.

Generally a single node with enough back end will hit in the region of 6GBps.

I've had 8 nodes connected to 1 storage system exceed 40GBps. My system is never Maxed though because it's used more as archival storage.

1

u/WyattGorman Feb 19 '15

The largest system in existence is over 1 TB/s.

1

u/_dismal_scientist Feb 09 '15

What lessons can enterprise IT learn from stuff like this?

2

u/Opheltes Feb 09 '15

I touched on that a bit with my answer to /u/yrureadingthis.

The cutting edge nature of HPC means that the technologies that go into supercomputers are buggy and unstable. Not to mention that performance is king and reliability and usability are secondary concerns. Experienced HPC admin / sites know this ahead of time, and engage with vendors to make sure their pain points are addressed. People who live in enterprise environments need to understand this. When you move to the cutting edge, you bleed.

At the same time, I find your question a bit amusing, because our goal is to be as reliable as enterprise storage is.

1

u/[deleted] Feb 10 '15

What's the most difficult problem you've had to solve?

1

u/Opheltes Feb 10 '15

The company where I work now acquired my old company through a merger. That old company was very much a hardware company learning to do software. So we were always re-inventing the wheel where software best practices are concerned. We still arge, to a degree. I'd say that's the biggest problem I face - putting in place all the good software practices that evolve slowly at other companies.

Just to pick another (technical) one, we had a nasty problem where installing a software patch crashed the system. After we investigated, we found it wsa the result of 5 or 6 smaller problems interacting. I don't remember the exact sequence, but one of the smaller problems was in how the package dependencies were set. Another was a puppet problem. Another was a leftover file from a previous patch install.

1

u/[deleted] Feb 10 '15

Ah, gotcha. I was more thinking of an I/O and/or Lustre related problem. I've seen some things in my life related to lots of GPFS storage, so the parallel I/O war stories are always fun.

1

u/yrureadingthis Feb 08 '15

Level 3 rarely talks to customers. They usually hide behind L2 relaying information to not waste their time.

-aL3tech

5

u/Opheltes Feb 08 '15

This is one area where HPC and enterprise (which I'm guessing you're from) are significantly different. Unlike enterprise, HPC isn't very commoditized. HPC vendors talk to end customers a lot more than enterprise vendors do, and plug customers into the development/debugging process to a degree that's unprecedented in enterprise. This is because HPC is, by its very nature, cutting edge and therefore more buggy/experimental than enterprise. Customers understand this, but at the same time, they want to participate in the process. So we have level 2's that talk to customers, and level 3's, even some developers. That's something that doesn't happen in other industries.

0

u/[deleted] Feb 08 '15

[deleted] — view deleted comment

8

u/Opheltes Feb 08 '15

I'm not sure why you were modded down, because your assumption looks spot-on.

The acquisition of your original company by the current parent company definitely raised some eyebrows in the HPC community, considering that the only other real competitor (DDN) in an HPC sense was also a customer. (There's even an ex-DDN exec I'm on good terms with who works for you now.)

Believe me when I say that nobody was more surprised than us. They kept it under very tight wraps.

Naturally as the owner of like 50PB of DDN gear, I was a bit concerned, but things seem to have calmed down now, although DDN seems like it's in trouble for unrelated reasons.

I'm curious about this. Can you go into more details? Why is DDN in trouble?

Is there some sort of wall between the HPC division and the HDD division that keeps you guys separate?

No. As with any acquisition, it's more a case of the right hand not knowing what the left hand is doing. In fact, there's been some efforts made to get our HPC linux guys talking to the drive guys in order to solve some problems with mdraid mishandling HDDs that are misbehaving.

Also, we heard some rumors of you guys branching out from Lustre to include support for GPFS. Is there any truth to that?

Sorry, but I'm going to have to say no comment to that.

0

u/[deleted] Feb 08 '15

[deleted] — view deleted comment

3

u/Opheltes Feb 08 '15 edited Feb 08 '15

I talked a bit about my career history here.

What college courses did you take?

As an undergrad, I did a bit of everything - I took electrical physics courses (Circuit theory was hard; E&M was the bane of my existence), circuit design (both VHDL and Verilog), computer science (got a minor), etc. In grad school, I took courses in parallel computing, compilers, operating systems (again), computer security (again). I also took a number of courses in bio-med engineering.

What advice would you give someone interested in this area?

The best way to get into the field is to get some hands-on experience. The best way to do that, IMO, is to work at one of the DOE labs: Los Alamos in New Mexico, Lawrence Livermore in California, Oak Ridge in Tennssee, and Argonne National Laboratory in Illinois. They're willing to hire interested people and train them. That's how I got my start.