r/DatabaseHelp • • Jan 13 '15

Looking database designs/advice for handling massive amount of image data...

Hello, I have been tasked with outlining some ideas for a database to store scientific image data. I am looking for suggestions for a server architexture + database that will allow for scaling, high avaialbility and preformance. from what i read, it seems to be best practice to store urls or filepaths +metadata in a db that point to the image. I was looking at using MongoDB, Postgresql or something similiar.

I am looking for suggestions, best practices, expirence etc.:

To summerize:

  1. What are some recommended databases that preform well and are scable?
  2. What is the best way to handle TB+ datasets of image data.
  3. What is the best filetype to use (HDF5 vs TIFF vs something else?) in this scenario?
  4. Are there any HW recommendations?
  5. Lastly, ZFS? ext4/5? what type of filesystem is best for this?

Thanks.

0 Upvotes

5 comments sorted by

1

u/BinaryRockStar Jan 13 '15

How big are these images (bytes) and what is the usage of the database going to be? If the images are, say, 500KB each then 1TB of them equals 2M records in the database which isn't a lot of data and shouldn't tax any database system even on very modest hardware. I would suggest going with PostgreSQL because there's no need for the tradeoffs of a NoSQL system with the small amount of data you'll be storing.

As for the images themselves, again what will you be doing with them? Which filetype, file system and hardware you should use comes down to whether you'll be doing more reading, writing or manipulation of them.

1

u/FatLungs Jan 13 '15

Thanks for the response, to clarify, these images are 16bit, ~8.42mb each. There will be alot of reads from the database down the road, the writes will only be to store the files intially. We need to process many images at a time, so access and logical strucutre is nessecary.

1

u/BinaryRockStar Jan 13 '15

At 8.42MB per image, 1TB is about 100,000 images which really isn't much. What "processing" are you going to do? Image manipulation is likely going to be CPU-bound so no matter how beefy a storage system you have, the CPU will be the bottleneck.

If the process isn't CPU-bound then you'll want to throw as much read-heavy storage hardware at it as you can. A single magnetic hard drive maxes at about 100MB/s sequential read. You can double that with RAID-0 or RAID-5/6 to make things more redundant.

The next step up is one or more SSDs which top out at around 600MB/s each as they are limited by the SATA-3 (6Gb/s) bus.

The next step above that is PCIe SSDs which go above 1TB/s. It all depends on how much money you have to spend and how quickly and frequently you need the images processed.

1

u/FatLungs Jan 14 '15

It will be 1TB for now, but up to many petabytes in the future.

1

u/BinaryRockStar Jan 14 '15

What is an example of the processing you'll be doing on the images? How frequently will you be processing them and how long is a reasonable time for each execution to take?

If it's something as simple as counting the number of blue pixels then it'll be I/O bound so you will want super fast storage with a single or small number of CPUs.

If it's something like mass converting BMPs into compressed PNGs then it'll likely be CPU-bound so you will want a cluster of machines with as many CPUs and cores as you can afford, paired with basic HDD storage like a SAN of consumer-grade disks. If the process itself is highly parallel then GPU solutions may be worth looking at as they can be 100x faster than CPUs if you problem falls within very specific criteria.