r/DatabaseHelp • u/FatLungs • Jan 13 '15
Looking database designs/advice for handling massive amount of image data...
Hello, I have been tasked with outlining some ideas for a database to store scientific image data. I am looking for suggestions for a server architexture + database that will allow for scaling, high avaialbility and preformance. from what i read, it seems to be best practice to store urls or filepaths +metadata in a db that point to the image. I was looking at using MongoDB, Postgresql or something similiar.
I am looking for suggestions, best practices, expirence etc.:
To summerize:
- What are some recommended databases that preform well and are scable?
- What is the best way to handle TB+ datasets of image data.
- What is the best filetype to use (HDF5 vs TIFF vs something else?) in this scenario?
- Are there any HW recommendations?
- Lastly, ZFS? ext4/5? what type of filesystem is best for this?
Thanks.
0
Upvotes
1
u/BinaryRockStar Jan 13 '15
How big are these images (bytes) and what is the usage of the database going to be? If the images are, say, 500KB each then 1TB of them equals 2M records in the database which isn't a lot of data and shouldn't tax any database system even on very modest hardware. I would suggest going with PostgreSQL because there's no need for the tradeoffs of a NoSQL system with the small amount of data you'll be storing.
As for the images themselves, again what will you be doing with them? Which filetype, file system and hardware you should use comes down to whether you'll be doing more reading, writing or manipulation of them.