A while back were evaluating different architecture to store data:
- Dump everything in a big NFS shared storage. It’s fast, it works,** IT knows it well**. Semantics are easy: it’s a filesystem! We can use usual GUI file explorers, or with a terminal
cdwherever we want. Building services on top is trivial. #oldInfra
- Use an S3-like blob storage, deployed internally on top of said filesystem. Our S3 would be running on a single node. #newInfra
- Using an artifact management system to mediate all requests for files, and either provides only
tar.gzfor reads, or uses the same above filesystem as a storage backend. #bestPractices
To weed out bad ideas you need one of those things:
- Enough political weight (but relying on it is dangerous because you’re not always right!)
- A engineering culture where pros and cons can be discussed fairly (but you need to have hired the right people, good luck otherwise).
- Reliable working code (but you don’t always have the task / time / budget to write it).
: when you have it, IT infrastructure support is great.