Talk @shapescience

Making decisions

A while back were evaluating different architecture to store data:

  • Dump everything in a big NFS shared storage. It’s fast, it works,** IT knows it well**[1]. Semantics are easy: it’s a filesystem! We can use usual GUI file explorers, or with a terminal cd wherever we want. Building services on top is trivial. #oldInfra
  • Use an S3-like blob storage, deployed internally on top of said filesystem. Our S3 would be running on a single node. #newInfra
  • Using an artifact management system to mediate all requests for files, and either provides only tar.gz for reads, or uses the same above filesystem as a storage backend. #bestPractices

To weed out bad ideas you need one of those things:

  • Enough political weight (but relying on it is dangerous because you’re not always right!)
  • A engineering culture where pros and cons can be discussed fairly (but you need to have hired the right people, good luck otherwise).
  • Reliable working code (but you don’t always have the task / time / budget to write it).

[1]: when you have it, IT infrastructure support is great.