PuppetConf 2015

As we architect our systems for greater demands, scale, uptime, and performance, the hardest thing to control becomes the environment in which we deploy our infrastructure and the subtle but crucial interactions between complicated systems. Chaos Patterns help us establish and implement a virtuous cycle that let’s us both prove AND improve our system along each of these dimensions. This is the story of the pitfalls we encountered, and how, through architecture, convention and common sense, we managed to build an infrastructure that is "Always Up" from the end user perspective and economical to build, scale & operate; using chaos testing, we learn more about how our system fails from a 10 second controlled failure than a multi-hour uncontrolled outage.