Back from DevOps Days Paris 2015
I attended DevOps Days
DevOps Days are mostly focused on the human side of DevOps, and therefore featured a lot of non-technical talks on the subject. The talks emphasized the importance of empathy and respect, of understanding peoples fears and wants, and of putting yourself in other's shoes. There was also a strong focus on challenging our assumptions and being mindful of our biases.
The Keynote was titled Containers, Germs, and Microservices, in reference to Jared Diamond's Guns, Germs and Steel. In this talk, John Willis explained how the World is divided into haves and have-nots : those who have power, and those who don't. Haves are created through the reduction of latency and distance that is brought about by cybernetic feedback loops between geography, civilization, agriculture, etc.At some point, a terminal velocity is reached that makes it impossible for the have-nots to compete with the haves. This was true for some indigenous people of the Americas when they encountered machine guns, it is now just as true for the non-digital companies encountering their digital competitors.
Willis believes that the new haves vs. have-nots front is data. He thinks acceleration to terminal velocity in this field will be achieved through containers and micro-services, as we need to move computing to data and not the converse. Data is indeed getting too heavy to move, a phenomenon he calls "data gravity". Data from IoT in particular is expected to come in huge quantities and therefore have a big impact on "data gravity".
In What Happens Without Traction, Steve Pereira gave ideas for introducing DevOps or bringing about similar changes in a company in the face of strong resistance. He suggests going for local changes, like automating yourself at the job, then using that as a demo for more, while always trying to have a holistic view. His preferred approach is finding gaps, hypothesizing fixes, changing something, measuring, and then sharing. He proposes a DevOps Checklist to identify gaps when it comes to DevOps.
He emphasizes the need to target messages and actions on very specific aspects, with clear and measurable benefits, and avoiding possibly scary and misunderstood terms like "DevOps".
One of his tips to overcome inertia is thinking of personas, to understand fears and what people care about, instead of always butting heads.
He view a lot of the responsibility for change as residing with the workers, not the bosses, since we work in a "knowledge field", where workers usually know more about the work than their bosses.
He cautions, though, against mistaking novelty for innovation, or motion for action.
Boris Feld, in his talk The importance of Why in DevOps insisted that people worry about the why of DevOps before they tackle the how, as the how is entirely contingent on the why. He thinks a lot people unfortunately worry about the how without having a clear idea of the why.
In Bizdevops – from development to the customer, Sabine Bernecker-Bendixen talked again about the relational aspects of DevOps, and how they can apply as well to business-tech relationships ("business" being whoever pays the final bill). To her, it's all about empathy, adapting your language, and translating emotions and feelings : "I'm OK, you're OK".
She insists we always communicate status in our work, even if there's nothing new, say what the problems are if any, so that people can emphasize. The alternative is people thinking you are incompetent or dumb. On the flip side, don't assume that people are incompetent or dumb, put yourself in their shoes.
Organizations should also use people's diversity as an asset, putting each in the place where they can have the most meaningful impact.
In DevOps Culture at BlaBlaCar - Keep CAMS and grow, Regis Allegre & Nicolas Blanc presented how DevOps work at BlaBlaCar, now that they have more than 10 million users and are experiencing exponential growth. One of the things I noted is that they don't use cloud infrastructure, except for very specific stuff like tests and emailing. An interesting practice they have is the "Weekly free speech", which is a whole company meeting where anyone can suggest subjects, and subjects to be discussed that day are then chosen by vote.
In Cognitive biases and our poor intuitions around probability, Nigel Kersten from Puppet Labs presented ways in which our cognitive biases negatively affect DevOps and change management in general. He makes an analogy between the shortcuts developers (rightfully) take, which can introduce bugs, and our brain's shortcuts, that can give us hard to overcome biases. The bugs introduced by developer shortcuts can be huge and incredible (his favorite example being the Xerox Copy Bug). The biases introduced by our brain shortcuts can be just as spectacular. Kersten insists that we take steps, particularly in postmortems, to avoid these biases.
Examples of known biases :
- More memorable events estimated as more frequent
- Hindsight bias, to avoid it, make predictions prior to results, review them after getting the results
- Attribution bias (exaggerating people's roles in outcomes). He gives the example of bad drivers : when you're the bad driver, there's an external factor (you're late, etc.), when others are bad drivers, it's because they're stupid
In Designing the Enterprise for Manufacturing, Scott Russell outlined how manufacturing concepts could be applied to DevOps. To summarize his proposition : in IT, we've only been making things for a few decades, let's learn from those who have been making things for centuries. He assimilates DevOps organizations to production lines, and sees the same kind of waste, in setup time, context switches, queuing issues...
Techniques from manufacturing he proposes we apply to DevOps include :
- taking into account non working hours where there might be zero use -> low use percentage, look at utilization of hardware, not capacity;
- distinguishing productions between high volume, low mix (variability in product type) and low volume, high mix;
- using statistical tests instead of systematic ones : check 1 item in 5 and if there's a failure check items produced around the failure;
He quipped about "Enterprise projects" being projects that have more developers than end-users :)
In Change management at scale: responsible agile delivery, Pierre-Yves Ritschard gave pointers for scaling the management of changes in production, such as opening firewall ports, while ensuring traceability, accountability and reversibility.
Of course, automation helps a lot, and infrastructure as code gives traceability and reversibility to infrastructure changes. In this vein, he insists we treat machines as cattle and not pets: they should be homogeneous and managed globally.
For things that can't be automated yet, he suggests keeping a text based log of changes (in Git). Pull requests can be used for peer review.In "Making the Elephant dance – Daily deliveries at SAP", Dirk Lehmann explains how he managed to bring DevOps ideas into the very rigid structure of a company like SAP. While his team is essentially a startup within SAP, he was expected to work with SAP processes as a way to test and stress these processes. He managed to get to deliveries every two weeks, from an SAP standard of at most quarterly deliveries, but in then trying to get down to daily deliveries, he encountered huge resistance. "No customer wants daily releases", he was told. He contrasts Innovation and processes, if processes are viewed as ways to ensure things are always done the way they were done before.He thinks breaking the rules a little bit is necessaries, asking for forgiveness later.Find allies, and split large processes in smaller ones so you can pick and choose the really necessary ones. Try to push trust over control.In "Screwing up for fun and profit", Oliver Hankeln talked about mistakes in corporate environments, and the patterns and anti-patterns he sees around this topic.
Patterns :
- transparency / trust / respect, proactive communication (professionalism implies being open about failure, and learning from postmortems)
- going to the root-cause
- accepting mistakes (postmortem is not a performance review. Focus on events not people, apply the prime directive)
- embracing failure: Whatever you do is an experiment, be an adventurer
- trying to predict failures to avoid them: have a look at what people are not confident about, to try to find patterns
Anti-patterns :
- hidden mistakes, which are wasted learning opportunities. This is linked to "CEO disease", i.e. only talking about successes.
- blaming: fear of blame encourages hiding of mistakes and/or of their root causes
- the "arc of escalation" : "escalating" issues forces a blame game, and information gets lost through the ensuing Chinese whispers game.
- information bias (decisions based on too little knowledge) : talk to people, try to switch perspectives...
- cowardice (situations where no one wants to be responsible)
He warns against incentives (financial incentives or even people getting fired for making mistakes) that encourage the anti-patterns, or make people avoid mistakes so badly that any meaningful changes are prevented.
Finally, for my part, I presented my open-source delivery tool Git-deliver. The talk went rather well, and generated quite an interest, as evidenced by the attendance and dynamism at the open space session I proposed after the talk. Questions centered around possible use cases and very concrete deployment issues people are experiencing with their current tooling. It gave me ideas for new features, for example when it comes to automated provisioning of new delivery targets.