DevOps tips and tricks, on the ops side

After applying as much as possible the DevOps principles for more than a year on the run of a highly business critical project of one of our customers, here are some golden rules we eventually found out and tried to stick with. Items are here presented from an Ops perspective.

One deployment tool to rule them all

Work hard on getting a nice deployment tool (scripts, capistrano, whatever) and use it to deploy on all your environments from development to UAT, perf and of course production. You'll make sure you'll apply the fail fast principle to deployment. Make sure it pulls all the artifacts it needs from your repo, Nexus or such.

If deployment is painfull do it more often, if it's fun, do it more often

Deployment must not be considered as a failure nor as an exception. It's fine, it's a standard and a frequent part of the job, it shows you have worked to bring out new features, improved performances, bugfixes. Don't be scared to deploy often, « practice makes perfect ». The more often you deploy, the smaller the changes are, the lesser the risks.

Monitoring is for everyone

From technical (sysadmins, developers, DBA, HelpDesk) to highly non-technical people (marketing, financial, top management), every one loves to see the heartbeat of the system. Propose several screens / dashboards to present the data to each perspective.

Commit developers in every steps of the production life, and give them feedback, loads of it

Make sure you can wake them up during a tricky update or even better, grab some extra pizza to make sure they stay during the upgrade process.

Give access to all data they need to investigate, either in real-time on the production environment or at least by giving access to data / logs with as few delay as possible

Use production-like database content, A.S.A.P.

Real-life data often contain  stranger stuff than you could ever imagine, use them. Anonymize them (yes, you really want to avoid emailing real users during your tests), reduce their size if needed, but do keep historical data. They will help with UAT, perf testing, and will secure your production roll-out.

Share tools and practices with Devs

Kanbans, scrums, retrospectives, Jira, Confluence are probably used into your dev team, ritualize such principles on the ops side, you'll share the same logic and delivery rhythm.

Express needs, act as a P.O. when needed

If you need something from the Devs that will help you make the run better / cheaper / faster, such as a better log classification, functionnal metrics to monitor, become a Product  Owner, propose user stories and make sure they take life. Negociate with the business to get a minimal amount of complexity points for ops user stories at each iteration.

Automatize everything that can be (especially boring stuff)

Script, script, script, keep your mind free from silly or time-consuming stuff, you have better things to do.

Manage your systems automatically

Use Chef, Puppet, cfEngine + a VCS (git, mercurial, SVN…) to store and historize your configurations.

Embrace the Test-Driven Infrastructure Approach

Try to make every change on your infrastructure testable, use the monitoring tool as a an automatic tool for it.

Elect a goalie to keep other ops from task-switching

This will ensure you that small tasks are handled as soon as possible and keep people working on long-term tasks focussed.