BOG’OPS’ challenge : our recipe for Team Spirit

When I arrived at OCTO Technology in 2014, I immediately loved and embraced its culture of team spirit and sharing and improvements of practices, which are an integral part of OCTO’s DNA. And thus the « I » rapidly turned into a « We ».

This culture also lies in the different community teams, a.k.a tribes. Each team organizes itself depending on its size and what they want to work on, to gather regularly and to share.

In this article, the OPS tribe will show how 3 of us organized an epic day called “The Bog’Ops” challenge, for over twenty people passionate by Cloud, “Infrastructure as Code”, automation and, most of all, by challenges.

One of the objectives of this event was to organize a friendly get-together while mobilizing in the same place at the same time all those brilliant minds on one single subject with an optimal cohesion effect.

And to spice up the whole thing: a competition between 2 teams.

Ingredients

  • a big room with the usual devices (projector, speakers and a network connectivity worthy of its name)
  • a comfortable work surface (tables, chairs, and computers)
  • an open-bar account on AWS as a playground
  • a great, motivated team
  • 3 crazy cooks ready for anything

Recipe

Our major concern was to ensure that nobody could solve the challenge by himself in one day, so we worked on a scenario that would require the implementation of several technical components:

  1. a data store layer that would need the implementation of a Ceph cluster
  2. a RESTful API
  3. a valid certificate recognized by an official CA to expose the API over HTTPS
  4. a continuous deployment with Concourse, that we’d been wanting to test for quite a while, using standardized job names exposed via its own API,
  5. the whole deployed in the AWS cloud while making the best use of its possibilities to make the application resilient and auto-scalable.

Architecture Générale

We reviewed the knowledge and motivations of everyone to be sure to build 2 teams able to take up that challenge, which will be called:

  1. The Cuddling Sea Otter Pups
  2. The Laughing Red Panda Cubs

Tasting

Overview of the menu

The challenge’s content was finally revealed to the whole tribe the afternoon before, via an 8-page PDF document containing the instructions and the description of the architecture to implement.

Starter

9.00am: everybody is already here

With the game scheduled to begin at 9.30 am, the tribe has never been so punctual. At 9.00, everybody was gathered in the room, waiting for the « go ». Considering all the tasks to do, it was to be the start of a full thrilling day. The amazing dashboard filled with red indicator perfectly complemented the challenge’s environment.

DéfiOPS  ops t-shirts  l'enthousiaste

To strengthen the team spirit, each participant received a customized T-shirt with the team’s name and logo, his trigram and a personal, funny nickname.

Main course

To meet that challenge, the « Pandas » and the « Otters » were free to organize themselves to create working groups and process the different tasks in parallel: API development, CI installation, certificates implementation, storage infrastructure, AWS infrastructure configuration...

Inspired by the DevOps’ culture, each team chose a kanban on the wall, then filled it with a backlog of the tasks to be done and with 1-hour iterations.

With 10 people by team and 5 bricks to implement, pairs naturally formed to share the tasks and define the target architecture.

Team decisionsThe Cuddling Sea Otter PupsThe Laughing Red Panda Cubs
LanguageNode.jsRuby
CertificateLet’s EncryptAWS Certificate Manager
Object StorageUse of S3 while waiting for the Ceph cluster to be readyUse of S3 while waiting for the Ceph cluster to be ready
Application packagingDockerDocker

10.00 am: time to work!

Kanban DevOpsPoint d'avancementAt the end of each 1-hour iteration, the team would present the progress report:

  • architecture implementation status
  • team organization (who is doing what)
  • a stand up to present the retrospective on the last iteration
  • next actions
  • difficulties encountered and attention points

With the first performance test expected at 1 pm, they had less than 3 hours left to develop the application, publish it and deploy it using the CI on AWS.

High availability, scalability and Ceph clusters? Let’s leave that for later!

Results weren’t long to come: the « Otters » took the lead!

Tableau de bord

Aide

During each iteration, the teams were facing several kinds of difficulties. When a pair was blocked, they would ask the rest of the team for some help, and by working together as a team, ideas emerged to solve the problem.

The goal of the morning was to bring up the entire infrastructure to make all the dashboard red indicators go green thus validating the proper functioning of implemented architectures. A table!

12.23 pm sharp: a well-deserved break

Quick lunch: between the last ones that need to be kicked out of the room and the first ones coming back from lunch, no more than 30 minutes had passed.

12.49 pm: resumption of hostilities

The first performance campaign initially scheduled at 1 pm was finally postponed to 2 pm in order to let the teams finish assembling the bricks. The « Otters » still had a slight advantage on the « Pandas ».

And the stressful (but not for everyone) performance campaign session could finally begin.

Stress test 

The stress scenario was as follows:

  • Ramp-up traffic injection on the platform for 30-40 minutes
    • PUT (to put files)
    • GET (to retrieve files and apply an unpredictable operation on them) will solicit each platform to pressurize it

During the performance tests, the organizers trigger the following events:

  • unexpected” VM crashes to disturb the platform and analyze its behavior (cf. design for failure) during load,
  • application deployments launched through Concours API to simulate an upgrade of the application version upgrade

Performances

Whereas the « Otters » were leading the race with a slight advantage for a while, the following performance campaign reversed the trend, and it was with goodwill that the « Pandas » were celebrating this event with modesty. Les pandas

18.00 19.00 pm: end of the game and results Fin de l'épreuveBogops

The tribe successfully met the challenge: by the end of the day, each team had an entirely working solution with only tiny differences.

Dessert

With a ROTI of 5 on this day, the objective was reached:

  • the tribe’s team spirit is even stronger than before,
  • each participant was able to work and improve himself thanks to the team, on the subjects we love
  • DevOps’ culture which drives us, has proved itself once again even on such a short term period:
    • communication, tasks and information sharing, mutual help and solidarity,
    • tests and measures,
    • team rituals (kanban, iterative development, stand-up),
    • infrastructure as code and tooling: Git, Concourse, Docker, Ceph, AWS with EC2, S3, Route 53, ELB, ECS...

On top of that

In addition, here are a few verbatims of the tribe who enjoyed the whole day:

Thank you, it was awesome. Congrats to all the OPS and to the organizers (Eric, Yann and Arnaud) who really rocked !” “Thank you guys it was really great ! Ok to do it again tomorrow ;) Big up to all the bogops !!!” “Simply exceptional. Thank you for everything, and congrats to panda team : we kicked some otters’ asses.“Thank you to the Otters team who kicked pandas’ asses the whole day, as expected :)“A great moment of open friendship and healthy friendly competition !“Great day :) Thanks to the organization team (which was magical and enlightening) and to the player team.

With a message shared several times and which still echoes: “It's awesome to learn while having fun !

Next time, come and play with us.

And if you behave, we will disclose to you the backstage of the preparation of such a great day.