When I arrived at OCTO Technology in 2014, I immediately loved and embraced its culture of team spirit and sharing and improvements of practices, which are an integral part of OCTO’s DNA. And thus the « I » rapidly turned into a « We ».
This culture also lies in the different community teams, a.k.a tribes. Each team organizes itself depending on its size and what they want to work on, to gather regularly and to share.
In this article, the OPS tribe will show how 3 of us organized an epic day called “The Bog’Ops” challenge, for over twenty people passionate by Cloud, “Infrastructure as Code”, automation and, most of all, by challenges.
One of the objectives of this event was to organize a friendly get-together while mobilizing in the same place at the same time all those brilliant minds on one single subject with an optimal cohesion effect.
And to spice up the whole thing: a competition between 2 teams.
Ingredients
a big room with the usual devices (projector, speakers and a network connectivity worthy of its name)
a comfortable work surface (tables, chairs, and computers)
Our major concern was to ensure that nobody could solve the challenge by himself in one day, so we worked on a scenario that would require the implementation of several technical components:
a data store layer that would need the implementation of a Ceph cluster
a RESTful API
a valid certificate recognized by an official CA to expose the API over HTTPS
a continuous deployment with Concourse, that we’d been wanting to test for quite a while, using standardized job names exposed via its own API,
the whole deployed in the AWS cloud while making the best use of its possibilities to make the application resilient and auto-scalable.
We reviewed the knowledge and motivations of everyone to be sure to build 2 teams able to take up that challenge, which will be called:
The Cuddling Sea Otter Pups
The Laughing Red Panda Cubs
Tasting
Overview of the menu
The challenge’s content was finally revealed to the whole tribe the afternoon before, via an 8-page PDF document containing the instructions and the description of the architecture to implement.
Starter
9.00am: everybody is already here
With the game scheduled to begin at 9.30 am, the tribe has never been so punctual. At 9.00, everybody was gathered in the room, waiting for the « go ». Considering all the tasks to do, it was to be the start of a full thrilling day. The amazing dashboard filled with red indicator perfectly complemented the challenge’s environment.
To strengthen the team spirit, each participant received a customized T-shirt with the team’s name and logo, his trigram and a personal, funny nickname.
Main course
To meet that challenge, the « Pandas » and the « Otters » were free to organize themselves to create working groups and process the different tasks in parallel: API development, CI installation, certificates implementation, storage infrastructure, AWS infrastructure configuration...
Inspired by the DevOps’ culture, each team chose a kanban on the wall, then filled it with a backlog of the tasks to be done and with 1-hour iterations.
With 10 people by team and 5 bricks to implement, pairs naturally formed to share the tasks and define the target architecture.
Use of S3 while waiting for the Ceph cluster to be ready
Use of S3 while waiting for the Ceph cluster to be ready
Application packaging
Docker
Docker
10.00 am: time to work!
At the end of each 1-hour iteration, the team would present the progress report:
architecture implementation status
team organization (who is doing what)
a stand up to present the retrospective on the last iteration
next actions
difficulties encountered and attention points
With the first performance test expected at 1 pm, they had less than 3 hours left to develop the application, publish it and deploy it using the CI on AWS.
High availability, scalability and Ceph clusters? Let’s leave that for later!
Results weren’t long to come: the « Otters » took the lead!
During each iteration, the teams were facing several kinds of difficulties. When a pair was blocked, they would ask the rest of the team for some help, and by working together as a team, ideas emerged to solve the problem.
The goal of the morning was to bring up the entire infrastructure to make all the dashboard red indicators go green thus validating the proper functioning of implemented architectures.
12.23 pm sharp: a well-deserved break
Quick lunch: between the last ones that need to be kicked out of the room and the first ones coming back from lunch, no more than 30 minutes had passed.
12.49 pm: resumption of hostilities
The first performance campaign initially scheduled at 1 pm was finally postponed to 2 pm in order to let the teams finish assembling the bricks. The « Otters » still had a slight advantage on the « Pandas ».
And the stressful (but not for everyone) performance campaign session could finally begin.
The stress scenario was as follows:
Ramp-up traffic injection on the platform for 30-40 minutes
PUT (to put files)
GET (to retrieve files and apply an unpredictable operation on them) will solicit each platform to pressurize it
During the performance tests, the organizers trigger the following events:
“unexpected” VM crashes to disturb the platform and analyze its behavior (cf. design for failure) during load,
application deployments launched through Concours API to simulate an upgrade of the application version upgrade
Whereas the « Otters » were leading the race with a slight advantage for a while, the following performance campaign reversed the trend, and it was with goodwill that the « Pandas » were celebrating this event with modesty.
18.0019.00 pm: end of the game and results
The tribe successfully met the challenge: by the end of the day, each team had an entirely working solution with only tiny differences.
Dessert
With a ROTI of 5 on this day, the objective was reached:
the tribe’s team spirit is even stronger than before,
each participant was able to work and improve himself thanks to the team, on the subjects we love
DevOps’ culture which drives us, has proved itself once again even on such a short term period:
communication, tasks and information sharing, mutual help and solidarity,
tests and measures,
team rituals (kanban, iterative development, stand-up),
infrastructure as code and tooling: Git, Concourse, Docker, Ceph, AWS with EC2, S3, Route 53, ELB, ECS...
On top of that
In addition, here are a few verbatims of the tribe who enjoyed the whole day:
“Thank you, it was awesome. Congrats to all the OPS and to the organizers (Eric, Yann and Arnaud) who really rocked !” “Thank you guys it was really great ! Ok to do it again tomorrow ;) Big up to all the bogops !!!” “Simply exceptional. Thank you for everything, and congrats to panda team : we kicked some otters’ asses.” “Thank you to the Otters team who kicked pandas’ asses the whole day, as expected :)” “A great moment of open friendship and healthy friendly competition !” “Great day :) Thanks to the organization team (which was magical and enlightening) and to the player team.”
With a message shared several times and which still echoes: “It's awesome to learn while having fun !”
