Archi & Techno

Serverless sweetness: enforcing strict EC2 instance tagging policies with Lambda

Hey everybody,

TL;DR

I coded something in order to enforce strict tagging policies on AWS EC2 instances using
Python and a bunch of AWS services (Lambda, Cloudtrail, SNS, and S3). If you keep reading,
I’m going to talk to you about AWS Lambda and Serverless computing, or FaaS (Function as a service).

You can check the source code and permission related template files here:

https://github.com/sebiwi/broom

If you want to use it, create a Cloudtrail trail, activate it on every region, create an S3 bucket
to store the Cloudtrail logs, create a SNS topic for notifications, create a Lambda function using
the Python code (adapted to your resources and use case), an IAM role using the policy that comes
with it, and finally activate events from the S3 bucket to the Lambda function on object creation.

I can teach you how to do all of these things if you don’t know how. Just keep reading.

Read more

Culture

Polar Expeditions and Agility: The 1911 Race to the South Pole and Modern Tales

www.octo.chA polar expedition and an IT project have much in common. They both share a goal, a team, and constraints. They share risk management issues, as failure is always a possibility even if the stakes are different. They also share a special relation with tooling and the influence of leadership style. But they mainly share the importance of the philosophy under which each project is undertaken.

We will see in this article how the approach taken to face a challenge can have tragic outcomes. And how the fantastic race to the South Pole in 1911, or modern polar explorers methodology, can relate to our daily IT experience.

My personal life as a software developer, a data scientist, a team leader and now an agile consultant is every day influenced by my ten years experience as a polar skier, and even more by the giants on whose shoulders I stood.

Vision, team, leadership, decision making, continuous improvement, tooling… The source of inspiration never ends. This post is an attempt to share some of it.

Slides are also available on slideshare.

Read more

Archi & Techno

How does it work? Kubernetes: Episode 5 – Master and Worker, at last!

Hey everybody,

TL;DR

I hacked something together in order to create a Kubernetes cluster on CoreOS (or Container Linux) using Vagrant and Ansible.

If you keep reading, I’m going to talk to you about Kubernetes, etcd, CoreOS, flannel, Calico, Infrastructure as Code and Ansible testing strategies. It’s gonna be super fun.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 5, regarding the actual installation and configuration of the Kubernetes Master and Worker nodes.

Read more

Archi & Techno

Visualizing massive data streams: a public transport use case

www.octo.ch

Public transport companies release more data every day and some of them are even opening their information system up to real time streaming (Swiss transport, TPG in Geneva, RATP in Paris are a couple of local ones). Vast lands are unveiled for technical experimentations!

Beside real time data, these companies also publish their full schedules. In Switzerland, it describes trains, buses, tramways, boats and even gondolas.

In this post, we propose to walk through an application built to visualize, in fast motion, one day of activity, as shown in this movie. As real time data are not yet available, they were simulated, based on available schedule information. This pretext is too good not to dig into a stack containing Play/Scala/Akka on the backend, Angular2/Pixi.js/D3.js/topojson in the browser, linked together by Server Side Events.

This prototype is intended to explore the possibility of doing massive geographical visualization in the browser, applying techniques described in a previous post.

The backend and frontend code is available on github, and tests continuously ran on travis-ci.

Read more

Archi & Techno

How does it work? Kubernetes: Episode 4 – How to Ansible your CoreOS, and etc(d)!

Hey everybody,

TL;DR

I hacked something together in order to create a Kubernetes cluster on CoreOS (or Container Linux) using Vagrant and Ansible.

If you keep reading, I’m going to talk to you about Kubernetes, etcd, CoreOS, flannel, Calico, Infrastructure as Code and Ansible testing strategies. It’s gonna be super fun.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 4, regarding the usage of Ansible on CoreOS, and etcd.

Read more

Archi & Techno

How does it work? Kubernetes: Episode 3 – Infrastructure as code: the tools of the trade

Hey everybody,

TL;DR

I hacked something together in order to create a Kubernetes cluster on CoreOS (or Container Linux) using Vagrant and Ansible.

If you keep reading, I’m going to talk to you about Kubernetes, etcd, CoreOS, flannel, Calico, Infrastructure as Code and Ansible testing strategies. It’s gonna be super fun.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 3, regarding Infrastructure as Code, and the tools of the trade.

Read more

Archi & Techno

How does it work? Kubernetes: Episode 2 – Kubernetes networking

Hey everybody,

TL;DR

I hacked something together in order to create a Kubernetes cluster on CoreOS (or Container Linux) using Vagrant and Ansible.

If you keep reading, I’m going to talk to you about Kubernetes, etcd, CoreOS, flannel, Calico, Infrastructure as Code and Ansible testing strategies. It’s gonna be super fun.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 2, regarding Kubernetes networking.

Read more

Archi & Techno

How does it work? Kubernetes: Episode 1 – Kubernetes general architecture

Hey everybody,

TL;DR

I hacked something together in order to create a Kubernetes cluster on CoreOS (or Container Linux) using Vagrant and Ansible.

If you keep reading, I’m going to talk to you about Kubernetes, etcd, CoreOS, flannel, Calico, Infrastructure as Code and Ansible testing strategies. It’s gonna be super fun.

The whole subject was way too long for a single article. Therefore, I’ve divided it into 5 parts. This is episode 1, regarding the Kubernetes general architecture.

If you want to try it:

git clone https://github.com/sebiwi/kubernetes-coreos

cd kubernetes-coreos

make up

This will spin up 4 VMs: an etcd node, a Kubernetes Master node, and two Kubernetes Worker nodes.  You can modify the size of the cluster by hacking on the Vagrantfile and the Ansible inventory.

You will need Ansible 2.2, Vagrant, Virtualbox and kubectl. You will also need molecule and docker-py, if you want to run the tests.

Read more

Big Data

A quick summary and some thoughts on the Scikit-learn workshop

On december 2nd was given at Telecom ParisTech the workshop : “Using Scikit-learn and Scientific Python at Scale” with top contributors from the project as speakers. This workshop was divided into four talks :

  1.    Scikit-learn for industrial applications, basic research and mind reading – Alexandre Gramfort
  2.    Distributed computing for predictive modeling in Python – Olivier Grisel
  3.    Scikit-learn at scale : out-of-core methods – Thierry Guillemot
  4.    An Industrial application at Airbus Group – Vincent Feuillard

Scikit-learn is currently the most widely used open source library for Machine Learning applications. It has been developed in Python (Cython and C/C++) and, with over 1000 documentation pages, has become the major contribution for democratizing machine learning for a large audience.

Read more