Author: Alexandre Masselot

Culture

Polar Expeditions and Agility: The 1911 Race to the South Pole and Modern Tales

www.octo.chA polar expedition and an IT project have much in common. They both share a goal, a team, and constraints. They share risk management issues, as failure is always a possibility even if the stakes are different. They also share a special relation with tooling and the influence of leadership style. But they mainly share the importance of the philosophy under which each project is undertaken.

We will see in this article how the approach taken to face a challenge can have tragic outcomes. And how the fantastic race to the South Pole in 1911, or modern polar explorers methodology, can relate to our daily IT experience.

My personal life as a software developer, a data scientist, a team leader and now an agile consultant is every day influenced by my ten years experience as a polar skier, and even more by the giants on whose shoulders I stood.

Vision, team, leadership, decision making, continuous improvement, tooling… The source of inspiration never ends. This post is an attempt to share some of it.

Slides are also available on slideshare.

Read more

Archi & Techno

Visualizing massive data streams: a public transport use case

www.octo.ch

Public transport companies release more data every day and some of them are even opening their information system up to real time streaming (Swiss transport, TPG in Geneva, RATP in Paris are a couple of local ones). Vast lands are unveiled for technical experimentations!

Beside real time data, these companies also publish their full schedules. In Switzerland, it describes trains, buses, tramways, boats and even gondolas.

In this post, we propose to walk through an application built to visualize, in fast motion, one day of activity, as shown in this movie. As real time data are not yet available, they were simulated, based on available schedule information. This pretext is too good not to dig into a stack containing Play/Scala/Akka on the backend, Angular2/Pixi.js/D3.js/topojson in the browser, linked together by Server Side Events.

This prototype is intended to explore the possibility of doing massive geographical visualization in the browser, applying techniques described in a previous post.

The backend and frontend code is available on github, and tests continuously ran on travis-ci.

Read more

Archi & Techno

D3.js transitions killed my CPU! A d3.js & pixi.js comparison

www.octo.chD3.js certainly is the most versatile JavaScript data rendering library available: turning data into mind blowing visualizations is only limited by your imagination. A key component to turn static pages into animated ones are the powerful selection transitions. However, too many simultaneous transitions on a web page will soon bring you CPU on its knees.
Hence this blog post.

We faced this problem when displaying swiss transport real time data on a map, within an SVG layout: rendering was lagging, event sourced data were not consumed consistently and laptop batteries were drowning at a dramatic speed. A video from a first attempt can be seen, and compared to a newer implementation with the technique presented in this article. Another surprise came from rendering a simple clock, burning 20% of CPU with a single transition.

If d3.js has no serious concurrents for many rendering problems, we decided to give try to a JavaScript library used for building games and leveraging the strengths of HTML5 and GPU: pixi.js.

At first, we will propose in this post a comparison between the two libraries in terms of rendering performance. For the sake of completeness, we will also discuss native CSS transitions. We will then dive into a couple of tricks to enhance dynamic visualizations with each of the two libraries and will even combine them to get the best of both worlds.

The project source code with benchmark data are hosted on github and a demo is available on github.io.

Read more

Big Data

A Journey into Industrializing the Writing and Deployment of Kibana Plugins (riding Docker)

by Alexandre Masselot (OCTO Technology Switzerland), Catherine Zwahlen (OCTO Technology Switzerland) and Jonathan Gianfreda.

The possibility of custom plugins is a strong Kibana promise. We propose an end to end tutorial to write such plugins. But this “end to end” approach also means “how to continuously deploy them?”, “how to share an environment with seeded data?” Those questions will bring us in a full fledged integration infrastructure, backed by Docker.

The Elasticsearch has grown from a Lucene evolution to a full fledged distributed document store, with powerful storage, search and aggregation capabilities. Kibana has definitely brought a strong component for interactive searching and visualization, transforming the data storage tier into an end user browser.

Customizable dashboards via a rich library of graphical components made its success, but soon, the need for real customization arose. If plugins were thought to be integrated from early on, the actual customization often lied into forking the master project and adapting to on particular purpose. Merging back fixes was soon to be a daunting effort to keep up with the high pace of the Github repository evolution .

Fortunately, as of version 4.3, the Kibana project took a more structured way to integrate custom plugins. The promise of maintainable external plugins became true. Those plugins, written in JavaScript, can be as simple as a standalone widget (e.g. a clock), a field formater (an up/down arrow instead of positive/negative number), a graphical representation of a search result (a chart) or a full blown application.

So, that should be easy. Just google and you would craft wonderful shiny visualizations.

But not fast, young Kibana Padavan! Documentation lacks, resources are valuable but scarce. But the promise is still shiny and we want to reach it.

In this post, we propose to share our journey into the writing of Kibana plugins, the little pitfalls we fell in and the setup of continuous deployment into a Docker environment. There is no dramatic discovery or stunning breakthrough today, but a tentative to write a map to make your journey easier.

Read more

Big Data

Geo localizing Medline citations

Where are the scientific publications coming from? Geolocalizing Medline citations

www.octo.chWhen and where are the scientific publications coming from? Which country are collaborating the most? To investigate those questions, we focused on Medline, the major biology and biomedical peer reviewed citations repository.

Big Data is not only a buzz word. A rich ecosystem of tools have emerged, together with new architectural paradigms, to tackle large problems. Open data are flowing around, waiting for new analysis angles. We have focused on the Medline challenge to demonstrate what can be achieved.

To provide some insights on how an interactive web application was built to explore such data, we will discuss the geographic localization method based on free text affiliation, Hadoop oriented treatment with Scala and Spark, interactive analysis with the Zeppelin notebook and rendering with React, a modern JavaScript framework. The code has been open sourced on github [1, 2] and the application is available on Amazon AWS.

Read more

Archi & Techno

New digital challenges in life science: the Swiss Lemanic area and beyond

www.octo.ch

The life science sector faces great challenges when the time comes to meet modern digital opportunities. This sector is incredibly dynamic, both from an economic and a scientific point of view, but innovation in the lab often comes in pair with a deep evolution in the information system strategy.

In this article, we will first focus on the Swiss Lemanic area landscape and see how the main computational trends encountered there are representative of the domain at large. We will then try to extract the most prevalent technical issues, either methodological or technological.

More importantly, we will see that life science has a particular culture and faces original computational issues. But a lot of the digital aspects are strikingly similar to those of other industries, like finance, retail or social media. Therefore, a great deal is to be learned from how companies in those sectors have undertaken their own transition towards a new digital era (and vice versa). Read more