Archi & Techno

Joyful wind of change: A software craftsmanship short tale

This is the story of a team. A bunch of 11 aspiring software craftsmen who decided to change things around and get their job done in a better way.
The story takes place between the 30th and the 50th iteration of the development process of a software. This software is a website serving over 2 million regular users and providing legal information and services to 65 million French citizens.

Chapter one:
Start from what hurts and set a direction

Leaky pipeline

It is normal that the build fails. There are some automated E2E tests that fail randomly. Just try again, hopefully it will work next time.

This was the common answer to all new developers who struggled passing their first user story. And for information, the build then used to be about 45 minutes long. Almost nobody would get shocked anymore, as if everyone just got used to the pain. Read more

Archi & Techno

How I removed two databases in one pull request

My first project right after school was called KISS which means  

Keep it simple, stupid

Three years and a few IT projects later, I thought that I got it right and applied the following rules:

  • keep your applications small and simple,
  • remove most of its complexity,
  • don’t over-engineer,
  • all code and user stories have to be maintained (YAGNI)

Yet a few weeks ago, I learned that you can always remove useless things. You just don’t know it yet.

Story time

I wanted to create an API above the drugs data set (freely provided by the French government as CSV files). I knew that I needed a script to parse and store the data and a web API to expose them. The code is on github and the application is live.

When I start a web application, I have 2 main choices to make:

  • Choose a platform: I picked nodejs, for the only reason that I am proficient in it
  • Choose a database: I took part to a similar project where I used readDB, commonly called mongoDB. It was painfully slow when upserting lots of data, so I chose couchbase instead, a key/value store with a document query language.

After a while I needed “full text” search, so I added elasticsearch, simply because I knew how to use it. It requires a few (ugly) scripts to automate its synchronization with couchbase.

And suddenly came the epiphany! Or to be more precise, I asked for a review and after detailing the context, my colleague simply asked: “why don’t you remove couchbase?”. He was spot on: the dataset fits in memory and it is readonly.
It was replaced with a simple import to load the JSON in memory ready to be queried by the application.
But it got me thinking, couldn’t I remove the second database too? And I learned that there are indeed multiple libraries to search into local documents: I just used lunr and could therefore remove both databases.

The application is now more simple and reliable:

  • The code base has been reduced: 122 untested lines of script, 227 lines of configuration
  • The application is now deployed with one image: less reasons to fail
  • No more scripts to load the data in the database (the file is read by the application)

Nonetheless, after the refactor, the application was a bit slower even if everything was done in memory. Having said that, response time is still below 20 milliseconds, so the advantages brought by this removal still exceed the loss of a few milliseconds.

This resulted in a pull request with twice as many deleted lines as added lines and my takeaway would be:
Following KISS principles is more difficult than it seems: you can always question yourself but you should more likely seek for feedback from your peers. And, instead of choosing a database straight away, I would now choose a place to store my data.

Happy coding!

Archi & Techno

GraphQL: A new actor in the API world

In 2012, Facebook teams were in the process of rebuilding their native mobile applications and they had to adapt their server queries to get the data they needed. When they started to refactor some parts of their code, some got frustrated and stepped back to think of a better way. GraphQL was born.

They changed their point of view in terms of resources, and they preferred the idea of a graph of objects instead. The whole idea is to call a single endpoint, passing a query as the content of the request.

Throughout this post, I will give an overview about what GraphQL is, including advantages and drawbacks. Then will come the best part with being how to create a GraphQL API.

Read more

Archi & Techno

A money pot in the blockchain

The blockchain is a trendy subject, and we believe one needs to apply its principle to an actual project to understand its technology and progress status. We chose to implement an online money pot in Ethereum, a kind of blockchain that focuses on smart-contracts instead of money transfer. This article demonstrates how we can leverage Ethereum to develop this use-case.

Read more

Big Data

A Journey into Industrializing the Writing and Deployment of Kibana Plugins (riding Docker)

by Alexandre Masselot (OCTO Technology Switzerland), Catherine Zwahlen (OCTO Technology Switzerland) and Jonathan Gianfreda.

The possibility of custom plugins is a strong Kibana promise. We propose an end to end tutorial to write such plugins. But this “end to end” approach also means “how to continuously deploy them?”, “how to share an environment with seeded data?” Those questions will bring us in a full fledged integration infrastructure, backed by Docker.

The Elasticsearch has grown from a Lucene evolution to a full fledged distributed document store, with powerful storage, search and aggregation capabilities. Kibana has definitely brought a strong component for interactive searching and visualization, transforming the data storage tier into an end user browser.

Customizable dashboards via a rich library of graphical components made its success, but soon, the need for real customization arose. If plugins were thought to be integrated from early on, the actual customization often lied into forking the master project and adapting to on particular purpose. Merging back fixes was soon to be a daunting effort to keep up with the high pace of the Github repository evolution .

Fortunately, as of version 4.3, the Kibana project took a more structured way to integrate custom plugins. The promise of maintainable external plugins became true. Those plugins, written in JavaScript, can be as simple as a standalone widget (e.g. a clock), a field formater (an up/down arrow instead of positive/negative number), a graphical representation of a search result (a chart) or a full blown application.

So, that should be easy. Just google and you would craft wonderful shiny visualizations.

But not fast, young Kibana Padavan! Documentation lacks, resources are valuable but scarce. But the promise is still shiny and we want to reach it.

In this post, we propose to share our journey into the writing of Kibana plugins, the little pitfalls we fell in and the setup of continuous deployment into a Docker environment. There is no dramatic discovery or stunning breakthrough today, but a tentative to write a map to make your journey easier.

Read more

Archi & Techno

Identify performance issues in your Android application

A smooth user experience is a common attribute of all the successful Android applications. It might sound as an obvious statement but many are the applications that are a little bit “laggy”. Moreover android developers are use to test their application mostly on high-end devices (generally their own devices), forgetting that their app will be run on cheaper devices, which will deteriorate the app smoothness a lot.

It is pretty easy to find documentation on the Internet that will give you advices to have an efficient android application (for instance the great android performance documentation page itself). In this article we propose the opposite: presenting tools that will help you benchmarking an existing Android application in order to find some room for improvement. It can also be a way to measure improvements when we are doing some performance related tasks. We will try to have first a macroscopic approach and then will go deeper and deeper in the analysis in order to find the hidden cause of a performance issue.
Some of the tools I will talk about are part of the android standard build tools, some are “external” products. All of them have proven their efficiency on the projects I have worked on, including Le Monde applications which are massively downloaded (over one million downloads) and need to be performant even on low end devices .

In the first part we will see how we can identify app launch slow downs thanks to two tools: Nimbledroid and AndroidDevMetrics. In the second part we will focus more generally on User Experience (UX) issues by using standard Android tools.

Read more

Archi & Techno

Protocol Buffers: Benchmark and mobile

Going faster on a mobile has become essential. Putting aside the actual choice on the means of communication, the data format used weighs in quite a bit when it comes to the speed. As of Today, JSON has proven to be a standard media for APIs. Still, is this data format suitable for mobile? For instance, handling JSON in an Android environment is difficult.

Other data formats are emerging in recent years like Thrift, Avro, Message Pack or Protocol Buffers.

Protocol Buffers has the ability to have a binary format that is easily adaptable and which can be manipulated. It also has a very basic structure to write and understand and easily generate source code for several languages.

This blog featured two articles on Protocol Buffers (protobuf) in 2012. The version used was 2.4.1, and the standard was proto2.

Read more

Archi & Techno

DockerCon 2016: “Nobody cares about containers!”

docker run busybox /bin/echo ‘hello boys and girls!’

As humans, we like new shinny things. And working in a wannabe devops world, that means solving problems with containers, too. We have been working with them for quite a while now, and we are more than happy with the results. We like them so much, that we decided to travel more than 8000 km, to Seattle, Washington, in order to go to this year’s DockerCon.

Dockercon16

So if you’re into containers as well, and you’re dying to hear the hot news, you’ve come to the right place. Fasten your seatbelts, and enjoy the ride!

Read more