10 tips for improving your business

The seminar will be held in French with support material in English.

Click here to register

Since 2007, OCTO intervenes on problems involving User eXperience as a key element in helping its clients achieve innovative products and applications, useful and usable in tune with the complexity of their business challenges. After our first meeting during USI 2012, we are delighted to welcome the Adaptive Path team for a workshop which will walk you through UX from strategy to practice, promising intesity despite a playful approach.

Jesse James Garrett’s team, Adaptive Path, will give for this workshop UX Essentials ( Programm details – pdf version) entirely dedicated to UX, provided for the first time in France on April the 10th, 11th and 12th !

« After my keynote at USI, I looked forward to working with OCTO again. I am happy we have this opportunity to work together to bring UX thinking to France! »

Jesse James Garrett, Chief Creative Officer, Adaptive Path.

Each day includes fun, hands-on activities designed to build skills and provide a framework that participants can immediately put into practice.

This workshop is led by Adaptive Path practitioners and will examine the key elements that contribute to creating great experiences.

Participants will spend a day immersed in strategy, followed by a day focussed on research, then wrapping it all up with a day devoted to putting those insights into action.

I want to subscribe to the UX Essentials worskhop

Nevertheless, up to now it was not possible to use Lint inside automated builds (i.e. on a Continuous Integration server like Jenkins). This situation just changed with release 3.5.1 of the Android Maven Plugin.

OCTO Contributed to release 3.5.1+ of Android Maven Plugin to offer integration of Lint as a simple maven goal : android:lint. The complete documentation of this goal can be found on the official documentation of the Maven Plugin goals. This articles provides a rapid overview of the android:lint goal’s usage and parameters.

Create a new Android project and its maven configuration

If you know how to create an Android project using Maven, or already have such a project, you can go directly to next section.

Let’s start a new Android project using acquinet maven archetypes. In a terminal, go to your working directory and type the following command :

1
2
3
4
5
6
7

mvn archetype:generate \
  -DarchetypeArtifactId=android-with-test \
  -DarchetypeGroupId=de.akquinet.android.archetypes \
  -DarchetypeVersion=1.0.9 \
  -DgroupId=com.foo.bar \
  -DartifactId=my-android-project \
  -Dpackage=com.foo.bar.android

In Eclipse, right-click in you package explorer view and select “Import” to import the project using the wizard “Import existing maven projects” :

Using the latest version of Eclipse (Juno), you will have to hack the imported projects to build them inside Eclipse :

• Right-click on the first project
• go to properties >> Java Build Path >> Sources tab
• Remove the src/test/java folder
• Apply the same procedure to the test project

An alternative, for now, is also to create a src/test/java folder in your project. As you can see, everything got much simpler than 2 years ago.

You can now go back to your terminal and build the project (both the app and its tests) using maven :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

>mvn clean install
...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] my-android-project - Parent ....................... SUCCESS [0.143s]
[INFO] my-android-project - Application .................. SUCCESS [3.228s]
[INFO] my-android-project-it - Integration tests ......... SUCCESS [7.966s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.234s
[INFO] Finished at: Mon Jan 07 14:57:48 CET 2013
[INFO] Final Memory: 22M/206M
[INFO] ------------------------------------------------------------------------

If you reach that point, then everything is setup properly and you now have a working Android Maven project that will also build inside Eclipse.

Add Lint checks to your Maven build

First, check that you are using the right version of the Android Maven Plugin. This plugin is managed by the parent pom of your project. It should look like :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

<project>
....
    <pluginRepositories>
        <pluginRepository>
            <id>oss.sonatype.org-jayway-snapshots</id>
            <name>Jayway OpenSource SNAPSHOTs on Sonatype.org</name>
            <url>http://oss.sonatype.org/content/repositories/jayway-snapshots/</url>
            <snapshots>
                <enabled>true</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>
    <build>
        <pluginManagement>
            <plugins>
                ....
                <plugin>
                    <groupId>com.jayway.maven.plugins.android.generation2</groupId>
                    <artifactId>android-maven-plugin</artifactId>
                    <version>3.5.1-SNAPSHOT</version>
                </plugin>
                ....
            </plugins>
        </pluginManagement>
    </build>
</project>

When you are sure to use the right version, you can now configure the Android Maven Plugin in your application’s project pom.xml file :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

<build>
    <plugins>
        ...
        <plugin>
            <groupId>com.jayway.maven.plugins.android.generation2</groupId>
            <artifactId>android-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>lint</id>
                    <goals>
                        <goal>lint</goal>
                    </goals>
                    <phase>install</phase>
                </execution>
            </executions>
            <configuration>
                <sdk>
                    <platform>17</platform>
                </sdk>
                <lint>
                    <skip>false</skip>
                    <failOnError>true</failOnError>
                </lint>
                <undeployBeforeDeploy>true</undeployBeforeDeploy>
            </configuration>
            <extensions>true</extensions>
        </plugin>
        ...
    </plugins>
</build>

You will now see that Lint is executed during the install phase of a maven for your project. These 3 lines should be logged during your maven build :

1
2
3
4

> mvn clean install | grep -i Lint
[INFO] Performing lint analysis.
[INFO] Writing Lint XML report in <path to your project>/my-android-project/my-android-project/target/lint/lint.xml
[INFO] Lint analysis completed successfully.

Customizing Lint usage via the Android Maven plugin

The new android:lint goal of the Android Maven Plugin supports quite a lot of features, most of them are directly related to the lint program (use lint --help to display all lint options).

You can use either command line parameters in the form of android.lint.* or, inside the <lint></lint> section of your pom.xml file you can use the following options :

Controlling Lint invocation

skip=true|false

whether or not to execute lint. False by default.

failOnError=true|false

if true, any lint error (not warnings) will stop the build. Defaults to false. This flag is useful for Continuous Integration builds as it allows you to enforce lint usage.

Controlling Lint granularity

ignoreWarnings=true|false

if true, don’t report lint warnings, only errors are reported. False by default.

warningsAsErrors=true|false

if true, all lint warnings will be treated as errors. Defaults to false.

Controlling Lint output

enableHtml=true|false

if true, lint will write a HTML report in the target/lint folder. False by default.

htmlOutputPath=<path>

path of the HTML report, only taken into account if previous property is set to true.

enableXml=true|false

if true, lint will write a XML report in the target/lint folder. True by default.

xmlOutputPath=<path>

path of the XML report, only taken into account if previous property is set to true.

Controlling Lint input

As usual : using lint, you can use a file named lint.xml at the root of your project to define the different level of severity of lint issues. This file will also be taken into account by the Android Maven Plugin.

enableSources=true|false

if true, lint will check the project’s source folder. True by default.

sources=<path>

path of the folder of the sources folder of your project. Defaults to ${project.build.sources}

enableClasspath=true|false

if true, lint will check the byte code of your project’s classes. False by default.

classpath=<path>

path of the folder of the classes of your project. Defaults to ${project.build.outputDirectory}

enableLibraries=<path>

if true, lint will check the byte code of your project’s dependencies. False by default.

libraries=<path>

path of the jars of the libraries of your project. Defaults to all jars of dependencies with a scope that is not “provided”.

Lint goal and maven phases

We must issue a warning here : the last two properties of the lint configuration are disabled by default. Using them is sensitive as they can have side-effects : both classes and libraries of a maven project are only present during certain phases of a maven build. For the classes, your project must have been compiled (during the compile phase). For the libraries, the dependencies' artifacts must have been resolved by maven (during the generate-sources phase of maven).

For this reason, we strongly encourage you to use the android:lint goal during the install phase of your application :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<build>
    <plugins>
        ...
        <plugin>
            <groupId>com.jayway.maven.plugins.android.generation2</groupId>
            <artifactId>android-maven-plugin</artifactId>
            <executions>
                <execution>
                    <id>lint</id>
                    <goals>
                        <goal>lint</goal>
                    </goals>
                    <phase>install</phase>
                </execution>
            </executions>
            <configuration>
                <!-- your lint config here-->              
            </configuration>
        </plugin>
        ...
    </plugins>
</build>

Conclusion

The new android:lint goal of the Android Maven plugin will let you add quality checks to your Android projects. You can either use it on the command line or inside an automated built on a continuous integration server. You can even make your build fail if you have any lint error or any lint warning. We hope Android developers will welcome this tool and that lint checks will be more and more widespread in the Android community.

At OCTO, we are proud to have contributed to the Android Maven Plugin. We think that this new maven goal is a good way to enhance the robustness of our applications. However, something is still missing according to us : a lint plugin for Sonar in order to visualize all lint issues directly inside our Sonar dashboards. That's our next R&D target, anyone interested in coordinating their efforts can just leave a message below this post.

Once again this year, we invite you to propose your session, with the deadline set on the 4th of March, 2013:

• Do you have a biography, a session title and resume, and a high-definition picture?
• Are you an opinion leader, a referent on one or more IT subjects, a super geek, a manager, or part of IS department?

Then you’re ready! We need your contibutions, your ideas, and your knowledge to make the 5th yearly USI an event as great as the reputation that proceeds it. As many as 700 guests are expected this year!

The USI team will determine, after having studied your proposition, the length of your session (20 or 40 minutes). For a callback, the first criteria will be the quality of the proposed session’s resume (depth and form). Thus we count on you to send your best work, one that will astound members of the jury.
The USI team will study every proposition and select those that seem the most interesting. You may be invited to come present your session beforehand, and the final list of accepted candidates will be announced at the end of March 2013.

You can also see the list of proposed sessions online, and vote for the ones that you most want to see at USI 2013. The USI site has everything you need to post sessions on popular social networks, allowing you to easily share your session with those around you. So don’t hesitate!

Click here to submit a session for USI

A glass to your candidacy then, and thank you!
The USI team

Key features of the “cloud machine”

This machine is made of:

• 3 motherboards, 2 with virtualization capability
• 1 managed switch
• 1 backplane with 4 disks (3 HD, 1 SSD)
• 1 DVD reader/burner
• 1 power supply
• 1 power supply control board
• Plenty of cables!

The architecture is as follows:

Choice of motherboards

Since my objective was to study cloud/virtualization and especially KVM, at least 2 CPU needed to support virtualization instructions. I was also looking for a small form factor because my home is rather small! I wanted all hardware Linux compatible and finally, my budget was also limited.

I chose to restrict my search to commodity hardware and to miniITX form factor. Most of miniITX available cards are based on Atom; so they don’t support virtualization instructions. I was disappointed by the lack of Atom Z series in commodity boards. But AMD saved me, by releasing the APU series. Finally, I chose the ASROCK E350M1.

This card provides very good value for the price, is simple to install and manage. But the key issue of commodity hardware is the lack of management functions: no IPMI, no ability to reset through the network, no SMB access… Then, I decided to use two ASROCK E350M1 as “compute” nodes, and to find a board with remote management capabilities as a “controller node”. Looking for such motherboard I chose the Supermicro X7SPA-HF. This board is rather expensive but has some interesting capabilities: IPMI onboard, access to SMB, 2 Ethernet ports, passive cooling…

Choice of the network cards

Transferring gigabytes of disk images on the network requires serious bandwidth. I could not afford 10Gbps cards for obvious price reasons, so I chose the Intel PRO/1000 CT which supports jumbo frames (useful for heavy payload) and some other interesting features. It is affordable and close to what is found in mid range servers environments.

Additionally their form factor is compatible with compact integration.

Choice of the switch

Ability to make VLANs was a primary requirement. But the price was a key issue since most managed switches are made for large installations and not for SoHo applications. I chose the Netgear ProSafe GS108T which offers great value for the price and allows to prototype most of usual network cloud configurations (VLAN, DHCP snooping…).

Choice of RAM and disks

I chose common low cost hardware that one finds on good shops on the Internet. Compute nodes can bear up to 8 GO.

The disk backplane, an ICY Box IB-2222SSK, is a very important device for the mechanical integration. However, I had some compatibility problems with the disks and the ICY Box. First, the mechanical aspect of the device (difficult to insert the disk in one slot) and second, Samsung disks that could not start rotating visibly for obscure power supply reasons. I managed to make it working with Hitatchi disks and I am finally pleased with this configuration. I advice to test some alternative solutions (not easy to find on the market however…). Nothing to say about the DVD reader/burner, except it’s a SATA interfaced device.

Choice of power supply and control

This is the most custom made device of the Cloud machine, as well as the mechanical structure. In fact, PC power supplies are not designed to drive more than one motherboard.

To catch the design stakes, you have to know how an ATX power supply is controlled. Look at ATX article on Wikipedia  and ATX12V PSDG2.01.pdf (search on Google).

There are two main issues:

• Make sure that Power on sequence (power and reset) is correct for the 3 motherboards using only one power on button;
• Shutdown of power supply shall be effective only if the 3 motherboards are in shutdown state.

I intended to design a control board to make all these controls. In fact, I only did a first version, very simple (based on TTL gates) which insures the correct power on sequence. But shutdown is done only from the Supermicro board. At the moment this design is sufficient, but I plan to make a control board based on Microchip PIC micro-controller that will implement all necessary logic.

The power supply is a 650W EPS, but a low cost 450W ATX would be sufficient.

Mechanical design and structure

Stacking motherboard is a pattern that makes me dreaming ;-) and I chose to reproduce such arrangement.

Overall dimensions of the cloud machine are 23cm x 26cm x 52 cm.

I chose to put the power supply on the bottom part of the structure for several reasons:

• The PSU fan blows up, pushing warm air in its upwards natural flow;
• The center of gravity is close to the bottom of the structure.

The drawback of this design is that dust falls into the PSU when it’s turned off.

Electric ground is connected to the whole structure through the PSU screwing.

All parts of the “Cloud machine” are in aluminum except steel thread bars that support the motherboards. I have a welding torch and aluminum solder which is fairly easy to use. A simple metal saw, a basic drill and some metal files make you able to do it as I did.

Further improvements

The next version of the control board will implement :

• Enhanced power supply control (common synchronized switch off);
• The ability to reset the ASROCK motherboards from the Supermicro motherboard (using USB port and a USB microcontroller on the control board).

Conclusion

It took me several week-ends to achieve this machine and I am really pleased to use it for my new software mocks. I spent around 2000 € in total. I had almost a heart attack when I saw that ASROCK motherboards didn’t support virtualization. But after BIOS update, they did as promised!!

The next blog article will present how this Cloud machine is installed with Linux and operated using PXE and SSH.

Have fun making your custom machines!

Good stress tests help us to:

• ensure that the software meets its performances requirements
• ensure that the service will deliver a fast response time even under a heavy load
• get to now the scalability limits which in turn is useful to plan the next steps of the development

Hadoop is not a web application, a database or a webservice. You don’t stress test a Hadoop job with a heavy load. Instead, you need to becnhmark the cluster which means assessing its performances by running a variety of jobs each focused on a specific field (indexing, querying, predictive statistics, machine learning, …).

Intel has released HiBench, a tool dedicated to run such benchmarks. In this article, we will talk about this tool.

What is HiBench?

HiBench is a  est un collection of shell scripts published under the Apache Licence 2 on GitHub : https://github.com/intel-hadoop/HiBench

It allows to stress test a Hadoop cluster according to several usage profile.

Micro Benchmarks

WordCount

This test dispatches the counting of the number of words from a data source.

The data source is generated by a preparation script of HiBench which relies on the randomtextwriter of Hadoop.

This test belongs to a class of jobs which extracts a small amount of information from a large data source.

It is a CPU bound test.

Sort

This test dispatches the sort of a data source.

The data source is generated by a preparation script which relies on the randomtextwriter d’Hadoop.

This test is the simplest one you can imagine. Indeed, both Map and Reduce stages are identity functions. The sorting is done automatically during the Shuffle & Merge stage of  MapReduce.

It is I/O bound.

TeraSort

This test too dispatches the sort of a data source.

The data source is generated by the Teragen jobs which creates by default  1 billion of 100 bytes lines.

These lines are then sorted by the Terasort. Unlike Sort, Terasort provides its own input and output format and also its own Partitioner which ensures that the keys are equally distributed among all nodes.

Therefore, it is an improved Sort which aims at providing an equal load between all nodes during the test.

With this specificity, this test is:

• CPU bound for the Map stage
• I/O bound for the Reduce stage

Enhanced DFSIO

This test is dedicated to HDFS. It aims at measuring the agregated I/O rate and throughput of HDFS during reads and writes.

During its preparation stage, a data source is generated and put on HDFS.

Then, two tests are run:

• A read of the generated data source
• A write of a large amount of data

The write test is basically the same thing as the preparation stage.

This test is I/O bound.

Web Search

Nutch indexing

This test focuses on the performances of the cluster when it comes to indexing data.

In order to do it, the preparation stage generates the data to be indexed.

Then, indexing is performed with Apache Nutch.

This test is I/O bound with a high CPU utilization during the Map stage.

Page Rank

This test measures the performances of the cluster for PageRanking jobs.

The preparation phase generates the data source in the form of a graph which can be processed using the PageRank algorithm.

Then, the actual indexing is performed by a chain of 6 MapReduce jobs.

This test is CPU bound.

Machine Learning

Naive Bayes Classifier

This test performs a probabilistic classification on a data source.

It is explained in depth on Wikipedia.

The preparation stage generates the data source.

Then, the test chains two MapReduce jobs with Mahout:

• seq2sparse transforms a text data source into vectors
• trainnb computes the final model using vectors

This test is I/O bound with a high CPU utilization during the Map stage of the seq2sparse.

When using this test, we didn’t observe a real load on the cluster. It looks like it is necessary to either provide its own data source or to greatly increase the size of the generated data during the preparation stage.

K-Means clustering

This test partitions a data source into several clusters where each element belongs to the cluster with the nearest mean.

It is explained in depth on Wikipedia.

The preparation stage generates the data source.

Then, the algorithm runs on this data source through Mahout.

The K-Means clustering algorithm is composed of two stages:

• iterations
• clustering

Each of these stages runs MapReduce jobs and has a specific usage profile.

• CPU bound for iterations
• I/O bound for clustering

Analytical Query

This class of tests performs queries that correspond to the usage profile of business analysts and other database users.

The data source is generated during the preparation stage.

Two tables are created:

• A rankings table
• A uservisits table

This is a common schema that we can meet in many web applications.

Once that the data source has been generated, two Hive  requests are performed:

• A joint
• An agregation

These tests are I/O bound.

Using HiBench

Run a stress test

Run HiBench is not very hard:

• retrieve the sources on GitHub
• Ensure that nobody is using the cluster
• Ensure that you have correctly configured your environment variables

Then, the file bin/hibench-config.sh contains all the options to tune before startting the stress test. It includes the HDFS directory where you want to write both source and result data, the path of the final report file on the local filesystem, …

Once configured, ensures that the HDFS directory where you want to write your data source and your results exists on the cluster and run the command bin/run-all.sh. Now you can take a coffee… or two.

Interpretation of the results

Results are written into the hibench.report file with the following CSV format:

test_name end_date <jobstart_timestamp,jobend_timestamp> size_in_bytes duration_ms throughput

Beware that the actual result file does not contain the column header  above.

The DFSIOE test also writes a CSV and an interpretation of its results in its subdirectory dfsioe.

Limitations

Latest Hadoop support

Presently, HiBench runs on Hadoop 1.0. This means that the latest versions of Cloudera or Hortons Works distributions for example won’t be able to run all tests since they rely on Hadoop 2.

However, the effort necessary to support Hadoop 2 is not that big for the majority of the tests since it is mainly a matter of updating configuration parameter names.

Also, HiBench alone is not enough for a good report of a stress test. It is necessary to also retrieve the informations provided by the JobTracker/ResourceManager like the mean execution time of Maps, Reduces, Shuffle and Merges of every job in order to build an accurate final report.

A public benchmark repository, a big lack

This is a lack that HiBench tried to address through its wiki page which invites you to post your results but with no success until now.

Building a public benchmark repository in order to provide a set of meaningful metrics to compare a cluster is still an uncover issue but  would be interesting and quite useful.

What are the alternatives?

An alternative exists to HiBench, but it is more focused on a specific usage profile.

GridMix

GridMix is included in Hadoop besides the example jobs like TeraSort, Sort, …

However, it generates MapReduce jobs which are focused on sorting large amount of data and does not cover other profiles like Machine Learning.

Conclusion

In spite of these drawbacks, HiBench greatly simplifies the benchmarking of a Hadoop cluster.

In the future, this domain will certainly see new tools with more functionalities and a better coverage or different usage profiles. It is only the beginning.

Next step now, planning the cluster… But Hadoop is a complex stack and you might have many questions:

• HDFS deals with replication and Map Reduce create files… How can I plan my storage needs?
• How to plan my CPU needs?
• How to plan my memory needs? Should I consider different needs on some nodes of the cluster?
• I heard that Map Reduce moves its job code where the data to process is located… What does it involve in terms of network bandwidth?
• At which point and how far should I consider what the final users will actually process on the cluster during my planning?

That is what we are trying to make clearer in this article by providing explanations and formulas in order to help you to best estimate your needs.

Some important aspects of the Hadoop architecture

Planning a Hadoop cluster requires a minimum knowledge the Hadoop architecture.

The distributed computation

At his heart, Hadoop is a distributed computation platform. This platform’s programming model is Map Reduce.

In order to be efficient, Map Reduce has two prerequisites:

• Datasets must be splitable in smaller and independant blocks
• Data locality: means that the code must be moved where the data lies, not the opposite.

The first prerequisite depends on both the type of input data which feeds the cluster and what we want to do with it.

The second prerequisite involves having a distributed storage system which exposes where exactly data is stored and allows the execution of code on any storage node.

This is where HDFS is useful.

Hadoop is a Master / Slave architecture:

• The JobTracker (ResourceManagerin Hadoop 2)
  • Monitor jobs that are running on the cluster
  • Needs a lot of memory and CPU (memory bound and cpu bound)
• The TaskTracker (NodeManager + ApplicationMasterin Hadoop 2)
  • Runs tasks of a jobs on each node of the cluster. Which means Maps and Reduces
  • Its jobs need a lot of memory and CPU (memory bound and cpu bound)

The critical component in this architecture is the JobTracker/ResourceManager.

The distributed storage

HDFS is a distributed storage filesystem. It runs on top of another filesystem like ext3 or ext4.

In order to be efficient, HDFS must satisfy the following prerequisites:

• Hard drives with a high throughput
• An underlying filesystem which supports the HDFS read and write pattern: one big read or write at a time (64MB, 128MB or 256MB)
• Network fast enough to cope with intermediate data transfer and block replication

HDFS is a Master / Slave architecture:

• The NameNode and the Secondary NameNode
  • Stores the filesystem meta informations (directory structure, names, attributes and file localization) and ensures that blocks are properly replicated in the cluster
  • It needs a lot of memory (memory bound)
• The DataNode
  • Manages the state of an HDFS node and interacts with its blocks
  • Needs a lot of I/O for processing and data transfer (I/O bound)

The critical components in this architecture are the NameNode and the Secondary NameNode.

These are two distinct but complementary architectures.

It is possible to not use HDFS with Hadoop. Amazon with their Elastic MapReduce for example rely on their own storage offer, S3 and a desktop tool like KarmaSphere Analyst embeds Hadoop with a local directory instead of HDFS.

Some important technicals facts to plan a cluster

No need to be an Hadoop expert but the following few facts are good to know when it comes to cluster planning.

How HDFS manages its files

HDFS is optimized for the storage of large files. You write the file once and access it many times.

In HDFS, a file is split into several blocks. Each block is asynchronously replicated in the cluster. Therefore, the client sends its files once and the cluster takes care of replicating its blocks in the background.

A block is a contigous area, a blob of data on the underlying filesystem, Its default size is 64MB but it can be extended to 128MB or even 256MB, depending on your needs.

The block replication, which has a default factor of 3, is useful for two reasons:

• Ensure data recovery after the failure of a node. Hard drives used for HDFS must be configured in JBOD, not RAID
• Increase the number of maps that can work on a bloc during a MapReduce job and therefore speedup processing

From a network standpoint, the bandwith is used at two moments:

• During the replication following a file write

• During the balancing of the replication factor when a node fails

How the NameNode manages the HDFS cluster

The NameNode manages the meta informations of the HDFS cluster.

This includes meta informations (filenames, directories, …) and the location of the blocks of a file.

The filesystem structure is entirely mapped into memory.

In order to have persistence over restarts, two files are also used:

• a fsimage file which contains the filesystem metadata
• the edits file which contains a list of modifications performed on the content of fsimage.

The trigger for this compaction process is configurable. It can be:

• The number of transactions performed on the cluster
• The size of the edits file
• The elapsed time since the last compaction

The following formula can be applied to know how much memory a NameNode needs:

<needed memory> = <total storage size in the cluster in MB> / <Size of a block in MB> / 1000000

In other words, a rule of thumb is to consider that a NameNode needs about 1GB / 1 million blocks.

How to determine my sizing needs?

Determine storage needs

Storage needs are split into three parts:

• Shared needs
• NameNode and Secondary NameNode specific needs
• DataNode specific needs

Shared needs

Shared needs are already known since it covers:

• The OS partition
• The OS logs partition

Those two partitions can be setup as usual.

NameNode and Secondary NameNode specific needs

The Secondary NameNode must be identical to the NameNode. Same hardware, same configuration.

A 1TB partition should be dedicated to  files written by both the NameNode and the Secondary NameNode. This is large enough so you won’t have to worry about disk space as the cluster grows.

If you want to be closer to the actual occupied size, you need to take into account the parameters of the NameNode  we explained above (a combination of the trigger for the compaction, the maximum fsimage size and the edits size) and to multiply this result by the number of checkpoints you want to be retained.

In any case, the NameNode must have an NFS mount point to a secured storage among its fsimage and edits directories.

This mount point has the same size than the local partition for fsimage and edits mentionned above.

The storage of  the NameNode and the Secondary NameNode is typically performed on RAID configuration.

DataNode specific needs

Hardware requirements for DataNodes storage is:

• SAS 6Gb/s controller configured in JBOD (Just a Bunch of Disk)
• SATA II  7200 rpm hard drives between 1Tb and 3Tb

Do not use RAID on a DataNode. HDFS provides its own replication mecanism.

The number of hard drive can vary depending on the  total desired storage capacity.

A good way to determine the latter is to start from the planned data input of the cluster.

It is also important to note that for every disk, 30% of its capacity is reserved to non HDFS use.

Let’s consider the following hypothesis:

•  Daily data input: 100Gb
• HDFS replication factor: 3
• Non HDFS reserved space per disk: 30%
• size of a disk: 3Tb

With these hypothesis, we are able to determine the storage needed and the number of DataNodes.

Therefore we have:

• Storage space used by daily data input : <daily data input> * <replication factor> = 300GB
• Size of a hard drive dedicated to HDFS : <Size of the hard drive > * (1 – <Non HDFS reserved space per disk>) = 2.1TB
• Number of DataNodes after 1 year (no monthly growth) : <storage space used by daily data input> * 365 / <HDFS reserved space in a disk> = 100TB / 2.1TB = 48 DataNodes

Two important elements are not included here:

• The monthly growth of the data input
• The ratio of data generated by jobs processing a data input

These informations depend on the needs of your business units and it must be taken into account in order to determine storage needs.

Determine your CPU needs

On both NameNode and Secondary NameNode, 4 physical cores running at 2Ghz will be enough.

For DataNodes, two elements help you to determine your CPU needs:

• The profile of the jobs that are going to run
• The number of jobs you want to run on each DataNode

Job profile

Roughly, we consider that a DataNode can perform two kind of jobs: I/O intensive and CPU intensive.

I/O intensive jobs

These jobs are I/O bound.

For example:

• indexing
• search
• clustering
• decompression
• data import/export

Here, a CPU running between 2Ghz and 2.4Ghz is enough.

CPU intensive jobs

These jobs are CPU bound.

For example:

• machine learning
• statistics
• semantic analysis
• language analysis

Here, a CPU running between 2.6Ghz and 3Ghz is enough.

The number of jobs

The number of physical cores determine the maximum number of jobs that can run in parallel on a DataNode. It is also important to keep in mind that there is a distribution between Map and Reduce tasks on DataNodes (typically 2/3 Maps and 1/3 Reduces).

To determine you needs, you can use the following formula:

(<number of physical cores> – 1) * 1.5 = <maximum number of tasks>

or, if you prefer to start from the number of tasks and adjust the number of cores according to it:

(<maximum number of tasks> / 1.5) + 1 = <number of physical cores>

The number 2 keeps 2 cores away for both the TaskTracker (MapReduce) and DataNode (HDFS) processes.

the number 1.5 indicates that a physical core, due to hyperthreading, might process more than one job at the same time.

Determine your memory needs

This is a two step process:

• Determine the memory of both NameNode and Secondary NameNode
• Determine the memory of DataNodes

In both cases, you should use DDR3 ECC memory.

Determine the memory of both NameNode and Secondary NameNode

As explained above the NameNode manages the HDFS cluster metadata in memory. The memory needed for the NameNode process and the memory needed for the OS must be added to it.

The Secondary NameNode must be identical to the NameNode.

Given these informations we have the following formula:

<Secondary NameNode memory> =  <NameNode memory> = <HDFS cluster management memory> + <2GB for the NameNode process> + <4GB for the OS>

Determine the memory of DataNodes

The memory needed for a DataNode is determined depending on the profile of jobs which will run on it.

For I/O bound jobs, between 2GB and 4GB per physical core.

For CPU bound jobs, between 6GB and  8GB per physical core.

In both cases, the following must be added:

• 2GB for the DataNode process which is in charge of managing HDFS blocks
• 2GB for the TaskTracker process which is in charge of managing running tasks on the node
• 4GB for the OS

Which leads to the following formulas:

<DataNode memory for I/O bound profile> = 4GB * <number of physical cores> + <2GB for the DataNode process> + <2GB for the TaskTracker process> + <4GB for the OS>

<DataNode memory for CPU bound profile> = 8GB * <number of physical cores> + <2GB for the DataNode process> + <2GB for the TaskTracker process> + <4GB for the OS>

Determine your network needs

The two reasons for which Hadoop generates the most network traffic are:

• The shuffle phase during which Map tasks outputs are sent to the Reducer tasks
• Maintaining the replication factor (when a file is added to the cluster or when a DataNode is lost)

In spite of it, network transfers in Hadoop follow an East/West pattern which means that even though orders come from the NameNode, most of the transfers are performed directly between DataNodes.

As long as these transfers do not cross the rack boundary, it is not a big issue and Hadoop does its best to perform only such transfers.

However, inter-rack transfers are sometimes needed, for example for the second replica of an HDFS block.

This is complex subject but as a rule of thumb, you should:

• Use a Spine Fabric network topology
  
  • Better throughput
  • Better resiliency to failures
• Avoid oversubscribtion of switches
  • A 1Gb 48 ports top of rack switch must have 5 ports at 10Gb on distribution switches
  • Avoids network slowdown for a cluster under heavy load (jobs + data input)
• If the cluster is I/O boundor if you plan to perform data input on recurrent data input which sature the 1GB and which cannot be performed outside of office hours, you should use:
  • 10Gb Ethernet intra rack
  • N x 10Gb Ethernet inter rack
• If the cluster is CPU bound, you should use:
  • 2 x 1Gb Ethernet intra rack
  • 10Gb Ethernet inter rack

We created an iPad app for Keytrade Bank, and we just launched it!, You can download it here.

Please contact us if you what to talk with us about this reference & our expertise!
 

Mixed feelings: happiness because it’s really a major achievement, nostalgia because we really loved building this app and now we are just slightly nervous as we hope users will love to consult their accounts, transfer money and trade using the Keytrade app as much as we enjoyed creating it.

For the banking part, this app allows one to customize accounts (rename accounts, assign pictures to each account), to consult the list of transactions, to transfer money and to visualize the monthly In & Out of an account with clear charts.

With respect to trading, our app offers a heatmap updated in real time with all the markets and all companies (when available from provider) and, of course, buying and selling stocks is possible from within the app. One can also consult the personal portfolio containing an overview of all trades.

To get a better idea of this project, here are some metrics:

• 8 months
• 2 Product Owners, one for Banking, one for Trading (from Keytrade Bank)
• 1 graphic designer (from Keytrade Bank)
• 1 UX designer
• 4,5 iOS developers (among them 1 from Keytrade Bank trained by OCTO) + 1 Scrum master
• 2 web-services developers (from Keytrade Bank)
• 21 Sprints
• 29 storyboards
• 1 user test session
• 302 User Stories & 2 300 complexity points
• 46 Releases of the app
• 3 languages
• 45 K lines of code (frameworks excluded)
• 17 K lines of comment
• 1 035 Unit Tests

Quality First

Quality has been the primary focus from the start.

For example, when a developer finished implementing a user story, another one had to validate the feature and to review the code. When peers validated the story, the product owner (PO) had to test the feature to ensure it matched what he had asked for.

To lower the numbers of bugs, we used the TDD (Test Driven Development) method. We tested the logic and the model of the app.

To change the state of a story from “in progress” to “done” we had to respect a checklist: this is called a Definition of Done (DOD). For example, we had to check if the code was clean, well documented and tested, if it didn’t add warnings etc.

We also had a continuous integration server (Jenkins). On each commit, an automatic build ran the unit tests and gave us metrics like number of lines of code, code coverage, code duplication. My colleague Cyril Picat described in this article how to add these metrics Jenkins for iOS.

With all these standards we dramatically decreased the number of bugs. However since the project is quite large, one may discover bugs from time to time that will be fixed in the next versions.

Each week, we released a new version of the app to Keytrade employees’ iPads. This process can be time consuming, that’s why we decided to automate it. So we just had to press a button and the users received a notification on their iPads to install the new version.

So, how did we do it?

It’s really simple, we created a new job on Jenkins, which build the app thanks to the Xcode plugin, and then we pushed it to Appaloosa, OCTO deployment solution available as SAAS.

UX Rules !

For us, User eXperience (UX) is really important. We believe a good UX changes everything. That’s why we had a UX designer during the whole project to challenge every usage, to create new patterns of banking and trading on tablets.

This was the first time that Keytrade worked with a UX designer. At first, people didn’t understand how and why creating user interfaces was complicated.

After a few months, we can see how UX improved our app, and how efficient it is to work with a UX designer during a project.

At the end of the project, we gave the opportunity to Keytrade’s real customers to test the app during a dedicated users testing session, before launching it on the Appstore. We received a lot of feedback, and of course, we worked one extra month to improve the app taking into account the remarks and recommendations!

Now, Keytrade Bank wants to hire a full time UX designer, … another proof that UX is not an option for quality apps !

The following images show us how UX helped us in our process.

This is an iteration process (Agile):

• First a UX designer drew the screen on a sheet of paper and in powerpoint
• Then the product owner validated the storyboard with the UX designer and added Business Rules on the powerpoint for the development team
• The graphic designer and the developers could start their work and iterated with the whole team.

Be Agile, be able to change everything, anytime.

At OCTO, we work using the Agile methodology. We believe Agile helps us to deliver better products that better fit the needs. It was the first project with Agile methodology for Keytrade.

At the project start, we had no idea of what we will create. We didn’t know how we should present data, what should be in the app, what shouldn’t.

The only thing we knew is that we wanted to change how people interacted in banking and trading on tablets. We wanted to be a reference, to be the #1 of the Appstore!

We created a lot of features, and we didn’t keep them all. We threw away a lot of concepts. We developed, designed, integrated the design, and fixed bugs on features that we didn’t ship.

Why ?

Because we wanted to give users the best we could deliver, a valuable product. We didn’t want to deliver useless features. Sometimes we deleted features; sometimes we rebuilt them from scratch.

It can be hard to accept, when you have worked for several hours and spent a lot of time and energy on a feature, but it’s important to go through with it for the quality of the final product.

Takeaways

To help us in our Agile approach, we had some tools. In particular we used Kanban Tools and Google Docs. These 2 tools were synchronized thanks to custom Google apps scripts. These scripts are really powerful when using Google Docs.

Our product backlog was a spreadsheet, which was hosted on Google docs. When we added or modified a task, it was automatically updated on our Kanban.

The most important point was the great collaboration with the Keytrade teams: the graphical designer, the back-end developers & the marketing team but also the business experts & the top management.
And first, to ease collaboration being in the same room helps a lot. We were in Keytrade premises 3 days a week. We were really close to the back office team, which is a good thing to quickly solve problems, and ease communication. This has been a strategic decision that saved us a lot of time.

Once a week, we demoed our progress to the Keytrade management team. This was important for us to have their feedback in order to know whether the product fitted their needs.

As a result, Keytrade wants to go further on Agile methodology and considers starting more Agile projects.

To conclude, our whole team Keytrade+OCTO is really proud of what we have created! This is probably one of the best apps OCTO have worked on. At this very moment, while writing this article, Keytrade Bank is the first app of the Appstore (in Belgium). And the score of the app is 4+ on the store.

You can have a quick overview of this with the following screenshots, but the best way, is to test it!

This is the End of the V1…. But just the beginning for this app!

This article is about testing private methods in android. This is a fairly common problem in android (even in Java at large) and can be solved easily. The technique proposed here provides the additionnal benefit of using a traditional way of solving the problem in the Java world. (suspense :) )

Using the android platform, you are used to divide your application into two projects :

• one for the main source code of your application,
• one for the tests

This is a mandatory structure on android, stongly emphasized by Google. Even, if you use the android-maven-plugin, you follow this pattern, and it is also the case if you use robolectric.

Moreover, it is a common practice to name the packages of your application using the following convention (inside the AndroidManifest.xml files of both projects) :

• foo.bar
• foo.bar.tests

Where foo.bar follows more or less your web domain name in the reverse order.

If you use the same package names to name the packages of your class, and this is a fairly common and intuitive practice, then you will face a big trouble : you will only be able to test your public methods : a test class inside a package named foo.bar.tests won’t be able to access the private and protected methods of a class in the package foo.bar. For instance :

In your app under test :

1
2
3
4
5

package foo.bar;
 
public class Foo {
    protected int m()  { return 2 };
}

In your test app :

1
2
3
4
5
6
7
8
9
10

package foo.bar.test;
 
import android.test.AndroidTestCase;
 
public class FooTest extends AndroidTestCase {
    public void testFoo() {
        //m() is not accessible from Foo
        assertEquals( 2, new Foo().m() );   //should not compile
    }
}

This code would generate a compile error. The method m() is only accessible to the classes that belong to the package foo.bar and the notion of subpackages doesn’t exist in the Java programming language.

Consequently you have several options.

1. Consider testing only public methods. Actually, this makes a lot of sense. Object-Oriented programming strongly emphasize the notion of contract and testing internals of a class can appear as violation of the notion of encapsulation.

Nevertheless, there are some cases where you really want to test some internal methods as they are complex, or just as they contain more of the logic of the class than public methods, or because you divided the different problems faced by public methods into some more specialized private methods (and this is a very good practice in IT, the famous divide and conquer principle).

2. Subclassing your class under test is an alternative : you can use protected methods in your class under test. And, in your tests, whenever you need to test a protected method, although you can’t invoke it directly, you can create a smaller inner class of the test class that extends the class under test and overrides every protected methods under test. This overridden method will just just invoke the super method, just to re-give it a package level visibility. The inner class protected methods will then be visible to the test class as they now belong to the same package.

Here is an example of this technique :

In your test app :

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

package foo.bar.test;
 
import android.test.AndroidTestCase;
 
public class FooTest extends AndroidTestCase {
    public void testFoo() {
        //m() is not accessible from Foo 
        //but it is from FooUnderTest
        assertEquals( 2, new FooUnderTest().m() );
    }
 
    private class FooUnderTest extends Foo {
        @Override
        protected int m() { super.m() };
     }
}

Although it is a common and accepted practice in Java to give a normally private method a protected visibility just to accomodate the case, purists are still under shock (and I belong to that camp), but that’s nevertheless a standard in Java. But this example demonstrates the drawback of this technique as well : it requires a huge, almost useless, amount of code, just a workaround and syntactic solution to a Java problem, but contains no interesting logic and is, indeed, a useless and thus painful programming effort. Nevertheless it could please purists as there is no need to alter the system under test only for testing purposes.

3. Using introspection is also possible. Reflexion allows to bypass the built-in visibility mechanism of Java. It allows to invoke any private / protected method on an object, or to use a private / protected field.

Here is an example :

In your test app :

1
2
3
4
5
6
7
8
9
10
11
12

package foo.bar.test;
 
import android.test.AndroidTestCase;
 
public class FooTest extends AndroidTestCase {
    public void testFoo() {
        Method method = targetClass.getDeclaredMethod("m", new Class[0]);
        method.setAccessible(true);
        int actual = method.invoke( new Foo(), new Object[0]);
        assertEquals( 2, actual );
    }
}

Although this leads to some much more cryptic syntax and make you loose the ability to refactor your code easily (as you will have to hard code method under test names in your tests), this technique is only one that can be used to test a really private method. It will please purists as there is no need to alter the system under test only for testing purposes.

4. Avoid naming your Java packages in the same way as you name you Application package :  in other words, there is absolutely no obligation to use the same name in your Java package and inside your AndroidManifest.xml.It was a bad idea from Google to call a “package” the app identifier inside an AndroidManifest file. This “package” is indeed an ID for your application inside google market for instance but it imposes no kind of constraints on Java packages inside your app and has indeed nothing to do with them.

1
2
3

 ...
 package="foo.bar.test"
 ...

1
2
3
4
5
6
7
8
9
10

package foo.bar; //<---That's the point !!
 
import android.test.AndroidTestCase;
 
public class FooTest extends AndroidTestCase{
    public void testFoo() {
        //m() is now accessible directly from Foo
        assertEquals( 2, new Foo().m() );
    }
}

During the presentation, we used the Monte Carlo method. It is, I must confess, an embarrassingly parallel algorithm. Perfect to demonstrate that parallelization can greatly improve an application performances.

The Monte Carlo method is usually explained by calculating Pi. It consists of calculating statistically something that we can’t calculate mathematically. Pi is in fact a bad example since it’s more efficient to calculate it using power series. I’ll let you read the Wikipedia article for a longer explanation but let just say that the idea is to randomly draw values and decide what they worth. The average worth value of all the draws will approximate the real answer. The more we draw, the closer we are from the real answer.

From where I stand, Pi just ain’t no fun. So I picked Batman instead. To calculate the area of the Batman logo to be precise. Much more amusing. I’ve based my calculation on the Batman equation” that was circling around Internet last year.

With a lot of minutia and helped by Wolfram Alpha, I’ve coded it in Java. Please note that the Wolfram Alpha equation is wrong. I’ll look at how to tell them without success. If someone knows how, please leave a comment. Here’s the fix:

The code is on Github. Visually, it looks like that:

batman

There’s also a Pi calculating version, but, since it’s just a circle, as I said, it ain’t no fun. But useful to test the Monte Carlo implementation. Back to Batman. Quite nice visually, but that was more useful in conference than here. What we want here is to compare the performances. For that, we use a 30 seconds execution of each. The goal is to perform as much draws as we can. I did three implementations:

• Sequential: A loop does on draw after the other
• Parallèle: A ForkJoinPool launch parallel loops on each available CPU
• GPU: Aparapi is used to calculate on the GPU

To give you an idea, here’s a snippet of the sequential code:

MonteCarloCalculator calculator = instantiateAlgorithm(constructor, 0);
new MonteCarloCmd(calculator).run();

Quite obvious. We call a calculator that loops. In parallel it’s a bit more complex but not much:

ForkJoinPool pool = new ForkJoinPool(); // pool creation
for (int i = 0; i < pool.getParallelism(); i++) {
   MonteCarloCalculator calculator = instantiateAlgorithm(constructor, i); // one calculator per CPU
   pool.execute(new MonteCarloCmd(calculator)); // we launch the parallel execution on the pool
}
try {
   latch.await(TIMEOUT, TimeUnit.SECONDS); // and wait for everything to be done
} catch (InterruptedException e) {
   throw new RuntimeException(e);
}
pool.shutdown(); // close the pool (that can be reused if needed)

The real code on GitHub is a bit more complicated because I’ve put in place a listener pattern to give some feedback while it’s running. This implementation is in fact penalizing the parallel algorithm since it requires data aggregation using atomic references and optimistic updates during the run. Than wouldn’t be necessary in a traditional implementation.

For the curious among you, here the Java version of the Batman equation:

// Wings bottom
if (pow(x, 2.0) / 49.0 + pow(y, 2.0) / 9.0 - 1.0 = 4.0 && -(3.0 * sqrt(33.0)) / 7.0 = 3.0 && -(3.0 * sqrt(33.0)) / 7.0 = 0) {
	return true;
}
// Tail
if (-3.0 = 0 && 3.0 / 4.0 = 0) {
	return true;
}
// Ears inside
if (1.0 / 2.0 = 0 && y >= 0) {
	return true;
}
// Chest
if (abs(x) = 0 && 9.0 / 4.0 - y >= 0) {
	return true;
}
// Shoulders
if (abs(x) >= 1.0
		&& y >= 0
		&& -(abs(x)) / 2.0 - 3.0 / 7.0 * sqrt(10.0) * sqrt(4.0 - pow(abs(x) - 1.0, 2.0)) - y + (6.0 * sqrt(10.0)) / 7.0
				+ 3.0 / 2.0 >= 0) {
	return true;
}
return false;

To compare the implementations, a benchmark have been done on a Quadruple Extra Large GPU cluster at Amazon. We had 16 CPU cores and 996 GPU cores. Here are the results for the 30 seconds execution:

• Sequential:179 786 000 draws
• Parallel: 709 731 000 draws
• GPU: 12 582 912 000 draws

In parallel, we get let improvement than I was hoping for. We surely can do a better job. As I said, the implementation is penalizing the parallel execution. And it’s not that terrible since it shows a nice demonstration of the Amdahl’s law. One thing to remember: The sequential version is using only one CPU while the parallel one is using them all. While keeping a relatively simple implementation.

On the GPU side, the simplicity effect is less obvious. The good news are that Aparapi is quite refreshing. The library is translating the kernel (the part that runs on the GPU) bytecode into an OpenCL implementation which is called through JNI. It’s developed by AMD. You can cook a matrix calculation example in about 10 minutes. However, doing a Monte Carlo is a bit more complicated. Here are some issues you might encounter.

• You can use only really simple JDK classes. For instance, java.util.Random just doesn’t work
• GPUs don’t like conditions (“if”). To get good performances, you have to modify your code into a linear computation
• For some reason, bitwise (&, |, ^) operations are silently not working

On top of that, you won’t get with Aparapi the performances as high as you would get with CUDA. But the gap is slowly closing. For instance, there are new annotations helping the memory management. Data localisation in memory is really important on a GPU. Still, you surely noticed that, without any optimisation effort, the GPU version is 17 times faster than the parallel one!

I’ll conclude like this:

• Sequential, you already know about it, it’s reassuring, no need to think too much… and it’s slow
• Parallel, not that complicated after all. A lot complexity hidden from the developer. Indeed, the GitHub code is in fact much more complicated than required
• GPU, I’ll say it’s refreshingly easier than 2 years ago, but we are not in the commodity realm yet. It’s not that hard, but you need to make sure you really need it before using it

Meanwhile, let Batman rise in parallel.