What are the common points? The main differences?

In the rest of the article, the term nosql will be used to talk about “transaction oriented” solutions like Cassandra, Riak, Voldemort, DynamoDB. On the other side, “Data Grid” will group solutions like Gigaspaces, Gemfire/SQLFire, Oracle Coherence.

Both Data Grids and NoSQL are talking about distributed storage

No surprise…this is a distributed vision of the “database” and of the storage which enable to support more throughputs, more volume under, generally, cost constraints. The main idea resides in the fact that instead of using high-end server to store and query data, you use several “low-end” servers to store more data, increase the throughput and being able to scale-out, address elasticity issues. Indeed, this article reminds us the unit cost of a transaction is approx.. 4 times cheaper on a cluster of low-end server than on a high end server, (the main drawback of distributed storage will be the bandwidth consumption that stays expensive).

So in both cases, the distributed vision of the storage implies:

• Partitioning the data based on a key. Then the key is associated to a server (or a specific buckets served by a server).
• Routing the queries to the server that holds the data…The main idea is to avoid asking to all the servers for a specific piece of data. So the storage system needs to route the query to the right server. There are different implementations of that routing. Solution like Cassandra will implement a server side routing, implying at least and almost systematically one network hop (except if you are lucky and hit the right server). Solution like Voldemort will use a client-side routing. “Data Grid” like Gemfire will also use a client-side routing, will learn from the cluster where the data is located (and thus avoid the network hops)
• All these distributed systems use replication mechanisms to ensure the availability and, in some cases, the durability (by mitigating the probability of losing a server before the data be replicated or by limiting the data corruption)

Data Grid and NoSQL comes from two distinct worlds: the latency oriented architecture or throughput architecture

I was reading again this article of Vogels. As he stated (talking about GPU and CPU):

“… the most fundamental abstraction trade-off has always been latency versus throughput. These trade-offs have even impacted the way the lowest level building blocks in our computer architectures have been designed. Modern CPUs strongly favor lower latency of operations with clock cycles in the nanoseconds ”

Even if it is quite difficult, should I say dangerous, to compare things, we can see the same trade-offs between NoSQL and data grid solutions. Data Grids come from a world where each milli-seconds (and now nano-seconds) count. These “data grids” have thus rapidly quit the disk and use the memory as main storage (even if it is configurable and we will discuss about that later).

On the other side, NoSQL solutions have been developed mainly for web-scale industries where the latency is not less important but let say around the second (because the end user is a human). In that case, you do not need to answer quicker and quicker, but you need to serve more and more requests.

NoSQL solutions have been developed to answer specific needs…whereas Data Grids are much more configurable: Bridging the gap

If you look at the history of these solutions you will, at 10 000 foot high view, see that NoSQL solutions are mainly clones of the Dynamo model that have been developed at Amazon to store session and virtual caddies.

The choices that were made fitted the Amazon.com needs : response time predictability, infrastructure elasticity and scaling out, multi-datacenter resiliency, design for failure…

“Data Grids” come from more heterogeneous (and thus richer) environments. You need to use them to relieve the RDBMS, to keep data in your local JVM, to scale…You need to integrate them with the historical part of your Information System, so you need SQL-like integration, you need java, .Net, ruby client APIs…

In short, data grids are clearly more configurable and so more adaptable than the NoSQL solutions. Without being exhaustive, we can think to:

• “Data Grids” enable you controlling the way data is stored. They all have default implementation (Gigaspaces offers RDBMS by default, Gemfire offers file and disk based storage by default….) but in all cases, you can choose the one that fits your needs: do you need to store data, do you need to relieve the existing databases….
• In order to minimize the latency, data grids enable you to store data synchronously (write-through) or asynchronously (write-behind) on disk. You can also define overflow strategies. In that case, data is store in memory up to a treshold where data is flushed on disk (following algorithms like LRU …). NoSQL solutions have not been designed to provide these features.
• Data grids enable you developing Event Driven Architecture. In some cases, an often-seen architecture could be a pure RDBMS plus a Message Oriented Middleware that is responsible to propagate the modifications (via events). Data Grids provide notification mechanisms for all or specified keys. Clients can so be notified on create / update / delete operations (like triggers)
• Querying is maybe the point on which pure NoSQL solutions and data grids are merging. Basically, both solutions provides pure map API which means you can put, get or delete an object by the key. Data grids already provide SQL queries (with limitations on joins) for a while but looking at the NoSQL market, you will see that these NoSQL solutions start providing secondary indexes that enable SQL-like queries, or MapReduce-like queries (with Brisk or DynamoDB that provides integration with Hadoop or Elastic MapReduce).
• Data grids enable near-cache topologies. The classical way of scaling readings with a RDBMS solution is to add a cache layer on top of the RDBMS (typically a memcached with difficulties like cache synchronization, cache partitioning to avoid collapsing the RDBMS in case of failure..). We do it so we know it works but data grids facilitates the deployment of “near cache topologies” – which means that data can be cached on the client side – ensuring the auto-eviction when the data is updated in the central storage, accelerating the readings…

Both systems have the same constraints especially when it comes to ACID

The funny thing is that both systems have the same constraints especially when it comes to ACID.

In fact, both solutions enables you to play with consistency and/or availability either using quorum based protocol or synchronous / asynchronous replication mechanisms. There is yet a subtility that can help data grids manage consistency (or at least conflict resolution): “data grid” use a master/slave topology per partition whereas nosql solutions use a master/master topology per partition.

Moreover, the “data grids” can offer you Atomicity and Transactions management BUT under certain constraints: typically, you will have to override the partitioning mechanism and ensure data colocation, you will not be able to rebalance the cluster while transactions are pending…These features come at a cost on the operability of the solution. This may not be blocking depending on the use cases but NoSQL solutions like Dynamo have made their choices: operability, elasticity with impacts on the dev side.

NoSQL offer a storage vision whereas data grids offer more cases

On that side, the match is not that simple.
Data Grids offer much more complicated deployment options. You can use them in a classical client / server architecture: the data is stored on dedicated JVM whereas the business processes have their own dedicated JVM. In that case, the data grid is seen as a pure storage layer. That case is finally quite close of the NoSQL solutions.

What data grids offer (and not NoSQL solutions) is a peer-to-peer deployment where business processes and data are deployed in the same JVM. What changes is that the unit of deployment is the sum of the business process and you data. From the distributed storage point of view, there are no real advantages if you use your data grid as storage: data will be partitioned across the cluster, based on the key. This is yet quite different if your business cases necessitate writing and reading a lot of local data that is specific to your business case and must not be shared with the other business processes. In that case, you will benefit from this kind of deployment (gains in latency and bandwidth consumption).

The main drawback of data grids is that they need to know the java object (typically in the classpath) and in the best case, you can choose between default java serialization (Serializable or Externalizable) or the specific serialization. That can complicate upgrading the model, adding indexes…without downtime…
On the opposite (and it solves the previous issues), NoSQL solutions work with a byte array: object versioning, serialization and unserialization must so be “manually” managed (even if these solutions often use protocols like Thrift, Avro…).

So what? what to conclude?

I am sure I forgot some points but in short that both systems are under the same constraints. The clear points is that “data grids”, due to their own story are certainly more adaptable.
That does not mean you must not look at NoSQL solutions because they can fit your requirements and these solutions work (see the works of Netflix with Cassandra)
More generally, sometimes I am asking myself: why will we move to distributed storage (if we move to)? May be the following elements:

• the variability of workload and the needed elasticity
• (maybe) the cost of this kind of infrastructure compared to traditional RDBMS infrastructure. I said maybe because I am convinced the economics equation must be explicitly posed.
• the resiliency of the systems (and the implied cost pressure) : we now need to address higher level of failure for cheaper (because the cost of mitigate the failure must not be higher than the cost of failure itself)
• the trivialization of infrastructure which will imply having less server diversity to serve more and more different use cases
• the platform vision that will push forward to develop multi-tenant architecture

Suggestion d'articles :

1. no:sql(eu) and NoSQL: What’s the deal?
2. Event Sourcing & noSQL

RestKit can be seen as yet another library to do it. However, its approach is radically different since it does not only address asynchronous networking but also object mapping and a seamless persistance of such mappings locally through CoreData.

This approach is quite interesting when it comes to interact with Restful web services because of the time (and code!) you can save.

In this article, we are going to look at RestKit’s main functionalities through the code to do it. The objective is to give you a precise idea of what can be done and how to do it.

Network communications through RKClient

In RestKit, network communications are managed by the RKClient class. This class encapsulates details on the HTTP level and provides control to the developer over it by simply editing properties.

RKClient is where you can find the functionalities of ASIHttpRequest, AFNetworking or MKNetworkKit.

From RestKit standpoint, it is an intermediate layer as we will see later in this article.

RKClient *client = [RKClient clientWithBaseURL:@"http://api.domain.com/v1"];
[RKClient setSharedClient:client];
NSLog(@"This is my default client: %@.", [RKClient sharedClient]);
if ([[RKClient sharedClient] isNetworkReachable]){
    // Do some work...
}

This code creates a new RKClient instance and set it as the default one. You can create as many instances as you want. Each instance is tied to a specific base URL which serves as the prefix to all network communications.
RKClient exposes several properties to configure HTTP communications.
Here is a review of the most common use case you might have.

Customize HTTP Headers

NSLog(@"HTTP headers added: %@.", [RKClient sharedClient].HTTPHeaders);
[[RKClient sharedClient].HTTPHeaders setObject:@"MyAppSpecialHeader" forKey:@"X-MyApp-Special-Header"];
[[RKClient sharedClient].HTTPHeaders removeObjectForKey:@"X-Old-Key"];

This allows you to control custom HTTP headers you want to use with your requests through this client.

SSL communications

RKClient *sslClient = [RKClient clientWithBaseURL:@"https://api.domain.com/v1"];
sslClient.disableCertificateValidation = YES;
SecCertificateRef myCustomRootCertRef = SecCertificateCreateWithData(NULL, certData);
[sslClient addRootCertificate:myCustomRootCertRef];
NSLog(@"NSSet of my additional root certs: %@.", sslClient.additionalRootCertificates);

As shown in the example, you can disable SSL certificate validation and add your own root certificates for a given RKClient.

Request queue and Cache policy control

RKClient is asynchronous, its concurrency level is customizable and requests can suspended in which case they are enqueued until the queuing limit is reached.

[RKClient sharedClient].requestQueue.concurrentRequestsLimit = 2;
[RKClient sharedClient].requestQueue.requestTimeout = 10;
[RKClient sharedClient].suspended  = YES;
NSLog(@"Number of requests in the queue: %d. Total number of request currently loading: %d.",
[RKClient sharedClient].loadingCount, [RKClient sharedClient].count);

It is also possible to access and control the cache of the RKClient instance.

Several policies such as use the cache only when offline or use the cache when the request timeouts as builtin. The developer can dynamically change the caching policy and also has a direct access to the cache implementation so she can invalidate entries, add others, …

[RKClient sharedClient].cachePolicy = RKRequestCachePolicyLoadIfOffline|RKRequestCachePolicyTimeout;
[RKClient sharedClient].requestCache.storagePolicy = RKRequestCacheStoragePolicyPermanently;

Builtin support for authentication schemes

RestKit natively supports 4 authentication schemes:

• HTTP Auth
• HTTP Basic
• OAuth1
• OAuth2

This is configured through properties of RKClient.
For example, to configure OAuth2 you would do the following.

[RKClient sharedClient].authenticationType = RKRequestAuthenticationTypeOAuth2;
[RKClient sharedClient].OAuth2AccessToken = @"YourOAuth2AccessTokenHere";
[RKClient sharedClient].OAuth2RefreshToken = @"YourOAuth2RefreshTokenHere";

Once configured this way, all network communications will use the credentials you specified.
Other authentication schemes are configured the same way. Each have its own set of properties in RKClient.

A real life example with RKClient

Now let’s consider a real life example. Let’s say that we want to retrieve an RSS feed of today from a website which requires an HTTP Basic authentication.
We don’t want to cache anything and we also don’t want to wait for more than 10 seconds to retrieve the feed.
The code to do it would look like this:

#import <Foundation/Foundation.h>
#import <RestKit/RestKit.h>

@interface MyClass : NSObject<RKRequestDelegate>

- (void)retrieveFeed;

@end

@implementation MyClass

- (void)retrieveFeed
{
  RKClient *client = [RKClient clientWithBaseURL:@"http://rss.domain.com"];
  client.requestQueue.requestTimeout = 10;
  client.cachePolicy = RKRequestCachePolicyNone;
  client.authenticationType = RKRequestAuthenticationTypeHTTPBasic;
  client.username = @"rssuser";
  client.password = @"rsspwd";
  NSDictionary *queryParameters = [NSDictionary dictionaryWithObjectsAndKeys:@"today", @"date", nil];
  NSString *getResourcePath = RKPathAppendQueryParams(@"/feed.xml", queryParameters);
  [client get:getResourcePath delegate:self];
}

- (void)request:(RKRequest *)request didLoadResponse:(RKResponse *)response
{
  if (request.method == RKRequestMethodGET) {
        id xmlParser = [[RKParserRegistry sharedRegistry] parserForMIMEType:RKMIMETypeXML];
        NSError *error = nil;
        id parsedResponse = [xmlParser objectFromString:[response bodyAsString] error:&error];
        if (error == nil)
            NSLog(@"GET:/user returned with HTTP Code %d and parsedContent: %@", [response statusCode], parsedResponse);
  }
}

- (void)request:(RKRequest *)request didFailLoadWithError:(NSError *)error
{
  if (request.method == RKRequestMethodGET) {
    NSLog(@"Failure of GET:/user with error %@.", error);
  }
}
@end

As you have probably noticed, a delegate is provided to handle the asynchronous results. RestKit relies a lot on delegate both internally and for its public interfaces. 

Another interesting point is how we parsed the response of the feed which is expected to be XML content.

Indeed, RestKit provides a parser registry that you can use to get an instance of a parser for one of these four mimetypes:

RKObjectManager *objectManager = [RKObjectManager objectManagerWithBaseURL:@"http://api.domain.com/v1"];
RKObjectManager *sslObjectManager = [RKObjectManager objecxtManagerWithBaseURL:@"https://api.domain.com/v1"];

// Initialize the managed object store because both will have Core Data managed objects.
objectManager.objectStore = [RKManagedObjectStore objectStoreWithStoreFilename:@"api.sql"];
sslObjectManager.objectStore = [RKManagedObjectStore objectStoreWithStoreFilename:@"apissl.sql"]; 

// We want the SSL Object Manager to be our sharedManager.
[RKObjectManager setSharedManager:sslObjectManager];

RKManagedObjectMapping *myMapping = [RKManagedObjectMapping mappingForClass:[MyClass class]];

// Map attributes
[myMapping mapAttributes:@"id", @"name", nil];

// These attributes have different name in objc and JSON.
[myMapping mapKeyPathsToAttributes:@"expirationTime", @"expiration_time", nil];

// Since it is a Core Data managed class, we can indicate to RestKit which property is the primary key (optional).
[myMapping setPrimaryKeyAttribute:@"id"];

// Set relationships. Previously mapped classes are used here.
[myMapping hasOne:@"user" withMapping:[[RKObjectManager sharedManager].mappingProvider objectMappingForKeyPath:@"userMapping"]];
[myMapping hasMany:@"sessions" withMapping:[[RKObjectManager sharedManager].mappingProvider objectMappingForKeyPath:@"sessionMapping"]];

// Set both the mapping and the serialization mapping.
[[RKObjectManager sharedManager].mappingProvider setMapping:myMapping forKeyPath:@"tokenMapping"];
[[RKObjectManager sharedManager].mappingProvider setSerializationMapping:[myMapping inverseMapping]  forClass:[MyClass class]];

// Register two routes for this class, a GET and a POST.
[[RKObjectManager sharedManager].router routeClass:[MyClass class] toResourcePath:@"/api/foo" forMethod:RKRequestMethodGET];

// The POST route has a dynamic parameter which is the objc property name prefixed by a semicolon.
[[RKObjectManager sharedManager].router routeClass:[MyClass class] toResourcePath:@"/api/foo/:id" forMethod:RKRequestMethodPOST];

// Our object is managed through CoreData. RestKit takes care of updating it.
@interface MyClass : NSManagedObject

@property (retain, nonatomic) NSInteger id;
@property (retain, nonatomic) NSString *name;

@end

@implementation MyClass

@dynamic id, name;

@end

@interface MyController : UIViewController<RKObjectLoaderDelegate>

- (void)loadData;

@end

@implementation MyController

- (void)loadData
{
  // Retrieve an instance stored in CoreData. RestKit provides a category to encapsulate Core Data interactions.
  MyClass *obj = [MyClass findFirstByAttribute:@"id" withValue:@"42"];
  obj.name = @"A new name to set remotely";

  // Perform the post request.
  [MyClass postObject:obj delegate:self];
}

- (void)objectLoader:(RKObjectLoader*)objectLoader didLoadObject:(id)object
{
  NSLog(@"Request succeed and  response loaded into CoreData: %@.", object);
}

- (void)objectLoader:(RKObjectLoader*)objectLoader didFailWithError:(NSError*)error
{
  NSLog(@"An error occurred: %@.", error);
}
@end

RKObjectMapping *userMapping = [[RKObjectManager sharedManager].mappingProvider objectMappingForKeyPath:@"sessionMapping"];

[[RKObjectManager sharedManager] loadObjectsAtResourcePath:@"/api/foo" objectMapping:userMapping delegate:self];

myMapping.setDefaultValueForMissingAttributes = YES;
myMapping.setNilForMissingRelationships = YES;

{ "users":
  {
    "foo": { "id": 1234, "email": "foo@domain.com" },
    "bar": { "id": 5678", "email": "bar@domain.com" }
  }
}

RKObjectMapping *myMapping = [RKObjectMapping mappingForClass:[User class]];
// RestKit cannot infer this is a collection, so we force it
myMapping.forceCollectionMapping = YES;
// Map our attributes.
[myMapping mapKeyOfNestedDictionaryToAttribute:@"firstName"];
[myMapping mapFromKeyPath:@"(firstName).id" toAttribute:"userID"];
[myMapping mapFromKeyPath:@"(firstName).email" toAttribute:"email"];

[[RKObjectManager sharedManager].mappingProvider setObjectMapping:myMapping forKeyPath:@"users"];

myMapping.preferredDateFormatter = [[[NSDateFormatter alloc] init] autorelease];
myMapping.dateFormatters  = [NSArray arrayWithObjects:[[[NSDateFormatter alloc] init] autorelease], [[[NSDateFormatter alloc] init] autorelease], [[[NSDateFormatter alloc] init] autorelease], nil];

// Set as the application wide default set of formatters.
[RKObjectMapping setDefaultDateFormatters:myMapping.dateFormatters];

RKManagedObjectMapping *myMapping = [RKManagedObjectMapping mappingForClass:[CoreDataManagerObject class]];
myMapping.primaryKeyAttribute = @"id";

RKObjectManager *objectManager = [RKObjectManager objectManagerWithBaseURL:@"http://api.domain.com/api"];

// ... Initializing mappings.
NSArray *objectsInStore = [MyObject allObjects];
if (objectsInStore count] == 0) {
  NSString *storeFilename = objectManager.objectStore.storeFilename;
  NSString *pathToSeedDB = [[NSBundle mainBundle] pathForResource:@"MySeedDB" ofType:@".db"];
  RKManagedObjectStore *objectStore = objectStoreWithStoreFilename:storeFilename usingSeedDatabaseName:pathToSeedDB managedObjectModel:nil delegate:self];
  objectManager.objectStore = objectStore;
}

@interface MyAppDelegate : UIResponder <RKManagedObjectStoreDelegate>
@end

@implementation MyAppDelegate

- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
  RKObjectManager *objectManager = [RKObjectManager   objectManagerWithBaseURL:@"http://api.domain.com/api"];
  objectManager.objectStore.delegate = self;
}

// Delegates

- (void)managedObjectStore:(RKManagedObjectStore *)objectStore didFailToCreatePersistentStoreCoordinatorWithError:(NSError *)error
{
  NSLog(@"Fail to create a persistent store. Error: %@", error);
}

- (void)managedObjectStore:(RKManagedObjectStore *)objectStore didFailToDeletePersistentStore:(NSString *)pathToStoreFile error:(NSError *)error;
{
  NSLog(@"Fail to delete a persistent store at path %@. Error: %@", pathToStoreFile, error);
}

- (void)managedObjectStore:(RKManagedObjectStore *)objectStore didFailToCopySeedDatabase:(NSString *)seedDatabase error:(NSError *)error
{
  NSLog(@"Fail to copy seed database: %@. Error: %@", seedDatabase, error);
}

- (void)managedObjectStore:(RKManagedObjectStore *)objectStore didFailToSaveContext:(NSManagedObjectContext *)context error:(NSError *)error exception:(NSException *)exception
{
  NSLog(@"Fail to save context: %@. Error: %@ (%@)", context, error, exception);
}
@end

- (void)fireASynchronousRequest
{
// From the original implementation.

[self prepareURLRequest];
NSString* body = [[NSString alloc] initWithData:[_URLRequest HTTPBody] encoding:NSUTF8StringEncoding];
NSLog(@"Sending %@ request to URL %@. HTTP Body: %@", [self HTTPMethod], [[self URL] absoluteString], body);
[body release]; 

_isLoading = YES;
if ([self.delegate respondsToSelector:@selector(requestDidStartLoad:)])
[self.delegate requestDidStartLoad:self]; 

RKResponse* response = [[[RKResponse alloc] initWithRequest:self] autorelease];
[[NSNotificationCenter defaultCenter] postNotificationName:RKRequestSentNotification object:self userInfo:nil];

// Now execute the mocking code.
NSString *bundlePath = nil;
NSInteger statusCode = 200;

id requestJsonObject = nil;
if (self.params != nil) {

  NSString *jsonString = [[[NSString alloc] initWithData:[self.params HTTPBody] encoding:NSUTF8StringEncoding] autorelease];

  // If you use the latest version of RestKit with Object Mapper 2.0:
  id parser = [[RKParserRegistry sharedRegistry] parserForMIMEType:RKMIMETypeJSON];
  NSAssert1(parser, @"Cannot perform object load without a parser for MIME Type '%@'", RKMIMETypeJSON);
  NSError **error = nil;
  requestJsonObject = [parser objectFromString:jsonString error:error];
}

// Switch to determine which to to use
…

// Prepare the results.
NSData *responseData = nil;
if (bundlePath == nil)
responseData = [NSData data];
else
{
// Load the mock file.
NSString *responseString = [NSString stringWithContentsOfFile:bundlePath encoding:NSUTF8StringEncoding error:nil];
responseData = [responseString dataUsingEncoding:NSUTF8StringEncoding];
}

// Send the response. This is a superclass to be able to return a custom statusCode.
MyHTTPURLResponse *urlResponse = [[[MyHTTPURLResponse alloc] initWithURL:_URL MIMEType:RKMIMETypeJSON expectedContentLength:[responseData length] textEncodingName:nil] autorelease];
[urlResponse setStatusCode:statusCode];

// Delegates in the order RestKit is expecting it.
[response connection:nil didReceiveResponse:urlResponse];
[response connection:nil didReceiveData:responseData];
if ([response isError] == YES)
[response connection:nil didFailWithError:[NSError errorWithDomain:@"test domain" code:statusCode userInfo:nil]];
else
[response connectionDidFinishLoading:nil];
}

Suggestion d'articles :

1. Hadoop Distributed File System : Overview & Configuration
2. How to rescue your data, 3/3
3. How to rescue your data, 1/3

Why untar?

Network connections cause latencies on your mobile app. Replace 10 downloads of 1Mb by 1 download of 10Mb is a good improvement to your app.

There’s a well-known unix tool for that: tar.

What is tar? Wikipedia says: “Tar is now commonly used to collect many files into one larger file for distribution or archiving, while preserving file system information such as user and group permissions, dates, and directory structures.” That’s exactly what we need with some extra features.

How to untar an archive file on your iOS device?

How to untar?

A goolge search about untar on iOS provide two solutions:

• BSD libarchive. This lib is already in your iOS device but without header files. Apple calls that a “private API”. You can’t use it in App Store. If you are doing an app for jailbroken device there isn’t any problem but I need to push my app to the App Store.
• Davepeck BSD libarchive implementation https://github.com/davepeck/iOS-libarchive. The code of libarchive is open source so davepeck has packaged this code for iOS and you can add it to your project. This libarchive implementation is the same than the tar command on your Mac. You can create or extract tar file and provide gzip or bzip compression. But this lib includes 617 files, 347 606 lines of code (without any Objective-C wrapper that you have to code by yourself) and weights 4.7MB when build. Just remember that you need wifi to download app heavier than 20MB.

Why not using libarchive?

With libarchive:

• We add code dependencies to our project. These dependencies need to be updated.
• More lines of code means more bugs. According to Steve McConnell’s “Code Complete”, there are on average between 15 and 50 bugs per KLOC (Kilo line of code), so if I take a calculator and make rocket science estimation, there are 5000 bugs in libarchive.
• If Apple changes an API and break libarchive compatibility, your code will not stay reliable.

Is there another way? What if I must push my app in the app store but don’t want to link this 4.7MB of hardly-maintainable code? Let’s code my own light and pragmatic untar implementation.

What do we really need?

Libarchive weights 4.7MB but do a lot of things we don’t really need.

We need to untar files but we don’t need:

• To create tar files.
• To uncompress files (gzip or bzip). If we want to compress our file we can use zlib and then untar. We don’t have to untar during inflate.
• To work with old fashion tar (from the 80’s).
• To work with unix ownership and right: iOS will not let us set this rights even if we wanted.
• To work with simlinks, hard links, FIFO or other special stuffs. We just want to handle files.

The solution

What’s a tar file?

A tar file is composed of 512 bytes data blocks.

The first block is a header for the first file; the next n blocks are the content of this file and then a new file start with a new header block.

A tar file looks like that: (H is header block, C content block, / partial content block)

At the end of the tar file there is two empty blocks. A new file starts always after an integer number of blocks. Some files haven’t any content block (like folder). Each block has 512 bytes.

A header block comports a lot of datas. Few of these data, and none of the new UStar header data are useful for us.

We just need:

• The file name: Encoded in ASCII, it’s a relative path like “mydir/myfile.pdf”
• The file size: Encoded in ASCII, it’s an octal value in bytes.
• The file type: we only need two file types, ‘0’ for regular files and ‘5’ for directories

With the file size we know when the header block of the next file starts. And where to strip the content of the last block of the file.

The result: Light-Untar for iOS

Not so complex isn’t it? I just implemented this code in Objective-C and push it to github: https://github.com/mhausherr/Light-Untar-for-iOS

This code is under BSD license; you can use it for your own project.

Is it better than libarchive? Light-Untar-for-iOS has only 2 files and 168 lines of code, which include the comment lines with the license text.

To use it, include the .h file and use this NSFileManager method:

[[NSFileManager defaultManager] createFilesAndDirectoriesAtPath:@"/path/to/your/extracted/files/" withTarData:tarData error:&error];

tarData is an NSData. You can create the NSData with this code:

NSData *tarData = [NSData dataWithContentsOfFile:@"/path/to/your/tar/file.tar"];

Limitations about Performances and Security

Performances

Libarchive  is written directly in C and does the job faster than my Objective-C code.

The question is: what’s the most important thing for you?

• To win 40ms on each untar?
• To save 4.7Mb of the size of your app?

If you choose the second one, Light-Untar-for-iOS is for you.

Security

What are major security issues of tar?

• Symbolic links: Someone can add links to give access to folders outside the working directory.
• Set root ownership or add execution right: Someone can add an executable file with root access and launch it.

Neither of these are implemented in my code, so you can use it safely.

And you know what? It’s an iOS lib. On iOS, applications are sandboxed. The system himself protects this code. We don’t need any other protection.

Conclusion

Mobile development is primarily development on low capacity devices. Size of the build app is also important.

If you have a simple problem to solve, ask you what is the best: A huge framework can do that for you. Do you need really all this framework? Can’t you just implement the requested feature?

To go further

• http://www.gnu.org/s/tar/manual/html_node/Standard.html
• http://en.wikipedia.org/wiki/Tar_(file_format)
• http://www.amazon.com/Code-Complete-Practical-Handbook-Construction/dp/0735619670

One deployment tool to rule them all

Work hard on getting a nice deployment tool (scripts, capistrano, whatever) and use it to deploy on all your environments from development to UAT, perf and of course production. You’ll make sure you’ll apply the fail fast principle to deployment. Make sure it pulls all the artifacts it needs from your repo, Nexus or such.

If deployment is painfull do it more often, if it’s fun, do it more often

Deployment must not be considered as a failure nor as an exception. It’s fine, it’s a standard and a frequent part of the job, it shows you have worked to bring out new features, improved performances, bugfixes. Don’t be scared to deploy often, « practice makes perfect ». The more often you deploy, the smaller the changes are, the lesser the risks.

Monitoring is for everyone

From technical (sysadmins, developers, DBA, HelpDesk) to highly non-technical people (marketing, financial, top management), every one loves to see the heartbeat of the system. Propose several screens / dashboards to present the data to each perspective.

Commit developers in every steps of the production life, and give them feedback, loads of it

Make sure you can wake them up during a tricky update or even better, grab some extra pizza to make sure they stay during the upgrade process.

Give access to all data they need to investigate, either in real-time on the production environment or at least by giving access to data / logs with as few delay as possible

Use production-like database content, A.S.A.P.

Real-life data often contain  stranger stuff than you could ever imagine, use them. Anonymize them (yes, you really want to avoid emailing real users during your tests), reduce their size if needed, but do keep historical data. They will help with UAT, perf testing, and will secure your production roll-out.

Share tools and practices with Devs

Kanbans, scrums, retrospectives, Jira, Confluence are probably used into your dev team, ritualize such principles on the ops side, you’ll share the same logic and delivery rhythm.

Express needs, act as a P.O. when needed

If you need something from the Devs that will help you make the run better / cheaper / faster, such as a better log classification, functionnal metrics to monitor, become a Product  Owner, propose user stories and make sure they take life. Negociate with the business to get a minimal amount of complexity points for ops user stories at each iteration.

Automatize everything that can be (especially boring stuff)

Script, script, script, keep your mind free from silly or time-consuming stuff, you have better things to do.

Manage your systems automatically

Use Chef, Puppet, cfEngine + a VCS (git, mercurial, SVN…) to store and historize your configurations.

Embrace the Test-Driven Infrastructure Approach

Try to make every change on your infrastructure testable, use the monitoring tool as a an automatic tool for it.

Elect a goalie to keep other ops from task-switching

This will ensure you that small tasks are handled as soon as possible and keep people working on long-term tasks focussed.

One of the biggest Java conference in the world took place in Antwerpen, Belgium on last November. This year, Devoxx main themes were (in no particular order)

• The future of Java
• Alternative languages on the JVM
• HTML5
• JavaFX
• Android
• A bit of Cloud, NoSQL and high performance architecture

We also had a great announcement about a new conference coming in 2012: Devoxx France!

Obviously, OCTO was there. In this article we won’t cover the sessions in details. A lot of blogs already did it. We’ll however summarize this edition’s main trends and give our impressions.

Java

Java SE & EE

All announcements from Oracle about Java were already made at JavaOne, but the two opening keynotes gave us a good reminder and here are the key points:

• Java SE 7 is out since last July and brings several new functionalities.
• Java SE 8 is planned for 2013 and will include mainly Jigsaw (Java modularity) and Lambda (closures in Java) projects.
• Java EE 7 is planned for Q3 2012 and will be cloud oriented. Standard metadata will be provided for easing provisioning, multi-tenancy support, elasticity, etc. We should see an emerging standard for solutions such as Cloud Foundry from vmWare or OpenShift from RedHat.
• Other standards should evolve, with by example a 2.0 version for JMS, which had no new version since 2003.

Mark Reinhold gave us a talk about Project Jigsaw, which aims to allow Java applications and the JDK itself to be natively modular. For example,  we should be able to deploy Java applications as RPM package. We will also be able to take only small parts of the JDK for specific applications. This project is interesting, but still  frightening on one point : they don’t seem to be really close from projects like Maven which already had to deal with many modularity and dependencies problems for years : can we hope they won’t fall in the same traps as Maven in the beginning?

Another important point is the great emphasis made on JavaFX 2, which is now the main GUI framework for thick clients and mobile. Therefore, no more investments will be done on Swing, even if Oracle will continue to support it. A lot of sessions were about Java FX, and it was quite conclusive. Let’s hope it won’t became the same flop as Java FX 1.

We also had two great retrospectives about Java by Joshua Bloch, who made a really in depth analysis of pretty much all the original JDK classes! Coming from the guy who gave us the generics, it’s a quite interesting talk. Watch it on Parleys.com when it will become available, you could discover some tricky details about Java.

The Oracle guys also tried to give us some messages:

• They (Oracle, but also the JCP) take care of Java, and developers shouldn’t worry about its future.
• Oracle’s behavior towards Java is all about transparency. First of all, Open JDK is now the reference implementation of Java. But the most important point is the new JCP 2.8 process, which made all JCP discussions public, and the arrival of new members such as Twitter, and London JUG and SouJUG (Brazil) in executive committees. We can hope that it will allow a lighter mood than last year (with Apache’s resignation).

Talking about mood, a good moment to feel opinions differ was the debate we watched on the last day. It involved Mark Reinhold and Brian Goetz from Oracle, Joshua Bloch from Google, Emmanuel Bernard from JBoss, Ben Evans from London JUG and Brian Prince from Microsoft. A lot of questions were really controversial and although most answers were quite evasive, we had a glimpse about the cold war between these companies :

First, it’s always interesting to see different companies’ philosophies, from Oracle, very corporate, Google, open minded with a startup spirit, to JBoss, with a full open-source model. It’s also really unfortunate to see how bad the relations between Oracle and Google are, and to see Oracle guys acting as Android simply doesn’t exist even if it is now the most prevalent mobile OS.

Apart from Java EE and Java SE, some other hot topics were addressed :

Java on the Cloud

While Java EE 7 should bring standards for a Java PaaS platform next year, these platforms are already here today, and seem really promising: Cloud Foundry from Spring Source and OpenShift from Redhat/JBoss.

Cloud Foundry is available in beta since april 2011. It provides an open PaaS platform under Apache license and offering multi languages support : Java, Ruby, NodeJS … Cloud Foundry can be used either on CloudFoundry.com or any other commercial provider, either on your private cloud, or locally for development purpose with Micro Cloud Foundry. It provides also integration with several third party technologies, and can be fully integrated with Spring Framework.

OpenShift is a promising concurrent to Cloud Foundry, using JBoss and Redhat products. We had a great demo, especially on how to manage applications with JBoss Forge and Git. Some differences from Cloud Foundry are that it is for now restricted to Java, not open sourced yet and that you can’t instantiate it in a private cloud.

A new generation of application servers

They made really great efforts in the new version of JBoss Application Server 7. We tested it, and it’s just amazing: fast start (1,7s), low memory usage, fast deploy and undeploy, easy to configure, and testable.

It sounds like other application servers such as Websphere or Weblogic are now far far behind: using this open source server for plain Java EE applications should be considered more than ever.

Other languages on the JVM 

Most of you already heard about Scala or Groovy. We will now have to count with Clojure, Ceylon, Kotlin and Fantom which are all new languages on the JVM. Why in the world is everyone creating a language? It seems that everyone now agrees that the JVM is an incredible piece of software but that Java still has space for improvement.

All of them are trying to solve something. Pick you favorite: less line of code, more modularity, having functional programming, make it easy to do parallel programming… My personal guess is that we haven’t found the real “Java killer” yet. But all these great language experiments will certainly give good ideas to its inventor. Meanwhile, these languages are to consider. They might help you on your current project. We had the “silver bullet framework”, now it’s time for the “silver bullet language”. Use them wisely.

Java & NoSQL

This year, we saw 2 emerging projects with quite the same goal, but different approaches: Spring Data and Hibernate OGM.

Both of them try to give an abstraction layer on top of NoSQL databases for Java application, particularly for the really tough task which is data denormalization (Chris Anderson showed used really well how hard this point can be).

Where Spring Data started a different project for each type of NoSQL database, Hibernate OGM tries to apply JPA concepts on top of NoSQL databases, starting with Infinispan (an in memory datagrid solution). This approach is quite ambitious but they take the problem step by step: for now, they only handle entity modeling and simple JPQL queries.

Web and HTML5

HTML5 is a really hot topic, and we saw some very appealing demonstrations. While HTML5 isn’t fully standardized yet, most web browsers already support it, except from Internet Explorer.

With HTML5, we can now easily do web offline, game programming, use a webcam and microphone, do voice recognition, but we also have many simplifications in HTML tags.

Google is pushing hard on it. For instance, Paul Kinlan (from Google) gave us a great demo around webkit and Chrome HTML5 features.

Linked to that, everyone is also trying the make it easier to code in the browser. It could be the old way, but easier (jQuery) and the new way (Dart… yes, even Javascript just got competition).

Mobile & Android

As we were five Java-ists we haven’t attended much sessions around mobile, but the Android keynote by Google was clear: the future is mobile. Everything is moving to the mobile so you, developers, should start to move to where the business will be. And you should hire a designer for you GUI…

Conclusion

We are living in interesting time my friends. Java and its developers took a five years vacation (healing from the generics?) but now they’re back. Jump in the train and keep learning. Because they are moving, they are moving fast.

In addition, the value of information may decay rapidly over time. For instance, some events that may help anticipate a production outage have no value once the outage happened. There is a need to process data as soon as events occur, with respect to latency constraints. We need to move away from traditional client-server (query-response) interaction models, to more asynchronous, event-oriented, loosely coupled push model, with applications able to take decisions based on events data.

Complex Event Processing (CEP) is a set of technologies that allows exploring temporal, causal, and semantic relationships among events to make sense of them in a timely fashion.

This article is the first of a serie exploring these technologies, their capabilities and possible applications.

Some business use cases

Use cases that may benefit from CEP are varied, but we can identify some patterns in them, showing the decisive contribution of this technology.

Manufacturing Execution systems

Event processing can be used in plants to detect anomalies or determine if significant changes require re-planning of production. Plant floor systems push events from numerous sensors to a centralized control system that will explore events patterns and emit new, aggregated, rich events to take decisions.

Patterns: Active diagnostics of problems, Real-time operational decision

Location-based services

RFID tags, mobile phones, and Wi-Fi enabled devices feed information about their spatial location into server-side systems. Applications can be tracking goods in the supply-chain, or pushing information to a customer, based on his mobile phone location.

Patterns: Information dissemination, Observation systems

Algorithmic trading

Heterogeneity of information sources and event rates impose an event processing approach on modern financial IT systems, in which quasi real-time market analytics can hardly be implemented in conventional client-server architectures.

Patterns: Information dissemination, Real-time operational decision

Telecommunications

Near real-time data coming from telecom subsystems could be analyzed together with business data from IT systems, or with historical data. With the use of predictive models, fraud detection can be improved.

Patterns: Predictive processing, Real-time operational decision

eCommerce

Clickstream analysis helps in optimizing user experience on commercial web sites, to adapt advertising, or drive page layout. This requires low latency decision, with immediate pattern recognition.

Patterns: Real-time operational decision

Power Utilities

The utility sector requires an efficient infrastructure for managing electric grids and other utilities. This requires immediate response to variations in consumption, using events coming from numerous data sources, aggregated along the grid.

Patterns: Real-time operational decision, Active diagnostics, Information dissemination

Criteria for adopting a complex event processing approach

“It’s all about time” !

The word “complex” in CEP refers mainly to the complexity of state management over time while processing the events. Typical examples are:

• Calculations over sliding windows
• Correlation of events along time, such as: determination that event X occurs before event Y within a precise timeframe ; or determination of a non-occurrence within a timeframe.

Most CEP implementations also provide advanced pattern detection, such as a non-deterministic finite state automaton, similar to a regular expression search over a flow of events, with influence of time in the search.

Another key influence of time is timeliness. Timeliness is the ability to handle events and produce output in a constrained time. It can be seen as end-to-end latency, and can reach the milli-second scale with CEP, or below (cf [perf 1, 3]). CEP tools also provide the ability to arbitrate between guaranteed time and correctness of output (eg. waiting or not for late or unordered events).

Then come event volumes and rates: CEP tools performance can exceed 10’000 and even reach 100’000 events/s [perf 1, 2].

Other complexity factors can also motivate a move towards CEP technologies as well :

• number and type of event sources
• if the application is expected to change over time (new event sources, new interactions and responses)
• richness of information in output events: such as counts, averages, composition of events from different sources
• context dependant situations: eg. detection of events occurring within a defined spatial distance, or within a defined group of customers (possibly querying an external systems to determine the context)
• correlation of real-time data with historical data
• intelligence in event processing: eg. inference models or machine learning capable models

This table sums up the areas in which event processing could particularly fit (from [Chandy et al 2011]) :
 

In other cases, more traditional messaging systems and/or transactional systems may be more suited than CEP.

Choosing a CEP technology

 
There are various vendors, having different approaches and paradigms in their event processing products. We can identify the following paradigms [Helmer et al 2011] :
 

One can see strong similarities with more traditional technologies, ranging from BRMS, to versatile messaging systems (JMS), or EAI.

Beside the ability of CEP to handle very high rates of incoming events, CEP above all brings a coherent set of specific features.

Here is a list that may help you refining your need around CEP :

Functional capabilities:

• Data Reduction : Filtering, Projection (discarding some attributes), Aggregation over time windows
• Modeling capabilities for event shape and payload, and more generally for query logic
• Reasoning: Transformation (eg. enrichment, change of shape…), Pattern detection (including detection of the absence of an event)
• Time handling : event timestamps, intervals of occurrence, particularly with respect to time windows and pattern detection, sliding time windows (or other kind of moving windows)
• Context awareness : taking into account the context in which the event occurs, capability to query external systems (eg. database with business data, or historical data)
• Logging and analysis : for audit purposes, or retrospective event processing (understanding precursor events that led to a particular output event)
• Prediction, learning and adaptation : pattern discovery, scoring against a data-mining model, or machine-learning capabilities
• Presence and capabilities of an integrated development environment (IDE)

Non-functional capabilities :

• Input and output connectivity to event sources and event sinks
• Routing (defined statically or dynamically) and partitioning for workload distribution
• Performance in response time (end-to-end latency)
• Predictability : low variance of latency (e.g. 95-percentile with guaranteed latency), acceptable rate of incoming events before latency increases
• Scalability and elasticity
• Availability and recoverability : fault tolerance, ability to recover, continuous operation (applying change while the application is running)
• Consistency and integrity in a distributed system : management of temporal granularity and potential network delays, clock synchronization
• Security and privacy : segregation of event streams, and user habilitations
• Usability, Maintainability, Manageability : depending on the kind of users that will be implementing and maintaining the logic (developers and/or business analysts)

Conclusion and future work

CEP is more an approach than a technology. On the contrary, there are several types of implementations available, and an even greater number of products on the market. Given the increasing importance of real-time information processing, choosing the best solution for your needs is not an easy task. For this purpose, our next series of articles on CEP will explore several CEP products in detail, and expose their key features.

Bibliography

[Chandy et al 2011]
The event processing manifesto, 2011
Authors : Mani K. Chandy ; Opher Etzion ; Rainer von Ammon

[Grabs et al 2009]
Introducing Microsoft StreamInsight, 2009
Authors : Torsten Grabs, Roman Schindlauer, Ramkumar Krishnan, Jonathan Goldstein

[Helmer et al 2011]
Reasoning in Event-Based Distributed Systems, 2011
Authors : Sven Helmer, Alexandra Poulovassilis, and Fatos Xhafa

[perf 1]
Sybase Aleri performance
http://m.sybase.com/files/Data_Sheets/SybaseAleri_CEPPlatform_PerfTesting_ds.pdf

[perf 2]
Esper performance
http://esper.codehaus.org/esper/performance/performance.html

[perf 3]
StreamBase performance at QCon 2011
http://qconlondon.com/dl/qcon-london-2011/slides/RichardTibbetts_ComplexEventProcessingDSLForHighFrequencyTrading.pdf

Suggestion d'articles :

1. Event Sourcing & noSQL

Bright future

Face recognition is recently getting more and more attention and we can anticipate bright future of this field.

Security was historically and will stay in the future the main application of face recognition in practice. Here face recognition can help with both: identification and authentication. Good example is the Frankfurt airport security system which uses face recognition to automatize passenger control. Another application can be the security analysis of videos purchased by external city cameras systems. Potential suspects can get identified before committing crime. Take a look at the integration of face recognition to London Borough of Newham, already in 1998.

Face recognition can be also used to speedup the identification of persons. We can image a systems which would recognize the client as soon as he walks into branch store (bank, assurance), and the front-office worker can than welcome the client by has name and prepare his folder before he actually gets to the counter.

Advertising companies are working on ad-boards which would adapt their content to the persons passing by. After analyzing the persons face, commercials would adapt to the gender, age, or even personal style. This usage however, might not conform to privacy laws. Private companies do not have rights to film persons in public places (of course, depending on the country.

Not to forgot, that Google and Facebook had both implemented algorithms to identify users in the huge database of photos which they maintain as part of their social network services. Third party services, such as Face.com offer Image base searching, which allow you search for example for picture which contain together your best friends.

One of the latest usages is coming also from Google and it is the Face Unlock feature, which will as the name says, enable you to unlock your phone after your face has been successfully recognized.

Latest news come to face recognition thanks to new hardware equipment, specially 3D cameras. 3D cameras obtain much better results, thanks to their ability to obtain three dimensional image of your face and solve problems which are the main issues in 2D face recognition (illumination, background detection). See the example of Microsoft Kinect , which can recognize you as soon as you walk in front of the camera.

We should keep in mind, that face recognition will be used more and more in the future. This applies not only to face recognition, but to the whole field of machine learning. The amount of data generated every second forces us to find ways to analyze the data. Machine learning will help us to find a way to get meaningful information from the data. Face recognition is just one concrete method from this big area.

How to get started?

Several methods and algorithms were developed since than, which makes the orientation in the field quite difficult for the developers or computer scientists coming to face recognition for the first time.

I would like this article to be a nice starter to the subject which will give you three pieces of information:

• What are the algorithms and methods used to perform face recognition.
• Fully describe the “Eigenfaces” algorithm.
• Show a fully functional example of face recognition using EmguCV library and Silverlight Web Camera features. Go directly to the second part of this article, describing the implementation.

Face recognition process

The process of recognizing a face in an image has two phases:

• Face detection – detecting the pixels in the image which represent the face. There are several algorithms for performing this task, one of these “Haar Cascade face detection” will be used later in the example, however not explained in the article.
• Face recognition – the actual task of recognizing the face by analyzing the part of the imaged identified during the face detection phase.

Face recognition brings in several problems which are completely unique to this domain and which make it one of the most challenging in the group of machine learning problems.

• Illumination problem – due to the reflexivity of human skin, even a slight change in the illumination of the image can widely affect the results.
• Pose changes – any rotation of the had of a person will affect the performance.
• Time delay – of course that due to the aging of the human individuals, the database have to be regulary updated.

Methods and algorithms

• Apearance based statistical methods are methods which use statistics do define different ways how to measure the distance between two images. In other words they try to find a way to say how similar two faces are to each other. There are several methods which fall into this group. The most significant are:
  • Principal Component Analysis (PCA) – described in this article.
  • Linear Discriminant Analysis (more information)
  • Independent Component Analysis (more information)

  PCA is described in this article, others are not. For comparison of these methods refer to this paper

• Gabor Filters – filters commonly used in image processing, that have a capability to capture important visual features. These filters are able to locate the important features in the image such eyes, nose or mouth. This method can be combined with the previously mentioned analytical methods to obtain better results.
• Neural Networks are simulating the behavior of human brain to perform machine learning tasks such as classification or prediction. In our case we need the classification of an image. The explication of Neural Networks would take at least one entire article (if not more). Basically Neural Network is a set of interconnected nodes. The edges which are between the nodes are weighted so the information which travels between two nodes is amplified. The information travels from set of input nodes, across a set of hidden nodes to a set of output nodes. The developer has to invent a way to encode the input (in this case an image) to a set of input nodes and decode the output (in this case a label identifying the person) from the set of output points.
  Commonly used method is to take one node for each pixel in the image on the input side of the network and one node for each person in the database on the output side as ilustrated on the following image:
  
  For more information about face recognition using Neural Networks visit this link

Eigenfaces algorithm and PCA

The eigenfaces algorithm follows the pattern which is followed by other statistical methods as well (LDA, ICA):

1. Compute the distance between the captured image and each of the images in the database.
2. Select the example from the database, which is closest to the processed image (the one with the smallest distance to captured image).
3. If the distance is not too big – label the image as concrete person.

What is the distance between two images?

The crucial question is: How to express the distance between two images? One possibility would be to compare the images pixel by pixel. But we can immediately feel, that this would not work. Each picture would contribute the same to the comparison, but not each pixel holds valuable information. For example background and hair pixels would arbitrary make the distance larger or smaller. Also for direct comparison we would need the faces to be perfectly aligned in all pictures and we would hope that the rotation of the head was always the same.

To overcome this issue the PCA algorithm creates a set of principal components, which are called eigenfaces. Eigenfaces are images, that represent the main differences between all the images in the database.
The recognizer first finds an average face by computing the average for each pixel in the image.
Each eigenface represents the differecens from the avarage face. First eigenface will represent the most significant differences between all images and the average image and the last one the least significant differences.

Here is the average image created by analyzing the faces of 10 consultants working at OCTO Technology, having 5 image of each consultant.

And here are the first 5 Eigenfaces:

Note that the consultants at OCTO are generally handsome guys and this ghost-look is characteristic to the eigenfaces of any group of persons. You can also notice that there is one face which is dominant. That is due to the fact that I had just limited number of pictures of my colleges and I could take a lot of myself using web cam. This way my face become a little dominant over others.

Now when we have the average image and the eigenfaces, each image in the database can be represented as composition of these. Let’s say:

Image1 = Avarage Image + 10% Eigenface 1 + 4% Eigenface 2 + … + 1% Eigenface 5

This basically means that we are able to express each image as a vector of percentages. The previous image becomes to
our recognizer just a vector [0.10, 0.4, … , 0.1]. The previous equation is a slight simplification of the subject. You might be asking yourself how are the coefficients of each eigenface computed. If we would enter into the details,

Now when we have expressed the image as a simple vector, we are able to say what is the distance between two images. Getting the distance of two vectors is not complicated and most of us remember it from school.

If in 2D space we have the vectors [x1,y1] and [x2,y2] we now that the distance between these two can be visualized and computed as shown in the following picture:

Now we can perform similar calculation for the vectors which represent our images.

Behind the scenes

This part will give you some more insights into how exactly are the eigenfaces computed and what is going on behind the scenes. Feel free to skip it if you do not want to enter more into the details.

The key point in the PCA is the reduction of dimensional space. The question we could pose us is, how would we compare the images without eigenfaces. We would simply have to compare each pixel. Having and image with resolution 50 x 50 pixels, that would give us 2500 pixels, in other words we would have space of 2500 dimensions. When comparing using the eigenvalues we arrived to reduce the dimensional space – the number of eigenfaces is the number of dimensions in our new space.

Remember the equation of the distance of two points – now in this equation each pixel would contribute to the distance between images. But not each pixel holds some meaningful information. The background behind the face, the cheeks, the forehead, hairs – these are the pixels which do not give meaningful information about the face. Instead the eyes, nose, ears are important. In the terms of Image processing lot of pixels just bring noise to the computation of the distance. One of the main ideas of PCA is the reduction of the noise by reducing the number of the dimensions.

Reducing the number of dimensions

You may be asking yourself, how exactly can we reduce the number of dimensions? It is not easy to imagine the transition from a space where each pixel is one dimension to a space where each eigenface is one dimension. To understand this transition, take a look at this example:

In two-dimensional space, each point is defined by its two coordinates. However if we know, that several points lay on the same line, we could identify the position of each point only by knowing it’s position along the line. To obtain the x and y coordinate of the point we would need to know the slope of the line and the position of the point on the line. We have reduced the dimension by one. In this example the slope of the line becomes the principal component, the same way our eigenfaces are principal components during the face recognition algorithm. Note that we can even estimate the position for the points which are not on the line, by using the projection of the point on the line (the case of the last point).

From mathematical point of view eigenfaces are graphical representation of eigenvectors (sometimes called characteristic vectors) of co-variance matrix representing all the images. That it how they got their name. For more details on Principal Component Analysis and computation of eigenvectors and eigenfaces visit the following links: PCA and Eigenvectors

Isn’t there an easy way?

You might be thinking, that this is too complex as introduction to the subject and you might be asking yourself whether there is an easy way to start with face recognition?

Well yes and no. Eigenfaces algorithm is the base of the research done in face recognition. Other analytic methods such as Linear Component Analysis and Independent Component Analysis build on the foundations defined by the eigenfaces algorithm. Gabor filters are used to identify the important features in the face and later eigenfaces algorithm can be used to compare these features.
Neural Networks are complex subject, but it has been shown that rarely they have better performance then eigenfaces algorithm. Sometimes the image is first defined as linear combination of eigenfaces and than it’s describing vector is passed to the Neural Network. In other words eigenfaces algorithm builds really the base of face recognition.
There are several open-source libraries which have implemented one or more of these methods, however as a developers we will not be able to use them without understanding how these algorithms work.

Enough of theory now! Visit the second part of the article to see how to include face recognition into your web application.

Suggestion d'articles :

1. Face recognition in RIA applications

But the time invested into this field by universities, companies and independent developers has brought its result in the means of several open-source libraries which any developer can use to perform image processing tasks including face recognition. Take a look at the complete overview of existing computer vision algorithms and libraries.

This article will show you how to incorporate face recognition into your web page using EmguCV image processing library and Silverlight Web Camera features. EmguCV is a .NET wrapper for OpenCV library, written in C++ by Intel and published as open-source. The method which will be used is Eigenfaces algorithm. If you are interested in the details of this algorithm, please refer to the previous article on this topic: The basics of face recognition.

This post describes quite simple application which consists of two parts – Silverlight client, which is responsible for the image capture and eventually pre-processing and the server part which performs the actual face detection and face recognition.

Capturing the image with Silverlight

Since the version 4 Silverlight gives us the possibility to use the camera to capture images or videos using the CaptureSource class. The following snippet shows how to access the camera and show directly the capture picture.

CaptureSource captureSource = new CaptureSource();
captureSource .VideoCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();
captureSource .CaptureImageCompleted += new EventHandler<CaptureImageCompletedEventArgs>(captureCompleted);

if (captureSource .State != CaptureState.Started)
{
    // Create video brush and fill the WebcamVideo rectangle with it
    var vidBrush = new VideoBrush();
    vidBrush.Stretch = Stretch.Uniform;
    vidBrush.SetSource(captureSource);
    WebcamVideo.Fill = vidBrush;

    // Ask user for permission and start the capturing
    if (CaptureDeviceConfiguration.RequestDeviceAccess())
        captureSource .Start();
}

Notice that I have to set the callback for the CaptureImageCompleted event. That should not surprise you, if you have worked with Silverlight before. Blocking of the user interface is not allowed, and so all operations depending on external sources are asynchronous. Now lets take a look at what happens when the image is actually captured:

private void captureCompleted(object sender, CaptureImageCompletedEventArgs e)
{
    var image = e.Result;
    var coll= new ObservableCollection<int>(image.Pixels);
    switch(appMode)
    {
        case AppMode.Recognition:
            client.RecognizeAsync(coll,image.PixelWidth);
            break;
        case AppMode.Training:
            client.TrainFaceAsync(coll, image.PixelWidth,TextBoxLabel.Text);
    }
}

As a result, we obtain WritableBitmap object, which has a simple Pixels property of type int[]. This property holds all the pixels of the image in one dimensional array created by aligning all the rows of the image to one array. Notice that each pixel is presented as Int– this means that that we have a classic 32bits representation of color for each pixel.

We take this array and we send it to the server. When the application is in training mode, we send also the label, which should be added to the Image. If the application is in recognition mode, we send just the image and we hope to receive the label describing the person on the image.

Face recognition using EmguCV on the server

As described in the previous article, face recognition has two phases: Face detection, and actual face recognition. We will use EmguCV to perform both of these tasks. EmguCV is a wrapper for OpenCV library. This basically means that we can call all the functions inside OpenCV library without the need of using constructs such as DLLImport directive and without the need of knowing the structure of the OpenCV library.

EmguCV is surely more friendly for C# developers than OpenCV (at least we do not have to treat the pointers), on the other hand the documentation is not perfect. Several times you will have to address directly the documentation of OpenCV to understand the structures and methods. For this point of view it is important to understand the algorithms which you want to use from EmguCV/OpenCV, because you will have hard-time finding the documentation to understand what each method does.

Detecting the face

I have created a function which takes the array of pixels, and the size of the image, converts the picture to gray scale, detects the face, trims the rest of the image and resizes the resulting image to demanded size, just as shown in the following picture:

public static Image<Gray, byte> DetectAndTrimFace(int[] pixels, int iSize, int oSize)
{
    var inBitmap = ConvertToBitmap(pixels, iSize);
    var grayframe = new Image<Gray, byte>(inBitmap);
    var haar = new HaarCascade(getHaarCascade());
    var faces = haar.Detect(grayframe,1.2,3, HAAR_DETECTION_TYPE.DO_CANNY_PRUNING, new Size(30,30));
    if (faces.Count() != 1)
        return null;

    var face = faces[0];
    var returnImage = grayframe.Copy(face.rect).Resize(oSize, oSize, INTER.CV_INTER_CUBIC);
    return returnImage;
}

The standard structure to treat images inside EmguCV is Image<ColorType, depth>. So from the previous code snippet you can see, that we are returning gray-scales image, where each pixel has 8 bits (byte structure). We first have to reconstruct an Bitmap image from pixels array and than create Image<Gray, byte> object from resulting Bitmap. In this step EmguCV converts automatically the image to gray-scale.

Next the face detection is performed using HaarCascade detection algrorithm. The description of this algorithm is not part this article.
Once the face is detected (and if there was only one face), we copy the rectangle wrapping the face and resize it to demanded output size.
It is important to use the same size, when adding the image to the training set and recognizing the face from the image.

For actual recognition EmguCV contains EigenObjectRecognizer class, which needs several arguments to be created:

• Array of images and corresponding array of labels.
• Eigen Distance Threshold – Eigenfaces algorithm measures the distance between images. This thresholds defines the maximal distance needed to classify the image as concrete person. Big values such as 5000 will make the classifer to return the closest match, even if the probability that the person has been recognized is quite small. Set it to 500 to obtain some reasonable results.
• MCvTermCriteria – is a class which represents OpenCV structure for terminating iterative algorithms. It is composed of two numbers the first being the number of iterations and the second one is demanded accuracy. For some algorithm it makes sense to iterate until the accuracy is not bellow certain threshold. For eigenfaces algorithm it is the number of iterations which is important and it will impact the number of eigenfaces being created. I have obtained some good results having around 40 eigenfaces for 100 images in the database, but this might depend on your scenario, quality and diversity of the images.

MCvTermCriteria termCrit = new MCvTermCriteria(40, accuracy);
EigenObjectRecognizer recognizer = new EigenObjectRecognizer(
                 trainedImages.ToArray(),
                 labels.ToArray(),
                 eigenDistanceThreshold,
                 ref termCrit);

Face recognition

The actual face recognition is a piece of cake if we already know how to create the recognizer. First we have to detact and extract the face from the image with the help of previously mentioned DetectAndTrimFace method. And the second step is to pass this image to the recognize method of the recognizer.

public String Recognize(int[] pixels, int size)
{
   var imgFace = ImagesProcessing.DetectAndTrimFace(pixels, size,
   var recognizer = ImagesProcessing.CreateRecognizer(folder, ...);
   String label = recognizer.Recognize(imgFace);
   if(!String.IsNullOrEmpty(label))
       return label;
   return "Could not recognize";
}

Adding the picture to the training set

This again is easy. Once we do the face detection and trimming, we will just see if there is a folder for the given label and if not, we will create one. One option is to use Guid as a name of file to be sure, that we wont have collisions.

[OperationContract]
public void AddToTraining(int[] pixels, int size, String label)
{
    var faceImg = ImagesProcessing.DetectAndTrimFace(pixels, size, fixedFaceSize);
    var directoryNames = GetDirectories();
    //no such label - create directory
    if (!directoryNames.Contains(label))
        Directory.CreateDirectory(DIR + "\\" + label);

    faceImg.Save(DIR+ "\\" + label + "\\" + Guid.NewGuid().ToString() + "_" + fixedFaceSize + ".jpg");
}

Performance

The face-detection phase is quite instantaneous. Using the HaarCascade algorithm to detect face in the image, we have a chance, that OpenCV already offers a set of HaarCascade features. The creation of these features is much more time consuming, than their usage.

The face-recognition phase is also not that time-consuming. In this example I was working with database of 100 images (10 persons, each having in average 10 images, each image being 80×80 pixels). In this setting the creation of the recognizer is almost instant. Surely the read access of images from the disk, takes more time than, the actual creation of recognizer (the eigenfaces algorithm).

If we have a database of pictures which does not change, we create the recognizer only once and than share it every-time there is a request for recognition. This of course is not possible when the image database changes. In that case we have to recreate the recognizer. There is no possibility to add new image to existing recognizer.

The bottleneck of this approach is still the network over which we have to pass the image from client to the server. To minimize the data which has to be transfer, take a look at the tip which shows you, how to perform the face detection on the client side at the end of the article.

Tricks and tips

Equalization of the image
Histogram equalization is a technique to improve the contrast and adjust the brightness of the image. EmguCV gives us the possibility to call the equalize function which resides in OpenCV:

public static Image<Gray, byte> EqualizeHist(Image<Gray, byte> input)
{
    Image<Gray, byte> output = new Image<Gray, byte>(input.Width, input.Height);
    CvInvoke.cvEqualizeHist(input.Ptr, output.Ptr);
    return output;
}

Take a look at the picture before and after equalization:

Creating additional images
Generally the results are better when the training set is large. If we do not have enough images, we can try to create new ones by slight rotation or horizontal flipping. EmguCV gives us the possibility to call Flip or Rotate method on each picture:

var fliped = faceImg.Flip(FLIP.HORIZONTAL);
var rotated = faceImg.Rotate(angle, new Gray(0.3));

Of course we have to be careful only to use Horizontal flipping (faces upside-down would just confuse the recognizer) and to rotate just to some angle.
More image treatment on the client side
In the presented code, you can notice, that I am sending the image to the server, right after image capture, without any preprocesing. Because the default size of the image is 640×480 pixels. Knowing that each pixel is represented by 32 bit Integer this means that we are sending roughly 1200 kB to the server for recognition or training. Since images are resized later anyway, we could resize them directly before sending to the server. Resizing the images to 320*240 would reduce the size of send data to 300 kB (but of course we can do more).
Unfortunately Silverlight does not have method for resizing images, however there is a great project on Codeplex, which contains several extensions methods for WritableBitmap, one of them being the resize method.
Face detection on client side
It is not possible to move all the face recognition process to the client side, from one simple reason: the recognizer has to, at some point, load and process all the images in the database. Performing this part on the client side, would demand us either to embark all the images into the client or transfer these images over the wire. Either of these options is not attractive for web developers.
However the process of face detection could be moved to the client side, while it is not dependent on any big amount of external data. Great article which explains face detection in Silverlight is available as FaceLight project on Channel 9

Will we log to our bank using only our web camera?

Well the answer is no, face recognition is still not completely trust-worthy. Eigenface algorithm presented above is too much sensitive on the brightness changes and face-pose changes. When testing the algorithm, I obtained successful results in 60-70% of the total tries. However new algorithms have been developed combining several image processing and face recognition techniques and of course many companies are specializing in this field and develop software with much better accuracy.

Another problem is the fact that it is far too easy to present a taken photograph to the web camera and unless the system is capable of recognizing the real person from taken picture it will be cheated.

We will probably see mutual systems where the login and password will still be the main security check and face recognition will be used instead of the third confirmatory information such as RSA generated number or confirmation code.

Suggestion d'articles :

1. The basics of face recognition

This year, we had the opportunity to go to the Droidcon in London, a great conference about mobility and Android. The two-day event was organized like in 2010: a barcamp the first day, and conferences with great speakers the second one. Main topics were user experience and interface, Android development, mobile apps business and distribution.

User eXperience

The number of Android users has incredibly risen! More than 300 000 applications can be downloaded on the market! What makes your app better and prevent it from user rejection ? Look and feel and responsiveness, of course!

Make your application smooth!

This year, Cyril Mottier talked about UI and gave pieces of advice to develop pretty and fluid applications. It starts with a simple order: do not block the UI thread because it is the one that displays elements on screen and handle interaction with the user. Using it for non UI process leads to slow animations, laggy interface and if five-second freeze occurs: Application Not Responding messages!
Android SDK offers several ways to handle secondary threads for non UI processes: execute it with simple Threads and Handlers (maybe with a Looper to do operation queues), create an AsyncTask or even launch a dedicated service with the IntentService class.
Likewise, access your data asynchronously: AsyncQueryHandler, Filter and SharedPreference.Editor.apply() can help you.

Use great components

All recent successful applications understood it: nice and customized views are the key! Cyril Mottier’s open-source library GreenDroid is one of the most known. It allows you to ease development by using its theme and numerous features for pre-Honeycomb application (and after with compatibility package): ActionBar, AsyncImageView, PagedView, QuickActions and more!

GDCatalog application is available on the Android market to demonstrate the capabilities of this very useful library.

Follow Google best practices to optimize your app for smartphones and tablets

ActionBar

Nick Butcher introduced some guidelines to design UIs for both kinds of devices. The most noticeable UI Pattern that has become unavoidable is the ActionBar. Its aim is to help the user accessing screen details and contextual actions. This action bar places elements in a way to help user interactions. See GMail example below.

On the left side, the application icon informs the user where he is. As on a website, a click on this button leads to the application main screen. A second way to navigate inside the application flow is offered by the in-app navigation button. This action can be different from traditional Android back button. The right side presents traditional application parameters button and contextual actions. More contextual informations can be added to the bar under certain conditions (items selection, etc.).

To implement an ActionBar on pre-Honeycomb Android version, you can use Google I/O application implementation, ActionBarSherlock or previously presented GreenDroid libraries. Upper versions (v3+) can use SDK ActionBar.

With the upcoming Ice Cream Sandwich version, ActionBar will be able to automatically layout the interface for phone or tablet. Little advice for this: provide text and icons for each Action item and set for each one the flag: showAsAction=“ifRoom|withText”.

Multi-pane layouts

Since Honeycomb, a single APK can be executed on both smartphones and tablets. On large screens, it is preferred to display more complete content than a simple ListView. With Fragments, it is possible to aggregate your code to use it on 2 smartphone screens and a single tablet screen. With this concept, you can imagine a lot of different screen dispositions depending of screen sizes, orientation, etc. We will talk about this subject in a next article.

Do’s and don’ts

Simple tips were given to improve development:

• aim for a single APK for both tablet and handset devices
• use compatibility library
• customize every view to represent your brand
• support both landscape and portrait modes
• extract dimensions for phones and tablets (e.g. values/dimens.xml and values-large/dimens.xml)
• use themes and styles to reduce code redundancy
• enable hardware acceleration

And other to avoid in your application:

• do not use small fonts
• tablets are not “big phones”: they fulfill a different need and require special layout optimizations
• do not assume that a device running an application with API level >= 11 is a tablet: Ice Cream Sandwich is coming for both tablet and phones!
• do not assume that all tablets have an xlarge resolution: 7” tablets have a large resolution! Detect tablets with the minimal width your application needs, generaly 600 dp:

         <manifest ... >
              <supports-screens android:requiresSmallestWidthDp="600" />
              ...
         </manifest>

A bit of technique

Testing

You may have seen comments such as “Doesn’t work on SGSII” or “Force close on HTC Wildfire” on Android Market. Indeed, with no less than 300 phone models, it is difficult to provide an application that works on each device. The best way to minimize bugs is to test your application. You can do it manually or with low cost outsourced hands but the best way is to automate your tests.

Several frameworks exist and cover different kinds of tests:

• Monkey, which ensures to simulate random actions on the UI (buttons, textedit, touch…). This is a good way to validate the robustness of the application.
• The Google test api (instrumentation) which can be laborious to implement although essential if you want to test the interface behavior.
• Robotium, a good testing framework, that simplifies the test writing over the Google’s instrumentation tests.

We also came across two other frameworks (plateforms):

• Robolectric comes with a great advantage over robotium and instrumentation: you don’t need an emulator to test your application, and everyone knows how much the emulator is slow! So, you run your tests in a standard JVM, in your favorite IDE and you gain a lot of time. Joe Moore (from Pivotal Labs) made a demo with a set of 20 tests. The standard instrumentation test took 45s on an emulator (already launched, so you can easily add one minute for the launch) while robolectric got the job done in only 12s! Moreover, it is very simple to use, simply add @RunWith(RobolectricTestRunner.class) on your test classes and that’s it! Give it a try, it’s free. You can get the slides here.

• All the previous solutions are either boring to implement (manual creation of the tests), nor realistic (emulator, JVM). Testdroid (developed by Bitbar) promises to answer to both issues with two services:
  Testdroid recorder, an eclipse plugin, ensures to capture your test while you execute the application. It generates robotium code, so you can replay the tests after.
  Testroid server is a cloud platform that allows to automatically execute the recorded tests on multiple Android physical devices. A lightweight DeviceAnywhere? Seems promising to test your applications on real devices, on different Android releases, with different form factors…

Do Not Repeat Yourself

In our projects, a lot of tasks are repeated. The session “Kick-starting Android Application Development” presented some interesting projects to improve Android development:

Android plugin for Gradle project allows to use Gradle to reduce time spent on APK build, packaging, integration, etc. It can be use in integration with Maven or Ant and also allows to create Android applications in Scala.

Roboguice, is a Google implementation of Guice to use dependency injection. It uses reflection to inject fields at runtime and allows to get rid of resources initialization, service creation,… A simple demonstrative example to get a TextView instance: @InjectView(R.id.name) TextView nameTextView;
Another similar project (AndroidAnnotations), based on subclass code generation instead of reflection, allows to inject content with no runtime impact on performances.

Application distribution

Several sessions were related to application distribution, and how to improve your application and its ranking on the Market. Here are some tips from Caroline Lewko (Wireless Industry Partnership), Thai Tran (LightBox) and Konstantinos Polychronis (Bugsense).

Do not forget marketing!

Nowadays, publishing an application and keeping it visible on the stores is a challenge. WIP shared several tips to make your application successful through time. First of all, as mobile is growing fast, you have to follow the same rule! A way to move at same speed is to make short Agile development cycles and for each, a marketing review of the product. Marketing’s role is to ensure that the application always fits the market’s needs, try different implementations (A/B testing). It keeps the product uptodate and brings opportunities to “fail fast” instead of achieving a useless application.

Once you have a product that grows fast, you have to make your customers like it! It starts with a good name. Use Google and social networks to find a name that will be easily adopted by the public. Then use those social networks, especially Twitter and Facebook, to make your name and your product known. The more it is known, the more it will be downloaded and visible on stores.

Beta test your application

Once you have developed it, and tested it, go to beta test. You can do it yourself, recruit some beta testers, send them the apk and get their feedback. You can go further, put your application on the Android Market and automatize feedback.

On the market, make your application the less visible possible: do not use your usual account, do not provide description, screenshots, keywords, use a different package name in the application to reset comments and notes when you will deploy the release, provide an invitation code to your beta testers.

Another solution, easier, is to get a service like Appaloosa to distribute your apps to a managed range of users. We got the opportunity to show this product during the barcamp and make a demo, which was well welcomed. We really felt the need of such a solution.

Get the maximum of feedback

In order to ensure customers engagement: give them a way to provide some feedback and use that data to improve your product’s features.

You can use frameworks like ACRA or Bugsense to retrieve crash reports (which phones and characteristics, which os version, what kind of error). Be sure to correct bugs after that.

Use Google Analytics or Flurry Analytics to get usage metrics (how many users, how many time, what features usage). Collect the maximum of information to improve your application navigation.

Use Android Market tools for developers

To conclude the conference, Richard Hyndmann from Google talked about developers facilities on the Android Market and how to get visible on it. To be at the top of rank, your app has to be fit and polished. Of course, avoid straight port from other plateform, it generally causes users rejection. If your application offers unique features for Android, it is more valuable and will be rated as so. Seasonality also assures regular update of your app’s position that can appear in several different categories like Top Free & Paid apps, top grossing apps, and others. Also, do not forget to provide new format images for add banner: it is a great way to get downloaded!

The Android market offers a new functionality: distribution of multiple APK. It allows a developer to provide different APK files for different kind of devices. For example, one application for handset devices and another for tablets. It can simplify adoption of new platform features and lighten your apps. However, Google advices to use this feature only in special cases and always prefer a single APK file when possible.

It appears that Honeycomb users are ready to pay for applications! It can generate some big revenues. Some attention was given to developers ways to make money. In-App Billing system was presented. It is another way to sell virtual content such as additional levels for games or download files. Fees are the same for in-app purchases as the transaction fee for application purchases on the market: 30%.

Another tool that can be useful to prevent piracy for paid applications on market is Licensing service. Its License Verification Library (LVL) allows your application to set up licensing status protection. The last feature presented is application Data Backup. Implement it to offer users to save application content and preferences, even private files in the cloud. All the data can then be restored, for example when switching to a new Android device. All these tools can be used to make your applications better! Just think to use them.

What’s next?

We want to finish with some interesting predictions given by Mark Murphy during the opening keynote:

• In 3 years, 1 application over 3 will be developed using cross platforms tools such as HTML 5 (with Phonegap or Rhodes), app generators (Appcelerator) or any other framework that could come to provide multi-platforms apps. Indeed, most companies want to provide their apps to the maximum of platforms with the smallest development cost. We even think it could be more than 1 over 3 applications…
• In 3 years, Amazon will be the second tablet manufacturer (after Apple). The new Kindle Fire offers today a full multimedia experience on an Android based system. Will Amazon still be using Android plateform in the following years ?
• In 3 years modded roms will be significantly more popular! Modded roms like CyanogenMod could diversify and meet enterprise needs in terms of security and professional tools… This is, with the rise of private application stores, one of the major points that may make Android become an important actor in enterprise.

DroidCon 2011 was a great time for us, we learnt a lot, especially that next year will be tablet year! Ice Cream Sandwich is coming and will give the opportunity to develop applications for both devices kinds. Be ready and always make better products!

Thanks to organizers and sponsors for this year.

Suggestion d'articles :

1. Droidcon London 2010

Functional testing works great with MVVM, but several issues prevent the use of functional testing tools (e.g. Fitnesse) with a Silverlight project out of the box.

If the previous article convinced you to implement automated functional testing on your application, but its GUI is in silverlight instead of WPF, let’s see how you can circumvent those issues.

The following has been implemented with Fitnesse, but should also work with Greenpepper or any other .Net automated testing framework.

How can Fitnesse load Silverlight code ?

Loading silverlight code in a standard .Net CLR is a matter of referencing the proper assemblies. Since the assemblies needed to load a ViewModel have the same name than .Net counterparts, Visual Studio does not allow to add them directly (it replaces it with the .Net assembly of the same name found in the GAC).

So this requires some .csproj editing, as such :

<Reference Include="System.ServiceModel">
  <HintPath>..\DLLs\System.ServiceModel.dll</HintPath>
</Reference>

Silverlight assemblies must be referenced with their location on the filesystem. The assemblies required to load a View Model are System.ServiceModel, System.Windows, System.Runtime.Serialization, System.Xml, and System.Xml.Linq.

Add those assemblies as references in your Visual Studio project holding your fixtures, as well as references to your Silverlight projects in your solution, and you’re good to go.

What if my view model calls a WCF service ?

Now we can call silverlight code from fitnesse. But if you’re writing a Silverlight application, chances are it is calling WCF services to do some work server-side. You can’t directly inject a WCF service implementation into a view model, since a WCF client proxy is generated to call it. Moreover, the client side interface only exposes asynchronous methods.

So the challenge is to be able to transform this :

// Server side WCF Service signature
int DoSomeWork(int i);

into these :

// Client side equivalent generated by Visual Studio
IAsyncResult BeginDoSomeWork(int i, AsyncCallback callback, object asyncState);
int EndDoSomeWork(IAsyncResult result);

Those are standard asynchonous methods any delegate can produce. A wrapper class can be created to translate the server-side service into the client-side one this way :

public class MyServiceWrapper : ClientSideNamespace.IService // Client-side interface {
 private ServerSideNamespace.IService service;
 private Func<int, int> SomeWorkDelegate;

 public MyServiceWrapper(ServerSideNamespace.IService service) // Server-side interface {
  this.service = service;
  this.SomeWorkDelegate = i => service.DoSomeWork(i);
 }
 public IAsyncResult BeginDoSomeWork(int i, AsyncCallback callback, object asyncState) {
  return SomeWorkDelegate.BeginInvoke(i, callback, asyncState);
 }
 public int EndDoSomeWork(IAsyncResult result) {
  return SomeWorkDelegate.EndInvoke(result);
 }
}

Use this wrapper as a service implementation in your ViewModel and you’re good to go.

What if my WCF service returns more than an integer ?

Just like method signatures differ between client and server side, data structures are not the same. Properties have the same name, but types can differ, especially Collections implementations. Automapper is an open source library able to solve this issue in a very convenient way, enabling the wrapper to translate a server-side dto into a client-side dto and vice versa.

Assuming the WCF service evolves to :

DoSomeWorkDto DoSomeWork(int i);

The wrapper becomes :

public class MyServiceWrapper : ClientSideNamespace.IService // Client-side interface {
 private ServerSideNamespace.IService service;
 private Func<int, ClientSideNamespace.DoSomeWorkDto> SomeWorkDelegate;

 public MyServiceWrapper(ServerSideNamespace.IService service) // Server-side interface {
  this.service = service;
  this.SomeWorkDelegate = i => Mapper.Map<ServerSideNamespace.DoSomeWorkDto, ClientSideNamespace.DoSomeWorkDto>(service.DoSomeWork(i));
 }
 public IAsyncResult BeginDoSomeWork(int i, AsyncCallback callback, object asyncState) {
  return SomeWorkDelegate.BeginInvoke(i, callback, asyncState);
 }
 public ClientSideNamespace.DoSomeWorkDto EndDoSomeWork(IAsyncResult result) {
  return SomeWorkDelegate.EndInvoke(result);
 }
 static MyServiceWrapper() {
  Mapper.CreateMap<ServerSideNamespace.DoSomeWorkDto, ClientSideNamespace.DoSomeWorkDto>();
 }
}

The Mapper class is pretty straightforward, just call the CreateMap method for all dto class couples (both ways if necessary). Then include a Mapper.Map call in your delegates in the wrapper and you’re good to go.

Conclusion

The practice of testing an application with Fitnesse (or any functional testing tool) through its ViewModels is really valuable. Matthieu’s feedback was already self-explanatory about this. Not being able to achieve it on a Silverlight application would have been a shame.

At the price of a little development overhead (the WCF services wrappers), this practice is applicable on Silverlight projects too.

A recent experience with a Silverlight application developed by Octo showed that this overhead is negligible compared to the benefits of automated functional testing involving the ViewModels.

Suggestion d'articles :

1. Design your Silverlight application for TDD !
2. Behavior Driven Development using MVVM pattern and GreenPepper tests