iOS dev: How to setup quality metrics on your Jenkins job?

iOS development projects are not first-in-class when it comes to managing the quality of the software produced.

Very short projects, very short time-to-market, it is not the kind of projects where you see a lot of attention towards quality, unfortunately.

Here at OCTO we try to do it differently, even for this kind of projects. Or above all, for that matter. But here comes another issue: the lack of tooling.

Here is one of our latest attempt to setup quality metrics on a short and budget constrained iOS project. And yes, a teaser, look at our dashboard at the end of this six weeks (six Agile iterations) project:

Screenshot of Jenkins dashboard on a iOS project

The following article will detail how to setup all these quality metrics in a integrated report in Jenkins (Continuous Integration).

Here is the breakdown of this article:

Tooling overview

When it comes to tooling, Objective-C is a mixed case because it is at the same time a proprietary platform and a language derived from C. In practice, here is the current situation on quality tools for Objective-C:

  • there is not much static analysis tools (ok there is Clang Static Analyzer but mostly for memory management, far less useful since the introduction of ARC for iOS). You could use clang AST to write your own checks for your project or company but I haven’t yet heard of anybody having done this. No FindBugs, PMD or Checkstyle for Objective-C yet. In fact by saying this I thought I’d better double check and I found OCLint that looks interesting but is in a very early stage.
  • even basic code metrics are not easy to compute. Most of the C tools for calculating LOCs work (for example SLOCCount or cloc) but you can’t get interesting metrics like number of classes, class complexity, method complexity etc.
  • one of my preferred quality metric, code duplication, is feasible but not that easy. You can use PMD. for this (with a specific grammar) or use Simian (that supports it out-of-the-box but is not free for commercial use)
  • code coverage is feasible but not easy to get it right. You can do it thanks to a gcc feature called GCOV.
  • there is no way to aggregate, consolidate and track all these quality metrics. No Sonar for iOS, said differently (or not yet, see the end of the article (LINK)). To fill in the gap, we have used directly our build server (Jenkins) to act both as the dashboard and the history of these metrics.

Let’s look at how to setup all those metrics.

Note: If you get lost on the way, do not hesitate to check the end of the article, I have put links to sample configuration files

Step 0: Setting-up Jenkins

We will not detail this point as there are a lot of posts around on doing this. For example you can go with this one in case Jenkins is not already installed at your place.
I recommend using the same user as your CI user because otherwise you will run into issues with Keychain certificates (the certificates you will install on your server will not be available to the Jenkins user).

Step 0bis: Setting-up your Jenkins job

As above, I refer you to this article if it is something new for you.
I also recommend using the Xcode plugin for Jenkins.

Step 1: Setting-up easy metrics

Let’s dive in interesting things. Setting up code metrics and code duplication is quite easy, so let’s start with it.

Code metrics (LOC)

I decided to go for SLOCCount as it already provides a Jenkins plugin. The documentation for it is here.

I followed the instructions on both the SLOCCount page and Jenkins plugin page, that is basically:

  • download latest archive of SLOCCount, unzip it somewhere, run ‘make install’. It will install sloccount in /usr/local/bin.
  • install the slocount plugin in your Jenkins (via Jenkins UI)
  • add a Shell Build Step in your Jenkins job to compute the code metrics. This is the step I added in my job:
    sloccount --duplicates --wide --details YOUR-PROJECT-ROOT-DIRECTORY | grep -v -e '.*ExternalFrameworks.*' | grep -v 'top_dir' > build/sloccount.sc This is how it looks in my Jenkins job configuration:
    Configure SLOCCount
    Notice two added commands compared to the standard sloccount command:

    • I excluded third-party source code with “grep -v -e ‘.*ExternalFrameworks.*’
    • I avoid an issue with SLOCCount plugin (see here) with “grep -v ‘top_dir’“. This issue has been fixed meanwhile so this step might not be necessary any more
  • Tell SLOCCount plugin where to find the report by filling the ‘Publish SLOCCount analysis result’ field with ‘build/sloccount.sc’ (see below)
    Configure SLOCCount plugin

Once this done, relaunch your build. You might have to modify your Jenkins configuration to have /usr/local/bin in Jenkins path (unfortunately Jenkins do not use your system path). I fixed this by setting the PATH variable in Jenkins configuration with the content of my path (‘echo $PATH’). You can do this in the ‘Global properties’ section of the configuration (not in your job configuration but in Jenkins configuration):

Configure PATH in Jenkins

If you go back on your dashboard, you should see the trend graph on it and have also access to the full report through the left menu. Actually, depending on the plugins, the trend graph does not appear from the start but once it has enough data to display (typically after 3 or 5 builds).

LOC trend graph

The detailed report gives you the details of LOC by file and by folder. You can do a first sanity check here to know if no file has been forgotten or is present by mistake. Here is an example of the report by folder:

LOC detailed report

Code duplication

The tool chosen here is PMD because it is free to use. In fact the Objective-C support is more ‘beta’ than it is in Simian, so you might choose the later.

Objective-C support in PMD is made possible thanks to Joshua Kennedy. Announcements and details on how to setup it are posted on his blog so I refer you to his post.

Basically you have to:

  • download PMD and unzip it somewhere (I went for ~/PMD so the extracted path is ~/PMD/pmd-4.2.5)
  • download the Objective-C grammar JAR and save it at the same place (~/PMD in my case)
  • install Jenkins DRY plugin (via Jenkins UI)
  • Add a Shell Build Step in your job and paste the following command:
    java -Xmx512m -classpath /Users/udd/PMD/pmd-4.2.5/lib/pmd-4.2.5.jar:/Users/udd/PMD/ObjCLanguage-0.0.6-SNAPSHOT.jar net.sourceforge.pmd.cpd.CPD --minimum-tokens 100 --language ObjectiveC --encoding UTF-8 --format net.sourceforge.pmd.cpd.XMLRenderer --files YOUR-PROJECT-ROOT-DIRECTORY --files YOUR-TEST-ROOT-DIRECTORY > build/cpd-output.xml This is how it looks in my Jenkins job configuration:
    Configure PMD
  • Configure DRY plugin to use the CPD output. For this you have to fill the field ‘Publish duplicate code analysis results’ with ‘build/cpd-output.xml’
    Configure DRY plugin

Just one comment: I had trouble to make it work and always get a blank report file first (so as few other if you read the comments).

<?xml version="1.0" encoding="MacRoman"?>
<pmd-cpd>
</pmd-cpd>

If your are in this case, here is the hint: you have to specify a full path for the JARs in the classpath, for some reason I ignore.

Well, if things went right, you should now have on your dashboard a nice trend graph like the following:

Duplication trend graph

and you can access from the left menu the detailed report, including the report by file which is quite interesting:

Duplication detailed report

and if you drill down to the duplication, the detail of each duplication per file:

Duplication detailed report by line

Nice, isn’t it?

Step 2: Setting up harder metrics – that’s where it hurts!

Let’s step to the harder part, getting test results and code coverage metrics in Jenkins. Getting test results is pretty easy thanks to the Xcode Jenkins plugin though, because the plugin will take care to generate the test results XML file in the expected JUnit format.

Test results

To achieve it, you need to, basically:

  • Add a Xcode Build Step to your job. This one should compile your test target. I recommend compiling in Debug for the simulator because this will be useful when we will want to add code coverage results. In our case, see the step configuration below:
    Configure Xcode plugin
  • enable the build setting ‘Test after build’ in your test target of your project in Xcode. Otherwise your tests will not be launched on the CI server. Notice that this will cause an ugly warning on the computer on which you develop when you run the tests (⌘ + u)
    Configure Xcode OCUnit for Jenkins
  • Configure Jenkins to publish the report. For this you have to fill the field ‘Publish JUnit test result report’ with the value ‘test-reports/*.xml’, as shown below:Configure Jenkins for JUnit

Once done, if you run your build again, you will get a new trend graph on your dashboard:

Test results trend graph

And you will be able to drill down to the individual tests or failures, in case there are some.

Test results detailed report

Alternative: Test results with GHUnit

Some of you might be using GHUnit instead of OCUnit (Xcode default).

Here is a summary of the differences:

  • you do not need to enable the build setting ‘Test after build’ in your test target of your project in Xcode
  • make sure that you have followed GHUnit documentation on command line usage. You should have added a script file RunTests.sh and a Run script Build Phase in your test target to launch the script. Notice that I didn’t use the latest version of the script since I had an issue when running in Jenkins. I used this older one. For reference, see this discussion on Google Groups
  • you should not add a Xcode Build Step to run the tests but rather a Shell Build Step in your Jenkins job configuration. The shell code to run is the following:
    GHUNIT_CLI=1 WRITE_JUNIT_XML=YES xcodebuild -target YOUR-TEST-TARGET-NAME -configuration Debug -sdk iphonesimulator clean buildThis is how it looks in my Jenkins job configuration:
    Run tests GHUnit Jenkins
  • The right path to the generated XML file for GHUnit is ‘build/test-results/*.xml’, as shown below:
    Configure Jenkins GHUnit

Test coverage

Take a breath now (or a coffee), this is the harder part. There is a few good posts around on how to compute code coverage for iOS, but not always up-to-date The way to do it has changed drastically with the different versions of Xcode. The following has been tested with Xcode 4.3.3.

To summarize, you should:

  • enable the two build settings ‘Generate Test Coverage Files’ and ‘Instrument Program Flow’ in the test target of your project in Xcode
    Configure coverage Xcode test target
  • enable the two build settings ‘Generate Test Coverage Files’ and ‘Instrument Program Flow’ in the main target of your project in Xcode, but only for Debug
    Configure coverage Xcode main target
  • add code for missing fopen$UNIX2003 and fwrite$UNIX2003 functions. This is more an Xcode bug. I added this code in my AppDelegate.h:
    #ifdef DEBUG
        FILE *fopen$UNIX2003(const char *filename, const char *mode);
        size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream);
    #endif
    

    and this one in my AppDelegate.m:

    #ifdef DEBUG
    FILE *fopen$UNIX2003(const char *filename, const char *mode) {
        return fopen(filename, mode);
    }
    
    size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream) {
        return fwrite(ptr, size, nitems, stream);
    }
    #endif
    

Once done, I recommend to check everything is working locally first. For that you should run your tests and go to: ~/Library/Developer/Xcode/DerivedData/YOUR-PROJECT-NAME/Build/Intermediates/YOUR-TEST-TARGET-NAME.build/Objects-normal/i386' and ~/Library/Developer/Xcode/DerivedData/YOUR-PROJECT-NAME/Build/Intermediates/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386'Both locations should contain several .gcno files if the project is properly setup and some .gcda files if your tests did generate coverage data. You can check that the coverage is correct by opening one of the files with CoverStory, a GUI for analyzing the coverage data.

Once your project properly set up, you have to configure Jenkins to generate this report automatically. This is explained in this blog post.

Basically you need to:

  • download GCOVR, a Python script that will be used to convert the coverage files to a Corbertura XML format that is understood by Jenkins. Copy it at a safe place in your server (I put it in my repo in scripts/ directory).
  • add a Shell Build Step to your Jenkins job with the following content:
    scripts/gcovr -r . --object-directory build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386 --exclude '.*Tests.*' --exclude '.*ExternalFrameworks.*' --xml > build/coverage.xmlThis is how it looks in my Jenkins job configuration:
    Configure GCOVR
  • install Cobertura plugin for Jenkins (using Jenkins UI)
  • Configure Cobertura plugin to use the right output file. For this you have to fill the field ‘Publish Cobertura Coverage Report’ with ‘build/coverage.xml’Configure Cobertura plugin

Launch your Jenkins job! It works? Congratulations, but don’t worry if it doesn’t, this part is a bit a makeshift job, it might need some trial-and-error.

Once done, you will get a new trend graph on your dashboard as below:

Coverage trend graph

and you will be able to drill down to see the coverage of all your different files:

Coverage detailed report

and event of the specific lines of your source code:

Coverage detailed report by line

Satisfied? If you try this step, you will notice that you don’t really get the same dashboard as the one shown above, right? In fact there is plenty to say about getting an useful code coverage measurement, and as this article was already long enough, I have detailed this in another blog article with all the tips and tricks that lead me here.

Alternative: Test coverage with GHUnit

For adding code coverage, the main differences with OCUnit (Xcode standard) are:

  • add the code for fopen$UNIX2003 and fwrite$UNIX2003 functions in the main.m file of your test target, not in the AppDelegate. This is how my main.m file looks like:
    #import 
    
    int main(int argc, char *argv[])
    {
        @autoreleasepool {
            return UIApplicationMain(argc, argv, nil, @"GHUnitIOSAppDelegate");
        }
    }
    
    FILE *fopen$UNIX2003(const char *filename, const char *mode) {
        return fopen(filename, mode);
    }
    	 
    size_t fwrite$UNIX2003(const void *ptr, size_t size, size_t nitems, FILE *stream) {
        return fwrite(ptr, size, nitems, stream);
    }
    
  • edit your test target .plist and add the property ‘Application does not run in background’ with value ‘YES’. This is depicted below. Notice that this is useful because the coverage files are only written when the application exits.
    Configure coverage Xcode GHUnit
  • add a Shell Build Step in your Jenkins job configuration between the launch of your unit tests and the launch of the GCOVR script, with the following content:
    cp -n build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects-normal/i386/*.gcno build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-TEST-TARGET-NAME.build/Objects-normal/i386 || true
    This is how it looks in my Jenkins job configuration:
    Add coverage for not tested application sources GHUnit

Step 3: Enjoy!

Congratulations!

But don’t forget that all of this is only useful if you analyze it and track it regularly. This is what the trends are made for, and this is how I use these metrics:

  • I manually mark important builds as ‘Keep forever’ in Jenkins and I enabled ‘Discard Old Builds’ in my Jenkins job configuration. In practice I only keep builds delivered at the end of each Agile iteration, to have a good vision of the coarse-grained variations (trends)Configuration discard unimportant builds

    This is why all the graphs shown in this article looks so nice and clearly outline trends.

  • I review them at the end of each iteration to add actions to the backlog of the next iteration. Example of such actions:
    • ‘the code duplication has gone red, we will fix all red during the next iteration’
    • ‘The service layer is no more tested at a 80% test coverage, add tests on XXX and YYY in the next iteration’
    • etc.

Conclusion

Honestly this is still a bit long/clumsy to setup, I think for a first installation it will take you about 2-4h. A lot more if you also need to install Jenkins and don’t know Jenkins yet.

The good news is that once done for a first project, it’s about 30 min of work for all the future mobile projects you will work on. I think this is acceptable even for very short projects.

I have an another good news for those that gave up because of time/pitfalls, we are currently working on a Sonar plugin for Objective-C. This will make all this easier and will bring new metrics. Stay tuned, we will announce it here as soon as we release the first useful version.

Last thing: I have put here the full configuration of my Jenkins job in case you want to compare with yours. I have also put a sample configuration for test and coverage for OCUnit and the same sample configuration for GHUnit, because this is the hardest part and I thought it deserves it.

And yes, I welcome your feedback if I forgot some steps or you think I should clarify a particular step.


21 commentsfor “iOS dev: How to setup quality metrics on your Jenkins job?”

  1. Very nice post and useful too! For the iOS jobs I manage I put all these small scripts (for PMD, uint tests, libs API documentation generation…) into a second repository. This repo is cloned into a “scripts-ci” directory of the workspace and therefore these scripts are shared between all iOS applications and library jobs.

  2. Nice write-up. You should also checkout the Clang scan-build Jenkins plugin.

  3. This is very helpful. Another item that may be worth adding to the reporting is cyclomatic complexity, using http://code.google.com/p/headerfile-free-cyclomatic-complexity-analyzer/

  4. @Josh, yeah I have used Clang a few times but since I have switched all my projects to ARC, it’s only throwing errors (internal), so I disabled it.

    From what I understood Clang is only focused on memory issues right? Doesn’t it become irrelevant with ARC?

    By the way thanks for your nice Objective-C grammar for PMD!

  5. @Jean-Christophe, you’re right it’s better to put these scripts in a common place either in a repository or on the server.

  6. @Jon, I didn’t know about it, thanks for the pointer!

    I will give it a try and update my post if I manage to integrate it nicely

  7. Fantastic Article – I’m currently struggling with the Code Coverage metrics for GHUnit however… Thing is (as you quite rightly point out above), the test app needs to be launched and shut down before the coverage files are generated. Unfortunately, the process above seems to run the batch script to create coverage.xml before any launch of the application and I’m left unsure as to how I can run the app via script… Any thoughts would be greatly appreciated!

    Liam

  8. A great post. Thank you.
    Have any body tried to get the coverage data on device instead of simulator? The problem I met is:
    If I change the option of build setting from simulator to device, there is no gnco file generated.

  9. @Mason No, I did not try to generate coverage on device, sorry.

    Interested in knowing the solution if you find one.

  10. @Liam It looks like you are not running your tests. Did you add the command line for it?

    You can have a look at http://blog.octo.com/wp-content/uploads/2012/08/configuration-jenkins-ghunit.png for a sample GHUnit configuration, it might help to check you get eveything right.

  11. Thanks for this post it helped a lot.

    Nevertheless i’m still getting issues with sloccount, the file generated seems “correct”, but the jenkins plugin has a lot of failures like :

    ERROR: Publisher hudson.plugins.sloccount.SloccountPublisher aborted due to exception
    java.lang.NullPointerException
    at hudson.plugins.sloccount.model.SloccountParser.parseLine(SloccountParser.java:92)
    at hudson.plugins.sloccount.model.SloccountParser.parse(SloccountParser.java:59)
    at hudson.plugins.sloccount.model.SloccountParser.parse(SloccountParser.java:50)
    at hudson.plugins.sloccount.model.SloccountParser.invoke(SloccountParser.java:40)
    at hudson.plugins.sloccount.model.SloccountParser.invoke(SloccountParser.java:17)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:2099)
    at hudson.remoting.UserRequest.perform(UserRequest.java:118)
    at hudson.remoting.UserRequest.perform(UserRequest.java:48)
    at hudson.remoting.Request$2.run(Request.java:287)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

    Do you have a sample file which is working ? i tried to replace the grep -v “top_dir” with grep -v ‘Have a non-directory at the top, so creating directory top_dir’, but does not change anything. Do you have any idea ?

  12. @LiohAu, I think there is a bug in the Sloccount plugin for Jenkins, version 1.8. Look at this: https://issues.jenkins-ci.org/browse/JENKINS-14135

  13. Yes, there is a bug with the version 1.8 of the SLOCcount plugin.

    It has been fixed, but the plugin is not update into Jenkins plugin. You can find the code here : https://github.com/jenkinsci/sloccount-plugin

    Once you have the code, you must build the plugin using maven (mvn package).

    Then drag & drop the sloccount-plugin.hpi into your jenkins/plugins folder.

    Don’t forget to delete the sloccount-plugin.jpi and his related folder.

    This worked for me.

    Rémy.

  14. very great article, it’s really helpful.
    but I got a question, I can’t get the code coverage file in the directory “build/YOUR-PROJECT-NAME.build/Debug-iphonesimulator/YOUR-MAIN-TARGET-NAME.build/Objects” as mentioned, because there’s no project produced in the “DerivedData/..” directory. My project can’t produce ***.app,I don’t know why. (XCode 4.5)

  15. Thank you for the write up. I’m using XCode 4.6dp3 and the default OCUnit for testing

    After setting:
    Test after build to YES

    I get the error:

    RunPlatformUnitTests:81: warning: Skipping tests; the iPhoneSimulator platform does not currently support application-hosted tests (TEST_HOST set)

    Are your instructions for GHUnit? Is using GHUnit required?

  16. I keep encountering this error . Set the class path but still didn’t help .. How did you do it

    Just one comment: I had trouble to make it work and always get a blank report file first (so as few other if you read the comments).

    GreatBlog to

  17. I have the problem that the my Jenkins PMD plugin fails with the following error message:

    org.xml.sax.SAXException: Input stream is not a PMD file.

    I checked the file to find a perfectly good XML file with some findings.

    Anyone encountering the same problem?

  18. @Cris: It seems that you try to run Application Tests. It’s quite tricky to get those running with an automated test environment. There used to be hacks but as of Xcode 4.6 the hack to modify apples apptest runscript won’t work anymore.
    Check this so thread for another approach: http://stackoverflow.com/questions/12557935/xcode-4-5-command-line-unit-testing

  19. For some reason my post didn’t make it through . I keep running into the empty xml Reports running PMD for code duplication. Tried setting the class path but that didn’t help either. Any clues ?

  20. Great article!

    I think I’m “almost there” with the code coverage. I can find all the gcno and gcda local in the i386 folder just like expected. I have tried to copy the gcovr script to that folder and if I run it like this “python gcovr -x > coverage.xml” I will get an xml file which actually contains things.
    But if I try “python gcovr -r . -x > coverage.xml” I get an empty file.

    If I try to add it as a shell script step on Jenkins I always end up with an empty coverage file.
    Any ideas what I might be missing to get the coverage into the coverage.xml file?

  21. I have been using sloccount a lot with Objective-C projects on OSX, never have a problem until recently that I upgraded to OSX 10.9 Mavericks. When I’m trying to run this simple script:

    #!/bin/sh
    sloccount –duplicates –wide –details WeatherApp > Build/sloccount.sc

    I’m getting this:

    /Applications/sloccount/compute_sloc_lang: line 52: c_count: command not found
    Warning! No ‘Total’ line in Models/ansic_outfile.dat.

    The output file has this:

    Creating filelist for Application
    Creating filelist for Controllers
    Creating filelist for Helpers
    Creating filelist for Managers
    Creating filelist for Models
    Creating filelist for Support
    Creating filelist for Views
    Categorizing files.
    Computing results.

    44 objc Application /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Application/AppDelegate.m
    11 objc Application /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Application/AppDelegate.h
    24 objc Controllers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Controllers/CitiesViewController.m
    10 objc Controllers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Controllers/CitiesViewController.h
    74 objc Helpers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Helpers/TranslatorHelper.m
    47 objc Helpers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Helpers/ValidatorHelper.m
    18 objc Helpers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Helpers/ErrorNotificationHelper.h
    21 objc Helpers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Helpers/TranslatorHelper.h
    14 objc Helpers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Helpers/ValidatorHelper.h
    85 objc Managers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Managers/WeatherAPIManager.m
    20 objc Managers /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Managers/WeatherAPIManager.h
    15 objc Support /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Support/main.m
    13 objc Support /Users/ruenzuo/Documents/GitHub/north-american-ironman/WeatherApp/Support/Includes.h

    And Sloccount Plugin for Jenkins is unable to parse it.

    Any thoughts on that?

Leave a comment