Releasing MObject

I've just released MObject. It's a tiny library to construct objects and object trees for mocking and stubbing in Python.

Go check it out!

Immutability in Python?

Where I dwell into the immutability concepts available in Python and make a shameless plug for Pyrsistent.

DISCLAIMER:

Some parts of this discussion is based on how CPython works. PyPy, Jython, IronPython and other Python implementations may behave differently in these areas.

What is immutability?

Lets keep it simple: If a thing is immutable it cannot be changed.

Why would you want immutability in programming?

These are the reasons I find most important:

• Cognitive offload for the developer. No need to ever worry about the content of an object once it has been created. It will remain the same until it is destroyed. You can pass it around confidently knowing that no harm will ever be done to it.
• Safe and simple invariance checking. It is only necessary to check the validity of the object state at construction time since there it is no way to change the object thereafter.
• Safe and fast sharing, within and between threads. No need to make use of locks or similar constructs to synchronized access to an object that can never change.
• Safe and fast reuse. If more objects with the same properties are required just use the same object over and over again. No need to ever restore anything to it's "original" state.
• Efficiency. Can make certain classes of problems more efficient, e.g. some types of comparisons. Makes object caching less error prone.

Why would you not want immutability in programming?

Of course there are also reasons that you might want mutable objects in your programming. These are some reasons I can find for that, and I think that they are all interrelated:

• Efficiency. It is often less efficient to create new objects than changing the value of existing ones. Both with regards to memory consumption and CPU cycles.
• Habits. It requires changes to programming style that may take some getting used to.
• Language support. Some languages (such as Clojure) embrace the use of immutability and make it the path of least resistance for the developer while others (such as Python) do not.

Immutability classes, (degrees of immutability?)

Are there really different classes of immutability? No, either it is possible to change something or not. End of discussion? Not really, for the continued discussion around immutability in Python we need to define a couple different classes.

• By convention.  The programmer who wrote the code has expressed the intention of immutability in some way. For example by comments or variable naming. There is nothing technically stopping anyone from mutating the object. Everything is fully flexible, nothing is known. This is what we love in Python, right?
• Apparent. The object appears to be immutable from the Python perspective. There are still moving things that are moving under the hood mainly related to reference counting / garbage collection in the CPython interpreter. There are also ways of mutating these objects from Python but you would have to use libraries such ctypes to achieve this.
• True immutability. The real thing, nothing moves once the object has been created. This is impossible in CPython.

The good thing is that in CPython there is a thing called the Global Interpreter Lock (GIL) which promotes apparently immutable objects to (almost*) truly immutable objects. It does this by excluding more than one thread from executing concurrently in the interpreter. This is why the GIL is also a bad thing since it makes certain types of tasks harder to parallelize.

* It is still possible to poke on the objects if you resign to writing C code that interacts with them.

The state of immutability in Python

There is no such thing as a true constant reference in Python. There are certain tricks that you can apply to mimic constants but these are crude workarounds.

There are  a  bunch of object types though that fall in the class of apparent immutability and hence are promoted to truly immutable objects in python.

Basic types

Strings and numbers are immutable. There’s no way to change a letter in the middle of a string without creating a new string. There’s no way to change the value of a number without creating a new number. CPython utilizes this fact to cache small numbers so that their underlying object can be reused.

>>> x = 17

>>> y = 17

>>> x is y

True

>>> y = 1700

>>> x = 1700

>>> x is y

False

Collections

Python comes with a couple of immutable collections, the well known tuple and the sligthly less known frozenset which are basically lists and sets but with all mutating operations removed.

There’s also the namedtuple which is a factory function to create classes inheriting from a tuple with a predifined, and fixed, set of members that cannot be changed once an object of the type has been instantiated. This is a very nice feature that has been around since Python 2.6.

These structures gives you immutability goodies such as hashability and to some extent also reuse. For example:

>>> tuple() is tuple()

True

>>> list() is list()

False

The problem with the above structures is that they are awkward to work with if you wish to actually manipulate data in your program. For example creating a new tuple based on an existing one but with an updated element at position 2 would require something like the below:

>>> t1 = (1, 2, 3, 4, 5)

>>> t2 = t1[:2] + tuple(17) + t1[3:]

# or

>>> l1 = list(t1)

>>> l1[2] = 17

>>> t2 = tuple(l1)

Not only is this a pain to write (and read) it’s also inefficient since we have to create several objects along the way that are immediately thrown away. Furthermore all references to items in t1 have to be copied to t2 even though only one reference was updated.

Enter Pyrsistent… Today v0.5.0 was released.

I love sets!

Yesterday was Global Day of Coderetreat and so we spent a couple of sessions at work to pound on solving Conways Game of Life with different approaches and/or languages for each session. Just like last time I was to a coderetreat we never really managed to finish the problem in any of the sessions (and that is not the point of the sessions really). But, during the day there was an idea of an implementation that emerged in my head and that I really wanted to try out. So I spent an hour yesterday evening getting it done. The result (in Python) can be found here https://gist.github.com/tobgu/7970432. I think it's quite nice, compact but still not entirely unreadable. ;-)

Once again I'm struck by how well sets are suited to solving certain types of problems.

Pyrsistent released

Just a short note. I've recently released my first package ever to PyPi. I call it Pyrsistent and it contains a vector, a map and a set that are all persistent. Persistent in this case means that the data structures are immutable. Changing the content of the data structure means creating a new object which contains the requested changes, the original object remains untouched

To not waste excessive memory structural sharing is used between the new and previous data structure. 

You can find it here https://pypi.python.org/pypi/pyrsistent/. The docs are available here http://pyrsistent.readthedocs.org/en/latest/. 

Game of Life in Python

I recently took a couple of hours to work through the Game Of Life kata trying to do it strictly TDD. I've had a couple of short goes on the kata before during a code retreat but we were never able to finish it during the 45 minute sessions.

My first implementation is shown below together with the tests developed n parallell. This is a fixed board implementation that wraps in both north<->south and east<->west direction. Originally I did it as a non-wrapping implementation but changing it to a wrapping one was only a matter of writing a couple o extra tests and updating the cell_at() implementation.
This implementation also holds a representation for all cells, both dead and live ones. The upside is that this makes it very straight forward to calculate the new state of all cells. The two large drawbacks is that the space and number of calculations needed for is proportional to the board size rather than the number of live cells. This is no problem for a board of reasonable size but may be if the size is large.

# gameoflife.py
"""
Simple Game of Life implementation for a fixed size board that
wraps at the edges. All cells, both alive and dead are part of
the representation.
"""
ALIVE = 1
DEAD  = 0

class GameAnalyzer(object):
    directions = ((-1,-1), (0,-1), (1,-1),
                  (-1, 0),         (1, 0),
                  (-1, 1), (0, 1), (1, 1)) 
    
    def __init__(self, board):
        self.board = board
    
    def cell_at(self, x, y):
        y_wrapped = y % len(self.board)
        x_wrapped = x % len(self.board[y_wrapped])
        return self.board[y_wrapped][x_wrapped]
    
    def neighbour_count(self, x, y):
        return sum([self.cell_at(x + dx, y + dy) for dx, dy in self.directions])
        
    def judge_cell(self, x, y):
        rules = (DEAD, DEAD, self.cell_at(x, y), ALIVE, DEAD, DEAD, DEAD, DEAD, DEAD)
        neighbour_count = self.neighbour_count(x, y)
        return rules[neighbour_count]


def evolve(board):
    a = GameAnalyzer(board)
    return [[a.judge_cell(x, y) for x, _ in enumerate(row)] for y, row in enumerate(board)]

# gameoflife_test.py
from gameoflife import evolve, GameAnalyzer, ALIVE as x, DEAD as o
from nose.tools import assert_equal

def test_empty_board_remains_empty():
    assert_equal(evolve([[o, o, o],
                         [o, o, o],
                         [o, o, o]]), [[o, o, o],
                                       [o, o, o],
                                       [o, o, o]])

def test_single_cell_dies():
    assert_equal(evolve([[o, o, o],
                         [o, x, o],
                         [o, o, o]]), [[o, o, o],
                                       [o, o, o],
                                       [o, o, o]])

def test_stable_formation_remains():
    assert_equal(evolve([[o, o, o],
                         [o, x, x],
                         [o, x, x]]), [[o, o, o],
                                       [o, x, x],
                                       [o, x, x]])

def test_blinker_blinks():
    assert_equal(evolve([[o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, x, x, x, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o]]), [[o, o, o, o, o],
                                             [o, o, x, o, o],
                                             [o, o, x, o, o],
                                             [o, o, x, o, o],
                                             [o, o, o, o, o]])
    
    assert_equal(evolve([[o, o, o, o, o],
                         [o, o, x, o, o],
                         [o, o, x, o, o],
                         [o, o, x, o, o],
                         [o, o, o, o, o]]), [[o, o, o, o, o],
                                             [o, o, o, o, o],
                                             [o, x, x, x, o],
                                             [o, o, o, o, o],
                                             [o, o, o, o, o]])


def test_board_wraps_around():
    # South -> North
    assert_equal(evolve([[o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, x, x, x, o]]), [[o, o, x, o, o],
                                             [o, o, o, o, o],
                                             [o, o, o, o, o],
                                             [o, o, x, o, o],
                                             [o, o, x, o, o]])

    # North -> South
    assert_equal(evolve([[o, x, x, x, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o],
                         [o, o, o, o, o]]), [[o, o, x, o, o],
                                             [o, o, x, o, o],
                                             [o, o, o, o, o],
                                             [o, o, o, o, o],
                                             [o, o, x, o, o]])

    # East -> West
    assert_equal(evolve([[o, o, o, o, o],
                         [o, o, o, o, x],
                         [o, o, o, o, x],
                         [o, o, o, o, x],
                         [o, o, o, o, o]]), [[o, o, o, o, o],
                                             [o, o, o, o, o],
                                             [x, o, o, x, x],
                                             [o, o, o, o, o],
                                             [o, o, o, o, o]])

    # East -> West
    assert_equal(evolve([[o, o, o, o, o],
                         [x, o, o, o, o],
                         [x, o, o, o, o],
                         [x, o, o, o, o],
                         [o, o, o, o, o]]), [[o, o, o, o, o],
                                             [o, o, o, o, o],
                                             [x, x, o, o, x],
                                             [o, o, o, o, o],
                                             [o, o, o, o, o]])


def check_neighbour_count(board, at, expected):
    (x, y) = at
    a = GameAnalyzer(board)
    assert_equal(a.neighbour_count(x, y), expected)
    
def test_correct_neighbour_count_to_the_right():
    check_neighbour_count([[o, o, o],
                           [o, o, x],
                           [o, o, o]], (1, 1), expected=1)

def test_correct_neighbour_count_to_the_right_and_right_down():
    check_neighbour_count([[o, o, o],
                           [o, o, x],
                           [o, o, x]], (1, 1), expected=2)

def test_zero_neighbour_count_with_no_cells():
    check_neighbour_count([[o, o, o],
                           [o, o, o],
                           [o, o, o]], (1, 1), expected=0)

def test_correct_neighbour_count_with_alive_cell_on_the_board_but_no_neighbours():
    check_neighbour_count([[o, o, o],
                           [o, x, o],
                           [o, o, o]], (1, 1), expected=0)

def test_correct_neighbour_count_with_alive_cells_in_all_neighbour_positions():
    check_neighbour_count([[x, x, x],
                           [x, x, x],
                           [x, x, x]], (1, 1), expected=8)
    
def test_correct_neighbour_count_for_corner_cell():
    check_neighbour_count([[o, o, o],
                           [o, x, x],
                           [o, x, o]], (2, 2), expected=3)

Representing the whole board makes the tests easy to read (no additional comments needed, both the existing and expected board are all fleshed out), but input data may be on the large size.
Driving the design with tests I experience as hard sometimes. The biggest problem I usually have is that I reach points in the development process were I would like to write a test that would require too much of a development effort in one step to make it pass (staying at red for too long in the red-green-refactoring cycle).

The number of tests for this implementation probably indicate some test redundancy and could probably be reduced. I have not gone over them. These are all the tests I wrote during my "design driving".

One common strategy to deal with this is to start writing tests for lower level functions that you've identified that you need. This is indeed a good way of moving forward. In the code below I took the decision to do this with the GameAnalyzer.neighbour_count() method. What I don't like about this strategy is that it locks your implementation to using certain classes and functions that shouldn't even be exposed in the API. In this example the GameAnalyzer class should probably be considered private and part of the implementation details that the user of this code shouldn't ned to know about.

Writing tests against it partly wrecks this idea I belive since it forces some of the tests in the regression suite to work against code that should not be exposed. It will also force you to keep the GameAnalyser class and the neighbour_count() function unless you want to rewrite the tests.

One solution to this may be to write tests on a low level while driving the implementation and than replace those with tests on an API level once the implementation is ready.
The not so good things with this strategy is, as I see it:

• You have to write the same test twice, but on different levels.
• The second time you write the test you do not have the same confidence that it actually tests what you intended it to test (unless you start removing the parts implementing what you want to test to simulate "test first").

One downside of writing the tests on higher (API) level is that you're likely to have to update alot of tests if you change the underlying functionality. This is will work against the idea of only needing to update one (or at least very few) tests when you update one piece of funtionality.

The answer to all these problems may (in part) be to write small and focused classes that corresponds well to the single responsibility principle and test those individually. Then complement those with a couple of high level integration tests that act as "guiding tests" to drive a top-down design.

For testing I used Nose with Rednose to colorize the output. I also used Nosier to have the tests run directly whenever I saved a file. This was a very nice setup: The command used (standing in the source folder) was:
nosier "nosetests --rednose" .

The second implementation was influenced by the implementation by Emily Bache in this video but I wanted to take the ideas of using sets a step or two further. This implementation uses an infinite board where only live cells are represented as (x, y) tuples. It's more efficient since both the number of calculations and space used is proportional only to the number of live cells.

# gameoflife2.py
"""
Game of Life implementation for an expanding game. Live cells are
represented as x and y tuples stored in a set.
"""
class Game(object):
    directions = ((-1,-1), (0,-1), (1,-1),
                  (-1, 0),         (1, 0),
                  (-1, 1), (0, 1), (1, 1)) 
        
    def __init__(self, live_cells):
        self.live_cells = live_cells
    
    def next_generation(self):
        return set.union(self.survivors(), self.births())
    
    def births(self):
        return set([(x, y) for x, y in self.birth_candidates() 
                    if len(self.live_neighbours(x, y)) == 3])
    
    def birth_candidates(self):
        return set.difference(self.all_neighbours(), self.live_cells)

    def all_neighbours(self):
        neighbours = [self.neighbours(x, y) for x, y in self.live_cells] or [set()]
        return set.union(*neighbours)

    def survivors(self):
        return set([(x, y) for x, y in self.live_cells 
                    if len(self.live_neighbours(x, y)) in (2, 3)])
    
    def live_neighbours(self, x, y):
        return set.intersection(self.live_cells, self.neighbours(x, y))
    
    def neighbours(self, x, y):
        return set([(x + dx, y + dy) for dx, dy in self.directions])
    
def game_generator(initial_state):
    state = initial_state
    while True:
        state = Game(state).next_generation()
        yield state
    

# gameoflife2_test.py
from gameoflife2 import game_generator, Game

def test_game_with_no_live_cells_remains_dead():
    assert game_generator(set()).next() == set()
    
def test_a_single_live_cell_dies():
    game = game_generator(set([(1, 1)]))
    assert game.next() == set()

def test_stable_formation_remains():
    """
    xxo    xxo
    xxo -> xxo
    ooo    ooo
    """
    stable = set([(0, 0), (0, 1), (1, 0), (1, 1)])
    game = game_generator(stable)
    assert game.next() == stable    
    
def test_blinker_blinks():
    """
    ooo    oxo    ooo
    xxx -> oxo -> xxx
    ooo    oxo    ooo
    """
    game = game_generator(set([(0, 1), (1, 1), (2, 1)]))
    assert game.next() == set([(1, 0), (1, 1), (1, 2)])
    assert game.next() == set([(0, 1), (1, 1), (2, 1)])

def test_game_plan_expands():
    """
           oxo
    xxx    oxo
    ooo -> oxo
    ooo    ooo
    """
    game = game_generator(set([(0, 0), (1, 0), (2, 0)]))
    assert game.next() == set([(1,-1), (1, 0), (1, 1)])
    
def test_Game_survivors_live_cell_with_two_and_three_neighbours_survive():
    """
    xxo              xxo
    xxo -survivors-> oox
    oxo              oxo
    """
    state = set([(0, 0), (0, 1), (1, 0), (1, 1), (1, 2)])
    assert Game(state).survivors() == set([(0, 0), (1, 0), (1, 2)])

def test_Game_births_dead_cell_with_three_neighbours_come_alive():
    """
    oxo           ooo
    oxo -births-> xox
    oxo           ooo
    """
    state = set([(1, 0), (1, 1), (1, 2)])
    assert Game(state).births() == set([(0, 1), (2, 1)])

def test_Game_birth_candidates_all_dead_cells_with_with_live_neighbour():
    """
    ooo                     ooo
    ooo -birth_candidates-> xxx
    oxo                     xox
    """
    state = set([(1, 2)])
    assert Game(state).birth_candidates() == set([(0, 1), (1, 1), (2, 1), (0, 2),
                                                  (2, 2), (0, 3), (1, 3), (2, 3)]) 

def test_Game_live_neighbours_counts_correctly():
    state = set([(0, 0), (0, 1), (1, 0)])
    game = Game(state)
    assert game.live_neighbours(0, 0) == set([(0, 1), (1, 0)]) 
    assert game.live_neighbours(1, 1) == set([(0, 0), (0, 1), (1, 0)]) 
    

The tests are a bit clumsier since the input and output is not so easy to read and may require additional comments.

I switched to pytest for this implementation. Still using Nosier to run the tests the command was as follows:
nosier py.test .

In this limited scenarion Nose and Py.Test are rather similar. Writing the tests in PyTest is a bit cleaner I think since you can use the standard assert and still get a nice printout of what caused the assertion to fail. In Nose you must use a special assert_equal() to get the same functionality. No big deal really. It fells like Nose is a tiny bit faster running the tests. I haven't done any measurements though.

Overall this was a nice and teaching session. I might return to the game of life in the future.

Five things that make Java painful (and some remedies)

I figured I'd present a list of the five things that disturbs me the most with Java as a language. Just give me nice standardized solutions to the following issues and I promise not to ask for more... ;-)

I will also present some third party libraries that will remedy some of the pain that these problems cause.

No closures or lambdas

No one on earth has missed the fact that clojures and lambdas are _the_ big thing these days (and have been for quite a while, even C++ has them in the latest version of the standard). Without them many of the nice "functional" (although one can argue about how functional they are) constructs that are present in quite a few modern languages (and old, Lisp for example) are ugly or impossible to implement.
I'm really not asking for that much more than what is already in the language today in the form of anonymous classes toay. It's just that the code gets so damn bloated when using these anonymous classes that you want to rip your eyes out when you look at it a week later.

Hopefully Java 8 will include closures through project lambda and hopefully it will be delivered on time 2013 as planned and hopefully it will be adopted as the industry java standard version really soon thereafter... Too many hopefully in that sentence to be really truthworthy.

Remedy until then

Check out JPropel. It is library that (among other things) can transform static functions into anonymous classes that implements a Function iterface by a simple annotation. This allows you to pass them around as objects. It ain't closures but it reduceds some of the boilerplate needed to pass function objects around. This actually mimics the way lambdas are implemented in Scala quite closely.

Setters and getters

What a pain to write all those setters and getters. Ok, they can be generated by your IDE... But what a pain to read all those setters and getters! Some claim that getters and definitely setters are a design smell. I can agree in some situations but in situations they may actually be required or at least useful. But I don't want to see all that noise they should just be there. Again, check out Scala. Getters can be automatically generated from the constructor parameters or we could just use public members and later replace these with methods as needed without breaking the interface.

As far as I know there is no short term remedy planned for this in Java.

Possible remedy

Project lombok has a couple of annotations that let you annotate the classes and fields for which setters and/or getters should be generated. Based on this lombok will then generate the necessary code. Check out the @Data annotation for example.

Painful declaration of variables

Declaring variables in java is a pain, especially when generics are involved. Up until Java 7 it was (is) a real pain with a previously unseen level of verbosity. Look at this:
final Map<String, List<Integer>> cpuLoad = new HashMap<String, List<Integer>>()

In Java 7 this was somewhat improved:

final Map<String, List<Integer>> cpuLoad = new HashMap<>()

So, what do I actually want? Pretty much what's in Scala today actually. A more powerful type inference system. We all know the compiler knows the types. Why can't it just infere them for us and let us focus on other things?
The other thing that annoys me about java reference declarations is the extra final keyword needed to make a variable immutable. I'm a big fan of immutable data. It makes the code easier to reason about and more maintainable. But even I cannot force myself to make the input parameters and local variables in a functions final because of all the clutter.
Also here I think Scala is a language to look at. Mutable variables are declared using the var keyword. Immutable "variables" are declared using the val keyword. The exact same number of letters!

A pain killer

At least when using Java <= 1.6 is the Guava library that removes some of the boilerplate in declaring generic collections. To my knowledge there is no  ongoing effort to make the type inference system in Java any better than it currently is. Guava is worth investigating for other nice features as well!

No persistent collections

No persitent collections exist in the standard java API. By persistent I mean collections that are immutable but supports efficient "modification" by returning a copy of itself with the applied changes. Since structural sharing is used most of the data stored in the new and old can often be shared between the two collections hence unnecessary copying is reduced.
The persistent collections bring with them all the good of immutable data (easier to reason about, thread safe, ...). They are used by default in JVM languages such as Clojure and Scala.

Librabries do exist that provide these type of collections for Java as well, and this may not be considered a big deal by some. The fact that they are not part of the standard library will limit their use though and most of the Java code written today does not utilize these lovely collections (I dare say) even if, in most cases, it would be benificial.

I don't know of any initiative to introduce a standardized alternative for persistent collections.

An option

The pcollections library is a nice player. Play with it!

No for comprehension

The lack of for comprehensions (as implemented in Python or Scala for example) is a shame. Although the for comprehension is just a syntactic sugar built with higher order functions such as map and filter it's a very sweet sugar that can lead to some beautiful solutions. It also makes it possible to introduce constructs close to those of Microsofts .NET LINQ to dig for data in collections. I don't know if there will ever be anything similar in Java but the introdution of lambdas and higher order functions will allow java to move one step closer to it.

An alternative

JPropel (light) has a really nice alternative here as well. Check it out!

Concluding thoughts

So in the end I think this was mostly a rant showing some of my frustration on the (non moving) state of Java. To me it seems like the biggest problem in the java development process is that there's to much focus on keeping things backward compatible. Of course you want your old Java 2 code to compile and run on a Java 8 JVM but do we really have to retro fit the massive API to make use of new features in the language? Can't it just rest in peace? The collections API for example, how about leaving that behind (maintenance should be minimal given it's maturity) and go for a new more up to date collections API?

In the case were interwork between old and new collections (or other APIs) are actually needed some adapter code could be introduced.

So, seems like there are a bunch of other languages running on the JVM that already include most of the stuff I'm asking for. Why not use them? Well, maybe I would in my day to day job but language mixes (no, I'm currently not on a green field project) often come with their own problems. Languages such as Scala also come with their own weak points (not always directly related to the language). IDE support that still lags behind compared to Java (although improving), slow compilation, some overly academic features, ...

Still, I would happily try out an alternative JVM language in a real world project! Given todays Java development pace it might just be the only way out.

Using Sonar to monitor a JAspect-based application

I started a small project a while ago to learn some more about AspectJ and aspect oriented programming in Java. Some days ago Recently I decided to try to apply Sonar on it to get some nice quality measures such as code complexity, test coverage and static code analysis to detect potential bugs.

This is a small writeup of my experiences from that experiment.

Installation

Installation was quite straight forward. I fetched the latest version Sonarsource and followed the installation instructions. I've opted not to install Sonar as a background service yet but instead start it manually whenever I need it. If I later find that I use it alot then installing it as a service should be as simple as running a batch script.

To run Sonar you must install a SQL database in which the analysis results are stored. I opted for the community version of MySQL since there was an example on the sonarsource web site for that particular DB. As I wanted to get started quickly that was my choice. In hindsight I don't think it would have been any more complicated to use another DB.

Installation of MySQL was equally straight forward. I just ran the supplied installer using the default installation.

To get Sonar working with MySQL was a simple two step procedure described below. I will do it only briefly as the installation instructions provided on the Sonar website are very good.

Create DB and sonar user in MySQL

Doing this was as simple as running the following lines (found through the installation instructions on the Sonar web page) in the mysql command shell (mysql -u root -p):

CREATE DATABASE sonar CHARACTER SET utf8 COLLATE utf8_general_ci;

CREATE USER 'sonar' IDENTIFIED BY 'sonar';

GRANT ALL ON sonar.* TO 'sonar'@'%' IDENTIFIED BY 'sonar';

GRANT ALL ON sonar.* TO 'sonar'@'localhost' IDENTIFIED BY 'sonar';

FLUSH PRIVILEGES;

Configure Sonar

The sonar configuration file must be updated with parameters to connect to the DB and port and address conlfiguration. It is found under <SONAR INSTALLATION DIR>/conf/sonar.properties.

# Listen host/port and context path (for example / or /sonar). Default values are 0.0.0.0:9000/.

sonar.web.host:                           0.0.0.0

sonar.web.port:                           9000

sonar.web.context:                        /

.

.

.

#----- Credentials

# Permissions to create tables and indexes must be granted to JDBC user.

# The schema must be created first.

sonar.jdbc.username:                       sonar

sonar.jdbc.password:                       sonar

.

.

.

#----- MySQL 5.x/6.x

# Comment the embedded database and uncomment the following line to use MySQL

sonar.jdbc.url:jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true

# Optional properties

sonar.jdbc.driverClassName:                com.mysql.jdbc.Driver

Setting up the project for sonar analyze

My initial plan was to use the Sonar Runner which is the recommended way to run a sonar analyze according to the Sonar website. Doing this proved to be easy. I just followed the instructions found here. Running the analyze was equally easy, see the instructions here. After running the analyze the results were browsable at the URL and port specified above (e.g. http://localhost:9000).

There was one cavaet using the Sonar runner though. It doesn't execute any code and can hence not be used for execution of automated tests and test coverage analysis. For this another method had to be used. The recommended approach is to use targets adjusted for your build system. Both Maven plugins and Ant tasks are available to run the analyze. For this particular project I use Maven which made the choice obvious.

Using a maven plugin to analyze the project

I'll start by dropping the entire POM here and will then discuss the details of it.

<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xmlns="http://maven.apache.org/POM/4.0.0"
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
   <modelVersion>4.0.0</modelVersion>
   <groupId>datalogger</groupId>
   <artifactId>DataLogger</artifactId>
   <packaging>jar</packaging>
   <version>1.0-SNAPSHOT</version>
   <name>DataLogger</name>
   <url>http://maven.apache.org</url>
   <build>
      <plugins>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-surefire-plugin</artifactId>
            <version>2.7.2</version>
            <configuration>
               <test>**/*.java</test>
            </configuration>
         </plugin>
         <plugin>
            <groupId>org.jacoco</groupId>
            <artifactId>jacoco-maven-plugin</artifactId>
            <version>0.5.10.201208310627</version>
            <executions>
               <execution>
                  <id>jacoco-initialize</id>
                  <goals>
                     <goal>prepare-agent</goal>
                  </goals>
               </execution>
            </executions>
         </plugin>
         <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>aspectj-maven-plugin</artifactId>
            <version>1.4</version>
            <executions>
               <execution>
                  <goals>
                     <goal>compile</goal>
                     <!-- use this goal to weave all your main classes -->
                     <goal>test-compile</goal>
                     <!-- use this goal to weave all your test classes -->
                  </goals>
               </execution>
            </executions>
            <configuration>
               <complianceLevel>1.6</complianceLevel>
            </configuration>
         </plugin>
         <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>sonar-maven-plugin</artifactId>
            <version>2.0</version>
         </plugin>
         <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.7</version>
            <executions>
               <execution>
                  <phase>package</phase>
                  <goals>
                     <goal>shade</goal>
                  </goals>
                  <configuration>
                     <artifactSet>
                        <excludes>
                           <exclude>junit:junit</exclude>
                           <exclude>org.aspectj:aspectjrt</exclude>
                           <exclude>org.slf4j:slf4j-simple</exclude>
                        </excludes>
                     </artifactSet>
                  </configuration>
               </execution>
            </executions>
         </plugin>
      </plugins>
   </build>

   <properties>
      <coverage.reports.dir>${basedir}/target/coverage-reports</coverage.reports.dir>
      <sonar.core.codeCoveragePlugin>jacoco</sonar.core.codeCoveragePlugin>
      <sonar.dynamicAnalysis>reuseReports</sonar.dynamicAnalysis>
   </properties>
   <dependencies>
      <dependency>
         <groupId>org.aspectj</groupId>
         <artifactId>aspectjrt</artifactId>
         <version>1.6.11</version>
      </dependency>
      <dependency>
         <groupId>junit</groupId>
         <artifactId>junit</artifactId>
         <version>4.10</version>
      </dependency>
      <dependency>
         <groupId>com.google.code.gson</groupId>
         <artifactId>gson</artifactId>
         <version>1.7.1</version>
      </dependency>
      <dependency>
         <groupId>org.slf4j</groupId>
         <artifactId>slf4j-api</artifactId>
         <version>1.6.4</version>
      </dependency>
      <dependency>
         <groupId>org.slf4j</groupId>
         <artifactId>slf4j-simple</artifactId>
         <version>1.6.4</version>
      </dependency>
   </dependencies>
</project>

There are four plugins of interest here. The surefire plugin which is used to execute the test cases. The jacoco plugin which is used to instrument the classes for coverage measurement (using a java agent). The sonar plugin which performs the actual analyse and the aspectj plugin which performs compile time weaving of my aspects.
I initially had some problems getting any coverage reports since the code I wanted to test was actually the aspects and they weren't woven into the code the way I wanted. The problem was that I never managed to get the load time weaving that I initially used (using a java agent) to work together with the code coverage agent of jacoco. Switching to compile time veawing using the AspectJ plugin did the trick.
Also, to get the AspectJ plugin to work properly I had to explicitly set the compliance level to 1.6.

With the above configuration and pom I now have full blown quality measures of my project without having to set up and configure every tool individually.