EasyMock: Facts and fallacies

There’s been a lot of talking going on on the web lately. About which mocking framework is the best. EasyMock and Mockito are frequently compared. Especially since Mockito is greatly inspired (and is reusing the same mocking code under the hood) by EasyMock with some tweaks in the syntax. Sadly, while reading comparisons, I’ve noticed things that are just plainly untrue. This article is meant to rectify this and give you my opinion on both frameworks. Being EasyMock lead developer, I’m of course biased. Nevertheless, I believe I did, as best as I could, an honest comparison.

EasyMock doesn’t mock classes by itself

This is false. Since EasyMock 3, standard EasyMock can now mock both classes and interfaces. It used to be true for historical reasons as there were EasyMock and EasyMock Class Extension. The class extension now only exists for backward compatibility reasons (which is an empty shell delegating to EasyMock).

EasyMock code is longer because you need to call replay()

Come on… Like if calling one more method would kill your productivity… And I always thought that it was bringing a nice separation between the preparation phase and actual testing phase. I will agree on one thing though. It used to be annoying to have to replay every single mock (replay(mock1, mock2, mock3)). That’s why EasyMockSupport was made. Using it will now require you to call replayAll() and that’s it.

Beside that, the original statement is a bit fallacious. Mockito requires you to add a bunch of verify clauses that would have already been included in the EasyMock expect calls. Here’s an example*:

EasyMock (using EasyMockSupport)

List mock = createNiceMock(List.class);

// My business code needs these
expect(mock.get(0)).andStubReturn("one");
expect(mock.get(1)).andStubReturn("two");
mock.clear();

replayAll();

// Actual business code
someCodeThatInteractsWithMock(mock);

// Make sure everything was call as expected
verifyAll();

Mockito

List mock = mock(List.class);

// My business code needs these
when(mock.get(0)).thenReturn("one");
when(mock.get(1)).thenReturn("two");

// Actual business code
someCodeThatInteractsWithMock(mock);

// Make sure everything was call as expected
verify(mock).get(0);
verify(mock).get(1);
verify(mock).clear();

You cannot “spy” with EasyMock

Mockito has a nice feature called spying. You create a real object (not a mock) and spy it. It basically means that the object will behave as usual but that all the method calls will be recorded allowing to verify them after. Here’s an example:

   List list = new LinkedList(); // real object
   List spy = spy(list); // spy wrapper

   //optionally, you can stub out some methods:
   when(spy.size()).thenReturn(100);

   //this method call is spied
   spy.add("one");

   // since a real add was done, get(0) will return "one"
   assertEquals("one", spy.get(0));

   //size() method was stubbed and to return 100
   assertEquals(100, spy.size());

   //optionally, you can verify that add was called as expected
   verify(spy).add("one");

In EasyMock there’s no spying per se. However, I tried to think about the use-cases and it brought me to the conclusion that by using a subset of EasyMock capture, partial mocking and delegation features, you should be able to cover them all. It might be more cumbersome though. But I think the use-cases are quite rare. I’m not against adding a spy feature if proved wrong.

About the example above, we need to think about the purpose of this test. Let’s say that in this case we have some old code for which we want to make sure the add method is called with the right parameter but we want to keep its usual behavior. This is quite rare but could happen. We would do:

    // Real object
    List real = new LinkedList();
    // create the mock
    List list = createMock(List.class);

    // spy the parameter and perform the normal behavior
    Capture c = new Capture();
    expect(list.add(capture(c))).andDelegateTo(real);

    replayAll();

    // the actual test
    list.add("one");

    // get will return what was expected
    assertEquals("one", real.get(0));

    // check the capture (will throw an error if nothing was captured)
    assertEquals("one", c.getValue());

    // this is unnecessary since checking the capture is enough
    verifyAll();

Better void handling in Mockito

The problem with void method is that they do not return anything. So you cannot do expect(myVoidMethod()) since it won’t compile. Both EasyMock and Mockito won’t require you to “expect” anything since you most of the time don’t need to return anything anyway. You will just do

 mock.myVoid();

It’s wrong to think that you need to call expectLastCall() with EasyMock. However, it can be done just to make it clear in your code that you are recording. There’s also no need to call once() since EasyMock expects a single call by default.

   // this is enough by itself
   mock.myVoid();
  // no need to do any of this
  expectLastCall().once();

The main difference is in how you “return” something from a void method. Like throwing an exception. Both frameworks have a special syntax for that. Mockito will use

doThrow(new RuntimeException()).when(mockedList).clear();

which is a kind of backward from the usual syntax. EasyMock will instead do

mockedList.clear();
expectLastCall().andThrow(new RuntimeException());

One line longer but keeps the forward syntax (the return is after the call).

Obviously, it amounts to the same thing, as we say in French, “blanc bonnet et bonnet blanc”. And both frameworks are forced to do so for technical reasons.

Mockito errors are clearer

This used to be true. A lot was done in EasyMock 3 to improve error messages. I hope I’ve now filled the gap.

EasyMock breaks tests more often

Yes. That’s the way it should be. That’s my point of view at least. It’s really one of the main differences. With EasyMock, you will be required to expect everything. So your tests will break as soon as you change something. With Mockito, you will verify what you want and only these verifications can break.

What’s the difference? EasyMock forces you to expect everything. Your tests will then break and you will be forced to go back to each test and ask yourself “Hum, was it suppose to break?”. Mockito won’t force you to expect anything. So the test will verify only what the original developer thought would be relevant. But, then, working on some other part of the code, you won’t know a verification is in fact missing. You won’t be forced to have a look at old tests. All tests will stay green and nasty bugs will sneak in. To me, that’s the main difference. A trade-off between a quicker test to write and a test testing the unexpected. Let’s look at an example demonstrating this.

Mockito

 //mock creation
 List mockedList = mock(List.class);

 // call the mock in some business code
 mockedList.add("one"); // the return value isn't used nor specified
 // mockedList.clear(); // adding this call in the business code WON'T break the test

 //verification that add() was called with the right parameter
 verify(mockedList).add("one");

EasyMock

 //mock creation
 List mockedList = createMock(List.class);
 // We are forced to return a meaningful value
 expect(mockedList.add("one").andReturn(true);
 replayAll();

 // call the mock in some business code
 mockedList.add("one"); // the return value isn't used by was specified
 // mockedList.clear(); // adding this call in the business code will break the test

 //verification that add() was called with the right parameter
 // and that no other methods were called
 verifyAll();

EasyMock example is testing what was intended but it is also doing two things behind the scene.

  1. Making sure the mock is accurately mocking a List by forcing to return a value
  2. Preventing the test from working if other calls to the mocks are made by the business code. This will force you to wonder if breaking this test was expected by your refactoring.

Still, I’ve give you one trick to prevent your tests to break all over the place. Think about what is really important. Is it really important that this getter is called only once by the tested method? No? Then record it as a stub. Do I care about this passed parameter? No? Just use the anyObject matcher. That seems silly but that’s how you get resilient tests. It’s a common mistake. Done usually while fighting to make the test work with the inner implementation of the method instead of thinking about what the tested method should really do (do I hear TDD?)

What do I think of those Mockito guys?

Some of you might think I just hate them. They stole my code and my users. Fortunately, while in other spheres people keep suing each other these days, that’s not the way it goes in the open source world. Everybody is using each other’s ideas and tries to improve them for the benefit of all. The ethic however dictates us to notice each other when we do so. Which they did. I was probably one of the first person aware of Mockito existence and they are not hiding that a lot of code and ideas are coming from EasyMock. These guys did some nice things that made me think. And they have a nice logo. Or course, it ain’t a unicorn world. For sure there is competition. But that’s what pushes us to evolve.

Last Thoughts

I must confess that EasyMock development has been slower than I would have liked these last two years. Still, EasyMock 3 has brought a lot of improvements. EasyMockSupport, new class mocking API, improved capture, delegation, etc. Also, the merge between EasyMock and EasyMock Class Extension should have been done a long time ago. However, lots of improvements are in the pipe. I’m even looking for help! So feel free to contact me (more on that soon on the EasyMock mailing list. In the future, I would like a better integration with testing frameworks (JUnit, TestNG) and with Hamcrest (coming soon in EasyMock 3.1). Also, some power features to allow crossing the last mocking boundaries like final and private methods would be nice too. And still reduce the boilerplate code and so on. I’m also not closed to add a Mockito kinda API. I still don’t think it’s the best approach to get long term quality but heck… we used to think only interfaces should be mocked…

*: Part of the examples are modified examples taken from the Mockito website

10 commentsfor “EasyMock: Facts and fallacies”

  1. Hi Henri, My name is Hamlet and I’m the author of said fallacious article about the comparison. First off, I have no experience with EasyMock 3.0. In fact, I had no idea it was even out. Sorry that my article unfairly compared Mockito to an older version of EasyMock. However, in general, I do stand by my main points. Mockito is an incremental improvement in some areas over EasyMock and in some areas they are equals. I’m unaware of any areas where EasyMock is superior, but would love to see your reply on that (I am not claiming there are none). Anyway, my three main points stand: 1) an over-reliance on the verify() method means your code has a lot of side effects. A better design is to have side effects limited to only a few places in your system. 2) a natural ordering of test code is arrange-act-assert, and getting away from this ordering obscures the essence of test methods. Making expectations serve as asserts at the top of a test method means your method becomes arrange-assert-act-assert which is more confusing. 3) The best way to organize/manage mock code and replay state is to define factory objects for types and try to hide the replay state and MockFramework as much as possible from the test methods.

    It is #3 that I am most interested in a discussion of, the other seem smaller in comparison.

  2. Hi Hamlet,

    Discussion is always good, the mockito newsgroup is always open for discussion, so it is for many other places :)

    By the way I was putting a bit of emphasis on BDD on the french article comment, and I believe that BDD comes pretty well along AAA.

    And I must say that I agree with your 3 points, however context is king. So you won’t write test the same way for every possible test.

  3. @Hamlet: Thank you for your message. First, I must clarify that I haven’t said your article was fallacious. That said, the purpose of my article was mainly to rectify wrong ideas, in your article and in general on the Net. Most of them are indeed coming from a comparison between Mockito and EasyMock 2 since too many people are sadly unaware of EasyMock 3.

    About your two first points. They are true. EasyMock relies on verify a lot and the arrange part includes an assert part. If both are a good or bad thing differs according to who you ask about it.

    About the third one, I’m not sure I getting it right but I do agree that the mocking code should have as less boilerplate code as possible. Most of EasyMock developments will now be on this since I will agree Mockito is a bit ahead.

  4. We are using easymock 2.3 together with Powermock to be able to mock private and static methods. This is sometimes needed when writing tests for legacy code.
    Any plans on incorporating features like those found in Powermock into Easymock ?

  5. verifyAll and replayAll are some really nice features I always desire for but again I just realize developers will be forced to extend EasyMockSupport in order to use them, which is a big turn-off.

  6. I have used both EasyMock 3.0 and Mockito a great deal. I do hope that both projects maintain their current direction. Although I often use Mockito for hand-written mock objects because the default behavior is more permissive, I find that it has a tendency to make me lazy in the sense that I can get a test running and not actually verify a lot of behavior. I find the default behavior of EasyMock is much more strict, so if I have a unit test running with EasyMock I have a good deal of confidence that I have the behavior locked down with the test. I wrote a “mock recorder” which generates mock objects to reproduce method calls on an object tree at test time using EasyMock. I could not figure out how to do the same thing with Mockito.

  7. @Georges: It’s on the roadmap but I’m not sure it will be done in the end. Powermock did a good job at it and maybe these powerful features should stay separated from EasyMock… or not… Haven’t decided yet.

    @Larry Fang: The easiest is to have a base class for your tests.

    @Matt: I totally agree with you

  8. This post is to follow up… and also a shameless plug. I recently added documentation and tutorials for the Java Test Object Recorder (please, someone help me come up with a better name!) at http://jtor.sourceforge.net/. The project uses EasyMock 3.0 to record the behavior of a live object and automatically generate mocks. I re-read the article above and you could think of this project as spying on an entire tree of objects and recording the results for playback in a unit test. I use the technique to record database connections, web service calls, etc. and convert them into unit tests.

  9. Here is a users perspective.

    I’ve used EasyMock and Mockito both and I find Mockito much more intuitive and easy to use. I find stubbing in Mockito to be easier than with EasyMock and find the replay() idea confusing and fluff that I dont care. I also dont care about the verify() in either frameworks and would rely on JUnit assertions instead. When it comes to mocking big data objects EasyMock is way too complicated.

    The last example you gave is simply wrong IMO and something you made up to make a point. How I would write this with Mockito is as follows.

    //mock creation
    List mockedList = mock(List.class);

    // call the mock in some business code
    mockedList.add(“one”); // the return value isn’t used nor specified
    // mockedList.clear(); // adding this call in the business code WON’T break the test

    //verification that add() was called with the right parameter
    //verify(mockedList).add(“one”);

    //Using JUnits
    assertEquals(1, list.size());
    assertEquals(“one”, list.get(0));

    Mockito should be re-named EasyMock and EasyMock be renamed UnintutiveMock!

  10. @Mahesh: An EasyMock nice mock is basically a Mockito mock. So stubbing is quite the same. Why do you think the last example is wrong?

Leave a comment