Acceptance testing with Yatspec

Working at Sky I’ve learnt a lot of things. One of the most interesting ones is writing acceptance tests with Yatspec. Yatspec has a role similar to Concordion and Fit in the sense that it allows developer to write automated acceptance tests that generate html files with the output.

There are two things I like about Yatspec. The first one is that it generates the sentences of the acceptance tests from the code, making it easy to keep test code and html output in sync. The second thing is how the output logs nicely all the interactions in the system, providing a low-level documentation of the API for the application.

In the previous post about embedded Jetty we created a basic server. In this post we will set up a minimalistic Yatspec test suite and use it to acceptance test a feature in our server.

The skeleton

<dependency>
    <groupId>com.googlecode.yatspec</groupId>
    <artifactId>yatspec</artifactId>
    <version>1.23</version>
    <scope>test</scope>
</dependency>

 

import com.googlecode.yatspec.junit.SpecRunner;
import org.junit.Test;
import org.junit.runner.RunWith;

/**
 * Created by Omar on 20/07/16.
 */
@RunWith(SpecRunner.class)
public class AcceptanceTest {

    @Test
    public void shouldShowASimpleOutput(){
        givenEverythingWillGoWell();
        whenWeDoSomething();
        thenTheResponseIsOk();
    }

   @Test
    public void shouldFail(){
        givenEverythingWillGoWell();
        whenSomethingFails();
    }

    private void whenSomethingFails() {  fail(); }

    private void givenEverythingWillGoWell() { }
    private void whenWeDoSomething() {}
    private void thenTheResponseIsOk() {}
}

This is not really testing anything, but it is enough to see a Yatspec output. Running the tests prints the following on the console:

Yatspec output:
/tmp/AcceptanceTest.html

yatspec basic test

We can see that it autogenerates an index, highlights the “Given” “When” “Then” words and tells me which tests passed and which didn’t. Let’s do something more interesting and make it actually test our app.

Repo url: https://github.com/obprado/EmbeddedJetty/tree/44284b5f38d412b192be8239f62f773d0967dfe4

Test against a running server

 

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.2</version>
            <scope>test</scope>
        </dependency>

 

&amp;amp;amp;amp;amp;amp;lt;/pre&amp;amp;amp;amp;amp;amp;gt;
&amp;amp;amp;amp;amp;amp;lt;pre&amp;amp;amp;amp;amp;amp;gt;import com.googlecode.yatspec.junit.SpecRunner;
import org.apache.http.HttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;
import org.apache.http.util.EntityUtils;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;
import org.junit.runner.RunWith;

import java.io.IOException;

import static org.assertj.core.api.Assertions.assertThat;

@RunWith(SpecRunner.class)
public class AcceptanceTest {

    private ExampleApp exampleApp = new ExampleApp();
    private HttpResponse response;

    @Before
    public void setUp() throws Exception {
        exampleApp.run();
    }

    @After
    public void tearDown() throws Exception {
        exampleApp.stop();
    }

    @Test
    public void shouldReturnHelloWorld() throws Exception {
        whenWeMakeARequestTo(&amp;amp;amp;amp;amp;amp;quot;http://localhost:8080/hello&amp;amp;amp;amp;amp;amp;quot;);
        thenTheResponseCodeIs200AndTheBodyIs(&amp;amp;amp;amp;amp;amp;quot;Hello from a servlet!!!!&amp;amp;amp;amp;amp;amp;quot;);
    }

    @Test
    public void shouldFail() throws Exception {
        whenWeMakeARequestTo(&amp;amp;amp;amp;amp;amp;quot;http://localhost:8080/a/bad/url&amp;amp;amp;amp;amp;amp;quot;);
        thenItReturnsAStatusCodeOf(404);
    }

    private void thenItReturnsAStatusCodeOf(int expected) {
        assertThat(response.getStatusLine().getStatusCode()).isEqualTo(expected);
    }

    private void whenWeMakeARequestTo(String uri) throws IOException {
        CloseableHttpClient httpClient = HttpClientBuilder.create().build();
        HttpGet request = new HttpGet(uri);
        response = httpClient.execute(request);
    }

    private void thenTheResponseCodeIs200AndTheBodyIs(String expected) throws IOException {
        thenItReturnsAStatusCodeOf(200);
        assertThat(EntityUtils.toString(response.getEntity())).isEqualTo(expected);
    }
}

 

Now the tests spin up a fresh server, execute an http request against it and then asserts against the response. After a few runs, it seems like they take 450 – 550 ms to run most of the time, with an average close to 500 ms.
test timer1

How much of that is spinning the server and how much executing a test?

We add 10 dummy tests that are replicas of ‘shouldReturnHelloWorld()’ and run them several times in a row.

test timer 2

Most times they take 550 – 650 ms to run, with an average around 600. It seems that 10 extra tests tend to take around 100 ms extra. That suggests that spinning up a server takes around 500 ms and running a test around 10 ms. Of course, a real world test is expected to be slower due to database interactions and interactions with stubbed third parties.

In a more realistic project I got a suite for 23 tests run in 13 seconds, averaging around half a second per test. That is around 1 extra minute to run the suit for every 120 tests you add.

Repo url: https://github.com/obprado/EmbeddedJetty/tree/f32d532577a7147dff784bb65d329e18398beb1b

Capturing interactions with the application

One of the things I like the most about using yatspec for acceptance testing is capturing the web messages exchanged with it, having an automated and executable documentation.

public class AcceptanceTest extends TestState {

...

@Test
public void shouldFail() throws Exception {
 when(weMakeAGetRequestTo("http://localhost:8080/a/bad/url"));
 thenItReturnsAStatusCodeOf(404);
}

private ActionUnderTest weMakeAGetRequestTo(String uri) {
 return (interestingGivens, capturedInputAndOutputs) -> whenWeMakeARequestTo(capturedInputAndOutputs, new HttpGet(uri));
}

private void thenItReturnsAStatusCodeOf(int expected) {
 assertThat(response.getStatusLine().getStatusCode()).isEqualTo(expected);
}

private CapturedInputAndOutputs whenWeMakeARequestTo(CapturedInputAndOutputs capturedInputAndOutputs, HttpGet request) throws IOException {
 capturedInputAndOutputs.add("Request", request);
 response = HttpClientBuilder.create().build().execute(request);
 responseBody = EntityUtils.toString(response.getEntity());
 capturedInputAndOutputs.add("Response", responseBody);
 return capturedInputAndOutputs;
}

Note that the “when()” method comes from the TestState class.  The code will need to be refactored if we want to grow the test suite and the printing of the request and response could be improved, but the essential bit is that any interaction between the test and the application will be logged. This will constitute a living, up to date and executable documentation for our API.testCapturedInputsOutputs

Repo: https://github.com/obprado/EmbeddedJetty/tree/740797fec72baa4d118593ee69e9d4def7b67902

In the next blog post Automatic Generation of Sequence Diagrams with Yatspec we will generate a uml diagram automatically from our acceptance test.

Advertisements

One Comment Add yours

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s