Speed up your Spring Boot integration tests with Testcontainers

Using docker containers for integration testing is a testing practice that makes sense in larger projects where InMemory databases are not feasible but you still want to have consistent, never changing test data so your tests don’t change their results just because you use production data that changes regularly. With the java framework Testcontainers it is very easy to integrate docker containers into your integration tests. In this article I will try to answer the following questions and propose solutions to possible pitfalls with various code examples:

  • What is Testcontainers?
  • How can it be integrated into a spring boot application?
  • Can you use custom images?
  • How do you use the same docker container across multiple test classes?
  • How to use different datasources?

Testcontainers

Testcontainers is a Java framework which makes the use of Docker for integration tests very easy. The integration into an existing tech stack (e.g. Java, Spring Boot, …) can be done via a few simple lines. The framework also reduces the amount of work you have to do due to the large number of included modules, functionalities and automatisms. Let me explain what I mean:

  • modules: For Docker images that are required very often (e.g. MySQL, PostgresSQL, Elasticsearch, RabbitMQ, …) and Docker Compose Testcontainers contain ready-made modules that make working with these images even easier. Take here a look which modules are available.
  • functionalities: First example: Modules using JDBC (e.g. MySQL) support specifically modified JDBC urls. With this url you don’t have to worry about to use the right host and port anymore. Read more about the JDBC url feature here. Second Example: You are not forced to use images from docker hub or a different repository. With Testcontainers you can create images on the fly. Read more about how you can creating images on-the-fly here.
  • automatisms: Testcontainers automatically takes care of creating, starting, stopping and deleting the containers. Each freshly started container gets a random free port which avoids port clashes from the get go and containers that are no longer used are automatically deleted. Read more about the lifecycle control here.

Unfortunately even with Testcontainers your integration tests will not be faster than unit tests or other integration tests with an InMemory database.

But regardless of this, Docker based integration tests with Testcontainers, especially in comparison to other frameworks and workarounds that allow the use of Docker containers for integration tests, are still much easier to use and much faster to execute.

The example application

Our example project is a small application with which book titles can be saved and displayed. The project is implemented with JDK11, Spring Boot (Dependencies: Spring Web, Spring Data JPA, MySQL Driver), Gradle, JUnit5 and Testcontainers. There is a small entity class Book, a BookController class, a BookService class and a BookRepository interface for the books. There is also an application.yml. Download all data from the example here.

@Entity
public class Book {

    @Id
    @GeneratedValue(strategy = GenerationType.AUTO)
    private Long id;

    private String title;

    public Long getId() {
        return id;
    }

    public void setId(final Long id) {
        this.id = id;
    }

    public String getTitle() {
        return title;
    }

    public void setTitle(final String title) {
        this.title = title;
    }

}

Sample 1: Book.java

@Controller
@RequestMapping(path = "/book")
public class BookController {

    private final BookService bookService;

    public BookController(final BookService bookService) {
        this.bookService = bookService;
    }

    @PostMapping("/add")
    public @ResponseBody
    String addBook(@RequestParam String title) {
        bookService.add(title);

        return "The book with the title '" + title + "' has been saved.";
    }

    @GetMapping("/all")
    public @ResponseBody
    Iterable<Book> getAllBooks() {
        return bookService.findAll();
    }

}

Sample 2: BookController.java

@Service
public class BookService {

    private final BookRepository bookRepository;

    public BookService(final BookRepository bookRepository) {
        this.bookRepository = bookRepository;
    }

    public void add(final String title) {
        final var book = new Book();
        book.setTitle(title);
        bookRepository.save(book);
    }

    public Iterable<Book> findAll() {
        return bookRepository.findAll();
    }

}

Sample 3: BookService.java

public interface BookRepository extends CrudRepository<Book, Long> {

}

Sample 4: BookRepository.java

spring:
  jpa:
    hibernate:
      ddl-auto: none
  datasource:
    url: jdbc:mysql://${MYSQL_HOST:localhost}:3306/db_example
    username: user
    password: my-secret
    driver-class-name: com.mysql.cj.jdbc.Driver

Sample 5: application.yml

When the application is started a book title can be added and queried.

curl localhost:8080/book/add -d title=bible
curl localhost:8080/book/all

First test with Testcontainers

Now we want to write our first test with Testcontainers. For this we have to add the following dependencies in our build.gradle:

testImplementation "org.testcontainers:junit-jupiter:1.15.3"
testImplementation "org.testcontainers:mysql:1.15.3"

Then we create the test class TestBookServiceWithTestcontainers and an application-test.yml.

@SpringBootTest
@Testcontainers
@ActiveProfiles("test")
class TestBookServiceWithTestcontainers {

    @Container
    private static final MySQLContainer<?> MY_SQL_CONTAINER = new MySQLContainer<>("mysql:latest");

    @Autowired
    private BookService bookService;

    @Test
    void docker() {
        assertTrue(MY_SQL_CONTAINER.isRunning());
    }

    @Test
    void add() {
        final var bookTitle = "The Hitchhiker's Guide to the Galaxy";

        bookService.add(bookTitle);

        final var books = bookService.findAll();
        final var titles = StreamSupport.stream(books.spliterator(), false)
                .map(Book::getTitle)
                .collect(Collectors.toList());

        assertThat(titles).contains(bookTitle);
    }

}

Sample 6: TestBookServiceWithTestcontainers.java

This small example already shows the powerful functionality of Testcontainers.

The @Testcontainers annotation takes care of the automatic starting and stopping of containers during the tests.

@Container
private static final MySQLContainer<?> MY_SQL_CONTAINER = new MySQLContainer<>("mysql:latest");

With the @Container annotation we mark containers which are to be managed by Testcontainers. Such a container is created in the next line. As the name suggests, it is a MySQL container. The name and tag from the container is given via the constructor parameter. That is all you need to use MySQL in your integration tests.

The add() Method contains an actual test. The second test method docker is only here to demonstrate the behaviour of the docker container when several tests use the same container.

If MY_SQL_CONTAINER is static the container will be shared between the test methods. Without static the container will be started before and stopped after each test method.

It’s best to try out the difference for yourself. Start the tests once with static and once without and observe the different behaviour, e.g. by checking docker every few seconds with docker ps. P.S .: I recommend using the command line tool watch. This tool executes a program periodically. For example, to see the result of docker ps every second, the following command can be used: watch -n1 docker ps.

Now one more question remains: How does my code know the correct jdbc url? To connect to a database normally the host and port are usually required. But Testcontainers randomly selects and assigns free ports every time a new container is started.

As mentioned above Testcontainers supports a specially modified JDBC url which we can use in this case. The jdbc url looks like this jdbc:tc:mysql:latest:///databasename and can be used in the application-test.yml. Testcontainers then handles the connection to the MySQL database fully automatically for all containers that derive from JdbcDatabaseContainer.

spring:
  jpa:
    hibernate:
      ddl-auto: update
  datasource:
    url: jdbc:tc:mysql:latest:///databasename
    username:
    password:
    driver-class-name:

Sample 7: application-test.yml

Summary:

  • with Testcontainers it’s very simple to implement docker containers for integration testing
  • the administration (e.g. starting/stopping the containers) is fully automated and very fast
  • you can choose if the containers run for all tests or restart for each test
  • no more port clashes, because Testcontainers automatically chose a random free port
  • with the jdbc url feature you have no implementation work regarding hostname, port, username and password

With that in mind, it’s time to look at a few more features.

Use other images

In the example above MySQL was required for the tests. What if you need a different kind of database e.g. Elasticsearch or Redis, or multiple databases at the same time?

As described above, Testcontainers offers ready-made modules for many Docker images from Docker Hub (e.g. MySQL, Elasticsearch, RabbitMQ, …). Every used module also needs the corresponding dependency to be added to the build.gradle (e.g. testImplementation "org.testcontainers: elasticsearch: 1.15.3" for Elasticsearch) to function. If you want to use an image for which there is not a separate module you can simply create a GenericContainer (e.g. for Redis). The image name and tag depend on what is available on the Docker Hub. Of course, several modules can also be used in one test class. The use of the modules is basically the same as in the example shown above with MySQL.

@Container
final MySQLContainer<?> mySQLContainer = new MySQLContainer<>("mysql:8.0.25");

@Container
final ElasticsearchContainer elasticsearchContainer = 
        new ElasticsearchContainer(DockerImageName
        .parse("docker.elastic.co/elasticsearch/elasticsearch")
        .withTag("7.13.1"));

@Container
public GenericContainer<?> redisContainer = 
        new GenericContainer<>(DockerImageName.parse("redis:5.0.3-alpine"))
        .withExposedPorts(6379);

Sample 8: Generate different containers

Custom images

Testcontainers also supports the use of your own images. On the one hand Testcontainers offers the option to generate images on the fly (more information can be found in the official documentation).

On the other hand you can use images from your own repository with Testcontainers like this:

final MySQLContainer<?> my_sql_container = new MySQLContainer<>(
    DockerImageName.parse("repository.mycompany.com/example/mysql:8.0.24")
                   .asCompatibleSubstituteFor("mysql")
)

Sample 9: Create a MySQL container from a custom repository

Use multiple images with Docker Compose

If different images are needed for a test Docker Compose can be used. Testcontainers, you guessed it, also offers a module for this. Everything that is required for this is contained in the core Testcontainers library which can be integrated in the build.gradle with the following line: testImplementation "org.testcontainers:testcontainers:1.15.3".

An example usage could look like this:

@Testcontainers
class TestcontainersDockerCompose {

    private static final String SERVICE_NAME_MYSQL = "database_1";
    private static final String SERVICE_NAME_ELASTICSEARCH = "elasticsearch_1";
    private static final String SERVICE_NAME_REDIS = "redis_1";

    private static final int SERVICE_PORT_MYSQL = 3306;
    private static final int SERVICE_PORT_ELASTICSEARCH = 9200;
    private static final int SERVICE_PORT_REDIS = 6379;

    @Container
    private static final DockerComposeContainer<?> environment = new DockerComposeContainer<>(
            new File("src/test/resources/docker-compose.yml"))
            .withLocalCompose(true)
            .withExposedService(SERVICE_NAME_MYSQL, SERVICE_PORT_MYSQL,
                                Wait.forListeningPort().withStartupTimeout(Duration.ofSeconds(30)))
            .withExposedService(SERVICE_NAME_ELASTICSEARCH, SERVICE_PORT_ELASTICSEARCH,
                                Wait.forHttp("/_cluster/health?wait_for_status=green&timeout=50s&pretty").forStatusCode(200))
            .withExposedService(SERVICE_NAME_REDIS, SERVICE_PORT_REDIS);

    @Test
    void mysql() {
        final var containerStateOptional = environment.getContainerByServiceName(SERVICE_NAME_MYSQL);
        List<Boolean> isRunning = containerStateOptional.stream().map(ContainerState::isRunning).collect(Collectors.toList());

        assertThat(isRunning)
                .isNotEmpty()
                .doesNotContain(false);
    }

    @Test
    void elasticsearch() {
        final var containerStateOptional = environment.getContainerByServiceName(SERVICE_NAME_ELASTICSEARCH);
        List<Boolean> isRunning = containerStateOptional.stream().map(ContainerState::isRunning).collect(Collectors.toList());

        assertThat(isRunning)
                .isNotEmpty()
                .doesNotContain(false);
    }

    @Test
    void redis() {
        final var containerStateOptional = environment.getContainerByServiceName(SERVICE_NAME_REDIS);
        List<Boolean> isRunning = containerStateOptional.stream().map(ContainerState::isRunning).collect(Collectors.toList());

        assertThat(isRunning)
                .isNotEmpty()
                .doesNotContain(false);
    }

}

Sample 10: TestcontainersDockerCompose.java

The associated docker-compose.yml can look like this:

database:
  image: mysql:8.0.25
  environment:
    MYSQL_ROOT_PASSWORD: mySecret.COM
elasticsearch:
  image: elasticsearch:6.8.16
redis:
  image: redis:6.2.4

Sample 11: docker-compose.yml

Or with the use of custom images:

mysql:
  image: repository.mycompany.com/example/my-database:1.0.14
  environment:
    MYSQL_ROOT_PASSWORD: mySecret.COM
elasticsearch:
  image: repository.mycompany.com/example/my-search-index:2.0.15
redis:
  image: repository.mycompany.com/example/my-key-value-store:3.0.16

Sample 12: docker-compose.yml with custom repository

A pitfall to avoid: If you already have a comprehensive docker-compose.yml for your tests, you will probably have to rework it since Testcontainers inherently provides a lot of features (e.g. wait strategy for the container health-checks) that you won’t have to manually configure anymore. Only declare what you need in the docker-compose.yml. This will save you a lot of errors, trouble and time ;-).

Use containers across multiple test classes

So far we have learned how we can use Docker Containers with Testcontainers for a single test class. However, there are advantages to using the same container for several test classes. E.g. the tests are faster if the containers only have to be started once.

And even if the containers are restarted for each test or for each test class, it saves lines of code if the containers are only defined in one place. In this case, however, the question arises as to how the necessary properties of the container are made available to the code. At certain points, the great jdbc url support unfortunately reaches its limits (e.g. when it is not a JDBC resource like Elasticsearch).

In order to answer these questions, I would like to present to you three different solutions that solve these problems in different ways. With the knowledge acquired so far, it should no longer be difficult to understand the code.

All three examples are similar in that they are used for creating, starting and stopping the containers with the help of Docker Compose. In addition, all three variants enable the same properties to be used in the integration tests. However, there are differences in the integration, the way in which the containers are managed and how the properties are accessed.

Testcontainers with Docker Compose and DynamicProperties

@SpringBootTest
public class TestContainersWithDynamicProperties {

    private static final String serviceNameMysql = "database_1";
    private static final String serviceNameElasticsearch = "elasticsearch_1";
    private static final String serviceNameRedis = "redis_1";

    private static final int servicePortMysql = 3306;
    private static final int servicePortElasticsearch = 9200;
    private static final int servicePortRedis = 6379;

    @Container
    private static final DockerComposeContainer<?> environment = new DockerComposeContainer<>(
            new File("src/test/resources/docker-compose.yml"))
            .withLocalCompose(true)
            .withExposedService(serviceNameMysql, servicePortMysql,
                                Wait.forListeningPort().withStartupTimeout(Duration.ofSeconds(30)))
            .withExposedService(serviceNameElasticsearch, servicePortElasticsearch,
                                Wait.forHttp("/_cluster/health?wait_for_status=green&timeout=50s&pretty").forStatusCode(200))
            .withExposedService(serviceNameRedis, servicePortRedis);

    @DynamicPropertySource
    static void registerProperties(DynamicPropertyRegistry registry) {
        var db_host = environment.getServiceHost(serviceNameMysql, servicePortMysql);
        var db_port = environment.getServicePort(serviceNameMysql, servicePortMysql);

        var es_host = environment.getServiceHost(serviceNameElasticsearch, servicePortElasticsearch);
        var es_port = environment.getServicePort(serviceNameElasticsearch, servicePortElasticsearch);

        var redis_host = environment.getServiceHost(serviceNameRedis, servicePortRedis);
        var redis_port = environment.getServicePort(serviceNameRedis, servicePortRedis);

        //If you use different data sources (e.g. with HikariCP) setting the properties could look like this:
        registry.add("datasource.first.jdbc-url", () -> createJdbcUrl(db_host, db_port, "it_first"));
        registry.add("datasource.second.jdbc-url", () -> createJdbcUrl(db_host, db_port, "it_second"));
        registry.add("datasource.third.jdbc-url", () -> createJdbcUrl(db_host, db_port, "it_third"));

        registry.add("elasticsearch.host", () -> es_host);
        registry.add("elasticsearch.port", () -> es_port);

        registry.add("redis.host", () -> redis_host);
        registry.add("redis.port", () -> redis_port);
    }

    private static String createJdbcUrl(final String host, final Integer port, final String urlPart) {
        return "jdbc:mysql://" + host + ":" + port + "/" + urlPart + "?serverTimezone=Europe/Vienna&useUnicode=yes&characterEncoding=utf8";
    }

}

Sample 13: TestContainersWithDynamicProperties.java

The containers are created with Docker Compose. Since they are static, they remain available during the runtime of all test classes and are automatically terminated by Testcontainers at the end. In order to make the containers available in a test class, it must extend TestContainersWithDynamicProperties. The properties are available via DynamicPropertySource and overwrite any existing properties.

@SpringBootTest
public class MyTest extends TestContainersWithDynamicProperties {
    // test code
}

Sample 14: Usage of TestContainersWithDynamicProperties

Testcontainers with Docker Compose and Initializer

@SpringBootTest
@ContextConfiguration(initializers = TestContainersWithInitializer.Initializer.class)
public class TestContainersWithInitializer {

    private static final String serviceNameMysql = "database_1";
    private static final String serviceNameElasticsearch = "elasticsearch_1";
    private static final String serviceNameRedis = "redis_1";

    private static final int servicePortMysql = 3306;
    private static final int servicePortElasticsearch = 9200;
    private static final int servicePortRedis = 6379;

    @Container
    private static final DockerComposeContainer<?> environment = new DockerComposeContainer<>(
            new File("src/test/resources/docker-compose.yml"))
            .withLocalCompose(true)
            .withExposedService(serviceNameMysql, servicePortMysql,
                                Wait.forListeningPort().withStartupTimeout(Duration.ofSeconds(30)))
            .withExposedService(serviceNameElasticsearch, servicePortElasticsearch,
                                Wait.forHttp("/_cluster/health?wait_for_status=green&timeout=50s&pretty").forStatusCode(200))
            .withExposedService(serviceNameRedis, servicePortRedis);

    public static class Initializer implements ApplicationContextInitializer<ConfigurableApplicationContext> {

        @Override
        public void initialize(final ConfigurableApplicationContext applicationContext) {

            var db_host = environment.getServiceHost(serviceNameMysql, servicePortMysql);
            var db_port = environment.getServicePort(serviceNameMysql, servicePortMysql);

            var es_host = environment.getServiceHost(serviceNameElasticsearch, servicePortElasticsearch);
            var es_port = environment.getServicePort(serviceNameElasticsearch, servicePortElasticsearch);

            var redis_host = environment.getServiceHost(serviceNameRedis, servicePortRedis);
            var redis_port = environment.getServicePort(serviceNameRedis, servicePortRedis);

            TestPropertyValues.of(
                    //If you use different data sources (e.g. with HikariCP) setting the properties could look like this:
                    "datasource.first.jdbc-url=" + createJdbcUrl(db_host, db_port, "it_first"),
                    "datasource.second.jdbc-url=" + createJdbcUrl(db_host, db_port, "it_second"),
                    "datasource.third.jdbc-url=" + createJdbcUrl(db_host, db_port, "it_third"),
                    "elasticsearch.host:" + es_host,
                    "elasticsearch.port: " + es_port,
                    "redis.host:" + redis_host,
                    "redis.port:" + redis_port
            ).applyTo(applicationContext.getEnvironment());
        }

    }

    private static String createJdbcUrl(final String host, final Integer port, final String urlPart) {
        return "jdbc:mysql://" + host + ":" + port + "/" + urlPart + "?serverTimezone=Europe/Vienna&useUnicode=yes&characterEncoding=utf8";
    }

}

Sample 15: TestContainersWithInitializer.java

The containers are created with Docker Compose. Since they are static, they remain available during the runtime of all test classes and are automatically terminated by Testcontainers at the end. In order to make the containers available in a test class, the class must be integrated with extends. The properties are injected into the application-test.yml with the ApplicationContextInitializer and are immediately available if @ActiveProfiles("test") is available.

@SpringBootTest
public class MyTest extends TestContainersWithInitializer {
    // test code
}

Sample 16: Usage of TestContainersWithInitializer

Testcontainers with Docker Compose and JUnit 5 Extension

@SpringBootTest
public class TestContainersExtension implements BeforeAllCallback, AfterAllCallback {

    public DockerComposeContainer<?> environment;

    @Override
    public void beforeAll(final ExtensionContext context) {
        final var serviceNameMysql = "database_1";
        final var serviceNameElasticsearch = "elasticsearch_1";
        final var serviceNameRedis = "redis_1";
        final var servicePortMysql = 3306;
        final var servicePortElasticsearch = 9200;
        final var servicePortRedis = 6379;

        environment = new DockerComposeContainer<>(
                new File("src/test/resources/docker-compose.yml"))
                .withLocalCompose(true)
                .withExposedService(serviceNameMysql, servicePortMysql,
                                    Wait.forListeningPort().withStartupTimeout(Duration.ofSeconds(30)))
                .withExposedService(serviceNameElasticsearch, servicePortElasticsearch,
                                    Wait.forHttp("/_cluster/health?wait_for_status=green&timeout=50s&pretty").forStatusCode(200))
                .withExposedService(serviceNameRedis, servicePortRedis);

        environment.start();

        var db_host = environment.getServiceHost(serviceNameMysql, servicePortMysql);
        var db_port = environment.getServicePort(serviceNameMysql, servicePortMysql);

        var es_host = environment.getServiceHost(serviceNameElasticsearch, servicePortElasticsearch);
        var es_port = environment.getServicePort(serviceNameElasticsearch, servicePortElasticsearch);

        var redis_host = environment.getServiceHost(serviceNameRedis, servicePortRedis);
        var redis_port = environment.getServicePort(serviceNameRedis, servicePortRedis);

        //If you use different data sources (e.g. with HikariCP) setting the properties could look like this:
        System.setProperty("datasource.first.jdbc-url", createJdbcUrl(db_host, db_port, "it_first"));
        System.setProperty("datasource.second.jdbc-url", createJdbcUrl(db_host, db_port, "it_second"));
        System.setProperty("datasource.third.jdbc-url", createJdbcUrl(db_host, db_port, "it_third"));

        System.setProperty("elasticsearch.host", es_host);
        System.setProperty("elasticsearch.port", es_port.toString());

        System.setProperty("redis.host", redis_host);
        System.setProperty("redis.port", redis_port.toString());
    }

    @Override
    public void afterAll(final ExtensionContext context) {
        environment.stop();
    }

    private static String createJdbcUrl(final String host, final Integer port, final String urlPart) {
        return "jdbc:mysql://" + host + ":" + port + "/" + urlPart + "?serverTimezone=Europe/Vienna&useUnicode=yes&characterEncoding=utf8";
    }

}

Sample 17: TestContainersExtension.java

Here, too, the containers are created with Docker Compose. However, the class derives from BeforeAllCallback and AfterAllCallback and starts the container with beforeAll and terminates it automatically with afterAll. The use of JUnit5 means that the availability of the containers differs from the other two examples.

With beforeAll and afterAll, the containers are used across all test classes. But only until the entire application is running. In a Spring Boot test, however, the application restarts with each test class. While the containers of the above two variants survive this, the containers in this example are terminated. This can be viewed as a feature. The two variants shown above are equivalent and are suitable if the containers are required across all test classes. If you want to restart the containers for each test class without defining them in each test class, this variant can be used.

The containers created here are available with @ExtendWith (...). The properties are made available with System.setProperty.

@SpringBootTest
@ExtendWith(TestContainersExtension.class)
public class MyTest {
    // test code
}

Sample 18: Usage of TestContainersExtension

And further?

Of course, this article cannot show all functionalities of the Testcontainers framework. I have already referred to the documentation in appropriate places, and it is advisable to read it through carefully and completely. I also advice you to just try out Testcontainers. If you are of the opinion that this is a nice Java framework, but you also need it outside the JVM, you should take a look at GitHub. Testcontainers are also available in native implementation for other languages. When porting the framework to other programming languages Testcontainers tries to mainly port the concept of what is implemented rather than the specific implementation itself. You can take a look for other implementations here.

The code presented in this article can be downloaded here.


Alexander

Alexander likes to do everything that can be done on foot: hiking, walking … or even running after the kids. If his family doesn’t keep him busy, the Java programmer volunteers for the fire and rescue service. Alex finds relaxation in strategy games, reading and listening to podcasts and as it should be IT stuff.


We're a team of makers, thinkers, organisers and digital explorers and we're always on the lookout for talented people.

Are you a good fit?