CarcFor Java applications using MongoDB, we can leverage Testcontainers to set up and manage our test environment. This tool allows us to create and configure lightweight, disposable containers for our tests, offering a more isolated and reliable testing experience.
Testcontainers provides a built-in MongoDB container that consists of a single-node replica set. This is convenient for simple tests. However, for scenarios requiring a multi-node MongoDB cluster, we encounter a challenge. While Testcontainers recommends an external library—https://github.com/silaev/mongodb-replica-set—this solution appears to be abandoned at the time of writing.
In this article, we'll explore how to create a simple multi-node MongoDB replica set using Testcontainers and discuss the challenges we'll face along the way.
The Challenges
Creating a multi-node MongoDB replica set looks like a straightforward task. We start several MongoDB instances using the parameter --replSet <replSetID>. Once all servers are running, we execute the rs.initiate(<rsConfig>) command on one of the servers. The rsConfig we pass to this command depends on the number of servers we've started. For a three-server replica set, it would be something as follows:
rs.initiate({
"_id": "rs0",
"members": [
{"_id": 0, "host": "mongo1:27017"},
{"_id": 1, "host": "mongo2:27017"},
{"_id": 2, "host": "mongo3:27017"}
]
});
Challenge 1
In this example, we have three MongoDB servers started with the parameter--replSet rs0. Each server has a different DNS name:mongo1,mongo2, andmongo3, all listening on the default MongoDB port27017. When we run thers.initiate(<rsConfig>)command, MongoDB checks that the MongoDB address (host:port) of the service executing the command is part of the members list, it will also check that all address in the list are resolvable from that server. It won't allow the creation of the replica set if this precondition is not met.
This is the first thing we need to take into account when creating a multi node MongoDB replica set.
Challenge 2
Once we have our MongoDB replica set ready we can connect to it. When connecting to our MongoDB replica set we have to use a connection string like the following: mongodb://mongo1:27017,mongo2:27017,mongo3:27017/?replicaSet=rs0 . The driver using that connection string will pickup one of the servers in the list, connect to it and ask for the members list, once got the list, the driver will try to resolve the address of each member of the list. So the host connecting to the replica set must be able to resolve the addresses used to configure the replica set (hosts in members ).
This is the second thing we need to take into account when creating a multi node MongoDB replica set.
Approaching the problem using Docker
Docker custom netwok and aliases
Docker networking and aliases allow containers to communicate with each other within isolated networks, making it possible to create clusters of servers. Containers in the same network can communicate using their aliases as if they were hostnames, enabling easy inter-container communication for clustering. It will allow to overcome the first limitation we found when initiating the replica set.
However, it's important to note that the host network is typically not part of the custom network where the MongoDB containers reside. Consequently, when the driver running on the host attempts to connect to the replica set, it may fail to resolve the member list addresses due to its lack of access to the custom network.
Docker host network driver
Docker offers the option to allow containers to share the host's network, providing direct access to the host's network interfaces. While this feature eliminates the need for port mapping, it can lead to conflicts if multiple containers attempt to use the same port. To mitigate this risk, it's essential to carefully assign unique ports to each container. By doing so, you can minimize the likelihood of encountering port conflicts and ensure smooth container operation.
One significant advantage of using the host network is the ability for containers to communicate directly with each other and for the host to resolve their addresses. This can simplify network configuration and facilitate communication between containers and the host.
This feature is supported out of the box in Linux. For Mac/Windows, Docker Desktop supports it from version 4.34 and later, however it needs to be explicitly enabled.
Testing our solution
Before jumping into implementing our multi-node MongoDB replica set in java using Testcontainers, let’s use docker-compose to try to create a PoC using the Docker host network driver approach.
version: "3.8"
services:
mongo1:
container_name: mongo1
image: mongo:6.0
command: ["--replSet", "rs0", "--bind_ip_all", "--port", "27020"]
restart: always
network_mode: "host"
healthcheck:
test: echo "try { rs.status() } catch (err) { rs.initiate({_id:'rs0',members:[{_id:0,host:'localhost:27020',priority:1},{_id:1,host:'localhost:27021',priority:0.5},{_id:2,host:'localhost:27022',priority:0.5}]}) }" | mongosh --port 27020 --quiet
interval: 5s
timeout: 30s
start_period: 0s
start_interval: 1s
retries: 30
mongo2:
image: mongo:6.0
container_name: mongo2
command: ["--replSet", "rs0" ,"--bind_ip_all", "--port", "27021"]
restart: always
network_mode: "host"
mongo3:
image: mongo:6.0
container_name: mongo3
command: ["--replSet", "rs0" ,"--bind_ip_all", "--port", "27022"]
restart: always
network_mode: "host"
Observe the use of network_mode: "host" for each container, which allows them to share the host's network. This configuration, in conjunction with the specified ports (--port 27020, --port 27021, --port 27022), enables the containers to communicate directly using their respective port numbers. These port assignments are reflected in the replica set configuration: rs.initiate({_id:'rs0',members:[{_id:0,host:'localhost:27020',priority:1},{_id:1,host:'localhost:27021',priority:0.5},{_id:2,host:'localhost:27022',priority:0.5}]})
Let’s start and test our MongoDB replica set using docker-compose
$ docker-compose up -d
[+] Running 3/3
✔ Container mongo1 Started 0.1s
✔ Container mongo2 Started 0.1s
✔ Container mongo3 Started
$ mongosh "mongodb://localhost:27020,localhost:27021,localhost:27022/?replicaSet=rs0"
Current Mongosh Log ID: 6719fb60b65eaef268d51880
Connecting to: mongodb://localhost:27020,localhost:27021,localhost:27022/?replicaSet=rs0&serverSelectionTimeoutMS=2000&appName=mongosh+2.3.2
Using MongoDB: 6.0.18
Using Mongosh: 2.3.2
For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/
------
The server generated these startup warnings when booting
2024-10-24T06:48:05.687+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2024-10-24T06:48:06.371+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2024-10-24T06:48:06.371+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never' in this binary version
2024-10-24T06:48:06.371+00:00: vm.max_map_count is too low
------
rs0 [primary] test>
The java code
Once we’ve proven our approach works using docker-compose let’s write the Testcontainers java version. It will use jbang to make prototyping easier.
///usr/bin/env jbang "$0" "$@" ; exit $?
//JAVA 21
//DEPS org.testcontainers:testcontainers:1.20.2
//DEPS org.slf4j:slf4j-simple:1.7.36
import static java.lang.String.format;
import org.testcontainers.containers.GenericContainer;
import org.testcontainers.containers.wait.strategy.Wait;
import java.io.IOException;
import java.net.ServerSocket;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
public class MongoDBReplicaSet {
private static final String MONGO_IMAGE = "mongo:6.0.13";
private static final String RS_NAME = "rs0";
private List<PortAndContainer> nodes = new ArrayList<>();
record PortAndContainer(int port, GenericContainer container) {}
public MongoDBReplicaSet() { this(3);}
public MongoDBReplicaSet(int n) {
if (n < 1) {
throw new IllegalArgumentException("At least one node is required");
}
for (int i = 0; i < n; i++) {
var port = findAvailableTcpPort();
var node = new GenericContainer(MONGO_IMAGE)//
.withNetworkMode("host")//
.waitingFor(Wait.forLogMessage(".*Waiting for connections.*\\n", 1))//
.withCommand("mongod", "--replSet", RS_NAME, "--bind_ip_all", "--port", String.valueOf(port));
nodes.add(new PortAndContainer(port, node));
}
}
public String getConnectionString() {
return nodes.stream()//
.map(node -> String.format("%s:%d", node.container().getHost(), node.port()))
.collect(Collectors.joining(",", "mongodb://", "/?replicaSet=" + RS_NAME));
}
public void start() {
nodes.forEach(node -> node.container().start());
var replicaMembersCfg = nodes.stream()//
.map(node -> format("{\"_id\": %d, \"host\": \"%s:%d\"}", nodes.indexOf(node),
node.container().getHost(), node.port()))
.collect(Collectors.joining(","));
var rsInitiate = format("rs.initiate({\"_id\": \"%s\", \"members\": [%s]})", RS_NAME, replicaMembersCfg);
var node = nodes.get(0);
var nodeUrl = node.container().getHost() + ":" + node.port();
execInContainer(node.container(), "mongosh", nodeUrl + "/admin", "--quiet", "--eval", rsInitiate);
execInContainer(node.container(), "/bin/bash", "-c",
"until mongosh " + nodeUrl + "/admin"
+ " --eval \"printjson(rs.isMaster())\" | grep ismaster | grep true > /dev/null 2>&1;"
+ "do sleep 1;done");
}
private void execInContainer(GenericContainer container, String... command) {
try {
var result = container.execInContainer(command);
if (result.getExitCode() != 0) {
throw new RuntimeException(format("Failed execution of command. Code: %s, output: %s ",
result.getExitCode(), result.getStdout() + "\n" + result.getStderr()));
}
} catch (UnsupportedOperationException | IOException | InterruptedException e) {
throw new RuntimeException("Failed to execute command", e);
}
}
public void stop() {
nodes.forEach(node -> node.container().stop());
}
private int findAvailableTcpPort() {
try (ServerSocket socket = new ServerSocket(0)) {
return socket.getLocalPort();
} catch (IOException e) {
throw new IllegalStateException("Failed to find available port", e);
}
}
public static void main(String[] args) throws Exception {
var replicaSet = new MongoDBReplicaSet();
try {
replicaSet.start();
System.out.println("MongoDB Replica Set started. Connection string: " + replicaSet.getConnectionString());
Thread.currentThread().join();
} finally {
replicaSet.stop();
}
}
}
In the code, we employed the withNetworkMode("host") configuration to allow the containers to utilize the host's network. Additionally, we dynamically determined an available port for each container using var port = findAvailableTcpPort(). This port was then incorporated into the server startup command as "--port", String.valueOf(port). Subsequently, when initiating the MongoDB replica set, we utilized these port values to populate the members list for each container.
Let’s start and test our Testcontainers MongoDB replica set using jbang .
$ jbang MongoDBReplicaSet.java
MongoDB Replica Set started. Connection string: mongodb://localhost:54303,localhost:54304,localhost:54305/?replicaSet=rs0
$ mongosh "mongodb://localhost:55039,localhost:55040,localhost:55041/?replicaSet=rs0"
Current Mongosh Log ID: 6719fbc79230eaae68149f3a
Connecting to: mongodb://localhost:55039,localhost:55040,localhost:55041/?replicaSet=rs0&serverSelectionTimeoutMS=2000&appName=mongosh+2.3.2
Using MongoDB: 6.0.13
Using Mongosh: 2.3.2
For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/
------
The server generated these startup warnings when booting
2024-10-24T07:47:35.720+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem
2024-10-24T07:47:36.383+00:00: Access control is not enabled for the database. Read and write access to data and configuration is unrestricted
2024-10-24T07:47:36.383+00:00: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never'
2024-10-24T07:47:36.383+00:00: vm.max_map_count is too low
------
rs0 [primary] test>
Summing up
We have successfully demonstrated the feasibility of creating a Testcontainers multi-node MongoDB replica set using Docker host network driver approach. However, this is merely a foundational example. While the provided code may be suitable for certain use cases, it might require further refinement or customization to meet specific requirements.
References
https://java.testcontainers.org/modules/databases/mongodb/
