Persisting Data with Nomad

Sebastian - May 25 '20 - - Dev Community

When you run stateful container with Nomad, you need to consider how to persist data between Docker containers that run on different hosts. What you want is this: Independent of where the container runs, it should have access to the same data. This sounds obvious, and yet I had severe trouble finding the right solution. I spend half a day to get either docker volume or docker bind mounts to a shared NFS dir with no success, and then another half day to systematically try all options.

This article is about the journey to achieve data persisting. You will read painful lessons learned and see the working, final solution.

This article appeared originally at my blog.

Nomad Docker Bind Mounts

Nomad provides different mounting options at different places in the configuration. The first option is to use host volume mounts: You define a host-specific path which is mapped to any path in the Docker container. You need to provide three config stanzas:

  • The Nomad agent config needs a host_volume config
  • The job group needs a volume declaration
  • The tasks needs a volume_mount declaration

Let’s see this setup, assuming we want to mount a volume in which we persists the monitoring data captured by Prometheus.

nomad.conf.hcl 

client {
  host_volume "prometheus_vol" {
    path = "/mnt/nfs/nomad/volumes/prometheus"
    read_only = false
  }
}
Enter fullscreen mode Exit fullscreen mode

monitoring.job.hcl 

group "monitoring" {
  volume "prometheus_grp" {
    type = "host"
    source = "prometheus_vol"
    read_only = false
  }

task "grafana" {
  volume_mount {
    volume      = "prometheus_vol"
    destination = "/prometheus"
    readonly = false
  }
  ...
}
Enter fullscreen mode Exit fullscreen mode

Nomad Docker Volume Mounts

The second option is to use Docker volume mounts. Volumes are named, persistent data that resides in the host in the path /var/lib/docker/volumes. When the job is run, Docker will create the volume if it does not exist, or mount it 1. It’s the same as interacting with docker volume create.

>> docker volume ls

DRIVER              VOLUME NAME
local               1e9b98423c15f1aa833b131ab9f6c72b0433a212a89b5b75a30424ba917f09d9
local               prometheus_grp

>> docker volume inspect prometheus_grp
[
  {
    "CreatedAt": "2020-03-14T11:54:04Z",
    "Driver": "local",
    "Labels": null,
    "Mountpoint": "/var/lib/docker/volumes/prometheus_grp/_data",
    "Name": "prometheus_grp",
    "Options": null,
    "Scope": "local"
  }
]
Enter fullscreen mode Exit fullscreen mode

To create a Nomad Docker volume, you need only one config stanza directly in the docker.config path.

monitoring.job.hcl 

task "grafana" {
  driver = "docker"
  config {
    mounts = [
      {
        type = "volume"
        target = "/prometheus"
        source = "prometheus"
      }
    ]
  }
  ...
}
Enter fullscreen mode Exit fullscreen mode

Day 1 - Trial & Error

Step 1: Symlink Docker Volumes to NFS

Many Docker documentations recommend to use docker volumes. I thought it would be some specific binary format, but actually it is a normal, accessible dir on your machine:

>> ls -la /var/lib/docker/volumes/prometheus_grp/_data/

total 16
drwxr-xr-x 3 nobody nogroup  4096 Mar 14 11:54 .
drwxr-xr-x 3 root   root     4096 Mar 14 11:54 ..
-rw-r--r-- 1 nobody nogroup     0 Mar 14 11:54 lock
-rw-r--r-- 1 nobody nogroup 20001 Mar 14 12:13 queries.active
drwxr-xr-x 2 nobody nogroup  4096 Mar 14 12:13 wal
Enter fullscreen mode Exit fullscreen mode

Therefore, my first try was to set a symlink /var/lib/docker/volumes => /mnt/nfs/docker/volumes. I set the symlink on each machine, and then started spinning up Prometheus on different hosts. It did not work! I saw different data on different hosts. And all existing data was gone!

Here is why: When the Docker container starts on a different machine, and there is no Docker volume with the given name, it will create one. And because of the symlink, it will delete the target dir /mnt/nfs/docker/volumes/prometheus rigorously. This is the default behavior of docker volume create - you don’t get any warning when the target dir already exists!

Can I fix this by creating the Docker volumes manually? I deleted the symlink, deleted all docker volumes on each node, recreated the symlink and created docker volumes on all nodes. Then, I started the container on different nodes again - but Nomad overwrites the existing volume declaration rigorously.

Second Try: Docker NFS Volumes

Ok. Let’s try to create proper Docker NFS volumes. The command I used is this:

docker volume create --driver local \
  --opt type=nfs \
  --opt o=addr=192.168.2.203,rw \
  --opt device=:/mnt/nfs/volumes/test/ nfs-test
Enter fullscreen mode Exit fullscreen mode

When executing the following test command to mount the volume and create a file, I received an error.

docker run -v /mnt/nfs/volumes/test/:/nfs-test busybox touch /nfs-test/hello

docker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/nfs-test/_data': failed to mount local volume: mount /mnt/nfs/volumes/test:/var/lib/docker/volumes/nfs-test/_data, data: addr=192.168.2.203: invalid argument
Enter fullscreen mode Exit fullscreen mode

For whatever reason, this directory was not writable.

Third Try: Docker Bind Volumes

Ok, if Docker volumes will not work, I will instead mount the shared dir /mnt/nfs/volumes/prometheus into the container. This should work because this dir is available on all nodes. I got this solution to work, but then I was seeing a strange dir access error. Even as the root user, I could not create files inside shared directory, and also, I could not delete them!

This lead me to thinking: Maybe the problems originate from my NFS setup: autofs mounts, and an entry in /etc/fstab to unmount the dir when it is not used.

At this point, I spend 4 hours, and finally gave up for this day.

Day 2 - Systematic Trial & Success

Figuring Out What’s Wrong

What went wrong? The NFS mounts are available on the hosts, and they work. And I could get one Docker container to write to this share. So let’s assume the NFS itself works, but I could not figure out how to tell Nomad to use the network share.

First things first: Lets properly define a docker NFS volume. In a stackoverflow question I saw a Docker compose file for mounting NFS shares. This command uses more optional arguments then the one I have been using, so I adopted its.

I created a docker volume share with this command:

docker volume create
  --driver local
  --opt type=nfs
  --opt "o=nfsvers=4,addr=192.168.2.203,nolock,soft,rw"
  --opt device=:/mnt/nfs/volumes/test/ nfs-test
Enter fullscreen mode Exit fullscreen mode

And then I started a docker container, mounted the volume, and put a hello file in the share.

docker run -v /mnt/nfs/volumes/test/:/nfs-test busybox touch /nfs-test/hello
Enter fullscreen mode Exit fullscreen mode

That worked! Now let’s start again and systematically try the different options of mounting volumes inside Nomads job.group.task stanza.

Nomad Named Volume Declaration

On each node, I removed all existing volumes, and then created the NFS volume with the working command from above.

Then, the first try is to use the named volume inside the task.config stanza.

task "grafana" {
  driver = "docker"
  config {
    mounts = [
      {
        type = "volume"
        target = "/prometheus"
        source = "prometheus"
      }
    ]
  ...
  }
}
Enter fullscreen mode Exit fullscreen mode

However, the job could not be executed. I saw this error message:

failed to create container: API error (500): failed to mount local volume: mount :/mnt/nfs/docker/prometheus:/var/lib/docker/volumes/prometheus/_data, data: nfsvers=4,addr=192.168.2.203,nolock,soft: no such file or directory
Enter fullscreen mode Exit fullscreen mode

It seems that Nomad is confused with the extra options part of the Docker volume mount. I considered then to use the Docker NFS volume declaration in Nomad, but I could not find any example in the official documentation.

Nomad Bind Mount Declaration

Then I tried a direct bind mount declaration.

task "grafana" {
  driver = "docker"
  config {
    mounts = [
      {
        type = "bind"
        target = "/prometheus"
        source = "/mnt/nfs/docker/prometheus"
        readonly = false
        bind_options {
          propagation = "rshared"
        }
      }
    ]
  ...
  }
}
Enter fullscreen mode Exit fullscreen mode

The Nomad job executed, and then ... something strange happened! The volume prometheus was suddenly an ordinary volume on the node! Nomad overwrites the existing volume - this happened to me on day 1 too! To confirm this, I reapplied the same steps again - and yes, Nomad overwrites the existing docker volume declaration.

Before the job:

>> docker volume inspect prometheus
[
  {
    "CreatedAt": "2020-03-18T19:15:57Z",
    "Driver": "local",
    "Labels": {},
    "Mountpoint": "/var/lib/docker/volumes/prometheus/_data",
    "Name": "prometheus",
    "Options": {
        "device": ":/mnt/nfs/docker/prometheus",
        "o": "nfsvers=4,addr=192.168.2.203,nolock,soft,rw",
        "type": "nfs"
    },
    "Scope": "local"
  }
]
Enter fullscreen mode Exit fullscreen mode

After the job:

>> docker volume inspect prometheus

[
  {
    "CreatedAt": "2020-03-18T19:17:30Z",
    "Driver": "local",
    "Labels": null,
    "Mountpoint": "/var/lib/docker/volumes/prometheus/_data",
    "Name": "prometheus",
    "Options": null,
    "Scope": "local"
  }
]
Enter fullscreen mode Exit fullscreen mode

Nomad Volume Declaration

Then I tried this config.volumes declaration:

task "grafana" {
  driver = "docker"
  config {
    image = "prom/prometheus:v2.16.0"
    volumes = [
      "local/prometheus.yml:/etc/prometheus/prometheus.yml",
      "/mnt/nfs/docker/prometheus:/prometheus"
    ]
    ...
  }
}
Enter fullscreen mode Exit fullscreen mode

I was surprised. The Docker container started. I saw the data streaming. Then, I stopped the container, waited some time, and deployed it on another container. The data was still there! Stopping and restarting the job several times, the NFS mount steadily accumulated data from different nodes.

I got it working finally!

prometheus_data_accumulation

Solution

The final solution for data persisting boils down to:

  • Setup nfs share on nodes
  • Use a config.volumes declaration with direct mapping between host source path and container destination path 
volumes = [
  "/path/to/nfs/share:/path/to/docker/destination"
]
Enter fullscreen mode Exit fullscreen mode

No volume declaration needed. No docker volume creation on the host. Just one line per volume in your Nomad job!

Conclusion

When you want to persist data of Docker containers running on different nodes, and have the flexibility to move containers around, then the best solution is to setup an NFS share between the nodes and use direct host to docker mounts. It was hard and painful figuring out how to achieve this. But learning from mistakes can be very educational because you learn about why things are not working. The goal was indeed worth the journey!

Footnotes

  1. As I learned later, that’s not correct: It will overwrite the docker volume declaration if it is anything else then a normal volume! If you defined an NFS volume with the same name, it will get overwritten. ↩

Image
11 Creative Ways To Write About Attorney Injury Accident Attorney
Image
15 Things You've Never Known About Replace Lost Car Keys
Image
sejumlah hal yang jarang-jarang orang tahu mengenai slots online
Image
Understanding the Linux File System, Virtual Storage, and Mounting Storage
Image
토토사이트의 모든 것: 안전하게 즐기는 스포츠 베팅 가이드
Image
Three Of The Biggest Catastrophes In Lost My Car Keys History
Image
What Is New Crypto Casino And How To Make Use Of It
Image
How Asbestos Mesothelioma Life Expectancy Became The Hottest Trend In 2023
Image
management waktu dalam perjudian slots online
Image
Panduan Jadi Raja Judi Online yang Sukses
Image
10 Facts About Accident Lawyer Philadelphia That Will Instantly Put You In A Good Mood
Image
10 Facts About Houston Asbestos Attorney That Will Instantly Put You In A Good Mood
Image
Ten Crypto Games Casino-Related Stumbling Blocks You Should Not Share On Twitter
Image
cors policy
Image
Medle do Spyialni: Odkrywając Nieznane Światy Wyjątkowej Muzyki
Image
situs slots online bonus new peserta yang baik
Image
Watch Out: How Asbestos Lawsuit Settlement Is Taking Over And What To Do About It
Image
14 Creative Ways To Spend The Remaining Asbestos Attorney In Virginia Budget
Image
20 Fun Informational Facts About Filter Coffe Machine
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Terabox Video Player