Consul docker image doesn't work with hyperctl or docker+runv #722
Description
I've been struggling to debug an issue with the standard consul image when running on docker+runv and/or pure hyperctl stack.
First of all, when running docker+runc (on the same fresh Ubuntu 16.04 bare metal box for all these examples -- I also tried on centos 7 bare metal and two other ubuntu 16.04 bare metal boxes), I get the following successful start:
root@hyperdev-02:~# docker run -d --name consul -p "8500:8500" -h "consul" consul agent -server -bootstrap -client "0.0.0.0" a991885eddd23412dfceb3b5bbd56d1e52b0ef67e29ae54405e3012da2576c7b root@hyperdev-02:~# docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES a991885eddd2 consul "docker-entrypoint..." 45 seconds ago Up 21 seconds 8300-8302/tcp, 8301-8302/udp, 8600/tcp, 8600/udp, 0.0.0.0:8500->8500/tcp consul root@hyperdev-02:~# docker logs a991885eddd2 bootstrap = true: do not enable unless necessary ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.1.0' Node ID: 'fb56a8fd-1dcc-70ae-231e-19f4fc012a14' Node name: 'consul' Datacenter: 'dc1' (Segment: '<all>') Server: true (Bootstrap: true) Client Addr: [0.0.0.0] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 172.17.0.2 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false ==> Log data will now stream in as it occurs: 2018/05/27 11:45:24 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID:fb56a8fd-1dcc-70ae-231e-19f4fc012a14 Address:172.17.0.2:8300}] 2018/05/27 11:45:24 [INFO] raft: Node at 172.17.0.2:8300 [Follower] entering Follower state (Leader: "") 2018/05/27 11:45:24 [INFO] serf: EventMemberJoin: consul.dc1 172.17.0.2 2018/05/27 11:45:24 [INFO] serf: EventMemberJoin: consul 172.17.0.2 2018/05/27 11:45:24 [INFO] agent: Started DNS server 0.0.0.0:8600 (tcp) 2018/05/27 11:45:24 [INFO] consul: Adding LAN server consul (Addr: tcp/172.17.0.2:8300) (DC: dc1) 2018/05/27 11:45:24 [INFO] agent: Started DNS server 0.0.0.0:8600 (udp) 2018/05/27 11:45:24 [INFO] consul: Handled member-join event for server "consul.dc1" in area "wan" 2018/05/27 11:45:24 [INFO] agent: Started HTTP server on [::]:8500 (tcp) 2018/05/27 11:45:24 [INFO] agent: started state syncer 2018/05/27 11:45:30 [WARN] raft: Heartbeat timeout from "" reached, starting election 2018/05/27 11:45:30 [INFO] raft: Node at 172.17.0.2:8300 [Candidate] entering Candidate state in term 2 2018/05/27 11:45:30 [INFO] raft: Election won. Tally: 1 2018/05/27 11:45:30 [INFO] raft: Node at 172.17.0.2:8300 [Leader] entering Leader state 2018/05/27 11:45:30 [INFO] consul: cluster leadership acquired 2018/05/27 11:45:30 [INFO] consul: New leader elected: consul 2018/05/27 11:45:30 [INFO] consul: member 'consul' joined, marking health alive 2018/05/27 11:45:30 [INFO] agent: Synced node infoAnd subsequently, the consul server works as expected.
Now, with docker+runv, I run the same docker run ... command as above, but I get the following output in the docker logs and the container exits:
root@hyperdev-02:~# docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES bcadbb56da7c consul "docker-entrypoint..." 10 seconds ago Exited (1) 6 seconds ago consul root@hyperdev-02:~# docker logs bcadbb56da7c bootstrap = true: do not enable unless necessary ==> Starting Consul agent... ==> Error starting agent: Failed to start Consul server: Failed to start Raft: invalid argumentThe process (and container) exit with the mysterious "Failed to start Consul server" error...
Trying it with hyperctl (slightly different command as hyperctl doesn't support the -h host flag):
root@hyperdev-02:~# hyperctl run -d --name consul --publish 8500:8500 consul agent -server -bootstrap -client "0.0.0.0" Using default tag: latest latest: Pulling from library/consul ff3a5c916c92: Pull complete 4f7ceec429a3: Pull complete 479d56dd1427: Pull complete fe7ac0cf7b6b: Pull complete e2db5deeb099: Pull complete Digest: sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249 Status: Downloaded newer image for consul:latest sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249: Pulling from library/consul Digest: sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249 Status: Downloaded newer image for consul@sha256:1c7b20c43b933c34e52e56f33904dda64cd78f085cab670cfc7c3ee34429a249 POD id is consul Time to run a POD is 35350 ms root@hyperdev-02:~# hyperctl list POD ID POD Name VM name Status consul consul vm-taqgLYUXKv running root@hyperdev-02:~# hyperctl logs consul bootstrap = true: do not enable unless necessary ==> Starting Consul agent... ==> Error starting agent: Failed to start Consul server: Failed to start Raft: invalid argumentThe first thing that I noticed is that when running with runv, the PORTS in the docker ps command are missing... So I have also tried running the container with --net=host which doesn't seem to work with runv for whatever reason... And then I also tried to just --publish all of the ports that are exposed in the consul:latest Dockerfile, but that yielded the same error...
On a separate note, it'd be great to have some sort of a lookup table/doc article with a list of known incompatabilities+workarounds for normal docker containers running on the runv stack... Maybe this exists, but I haven't been able to uncover it. Thanks!