# cntr Say no to `$ apt install vim` in containers! `cntr` is a replacement for `docker exec` that brings all your developers tools with you. This is done by mounting the file system from one container or the host into the target container by creating a nested container with the help of a FUSE filesystem. This allows to ship minimal runtime image in production and limit the surface for exploits. Cntr was also published in [Usenix ATC 2018](https://www.usenix.org/conference/atc18/presentation/thalheim). See [bibtex](#bibtex) for citation. ## Demo In this two minute recording you learn all the basics of cntr: [![asciicast](https://asciinema.org/a/PQB5RnRGLIPn88a1R2MtPccdy.png)](https://asciinema.org/a/PQB5RnRGLIPn88a1R2MtPccdy) ## Features - For convenience cntr supports container names/identifier for the following container engines natively: * docker * podman * LXC * LXD * rkt * systemd-nspawn * containerd - For other container engines cntr also takes process ids (PIDs) instead of container names. ## Installation Cntr can be only supports linux. ### Pre-build static-linked binary For linux x86_64 we build static binaries for every release. More platforms can added on request. See the [release tab](https://github.com/Mic92/cntr/releases/) for pre-build tarballs. At runtime only commandline utils of the container engine in questions are required. ### Build from source All you need for compilation is rust + cargo. Checkout [rustup.rs](https://rustup.rs/) on how to get a working rust toolchain. Then run: Either: ```console $ cargo install cntr ``` Or the latest master: ```console $ cargo install --git https://github.com/Mic92/cntr ``` For offline builds we also provided a tarball with all dependencies bundled [here](https://github.com/Mic92/cntr/releases) for compilation with [cargo-vendor](https://github.com/alexcrichton/cargo-vendor). ## Usage At a high-level cntr provides two subcommands: `attach` and `exec`: - `attach`: Allows you to attach to a container with your own native shell/commands. Cntr will mount the container at `/var/lib/cntr`. The container itself will run unaffected as the mount changes are not visible to container processes. - Example: `cntr attach ` where `container_id` can be a container identifier or process id (see examples below). - `exec`: Once you are in the container, you can also run commands from the container filesystem itself. Since those might need their native mount layout at `/` instead of `/var/lib/cntr`, cntr provides `exec` subcommand to chroot to container again and also resets the environment variables that might have been changed by the shell. - Example: `cntr exec ` where `command` is an executable in the container **Note**: Cntr needs to run on the same host as the container. It does not work if the container is running in a virtual machine while cntr is running on the hypervisor. ```console $ cntr --help Cntr 1.5.1 Jörg Thalheim Enter or executed in container USAGE: cntr FLAGS: -h, --help Prints help information -V, --version Prints version information SUBCOMMANDS: attach Enter container exec Execute command in container filesystem help Prints this message or the help of the given subcommand(s) ``` ```console $ cntr attach --help cntr-attach 1.5.1 Jörg Thalheim Enter container USAGE: cntr attach [OPTIONS] [command]... FLAGS: -h, --help Prints help information OPTIONS: --effective-user effective username that should be owner of new created files on the host -t, --type Container types to try (sperated by ','). [default: all but command] [possible values: process_id, rkt, podman, docker, nspawn, lxc, lxd, containerd, command] ARGS: container id, container name or process id ... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent parsing of '-x'-like flags. [default: $SHELL] ``` ```console $ cntr exec --help cntr-exec 1.5.1 Jörg Thalheim Execute command in container filesystem USAGE: cntr exec [command]... FLAGS: -h, --help Prints help information -V, --version Prints version information ARGS: ... Command and its arguments to execute after attach. Consider prepending it with '-- ' to prevent parsing of '-x'-like flags. [default: $SHELL] ``` ### Docker 1: Find out the container name/container id: ```console $ docker run --name boxbusy -ti busybox $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 55a93d71b53b busybox "sh" 22 seconds ago Up 20 seconds boxbusy ``` Either provide a container id... ```console $ cntr attach 55a93d71b53b [root@55a93d71b53b:/var/lib/cntr]# echo "I am in a container!" [root@55a93d71b53b:/var/lib/cntr]# ip addr 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever 40: eth0@if41: mtu 1500 qdisc noqueue state UP group default link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0 inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0 valid_lft forever preferred_lft forever [root@55a93d71b53b:/var/lib/cntr]# vim etc/resolv.conf ``` ...or the container name. Use `cntr exec` to execute container native commands (while running in the cntr shell). ```console $ cntr attach boxbusy [root@55a93d71b53b:/var/lib/cntr]# cntr exec -- sh -c 'busybox | head -1' ``` You can also use Dockerfile from this repo to build a docker container with cntr: ``` console $ docker build -f Dockerfile . -t cntr # boxbusy here is the name of the target container to attach to $ docker run --pid=host --privileged=true -v /var/run/docker.sock:/var/run/docker.sock -ti --rm cntr attach boxbusy /bin/sh ``` ### Podman See docker usage, just replace `docker` with the `podman` command. ### LXD 1: Create a container and start it ```console $ lxc image import images:/alpine/edge $ lxc launch images:alpine/edge $ lxc list +-----------------+---------+------+------+------------+-----------+ | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS | +-----------------+---------+------+------+------------+-----------+ | amazed-sailfish | RUNNING | | | PERSISTENT | 0 | +-----------------+---------+------+------+------------+-----------+ ``` 2: Attach to the container with cntr ```console $ cntr attach amazed-sailfish $ cat etc/hostname amazed-sailfish ``` ### LXC 1: Create a container and start it ```console $ lxc-create --name ubuntu -t download -- -d ubuntu -r xenial -a amd64 $ lxc-start --name ubuntu -F ... Ubuntu 16.04.4 LTS ubuntu console ubuntu login: $ lxc-ls ubuntu ``` 2: Attach to container with cntr: ```console $ cntr attach ubuntu [root@ubuntu2:/var/lib/cntr]# cat etc/os-release NAME="Ubuntu" VERSION="16.04.4 LTS (Xenial Xerus)" ID=ubuntu ID_LIKE=debian PRETTY_NAME="Ubuntu 16.04.4 LTS" VERSION_ID="16.04" HOME_URL="http://www.ubuntu.com/" SUPPORT_URL="http://help.ubuntu.com/" BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" VERSION_CODENAME=xenial UBUNTU_CODENAME=xenial ``` ### rkt 1: Find out the container uuid: ```console $ rkt run --interactive=true docker://busybox $ rkt list UUID APP IMAGE NAME STATE CREATED STARTED NETWORKS c2d2e87e busybox registry-1.docker.io/library/busybox:latest running 6 minutes ago 6 minutes ago default:ip4=172.16.28.3 ``` 2: Attach with cntr ```console # make sure your container is still running! $ cntr attach c2d2e87e # Finally not the old ugly top! [gen0@rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017:/var/lib/cntr]# htop ... ``` With cntr you can also debug stage1 of rkt - even there is no support from rkt itself. ```console $ ps aux | grep stage1 joerg 13546 0.0 0.0 120808 1608 pts/12 S+ 11:10 0:00 grep --binary-files=without-match --directories=skip --color=auto stage1 root 22232 0.0 0.0 54208 2656 pts/7 S+ 10:54 0:00 stage1/rootfs/usr/lib/ld-linux-x86-64.so.2 stage1/rootfs/usr/bin/systemd-nspawn --boot --notify-ready=yes --register=true --link-journal=try-guest --quiet --uuid=c2d2e87e-e798-4341-ae93-26f6cbb7c017 --machine=rkt-c2d2e87e-e798-4341-ae93-26f6cbb7c017 --directory=stage1/rootfs --capability=CAP_AUDIT_WRITE,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FSETID,CAP_FOWNER,CAP_KILL,CAP_MKNOD,CAP_NET_RAW,CAP_NET_BIND_SERVICE,CAP_SETUID,CAP_SETGID,CAP_SETPCAP,CAP_SETFCAP,CAP_SYS_CHROOT -- --default-standard-output=tty --log-target=null --show-status=0 ``` Therefore we use the process id instead of the container uuid: ```console $ cntr attach 22232 # new and exiting territory! [root@turingmachine:/var/lib/cntr]# mount | grep pods sysfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys type sysfs (ro,nosuid,nodev,noexec,relatime) tmpfs on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755) cgroup on /var/lib/cntr/var/lib/rkt/pods/run/c2d2e87e-e798-4341-ae93-26f6cbb7c017/stage1/rootfs/sys/fs/cgroup/memory type cgroup (ro,nosuid,nodev,noexec,relatime,memory) ``` ### systemd-nspawn 1: Start container ```console $ wget https://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-16.04-server-cloudimg-amd64-root.tar.xz $ mkdir /var/lib/machines/ubuntu $ tar -xf ubuntu-16.04-server-cloudimg-amd64-root.tar.xz -C /var/lib/machines/ubuntu $ systemd-nspawn -b -M ubuntu $ machinectl list MACHINE CLASS SERVICE OS VERSION ADDRESSES ubuntu container systemd-nspawn ubuntu 16.04 - ``` 2: Attach ```console $ cntr attach ubuntu ``` ### Generic process id The minimal information needed by cntr is the process id of a container process you want to attach to. ```console # Did you now chromium uses namespaces too? $ ps aux | grep 'chromium --type=renderer' joerg 17498 11.7 1.0 1394504 174256 ? Sl 15:16 0:08 /usr/bin/chromium ``` In this case 17498 is the pid we are looking for. ```console $ cntr attach 17498 # looks quite similar to our system, but with less users [joerg@turingmachine cntr]$ ls -la / total 240 drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 . drwxr-xr-x 23 nobody nogroup 23 Mar 13 15:05 .. drwxr-xr-x 2 nobody nogroup 3 Mar 13 15:14 bin drwxr-xr-x 4 nobody nogroup 16384 Jan 1 1970 boot drwxr-xr-x 24 nobody nogroup 4120 Mar 13 14:56 dev drwxr-xr-x 52 nobody nogroup 125 Mar 13 15:14 etc drwxr-xr-x 3 nobody nogroup 3 Jan 8 16:17 home drwxr-xr-x 8 nobody nogroup 8 Feb 9 22:10 mnt dr-xr-xr-x 306 nobody nogroup 0 Mar 13 09:38 proc drwx------ 22 nobody nogroup 43 Mar 13 15:09 root ... ``` ### Containerd For containerd integration the `ctr` binary is required. You can get a binary by running: ``` console $ GOPATH=$(mktemp -d) $ go get github.com/containerd/containerd/cmd/ctr $ $GOPATH/bin/ctr --help ``` Put the resulting `ctr` binary in your `$PATH` 1: Start container ```console $ ctr images pull docker.io/library/busybox:latest $ ctr run docker.io/library/busybox:latest boxbusy $ ctr tasks lists TASK PID STATUS boxbusy 24310 RUNNING ``` 2: Attach ```console $ cntr attach boxbusy ``` It's also possible to run cntr from a container itself. This repository contains a example Dockerfile for that: ```console $ docker build -f Dockerfile.example . -t cntr $ docker save cntr > cntr.tar $ ctr images import --base-name cntr ./cntr.tar ``` In this example we attach to containerd by process id. The process id of a task is given in `ctr tasks list`. ```console $ ctr run --privileged --with-ns pid:/proc/1/ns/pid --tty docker.io/library/cntr:latest cntr /usr/bin/cntr attach 31523 /bin/sh ``` To resolve containerd names one also would need to add the `ctr` binary (~12mb) to the Dockerfile. ## Additional Config ### ZFS `cntr` requires POSIX ACLs be enabled under ZFS. By default, Linux ZFS doesn't have POSIX ACLs enabled. This results in the following error when trying to `attach`: ```console unable to move container mounts to new mountpoint: EOPNOTSUPP: Operation not supported on transport endpoint ``` To enable POSIX ACLs on the ZFS dataset: ```console $ zfs set acltype=posixacl zpool/media $ zfs set xattr=sa zpool/media # optional, but encouraged for best performance ``` # How it works Cntr is container-agnostic: Instead of interfacing with container engines, it implements the underlying operating system API. It treats every container as a group of processes, that it can inherit properties from. Cntr inherits the following container properties: * Namespaces (mount, uts, pid, net, cgroup, ipc) * Cgroups * Apparamor/selinux * Capabilities * User/group ids * Environment variables * The following files: /etc/passwd, /etc/hostname, /etc/hosts, /etc/resolv.conf Under the hood it spawns a shell or user defined program that inherits the full context of the container and mount itself as a fuse filesystem. We extensively evaluated the correctness and performance of cntr's filesystem using [xfstests](https://github.com/Mic92/xfstests-cntr) and a wide range of filesystem performance benchmarks (iozone, pgbench, dbench, fio, fs-mark, postmark, ...) # Related projects - [nsenter](https://manpages.debian.org/testing/manpages-de/nsenter.1.de.html) - Only covers linux namespaces and the user is limited to tools installed in the containers - [toolbox](https://github.com/coreos/toolbox) - Does attach from a container to the host, this is the opposite of what Cntr is doing # Bibtex We published a paper with all technical details about Cntr in [Usenix ATC 2018](https://www.usenix.org/conference/atc18/presentation/thalheim). ```bibtex @inproceedings{cntr-atc18, author = {J{\"o}rg Thalheim and Pramod Bhatotia and Pedro Fonseca and Baris Kasikci}, title = {Cntr: Lightweight {OS} Containers}, booktitle = {2018 {USENIX} Annual Technical Conference ({USENIX} {ATC} 18)}, year = {2018}, } ```