.. highlight:: yaml .. _container_config: ======================= Container Configuration ======================= Container configuration is a YAML file which is usually put into ``/config/.yaml`` into container image itself. .. note:: Curently container configuration may be put into any folder inside the image, but we may fix this folder later. The arbitrary path for container configuration may be a security vulnerability. The somewhat minimal configuration is looks like following: .. code-block:: yaml kind: Daemon user-id: 1 volumes: /tmp: !Tmpfs { size: 100m } executable: /bin/sleep arguments: [60] .. _container_variables: Variables ========= Container can declare some things, that can be changed in specific instantiation of the service, for example: .. code-block:: yaml variables: tcp_port: !TcpPort kind: Daemon user-id: 1 volumes: /tmp: !Tmpfs { size: 100m } executable: /bin/some_program arguments: - "--listen=localhost:@{tcp_port}" The ``variables`` key declares variable names and types. Value for these variables can be provided in ``variables`` in :ref:`process_config`. There are the following types of variables: TcpPort Allows a number between 1-65535 and ensures that the number matches port range allowed in sandbox (see :opt:`allow-tcp-ports`) .. _activation: .. versionchanged:: 0.17.4 Added ``activation`` parameter as a shortcut to support systemd activation protocol. I.e. the following (showing two ports for more comprehensive example): .. code-block:: yaml variables: port1: !TcpPort { activation: systemd } port2: !TcpPort { activation: systemd } Means to add something like this: .. code-block:: yaml variables: port1: !TcpPort port2: !TcpPort tcp-ports: "@{port1}": fd: 3 "@{port2}": fd: 4 environ: LISTEN_FDS: 1 LISTEN_FDNAMES: "port1:port2" LISTEN_PID: "@{lithos:pid}" This works for any number of sockets. And it requires that ``LISTEN_FDS`, ``LISTEN_FDNAMES``, ``LISTEN_PID`` were absent in the ``environ`` as written in the file. Also it doesn't allow fine-grained control over parameters of the socket and file descriptor numbers. Use full form if you need specific options. Choice Allows a value from a fixed set of choices (example: ``!Choice ["high-priority", "low-priority"]``) Name Allows a value that matches regex ``^[0-9a-zA-Z_-]+$``. Useful for passing names of things into a script without having a chance to keep value unescaped when passing somewhere within a script or using it as a filename. .. versionadded:: 0.10.3 DottedName Allows arbitrary DNS-like name. It's defined as dot-separated name with only alphanumeric and underscores, where no component could start or end with a dash and no consequent dots allowed. .. versionadded:: 0.17.4 All entries of ``@{variable_name}`` are substituted in the following fields: 1. :opt:`arguments` 2. The values of :opt:`environ` (not in the keys yet) 3. The key in the :opt:`tcp-ports` (i.e. port number) The expansion in any other place does not work yet, but may be implemented in the future. Only **declared** variables can be substituted. Trying to substitute undeclared variables or non-existing built-in variable results into configuration syntax error. There are the number of builtin variables that start with ``lithos:``: lithos:name Name of the process, same as inserted in ``LITHOS_NAME`` environment variable lithos:config_filename Full path of this configuration file as visible from within container lithos:pid Pid of the process as visible inside of the container. Note: this variable can only be in environment and can only be full value of the variable. I.e. `PID: "@{lithos:pid}"` is fine, but `PID: "pid is @{lithos:pid}"` is **not allowed**. (In most cases this variable is exaclty ``2``, this is expected but might not be always true in some cases). More built-in variables may be added in the future. Built-in variables doesn't have to be declared. Reference ========= .. opt:: kind One of ``Daemon`` (default), ``Command`` or ``CommandOrDaemon``. The ``Daemon`` is long-running process that is monitored by supervisor. The ``Command`` things are just one-off tasks, for example to initialize local file system data, or to check health of daemon process. The ``Command`` things are run by ``lithos_cmd`` utility The ``CommandOrDaemon`` may be used in both ways, based on how it was declared in :ref:`Process Config `. In the command itself you can distinguish how it is run by ``/cmd.`` in ``LITHOS_NAME`` or cgroup name or better you can pass :ref:`variable ` to a specific command and/or daemon. .. versionadded:: 0.10.3 ``ContainerOrDaemon`` mode .. opt:: user-id The numeric user indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of ``0`` is not allowed. .. opt:: group-id The numeric group indentifier for the process. It must be one of the allowed values in lithos configuration. Usually value of ``0`` is not allowed. .. opt:: memory-limit The memory limit for process and it's children. This is enforced by cgroups, so this needs `memory` cgroup to be enabled (otherwise its no-op). See :opt:`cgroup-controllers` for more info. Default: nolimit. You can use ``ki``, ``Mi`` and ``Gi`` units for memory accounting. See integer-units_. .. versionchanged:: 0.14.0 Previously it only set ``memory.limit_in_bytes`` but now it also sets ``memory.memsw.limit_in_bytes`` if the latter exists (otherwise skipping silently). This helps to kill processes earlier instead of swapping out to disk. .. opt:: cpu-shares The number of CPU shares for the process. Default is ``1024`` which means all processes get equal share. You may split them to different values like ``768`` for one process and ``256`` for another one. This is enforced by cgroups, so this needs `cpu` cgroup to be enabled (otherwise its no-op). See :opt:`cgroup-controllers` for more info. .. opt:: fileno-limit The limit on file descriptors for process. Default ``1024``. .. opt:: restart-timeout The minimum time to wait between subsequent restarts of failed processes in seconds. This is to ensure that it doesn't boggles down CPU. Default is ``1`` second. It's enough so that lithos itself do not hang. But it should be bigger for heavy-weight processes. Note: this is time between restarts, i.e. if process were running more than this number of seconds it will be restarted immediately. .. opt:: kill-timeout (default ``5`` seconds) The time to wait for application to die. If it is not dead by this number of seconds we kill it with ``KILL``. You should not rely on this timeout to be precise for multiple reasons: 1. Unidentified children are killed with a default timeout (5 sec). This includes children which are being killed when their configuration is removed. 2. When lithos is restarted (i.e. to reload a configuration) during the timeout, the timeout is reset. I.e. the process may hang more than this time. .. opt:: executable The path to executable to run. Only absolute paths are allowed. .. opt:: arguments The list of arguments for the command. Except argument zero. .. opt:: environ The mapping of values that are set for process. You must set all needed environment variables here. The only variable that is propagated by default is ``TERM``. Also few special ``LITHOS_`` variables may be set. This means you must set all the basic ``LANG``, ``HOME`` and so on explicitly. This is to ensure that your environment is always the same regardless of where you run process. .. opt:: secret-environ Similarlty to ``environ`` but contains encrypted environment variables. For example:: secret-environ: DB_PASSWORD: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc Note: if environment variable is both in ``environ`` and ``secret-environ`` which one overrides is not specified for now. You can encrypt variables using ``lithos_crypt``:: lithos_crypt encrypt -k key.pub -d "secret" -n "some.namespace" You only need public key for encryption. So the idea is that public key is published somewhere and anyone, even users having to access to server/private key can add a secret. The ``-n`` / ``--namespace`` parameter must match one of the :opt:`secrets-namespaces` defined for project's sandbox. Usually there is only one private key for every deployment (cluster), and a single namespace per project. But in some cases you might need single lithos config for multiple destinations or just want to rotate private key smoothly. So you can put secret(s) encoded for multiple keys and/or namespaces: .. code-block:: yaml secret-environ: DB_PASSWORD: - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE - v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q== - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE Note: technically you can encrypt different secrets here, we can't enforce that, but it's very discouraged. The underlying encyrption is curve25519xsalsa20poly1305 which is compatible with libnacl and libsodium. See :ref:`encrypted-vars` for more info. This option conflicts with :opt:`secret-environ-file`. .. opt:: secret-environ-file Path to the file where to read secret environ from. Instead of including ``secret-environ`` in the container config itself you can use a separate file where data is contained. This is useful to keep single set of secrets shared between multiple containers. The target file is also yaml, but it containers just mapping of names of the secrets to their values (or lists). For example: .. code-block:: yaml PASSWD1: v2:ROit92I5:82HdsExJ:Gd3ocJsr:Hp3pngQZUos5b8ioKVUx40kegM1uDsYWwsWqC1cJ1/1KmQPQQWJZe86xgl1EOIxbuLj6PUlBH8yz5qCnWp//Ofbc PASSWD2: - v2:h+M9Ue9x:82HdsExJ:Gd3ocJsr:/+f4ezLfKIP/mp0xdF7H6gfdM7onHWwbGFQX+M1aB+PoCNQidKyz/1yEGrwxD+i+qBGwLVBIXRqIc5FJ6/hw26CE - v2:ROit92I5:cX9ciQzf:Gd3ocJsr:LMHBRtPFpMRRrljNnkaU6Y9JyVvEukRiDs4mitnTksNGSX5xU/zADWDwEOCOtYoelbJeyDdPhM7Q1mEOSwjeyO317Q== Absolute paths here interpreted relative to the container root and relative paths are interpreted relative to the container config itself. Note: we currently support reading file from container's filesystem only, whether reading from a volume works or not is *unspecified* at the moment. This option conflicts with :opt:`secret-environ`. .. versionadded: v0.18.2 .. opt:: workdir The working directory for target process. Default is ``/``. Working directory must be absolute. .. opt:: resolv-conf Parameters of the ``/etc/resolv.conf`` file to generate. Default configuration is: .. code-block:: yaml resolv-conf: mount: nil # which basically means "auto" copy-from-host: true Which means ``resolv.conf`` from host where lithos is running is copied to the "state" directory of the container. Then if ``/etc/resolv.conf`` in container is a file (and not a symlink) resolv conf is mounted over the ``/etc/resolv.conf``. More options are expected to be added later. .. versionchanged:: 0.15.0 ``mount`` option added. Previously to make use of ``resolv.conf`` you should symlink ``ln -s /state/resolv.conf /etc/resolv.conf`` in the container's image. Another change is that ``copy-from-host`` copies file that is specified in sandbox's ``resolv.conf`` which default to ``/etc/resolv.conf`` but may be different. Parameters: copy-from-host (default ``true``) Copy ``resolv.conf`` file from host machine. Note: even if ``copy-from-host`` is ``true``, :opt:`additional-hosts` from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places. .. versionchanged:: v0.11.0 The parameter used to be ``false`` by default, because we were thinking about better (perceived) isolation. mount (default ``nil``, which means "auto") Mount copied ``resolv.conf`` file over ``/etc/resolf.conf``. `nil` enables mounting if ``/etc/resolv.conf`` is present in the container and is a file (not a symlink) and also ``copy-from-host`` is true .. versionadded:: 0.15.0 .. opt:: hosts-file Parameters of the ``/etc/hosts`` file to generate. Default configuration is:: hosts-file: mount: nil # which basically means "auto" localhost: true public-hostname: true copy-from-host: false .. versionchanged:: 0.15.0 ``mount`` option added. Previously to make use of ``resolv.conf`` you should symlink ``ln -s /state/resolv.conf /etc/resolv.conf`` in the container's image. Another change is that ``copy-from-host`` copies file that is specified in sandbox's ``resolv.conf`` which default to ``/etc/resolv.conf`` but may be different. Parameters: copy-from-host (default ``true``) Copy hosts file from host machine. Note: even if ``copy-from-host`` is ``true``, :opt:`additional-hosts` from sandbox config work, which may lead to duplicate or conflicting entries if some names are specified in both places. .. versionchanged:: v0.11.0 The parameter used to be ``false`` by default, because we were thinking about better (perceived) isolation. And also because hostname in Ubuntu doesn't resolve to real IP of the host. But we find those occassions where it matters to be quite rare in practice and using ``hosts-file`` as well as ``resolv.conf`` from the host system as the most expected and intuitive behavior. mount (default ``nil``, which means "auto") Mount produced ``hosts`` file over ``/etc/hosts``. `nil` enables mounting if ``/etc/hosts`` is present in the container and is a file (not a symlink). Value of ``true`` fails if ``/etc/hosts`` is not a file. Value of ``false`` leaves ``/etc/hosts`` intact. .. versionadded:: 0.15.0 localhost (default is true when ``copy-from-host`` is false) A boolean which defines whether to add ``127.0.0.1 localhost`` record to ``hosts`` public-hostname (default is true when ``copy-from-host`` is false) Add to ``hosts`` file the result of ``gethostname`` system call along with the ip address that name resolves into. .. opt:: uid-map, gid-map The list of mapping for uids(gids) in the user namespace of the container. If they are not specified the user namespace is not used. This setting allows to run processes with ``uid`` zero without the risk of being the ``root`` on host system. Here is a example of maps:: uid-map: - {inside: 0, outside: 1000, count: 1} - {inside: 1, outside: 1, count: 1} gid-map: - {inside: 0, outside: 100, count: 1} .. note:: Currently you may have uid-map either in a sandbox or in a container config, not both. .. opt:: stdout-stderr-file This redirects both stdout and stderr to a file. The path is opened inside the container. So must reside on one of the mounted writeable :ref:`volumes`. Probably you want :volume:`Persistent` volume. While it can be on :volume:`Tmpfs` or :volume:`Statedir` the applicability of such thing is very limited. Usually log is put into the directory specified by :opt:`stdio-log-dir`. .. opt:: interactive (default ``false``) Useful only for containers of kind ``Command``. If ``true`` lithos_cmd doesn't clobber stdin and doesn't redirect stdout and stderr to a log file, effectively allowing command to be used for interactive commands or as a part of pipeline. .. note:: for certain use cases, like pipelines it might be better to use fifo's (see ``man mkfifo``) and a ``Daemon`` instead of this one because daemons may be restarted on death or for software upgrade, while ``Command`` is not supervised by lithos. .. versionadded:: 0.6.3 .. versionchanged:: ≥0.5 Commands were always interactive .. opt:: restart-process-only (default ``false``) If true when restarting process (i.e. in case process died or was killed), lithos restarts just the failed process. This means container will not be recreated, volumes will not be remounted, tmpfs will not be cleaned and some daemon processes may leave running. By default ``lithos_knot`` which is pid 1 in the container exits when process dies. Which means all other processes will die on ``KILL`` signal, and container will be removed and created again. It's a little bit slower but safer default. This leaves no hanging daemons, orphan files in state dir and tmpfs garbage. .. opt:: volumes The mapping of mountpoint to volume definition. See :ref:`volumes` for more info .. opt:: tcp-ports Binds address and provides file descriptor to the child process. All the children receive dup of the same file descriptor, so may all do ``accept()`` simultaneously. The configuration looks like:: tcp-ports: 7777: fd: 3 host: 0.0.0.0 listen-backlog: 128 reuse-addr: true reuse-port: false All the fields except ``fd`` are optional. Programs may require to pass listening file descriptor number by some means (usually environment). For example to run nginx with port bound (so you don't need to start it as root) you need:: tcp-ports: 80: fd: 3 set-non-block: true environ: NGINX: "3;" To run gunicorn you may want:: tcp-ports: 80: fd: 3 environ: GUNICORN_FD: "3" More examples are in :ref:`tcp-ports-tips` Parameters: *key* TCP port number. .. warning:: * The paramters (except ``fd``) do not change after socket is bound even if configuration change * You can't bind same port with different hostnames in a **single process** (previously there was a global limit for the single port for whole lithos master, currently this is limited just because ``tcp-ports`` is a mapping) Port parameter should be unique amongst all containers. But sharing port works because it is useful if you are doing smooth software upgrade (i.e. you have few old processes running and few new processes running both sharing same port/file-descriptor). *Running them on single port is not the best practices for smooth software upgrade but that topic if out of scope of this documentation.* fd *Required*. File descriptor number host (default is ``0.0.0.0`` meaning all addresses) Host to bind to. It must be IP address, hostname is not supported. listen-backlog (default ``128``) the value to pass to the `listen()` system call. The value is capped by ``net.core.somaxconn`` reuse-addr (default ``true``) Sets ``SO_REUSEADDR`` socket option reuse-port (default ``false``) If set to ``true`` this changes behavior of the lithos with respect of the socket. In default case lithos binds socket as quick as possible and passes to each child on start. When this set to ``true``, lithos creates a separate socket and calls bind for each process start. This has two consequences: * Socket is not bound when no processes started (i.e. they are failing) * Each process gets separate in-kernel queue of connections to accept This should be set to ``true`` only on very high performant servers that experience assymetric workload in default case. set-non-block (default ``false``) Sets socket into non-blocking mode. This is usually done by an application itself but some of them (especially ones, that don't expect socket to be created by an external utility, e.g. nginx) don't do it themselves. external (default ``false``) If set to ``true`` listen on the port in the external network (host network of the system not bridged network). This is only effective if :opt:`bridged-network` is enabled for container. .. versionchanged:: 0.17.0 Previously we only allowed external ports to be declared in lithos config. It was expected that container in bridged network can listen port itself. But it turned out file descriptors are still convenient for some use-cases even inside a bridge. .. opt:: metadata (optional) Allows to add arbitrary metadata to lithos configuration file. Lithos does not use and does not validate this data in any way (except that it must be a valid YAML). The metadata can be used by other tools that inspect lithos configs and extract data from it. In particular, we use metadata for our deployment tools (to keep configuration files more consolidated instead of keeping then in small fragments). .. opt:: normal-exit-codes (optional) A list of exit codes which are considered normal for process death. This currently only improves ``failures`` metric. See :ref:`Determining Failure `. Note: by default even ``0`` exit code is considered an error for daemons, and for commands (``lithos_cmd``) ``0`` is considered successful. This setting is intended for daemons which may voluntarily exit for some reason (soft memory limit, version upgrade, configuration reload). It's not recommended to add `0` or `1` into the list, as some commands threat them pretty arbitrarily. For example `0` is exit code of most utilities running `--help` so this mistake will not be detected. And `1` is used for arbitrary crashes in scripting languages. So the good idea is to define some specific code in range of `8..120` to define successful exit. .. _integer-units: http://rust-quire.readthedocs.io/en/latest/user.html#units