# Developer use of CI Docker images There are two available flavors of Envoy Docker images for Linux, based on Ubuntu and Alpine Linux and an image based on Windows2019. ## Ubuntu Envoy image The Ubuntu based Envoy Docker image at [`envoyproxy/envoy-build:`](https://hub.docker.com/r/envoyproxy/envoy-build/) is used for CI checks, where `` is specified in [`envoy_build_sha.sh`](https://github.com/envoyproxy/envoy/blob/master/ci/envoy_build_sha.sh). Developers may work with the latest build image SHA in [envoy-build-tools](https://github.com/envoyproxy/envoy-build-tools/blob/master/toolchains/rbe_toolchains_config.bzl#L8) repo to provide a self-contained environment for building Envoy binaries and running tests that reflects the latest built Ubuntu Envoy image. Moreover, the Docker image at [`envoyproxy/envoy-dev:`](https://hub.docker.com/r/envoyproxy/envoy-dev/) is an image that has an Envoy binary at `/usr/local/bin/envoy`. The `` corresponds to the master commit at which the binary was compiled. Lastly, `envoyproxy/envoy-dev:latest` contains an Envoy binary built from the latest tip of master that passed tests. ## Alpine Envoy image Minimal images based on Alpine Linux allow for quicker deployment of Envoy. The Alpine base image is only built with symbols stripped. To get the binary with symbols, use the corresponding Ubuntu based debug image. The image is pushed with two different tags: `` and `latest`. Parallel to the Ubuntu images above, `` corresponds to the master commit at which the binary was compiled, and `latest` corresponds to a binary built from the latest tip of master that passed tests. ## Windows 2019 Envoy image The Windows 2019 based Envoy Docker image at [`envoyproxy/envoy-build-windows2019:`](https://hub.docker.com/r/envoyproxy/envoy-build-windows2019/) is used for CI checks, where `` is specified in [`envoy_build_sha.sh`](https://github.com/envoyproxy/envoy/blob/master/ci/envoy_build_sha.sh). Developers may work with the most recent `envoyproxy/envoy-build-windows2019` image to provide a self-contained environment for building Envoy binaries and running tests that reflects the latest built Windows 2019 Envoy image. # Build image base and compiler versions Currently there are three build images for Linux and one for Windows: * `envoyproxy/envoy-build` — alias to `envoyproxy/envoy-build-ubuntu`. * `envoyproxy/envoy-build-ubuntu` — based on Ubuntu 18.04 (Bionic) with GCC 9 and Clang 10 compiler. * `envoyproxy/envoy-build-centos` — based on CentOS 7 with GCC 9 and Clang 10 compiler, this image is experimental and not well tested. * `envoyproxy/envoy-build-windows2019` — based on Windows 2019 LTS with VS 2019 Build Tools. The source for these images is located in the [envoyproxy/envoy-build-tools](https://github.com/envoyproxy/envoy-build-tools) repository. We use the Clang compiler for all Linux CI runs with tests. We have an additional Linux CI run with GCC which builds binary only. # C++ standard library As of November 2019 after [#8859](https://github.com/envoyproxy/envoy/pull/8859) the official released binary is [linked against libc++ on Linux](https://github.com/envoyproxy/envoy/blob/master/bazel/README.md#linking-against-libc-on-linux). To override the C++ standard library in your build, set environment variable `ENVOY_STDLIB` to `libstdc++` or `libc++` and run `./ci/do_ci.sh` as described below. # Building and running tests as a developer The `./ci/run_envoy_docker.sh` script can be used to set up a Docker container on Linux and Windows to build an Envoy static binary and run tests. The build image defaults to `envoyproxy/envoy-build-ubuntu` on Linux and `envoyproxy/envoy-build-windows2019` on Windows, but you can choose build image by setting `IMAGE_NAME` in the environment. In case your setup is behind a proxy, set `http_proxy` and `https_proxy` to the proxy servers before invoking the build. ```bash IMAGE_NAME=envoyproxy/envoy-build-ubuntu http_proxy=http://proxy.foo.com:8080 https_proxy=http://proxy.bar.com:8080 ./ci/run_envoy_docker.sh ' ``` ## On Linux An example basic invocation to build a developer version of the Envoy static binary (using the Bazel `fastbuild` type) is: ```bash ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev' ``` The Envoy binary can be found in `/tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild` on the Docker host. You can control this by setting `ENVOY_DOCKER_BUILD_DIR` in the environment, e.g. to generate the binary in `~/build/envoy/source/exe/envoy-fastbuild` you can run: ```bash ENVOY_DOCKER_BUILD_DIR=~/build ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev' ``` For a release version of the Envoy binary you can run: ```bash ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.release.server_only' ``` The build artifact can be found in `/tmp/envoy-docker-build/envoy/source/exe/envoy` (or wherever `$ENVOY_DOCKER_BUILD_DIR` points). For a debug version of the Envoy binary you can run: ```bash ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.debug.server_only' ``` The build artifact can be found in `/tmp/envoy-docker-build/envoy/source/exe/envoy-debug` (or wherever `$ENVOY_DOCKER_BUILD_DIR` points). To leverage a [bazel remote cache](https://github.com/envoyproxy/envoy/tree/master/bazel#advanced-caching-setup) add the http_remote_cache endpoint to the BAZEL_BUILD_EXTRA_OPTIONS environment variable ```bash ./ci/run_envoy_docker.sh "BAZEL_BUILD_EXTRA_OPTIONS='--remote_http_cache=http://127.0.0.1:28080' ./ci/do_ci.sh bazel.release" ``` The `./ci/run_envoy_docker.sh './ci/do_ci.sh '` targets are: * `bazel.api` — build and run API tests under `-c fastbuild` with clang. * `bazel.asan` — build and run tests under `-c dbg --config=clang-asan` with clang. * `bazel.asan ` — build and run a specified test or test dir under `-c dbg --config=clang-asan` with clang. * `bazel.debug` — build Envoy static binary and run tests under `-c dbg`. * `bazel.debug ` — build Envoy static binary and run a specified test or test dir under `-c dbg`. * `bazel.debug.server_only` — build Envoy static binary under `-c dbg`. * `bazel.dev` — build Envoy static binary and run tests under `-c fastbuild` with clang. * `bazel.dev ` — build Envoy static binary and run a specified test or test dir under `-c fastbuild` with clang. * `bazel.release` — build Envoy static binary and run tests under `-c opt` with clang. * `bazel.release ` — build Envoy static binary and run a specified test or test dir under `-c opt` with clang. * `bazel.release.server_only` — build Envoy static binary under `-c opt` with clang. * `bazel.sizeopt` — build Envoy static binary and run tests under `-c opt --config=sizeopt` with clang. * `bazel.sizeopt ` — build Envoy static binary and run a specified test or test dir under `-c opt --config=sizeopt` with clang. * `bazel.sizeopt.server_only` — build Envoy static binary under `-c opt --config=sizeopt` with clang. * `bazel.coverage` — build and run tests under `-c dbg` with gcc, generating coverage information in `$ENVOY_DOCKER_BUILD_DIR/envoy/generated/coverage/coverage.html`. * `bazel.coverage ` — build and run a specified test or test dir under `-c dbg` with gcc, generating coverage information in `$ENVOY_DOCKER_BUILD_DIR/envoy/generated/coverage/coverage.html`. * `bazel.coverity` — build Envoy static binary and run Coverity Scan static analysis. * `bazel.msan` — build and run tests under `-c dbg --config=clang-msan` with clang. * `bazel.msan ` — build and run a specified test or test dir under `-c dbg --config=clang-msan` with clang. * `bazel.tsan` — build and run tests under `-c dbg --config=clang-tsan` with clang. * `bazel.tsan ` — build and run a specified test or test dir under `-c dbg --config=clang-tsan` with clang. * `bazel.fuzz` — build and run fuzz tests under `-c dbg --config=asan-fuzzer` with clang. * `bazel.fuzz ` — build and run a specified fuzz test or test dir under `-c dbg --config=asan-fuzzer` with clang. If specifying a single fuzz test, must use the full target name with "_with_libfuzzer" for ``. * `bazel.compile_time_options` — build Envoy and run tests with various compile-time options toggled to their non-default state, to ensure they still build. * `bazel.compile_time_options ` — build Envoy and run a specified test or test dir with various compile-time options toggled to their non-default state, to ensure they still build. * `bazel.clang_tidy ` — build and run clang-tidy specified source files, if no files specified, runs against the diff with the last GitHub commit. * `check_format`— run `clang-format` and `buildifier` on entire source tree. * `fix_format`— run and enforce `clang-format` and `buildifier` on entire source tree. * `check_spelling`— run `misspell` on entire project. * `fix_spelling`— run and enforce `misspell` on entire project. * `check_spelling_pedantic`— run `aspell` on C++ and proto comments. * `docs`— build documentation tree in `generated/docs`. ## On Windows An example basic invocation to build the Envoy static binary and run tests is: ```bash ./ci/run_envoy_docker.sh './ci/windows_ci_steps.sh' ``` You can modify `./ci/windows_ci_steps.sh` to modify `bazel` arguments, tests to run, etc. as well as set environment variables to adjust your container build environment as described above. The Envoy binary can be found in `C:\Windows\Temp\envoy-docker-build\envoy\source\exe` on the Docker host. You can control this by setting `ENVOY_DOCKER_BUILD_DIR` in the environment, e.g. to generate the binary in `C:\Users\foo\build\envoy\source\exe` you can run: ```bash ENVOY_DOCKER_BUILD_DIR="C:\Users\foo\build" ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev' ``` Note the quotations around the `ENVOY_DOCKER_BUILD_DIR` value to preserve the backslashes in the path. If you would like to run an interactive session to keep the build container running (to persist your local build environment), run: ```bash ./ci/run_envoy_docker.sh 'bash' ``` From an interactive session, you can invoke `bazel` manually or use the `./ci/windows_ci_steps.sh` script to build and run tests. # Testing changes to the build image as a developer While all changes to the build image should eventually be upstreamed, it can be useful to test those changes locally before sending out a pull request. To experiment with a local clone of the upstream build image you can make changes to files such as build_container.sh locally and then run: ```bash DISTRO=ubuntu cd ci/build_container LINUX_DISTRO="${DISTRO}" CIRCLE_SHA1=my_tag ./docker_build.sh # Wait patiently for quite some time cd ../.. IMAGE_NAME="envoyproxy/envoy-build-${DISTRO}" IMAGE_ID=my_tag ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.whatever' ``` This build the Ubuntu based `envoyproxy/envoy-build-ubuntu` image, and the final call will run against your local copy of the build image. # macOS Build Flow The macOS CI build is part of the [Azure Pipelines](https://dev.azure.com/cncf/envoy/_build) workflow. Dependencies are installed by the `ci/mac_ci_setup.sh` script, via [Homebrew](https://brew.sh), which is pre-installed on the [Azure Pipelines macOS image](https://github.com/actions/virtual-environments/blob/main/images/macos/macos-10.15-Readme.md). The dependencies are cached and re-installed on every build. The `ci/mac_ci_steps.sh` script executes the specific commands that build and test Envoy. Note that the full version of Xcode (not just Command Line Tools) is required. # Coverity Scan Build Flow [Coverity Scan Envoy Project](https://scan.coverity.com/projects/envoy-proxy) Coverity Scan static analysis is not run within Envoy CI. However, Envoy can be locally built and submitted for analysis. A Coverity Scan Envoy project token must be generated from the [Coverity Project Settings](https://scan.coverity.com/projects/envoy-proxy?tab=project_settings). Only a Coverity Project Administrator can create a token. With this token, running `ci/do_coverity_local.sh` will use the Ubuntu based `envoyproxy/envoy-build-ubuntu` image to build the Envoy static binary with the Coverity Scan tool chain. This process generates an artifact, envoy-coverity-output.tgz, that is uploaded to Coverity for static analysis. To build and submit for analysis: ```bash COVERITY_TOKEN={generated Coverity project token} ./ci/do_coverity_local.sh ```