# HowTo: Reproduce CI jobs interactively.

We rely on Spack, driven by uberenv, to build MFEM dependencies automatically
in CI. Then we use a script that will be retrieve the configuration files
generated and use it to build MFEM before running the tests.

This process is straightforward to reproduce. However the steps are not easy
to extract from the CI configuration, hence this article.

**WARNING**

This will only work on the same machines as used for CI. To add a new machine
you will need to add a corresponding configuration in mfem-uberenv repo.

**NOTE**

The `build_and_test` script controlling the build of MFEM and its dependencies
has two modes:
- The CI mode will build the deps, then MFEM and then run tests for the
  specified spec.
- The Interactive mode, presented here, where we first build the dependencies
  with the `--deps-only` option and then use the spack generated configuration
  files to build MFEM.

Those modes are essentially the same, but we emphasize building the
dependencies as a first isolated step because it makes it clear what is
happening and how to use this workflow.

## Prerequisite: Retrieve Uberenv

```bash
tests/gitlab/get_mfem_uberenv
```

We have a script to automatically download and install an MFEM-tuned uberenv
instance.

Uberenv will be placed in `tests/uberenv`, along with spack configuration files
for each machine we currently support, and possibly some Spack packages we
patch.


## Install Dependencies

### Option 1: Using the CI script

```bash
./tests/gitlab/build_and_test --deps-only --spec "%gcc@6.1.0 +sundials"
```

The CI script has three steps that can be run individually with the options
`--deps-only`, `--build-only`, `--test-only`.

We ask to build only the dependencies, and we need to provide a spack spec
through the `--spec` option.

Virtually any spec can be provided, but you should check which compilers are
defined in the spack configuration
(`tests/uberenv/spack-configs/<sys_type>/compilers.yaml`).

As a result, dependencies will be installed under `uberenv_libs`.  The
configuration files `spack-config.mk` and `spack_config.hpp` will be generated
in `config`. They are just copies of the `config.mk` and `\_config.hpp`
generated by the MFEM spack package. This copy is meant to make sure in CI that
those files were generated, otherwise `make all` would just regenerate them.

**NOTE**

When `build_and_test` needs to build the dependencies, i.e. (a) `--deps-only` is
used, or (b) none of the `--XXX-only` options is used, then the script behaves
slightly differently. In case (b), it will build and install dependencies in
`/dev/shm` for better performance. However, this is only valid if we don't want
the installation to persist. Installation will happen locally to the uberenv
directory in case (a), i.e. if `--deps-only` is used.

### Option 2: Calling uberenv directly

```bash
python ./tests/uberenv/uberenv.py --spec="%gcc@6.1.0 +sundials"
```

This is essentially the command the CI script runs in the end.

**NOTE**

When using this command, the configuration files will not be copied (saved),
but still ready to use.

## Build and test MFEM

### Option 1: Without using scripts

```bash
\# optional: the spack config is saved in copies (in case you reconfigure MFEM
\# in the meantime).
cp config/spack-config.mk config/config.mk
cp config/spack_config.hpp config/_config.hpp
```

```bash
make all -j 8
make test
```

The key here is that the configuration files were generated during the spack
run: they will point to the dependencies location, and apply any option
selected with MFEM spec variants in Spack.

### Option 2: Using the CI script

```bash
./tests/gitlab/build_and_test --build-only
```

We could use `--test-only` option, which would also build MFEM to make
sure to use the spack generated configuration file.

**NOTE**

Using the CI script requires `config/spack-config.mk` and
`config/spack_config.hpp` to exist.