# Nydus Setup for Containerd Environment

This document will walk through how to setup a nydus image service to work with containerd. It assumes that you already have `containerd` installed. If not, please refer to [containerd documents](https://github.com/containerd/containerd/blob/master/docs/ops.md) on how to install and set it up.

## Install All Nydus Binaries

1. Get `nydus-image`, `nydusd`, `nydusify`, `nydusctl` and `nydus-overlayfs` binaries from [release](https://github.com/dragonflyoss/image-service/releases/latest) page.

```bash
sudo install -D -m 755 nydusd nydus-image nydusify nydusctl nydus-overlayfs /usr/bin
```

2. Get `containerd-nydus-grpc` (nydus snapshotter) binary from nydus-snapshotter [release](https://github.com/containerd/nydus-snapshotter/releases/latest) page.

```bash
sudo install -D -m 755 containerd-nydus-grpc /usr/bin
```

## Start Nydus Snapshotter

Nydus provides a containerd remote snapshotter `containerd-nydus-grpc` (nydus snapshotter) to prepare container rootfs with nydus formatted images.

1. Prepare a `nydusd` configuration to `/etc/nydus/nydusd-config.fusedev.json`:

```bash
$ sudo tee /etc/nydus/nydusd-config.fusedev.json > /dev/null << EOF
{
  "device": {
    "backend": {
      "type": "registry",
      "config": {
        "scheme": "",
        "skip_verify": true,
        "timeout": 5,
        "connect_timeout": 5,
        "retry_limit": 4,
        "auth": "YOUR_LOGIN_AUTH="
      }
    },
    "cache": {
      "type": "blobcache",
      "config": {
        "work_dir": "cache"
      }
    }
  },
  "mode": "direct",
  "digest_validate": false,
  "iostats_files": false,
  "enable_xattr": true,
  "fs_prefetch": {
    "enable": true,
    "threads_count": 4
  }
}
EOF
```

Please refer to the nydusd [doc](./nydusd.md) to learn more options.

⚠️ Note:

- The `device.backend.config.scheme` is the URL scheme for the registry. Leave it empty for automatic detection, or specify `https` or `http` depending on your registry server configuration.
- The `device.backend.config.auth` is the base64 encoded `username:password` required by nydusd to lazily pull image data from an authenticated registry. The nydus snapshotter will automatically read the authentication information from the `$HOME/.docker/config.json` configuration file. If you are using a registry that requires authentication, you should replace `YOUR_LOGIN_AUTH=` with your own login information.
- The `device.backend.config.skip_verify` allows you to skip the insecure https certificate checks for the registry, only set it to `true` when necessary. Note that enabling this option is a security risk for the connection to registry, so you should only use this when you are sure it is safe.
- The `fs_prefetch.enable` option enables nydusd to prefetch image data in a background thread, which can make container startup faster when it needs to read a large amount of image data. Set this to `false` if you don't need this functionality.

2. [Optional] Cleanup snapshotter environment:

Make sure the default nydus snapshotter root directory is clear.

```
sudo rm -rf /var/lib/containerd-nydus
```

3. Start `containerd-nydus-grpc` (nydus snapshotter):

```bash
sudo /usr/bin/containerd-nydus-grpc \
    --config-path /etc/nydus/nydusd-config.fusedev.json \
    --log-to-stdout
```

## Configure and Start Containerd

Nydus uses two features of containerd:

- remote snapshotter
- snapshotter annotations

To set them up, first add something like the following to your `containerd` configuration (default to `/etc/containerd/config.toml`):

```toml
[proxy_plugins]
  [proxy_plugins.nydus]
    type = "snapshot"
    address = "/run/containerd-nydus/containerd-nydus-grpc.sock"
```

Next you should change default snapshotter to `nydus` and enable snapshot annotations like below:

For version 1 containerd config format:

```toml
[plugins.cri]
  [plugins.cri.containerd]
    snapshotter = "nydus"
    disable_snapshot_annotations = false
```

For version 2 containerd config format:

```toml
[plugins."io.containerd.grpc.v1.cri".containerd]
   snapshotter = "nydus"
   disable_snapshot_annotations = false
```

Then restart containerd, e.g.:

```bash
sudo systemctl restart containerd
```

## Start a Local Registry Container

To make it easier to convert and run nydus images next, we can run a local registry service with docker:

```bash
sudo docker run -d --restart=always -p 5000:5000 registry
```

## Convert an Image to Nydus Format

Currently, nydus image must be created by converting from an existed OCI or docker v2 image.

Note: For private registry repo, please make sure you are authorized to pull and push the target registry. The basic method is to use `docker pull` and `docker push` to validate your access to the target registry.

```bash
sudo nydusify convert --source ubuntu --target localhost:5000/ubuntu-nydus
```

For more details about how to build nydus image, please refer to [Nydusify](https://github.com/dragonflyoss/image-service/blob/master/docs/nydusify.md) conversion tool, [Acceld](https://github.com/goharbor/acceleration-service) conversion service or [Nerdctl](https://github.com/containerd/nerdctl/blob/master/docs/nydus.md#build-nydus-image-using-nerdctl-image-convert).

## Try Nydus with `nerdctl`

Nydus snapshotter has been supported by [nerdctl](https://github.com/containerd/nerdctl)(requires >= v0.22), we can lazily start container with it.

```bash
$ sudo nerdctl --snapshotter nydus run --rm -it localhost:5000/ubuntu-nydus:latest bash
```

## Create Pod with Nydus Image in Kubernetes

For example, use the following `nydus-sandbox.yaml` and `nydus-container.yaml`

The `nydus-sandbox.yaml` looks like below:

```yaml
metadata:
  attempt: 1
  name: nydus-sandbox
  namespace: default
log_directory: /tmp
linux:
  security_context:
    namespace_options:
      network: 2
annotations:
  "io.containerd.osfeature": "nydus.remoteimage.v1"
```

The `nydus-container.yaml` looks like below:

```yaml
metadata:
  name: nydus-container
image:
  image: localhost:5000/ubuntu-nydus:latest
command:
  - /bin/sleep
args:
  - 600
log_path: container.1.log
```

To create a pod with the just converted nydus image:

```bash
$ sudo crictl pull localhost:5000/ubuntu-nydus:latest
$ pod=`sudo crictl runp nydus-sandbox.yaml`
$ container=`sudo crictl create $pod nydus-container.yaml nydus-sandbox.yaml`
$ sudo crictl start $container
$ sudo crictl ps
CONTAINER ID        IMAGE                                CREATED             STATE               NAME                      ATTEMPT             POD ID
f4a6c6dc47e34       localhost:5000/ubuntu-nydus:latest   9 seconds ago       Running             nydus-container           0                   21b91779d551e
```

## Integrate P2P with Dragonfly

Nydus is deeply integrated with [Dragonfly](https://d7y.io/) P2P system, which can greatly reduce the network latency and the single point of network pressure for registry server, testing in the production environment shows that using Dragonfly can reduce network latency by more than 80%, to understand the performance test data and how to configure Nydus to use Dragonfly, please refer to the [doc](https://d7y.io/docs/setup/integration/nydus).