Dogtail

Tail logs in Datadog to your local machine.

Dogtail is designed to let you use the powerful command line tools you already know and love for exploring the logs of your projects, because no matter how good Datadog is, there's no place like $HOME.

Usage

Running pods, an want to be able to read logs per pod? (note the use of RUST_LOG causes dogtail logs to be printed to stderr)

> RUST_LOG=dogtail=info dogtail "service:my-service env:production" -k attributes.tags.pod_name
2023-09-21 8:16:05.330957357  INFO dogtail Found 60 events to write, total written: 60
2023-09-21 8:16:05.331068639  INFO dogtail Started writing to file: my-service-deployment-bb5d459d4-hf7v9.log
2023-09-21 8:16:15.774211007  INFO dogtail Found 11 events to write, total written: 71

Got noisy logs? View only the unique messages

? dogtail "env:production service:my-service" -m stdout -s | jq .attributes.message | huniq

Refactoring, and want to see what logs are more noise than they're worth? This example uses nushell, and writes to file as an intermediate step because tools like uniq can't emit counts until the input stream ends.

> dogtail "env: production service:my-service" -s
# Run for a while, then kill with ctrl-c
> cat output.log | jq .attributes.message | lines | uniq -c | sort-by count

Installation

git clone ...
cargo install --path .

Configuration

Dogtail needs access to a Datadog API key and an APP key to query logs. These are pulled from the environment variables DD_API_KEY and DD_APP_KEY respectively.

Usage detail:

> dogtail --help
Tail datadog logs to files, or stdout

Usage: dogtail [OPTIONS] <QUERY_STRING>

Arguments:
  <QUERY_STRING>  A query string, the same as you would use in the UI, e.g. "service:my-service"

Options:
  -d, --domain <DOMAIN>            The domain to use for the API [default: api.datadoghq.eu]
  -m, --mode <MODE>                Mode - If file, log events will be partitioned by split_key and written to files, if stdout, logs will be written to stdout [default: file] [possible values: file, stdout]
  -k, --split-key <SPLIT_KEY>      If mode is file, this is the event attribute lookup key to use for partitioning logs. Uses json-pointer syntax, e.g. "attributes.tags.pod_name". Note that event tags are unpacked into a map, so you can use tags "attributes.tags.pod_name" for this purpose
      --format-file <FORMAT_FILE>  A file to load a formatting config from. The formatting config if a newline separated list of json-pointer keys - each output line will be the found value of each of those keys, joined by a space. If none is provided, a default logging format of "timestamp status message" will be used
  -s, --structured                 If true, structured json will be written to the output instead of formatted logs, with one event written per line
  -h, --help                       Print help
  -V, --version                    Print version