url-bot-rs

URL title fetching bot for IRC in Rust. The bot monitors all messages sent to it in any IRC channels it's joined to, if any messages contain URLs, the bot fetches the page and extracts the title, posting the result on the same channel, adding a certain je ne sais quoi to your IRC experience.

For example:

<user> http://rust-lang.org/
<url-bot-rs> ⤷ The Rust Programming Language

Dependencies

Currently, SQLite is a required dependency.

On Linux or OSX, you can install SQLite as follows:

Ubuntu also needs additional dependencies for TLS. These can be satisfied with:

On Windows, or if preferred, also for Linux or OSX, bundled SQLite can also be used, where it is built and linked statically without external dependencies, by specifying the sqlite_bundled feature when building with Cargo, e.g.:

cargo build --features "sqlite_bundled"
cargo install url-bot-rs --features "sqlite_bundled"

When using bundled SQLite, no external dependencies are required.

Quick install

cargo install url-bot-rs

Or, with bundled SQLite:

cargo install url-bot-rs --features "sqlite_bundled"

To get started quickly with a working configuration, run url-bot-rs with no parameters, and edit the file as shown below.

Packages

Debian packages for releases are available on Github.

Build and install local snap packages like so:

git clone https://github.com/nuxeh/url-bot-rs.git
sudo snap install snapcraft
sudo snap install url-bot-rs*.snap --dangerous

Build

Get Rust

https://www.rust-lang.org/en-US/install.html

Build

git clone https://github.com/nuxeh/url-bot-rs
cd url-bot-rs
cargo build

Run tests

cargo test

Configuration

A configuration file is required to specify IRC server details, features to enable, database setting, and other general settings for the bot; a path to this config can be specified manually with the --conf=<path> command line option.

If not provided, url bot will look in a default path for your platform, e.g. on Linux the XDG specification will be used.

First, the following directory will be searched for valid configurations:

~/.config/url-bot-rs/

or, if $XDG_CONFIG_PATH is set:

$XDG_CONFIG_HOME/url-bot-rs/

If no configurations are found in this directory, a default-valued configuration will be created at:

~/.config/url-bot-rs/config.toml

or, if $XDG_CONFIG_PATH is set:

$XDG_CONFIG_HOME/url-bot-rs/config.toml

The --conf parameter may be provided multiple times, in order to connect to multiple servers/networks.

Additionally, an additional search path may be specified by providing the --conf-dir=<dir> CLI argument, with the effect that any valid configurations existing non-recursively under this path will be loaded. This option may also be specified multiple times.

When searching for configurations using the --conf-dir option, any configurations in which network.enable is false will not be loaded.

Configuration file options

The configuration includes settings pertaining to the IRC server the bot will connect to, including among other things:

• Address of the IRC server
• Connection credentials
• The nick the bot will use when joining
• Channels to join

The [network] section gives some metadata for the network the configuration will connect to, including:

• name (string) an identifier for the network.
• enable (bool) whether to enable this network - only used when a configuration is found in a search path using the --conf-dir CLI argument, and ignored if the configuration is explicitly loaded.

It is also possible to configure a number of optional features for the bot's operation, specified in the [features] section:

• mask_highlights (bool) inserts invisible characters to defeat highlight regexes.
• send_notice (bool) causes the bot to respond with notices rather than private messages.
• report_metadata (bool) if enabled, causes image metadata to be reported.
• report_mime (bool) if enabled, causes mime types to be reported, if no other title or metadata is found.
• history (bool) enable previous post information using a database.
• cross_channel_history (bool) if enabled, only post pre-post information to the same channel as the original post.
• invite (bool) if enabled, /invite will cause the bot to join a channel.
• autosave (bool) if enabled, /invite and /kick will automatically write out the active configuration with an updated list of channels.
• send_errors_to_poster (bool) if enabled, sends any errors occurring when trying to resolve a link to the user posting the link, in a private message.
• reply_with_errors (bool) if enabled, always reply with error messages.
• partial_urls (bool) attempt to resolve titles for URLs without scheme, e.g. "docs.rs".
• nick_response (bool) respond with a message if bot is pinged in a message with no other action to perform.
• reconnect (bool) reconnect to the server after errors.

The [parameters] section includes a number of tunable parameters:

• url_limit (u8) max number of URLs to process for each message (default: 10).
• status_channels (list) channel(s) to create, join and message with any error messages produced from URL title retrieval.
• nick_response_str (string) the message to send for the nick response feature.
• reconnect_timeout (u32) amount of time to wait before reconnecting.
• ignore_nicks (list) nicknames, messages from whom will result in no titles being retrieved. For example to ignore messages from other bots in the same channel.

The [http] section contains options for HTTP requests used to obtain titles:

• timeout_s (u64) the timeout for any given request.
• max_redirections (u8) the maximum number of HTTP redirections to follow.
• max_retries (u8) the maximum number of times to retry after receiving an HTTP error response which may be temporary (e.g. 503 Service Unavailable).
• retry_delay_s (u64) the number of seconds to wait before retrying.
• accept_lang (string) the Accept-Lang HTTP request header to send when making a request (default: "en").
• user_agent (string) the user agent string to send in HTTP requests.

The [database] section contains options for the database, as follows:

• type (string) is the type of database to use, e.g. sqlite.
• path (string) is the path to a database file (for sqlite).

If no configuration file exists at the location specified with the --conf command line option, a default-valued configuration file will be created.

An example configuration is provided as example.config.toml in this repository.

Plugins

A plugin system currently caters for using a number of JSON APIs to get better title results. These each require additional configuration to be added to your configuration file(s).

• Imgur

  [plugins.imgur]
  api_key = "your API key"
  

• Youtube

  [plugins.youtube]
  api_key = "your API key"
  

• Vimeo

  [plugins.vimeo]
  api_key = "your API key"
  

In each case, once a valid API key is present in the configuration, the plugin will work automatically. For debugging, it may be helpful to use the url-bot-get tool, using the --plugin option.

Database

A database may be specified, which is used to cache posted links, so that if the same URL is posted again, the original poster and the time posted is added to the returned message. This feature can be enabled using the history feature within the [features] section of the configuration file.

Currently supported database type strings are:

• in-memory
• sqlite

The database type may be specified in the [database] configuration section, as field type. A corresponding path to use for the database may be given as field, path.

For SQLite, if no path is specified, a default path will be used, and a database will be created according to the network name specified in the [network] section of the configuration.

Install from source

Cargo

cargo install url-bot-rs

Debian/Ubuntu (Linux)

git clone https://github.com/nuxeh/url-bot-rs
cd url-bot-rs
cargo install cargo-deb
cargo deb --install

Nix

The following should be run on NixOS, or inside a Nix environment on another OS.

git clone https://github.com/nuxeh/url-bot-rs
cd url-bot-rs
nix-shell
cargo build

Static build

Debian/Ubuntu

url-bot-rs can be built statically using cargo and rustup. For example:

• Install dependencies

  apt install make musl-tools
  

• Install rustup

  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
  source $HOME/.cargo/env
  

• Add Musl target

  rustup update
  rustup target add x86_64-unknown-linux-musl
  

• Build url-bot-rs

  cargo build --release --features "openssl_vendored,sqlite_bundled" --target x86_64-unknown-linux-musl
  

Running as a service

The bot can be run automatically as a service by systemd. This is set up automatically in the case of a Debian package install, or alternatively can be set up manually.

Debian package install

If you install using the Debian package, a url-bot-rs user is created automatically. Additionally, the systemd unit is installed, and the service is enabled and started automatically, after installation.

The configuration should be customised as described in "Customising configuration" below.

When uninstalling the Debian package, the url-bot-rs user nor its home directory files are deleted, according to Debian packaging guidelines. This keeps UIDs more deterministic, and allows re-installation or upgrade without losing the bot's configuration.

Manual systemd install

To set up systemd manually, the unit file must be copied, and the url-bot-rs user must be created on the system. From inside the project repository:

sudo install -m 644 systemd/url-bot-rs.service /etc/systemd/system/
sudo useradd -m --system url-bot-rs --shell /usr/sbin/nologin
sudo systemctl enable --now url-bot-rs.service

The configuration should be customised as described in "Customising configuration" below.

Customising configuration

Once started once, a default configuration is created in /home/url-bot-rs/.config/url-bot-rs/config.toml, which should be edited, and the bot restarted:

sudo systemctl restart url-bot-rs.service

You can also place any configurations you wish to run under the default search path:

/home/url-bot-rs/.config/url-bot-rs/

Checking status

To check status or get logs:

systemctl status url-bot-rs.service
sudo journalctl -u url-bot-rs.service

url-bot-rs additional command line options

• Usage is printed by providing --help on run.
• To print additional runtime information, add -v or --verbose. The level of verbosity can be increased by adding extra vs; at higher levels of verbosity IRC messages received, HTTP response headers, and information regarding resolution of URLs, such as cookies set, can be printed.

Additional CLI tools

The crate comes with additional binary tools, to aid in testing URL title retrieval.

url-bot-get

Performs title retrieval using the same code that is used in the bot, but instead supplied with URLs via the command line, with tweakable request parameters, such as user agent, and others. It is intended to be useful for debugging cases where title retrieval fails for some reason, to assist in offline development.

IRC

There are IRC channels on Moznet, Freenode and OFTC, #url-bot-rs.