argtokens: Rust library to tokenize command-line arguments

argtokens extracts options and values out of command-line arguments.

[Source] [Crate info] [API docs]

Design

argtokens works like an iterator: you pull individual tokens from it (using next_token) and process them in sequence. If you want an option to take a value, you have to explicitly call next_value when encountering the option.

The API supports no_std, no-alloc usage. When used with std, there are extra helpers to interoperate with OsStr.

The supported syntax is similar to what you often find on GNU and FreeBSD:

• -abc is normally interpreted like -a -b -c. If you tell argtokens to extract a value after the -a, it is interpreted like -a bc instead.
• Long options are supported, in both --opt val and --opt=val forms.
• A freestanding -- produces a distinct token. It is up to the client code to decide what it wants to do in response to this token; by itself, the token does nothing special.

argtokens is cross-platform. It operates on &[Ch], where Ch is a generic type implementing From<u8>. You would normally use the helper type NChar here, which is an alias for the platform-specific native character type (u16 on Windows, u8 elsewhere).

While option names are limited to ASCII characters, option values and positional arguments can contain arbitrary characters which may or may not be valid in any encoding. For example, in --opt=val arg, opt can only consist of ASCII characters, while val and arg can contain any character sequence.

argtokens is suitable for use directly by applications or as the tokenizer/lexer of a higher-level argument parsing library.

Examples

The following example shows argtokens usage with std support enabled.

use argtokens::{long_opt, short_opt, ArgTokens, NChar, ToNString, ToOsString, Token};

const OPT_C: NChar = short_opt!('c');
const OPT_F: NChar = short_opt!('f');
const OPT_X: NChar = short_opt!('x');
const OPT_FILE: &[NChar] = long_opt!("file");

fn main() -> Result<(), &'static str> {
	let args = Vec::from_iter(
		std::env::args_os()
			.skip(1)
			.map(|a| a.to_n_string().into_owned()),
	);
	let mut tokens = ArgTokens::new(args.iter().map(Vec::as_slice));

	while let Some(token) = tokens.next_token() {
		match token {
			Token::Short(OPT_C) => println!("Mode: create"),
			Token::Short(OPT_X) => println!("Mode: extract"),
			Token::Short(OPT_F) | Token::Long(OPT_FILE) => {
				let value = tokens.next_value().ok_or("-f/--file requires value")?;
				println!("File: {:?}", value.to_os_string());
			}
			Token::Freestanding(value) => println!("Arg: {:?}", value.to_os_string()),
			Token::Hanging(_) => return Err("Unexpected value"),
			Token::DashDash => {
				while let Some(value) = tokens.next_value() {
					println!("Arg: {:?}", value.to_os_string());
				}
				break;
			}
			_ => return Err("Invalid option"),
		}
	}
	Ok(())
}

Refer to examples/realistic.rs in the source repository to see a much more complete example that you might use in a real application.

The examples/no_std directory demonstrates no_std, no-alloc usage.

Feature flags

• inline (default): Marks the ArgTokens::next_token method #[inline], which may affect code size and/or speed. This is most effective when you only call the method from one place in the code (usually the main parser loop).
• std (default): Adds helpers for dealing with OsStr and native strings.

Alternatives

lexopt is another library that uses a similar approach to argtokens. There are differences in the API but the iterator-based architecture is very close.

clap uses a mapping-based architecture, making it very different from argtokens.

License & contributions

This project is licensed under the Mozilla Public License Version 2.0 (MPL-2.0).

By submitting any contribution to this project, unless you specify otherwise, you agree to license that contribution under MPL-2.0.