argtokens: Rust library to tokenize command-line arguments

argtokens extracts options and values out of command-line arguments.

[Source] [Crate info] [API docs]

Design

argtokens works like an iterator: you pull individual tokens from it (using next) and process them in sequence. If you want an option to take a value, you have to explicitly call next_value when encountering the option.

argtokens is cross-platform and supports no_std, no-alloc usage. It operates on an Argument trait that is implemented for &[u8], &[u16], &OsStr, and &str depending on the feature flags enabled.

The supported syntax is similar to what you often find on GNU and FreeBSD:

• -abc is normally interpreted like -a -b -c. If you tell argtokens to extract a value after the -a, it is interpreted like -a bc instead.
• Long options are supported, in both --opt val and --opt=val forms.
• - and -- are interpreted as normal freestanding arguments, not as an options. argtokens does not otherwise treat -- in a special way; the client code is free to either process it like any other freestanding argument or to change parsing behavior in response.

While argtokens limits option names to ASCII characters, option values and positional arguments can contain arbitrary characters which may or may not be valid in any encoding. For example, in --opt=val arg, opt can only consist of ASCII characters, while val and arg can contain any character sequence.

argtokens never panics nor aborts when encountering invalid input; you are always able to handle user errors.

argtokens aims to be suitable for use directly by applications or as the tokenizer/lexer of a higher-level argument parsing library.

Examples

The following example shows argtokens usage with std support enabled.

use std::ffi::OsString;

use argtokens::{ArgTokens, Token};

fn main() -> Result<(), &'static str> {
	let args = Vec::from_iter(std::env::args_os().skip(1));
	let mut tokens = ArgTokens::new(args.iter().map(OsString::as_os_str));

	while let Some(token) = tokens.next() {
		match token {
			Token::Freestanding(value) => println!("Arg: {value:?}"),
			Token::Short(b'c') => println!("Mode: create"),
			Token::Short(b'x') => println!("Mode: extract"),
			Token::Short(b'f') | Token::Long(b"file") => {
				let value = tokens.next_value().ok_or("-f/--file requires value")?;
				println!("File: {value:?}");
			}
			_ => return Err("Error"),
		}
	}
	Ok(())
}

Refer to examples/realistic.rs in the source repository to see a much more complete example that you might use in a real application, with -- support and proper error handling.

The examples/no_std directory demonstrates no_std, no-alloc usage.

Feature flags

• inline-next (default): Marks the ArgTokens::next method #[inline], which may affect code size and/or speed. This is most effective when you only call the method from one place in the code (usually the main parser loop).
• std (default): Allows argtokens to work on OsStr arguments. This also adds a few helpers to convert to and from OsStr.
• str: Allows argtokens to work on str arguments. This is only useful in some niche situations (e.g. no_std WASI).

Alternatives

lexopt is another library that uses a similar approach to argtokens. There are differences in the API but the iterator-based architecture is very close.

clap uses a mapping-based architecture, making it very different from argtokens.

License & contributions

This project is licensed under the Mozilla Public License Version 2.0 (MPL-2.0).

By submitting any contribution to this project, unless you specify otherwise, you agree to license that contribution under MPL-2.0.