# rusty-daw-io Design Document *(note "rusty-daw-io" may not be the final name of this crate)* # Objective The goal of this crate is to provide a powerful, cross-platform, highly configurable, low-latency, and robust solution for connecting audio software to audio and MIDI devices. ## Why not contribute to an already existing project like `RTAudio` or `CPAL`? ### RTAudio - This API is written in a complicated C++ codebase, making it very tricky to bind to other languages such as Rust. - This project has a poor track record in its stability and ability to gracefully handle errors (not ideal for live audio software). ### CPAL In short, CPAL is very opinionated, and we have a few deal-breaking issues with its core design. - CPAL's design does not handle duplex audio devices well. It spawns each input and output stream into separate threads, requiring the developer to sync them together with ring buffers. This is inneficient for most consumer and professional duplex audio devices which already have their inputs and outputs tied into the same stream to reduce latency. - The API for searching for and configuring audio devices is cumbersome. It returns a list of every possible combination of configurations available with the system's devices. This is not how a user configuring audio settings through a GUI expects this to work. - CPAL does not have any support for MIDI devices, so we would need to write our own support for it anyway. Why not just fork `CPAL`? - To fix these design issues we would pretty much need to rewrite the whole API anyway. Of course we don't have to work completely from scratch. We can still borrow some of the low-level platform specific code in CPAL. # Goals - Support for Linux, Mac, and Windows using the following backends: (and maybe Android and iOS in the future, but that is not a gaurantee) - Linux - [ ] Jack - [ ] Pipewire - [ ] Alsa (Maybe, depending on how difficult this is. This could be unecessary if Pipewire turns out to be good enough.) - [ ] Pulseaudio (Maybe, depending on how difficult this is. This could be unecessary if Pipewire turns out to be good enough.) - Mac - [ ] CoreAudio - [ ] Jack (Maybe, if it is stable enough on Mac.) - Window - [ ] WASAPI - [ ] ASIO (reluctantly) - [ ] Jack (Maybe, if it is stable enough on Windows.) - Scan the available devices on the system, and present configuration options in a format that is intuitive to an end-user configuring devices inside a settings GUI. - Send all audio and midi streams into a single high-priority thread, taking advantage of native duplex devices when available. (Audio buffers will be presented as de-interlaced `f32` buffers). - Robust and graceful error handling, especially while the stream is running. - Easily save and load configurations to/from a config file. - A system that will try to automatically create a good initial default configuration. # Later/Maybe Goals - Support MIDI 2.0 devices - Support for OSC devices - C API bindings # Non-Goals - No Android and iOS support (for now atleast) - No support for using multiple backends at the same time (i.e trying to use WASAPI device as an input and an ASIO device as an output). This will just add a whole slew of complexity and stuff that can go wrong. - No support for tying multiple separate (non-duplexed) audio devices together. We will only support either connecting to a single duplex audio device *or* connecting to a single non-duplex output device. - This one is probably controversal, so let me explain the reasoning: - Pretty much all modern external audio devices (a setup used by most professionals and pro-sumers) are already duplex. - MacOS (and in Linux using JACK or Pipewire) already packages all audio device streams into a single "system-wide duplex device". So this is really only a Windows-specific problem. - Tying together multiple non-duplex audio streams requires an intermediate buffer that adds a sometimes unkowable amount of latency. - Allowing for multiple separate audio devices adds a lot of complexity to both the settings GUI and the config file, and a lot more that can go wrong. - Some modern DAWs like Bitwig already use this "single audio device only" system, so it's not like it's a new concept. - No support for non-f32 audio streams. - There is just no point in my opinion in presenting any other sample format other than `f32` in such an API. These `f32` buffers will just be converted to/from the native sample format that the device wants behind the scenes. # API Design The API is divided into four parts: Enumerating the available devices, creating a config, running the stream, and responding to messages after the stream is ran. ## Device Enumeration API: ```rust /// Returns the list available audio backends for this platform. /// /// These are ordered with the first item (index 0) being the most highly /// preferred default backend. pub fn available_audio_backends() -> &'static [&'static str] { ... } #[cfg(feature = "midi")] /// Returns the list available midi backends for this platform. /// /// These are ordered with the first item (index 0) being the most highly /// preferred default backend. pub fn available_midi_backends() -> &'static [&'static str] { ... } /// Returns the list of available audio devices for the given backend. /// /// This will return an error if the backend with the given name could /// not be found. pub fn enumerate_audio_backend(backend: &str) -> Result { ... } /// Returns the configuration options for the given device. /// /// This will return an error if the backend or the device could not /// be found. pub fn enumerate_audio_device( backend: &str, device: &DeviceID, ) -> Result { ... } #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] /// Returns the configuration options for "monolithic" system-wide Jack /// audio device. /// /// This will return an error if Jack is not installed on the system /// or if the Jack server is not running. pub fn enumerate_jack_audio_device() -> Result { ... } #[cfg(feature = "asio")] #[cfg(target_os = "windows")] /// Returns the configuration options for the given ASIO device. /// /// This will return an error if the device could not be found. pub fn enumerate_asio_audio_device(device: &DeviceID) -> Result { ... } #[cfg(feature = "midi")] /// Returns the list of available midi devices for the given backend. /// /// This will return an error if the backend with the given name could /// not be found. pub fn enumerate_midi_backend(backend: &str) -> Result { ... } /// Information about an audio backend, including its available devices /// and configurations pub struct AudioBackendOptions { /// The name of this audio backend pub name: &'static str, /// The version of this audio backend (if that information is available) pub version: Option, /// The available audio devices to select from pub device_options: AudioDeviceOptions, } /// The available audio devices to select from pub enum AudioDeviceOptions { /// Only a single audio device can be selected from this list. These /// devices may be output only, input only, or (most commonly) /// duplex. SingleDeviceOnly { /// The available audio devices to select from. options: Vec, }, /// A single input and output device pair can be selected from this list. LinkedInOutDevice { /// The names/IDs of the available input devices to select from input_devices: Vec, /// The names/IDs of the available output devices to select from output_devices: Vec, /// The available configurations for this device pair config_options: AudioDeviceConfigOptions, }, #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] /// There is a single "monolithic" system-wide Jack audio device JackSystemWideDevice, #[cfg(feature = "asio")] #[cfg(target_os = "windows")] /// A single ASIO device can be selected from this list. SingleAsioDevice { /// A single ASIO device can be selected from this list. options: Vec, }, } /// The name/ID of a device pub struct DeviceID { /// The name of the device pub name: String, /// The unique identifier of this device (if one is available). This /// is usually more reliable than just the name of the device. pub identifier: Option, } #[derive(Debug, Clone)] /// The available configuration options for the audio device/devices pub struct AudioDeviceConfigOptions { /// The available sample rates to choose from. /// /// If the available sample rates could not be determined at this time, /// then this will be `None`. pub sample_rates: Option>, /// The available range of fixed block/buffer sizes /// /// If the device does not support fixed block/buffer sizes, then this /// will be `None`. pub block_sizes: Option, /// The number of input audio ports available pub num_input_ports: usize, /// The number of output audio ports available pub num_output_ports: usize, /// The layout of the input audio ports pub input_channel_layout: ChannelLayout, /// The layout of the output audio ports pub output_channel_layout: ChannelLayout, /// If `true` then it means that the application can request to take /// exclusive access of the device to improve latency. /// /// This is only relevant for WASAPI on Windows. This will always be /// `false` on other backends and platforms. pub can_take_exclusive_access: bool, } #[non_exhaustive] #[derive(Debug, Clone, PartialEq)] /// The channel layout of the audio ports pub enum ChannelLayout { /// The device has not specified the channel layout of the audio ports Unspecified, /// The device has a single mono channel Mono, /// The device has multiple mono channels (i.e. multiple microphone /// inputs) MultiMono, /// The device has a single stereo channel Stereo, /// The device has multiple stereo channels MultiStereo, /// The special (but fairly common) case where the device has two stereo /// output channels: one for speakers and one for headphones StereoX2SpeakerHeadphone, /// Some other configuration not listed. Other(String), // TODO: More channel layouts } /// The range of possible block sizes for an audio device. #[derive(Debug, Clone)] pub struct BlockSizeRange { /// The minimum buffer/block size that can be used (inclusive) pub min: u32, /// The maximum buffer/block size that can be used (inclusive) pub max: u32, /// The default buffer/block size for this device pub default: u32, } #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] #[derive(Debug, Clone)] /// Information and configuration options for the "monolithic" system-wide /// Jack audio device pub struct JackAudioDeviceOptions { /// If this is `false`, then it means that Jack is not installed on the /// system and thus cannot be used. pub installed_on_sytem: bool, /// If this is `false`, then it means that Jack is installed but it is /// not currently running on the system, and thus cannot be used until /// the Jack server is started. pub running: bool, /// The sample rate of the Jack device pub sample_rate: u32, /// The block size of the Jack device pub block_size: u32, /// The names of the available input ports to select from pub input_ports: Vec, /// The names of the available output ports to select from pub output_ports: Vec, /// The indexes of the default input ports, along with their channel /// layout. /// /// If no default input ports could be found, then this will be `None`. pub default_input_ports: Option<(Vec, ChannelLayout)>, /// The indexes of the default output ports, along with their channel /// layout. /// /// If no default output ports could be found, then this will be `None`. pub default_output_ports: Option<(Vec, ChannelLayout)>, } #[cfg(feature = "asio")] #[cfg(target_os = "windows")] #[derive(Debug, Clone)] /// Information and configuration options for an ASIO audio device on /// Windows pub struct AsioAudioDeviceOptions { /// The configuration options for this ASIO audio device pub config_options: AudioDeviceConfigOptions, /// The path the the executable that launches the settings GUI for /// this ASIO device pub settings_application: std::path::PathBuf, } #[cfg(feature = "midi")] #[derive(Debug, Clone)] /// Information about a MIDI backend, including its available devices /// and configurations pub struct MidiBackendOptions { /// The name of this MIDI backend pub name: &'static str, /// The version of this MIDI backend (if that information is available) pub version: Option, /// The names of the available input MIDI devices to select from pub in_device_ports: Vec, /// The names of the available output MIDI devices to select from pub out_device_ports: Vec, /// The index of the default/preferred input MIDI port for the backend /// /// This will be `None` if no default input port could be /// determined. pub default_in_port: Option, /// The index of the default/preferred output MIDI port for the backend /// /// This will be `None` if no default output port could be /// determined. pub default_out_port: Option, } #[cfg(feature = "midi")] #[derive(Debug, Clone)] /// Information and configuration options for a MIDI device port pub struct MidiDevicePortOptions { /// The name/ID of this device pub id: DeviceID, /// The index of this port for this device pub port_index: usize, /// The type of control scheme that this port uses pub control_type: MidiControlType, } #[cfg(feature = "midi")] #[non_exhaustive] #[derive(Debug, Clone, Copy, PartialEq)] /// The type of control scheme that this port supports pub enum MidiControlScheme { /// Supports only MIDI version 1 Midi1, #[cfg(feature = "midi2")] /// Supports MIDI version 2 (and by proxy also supports MIDI version 1) Midi2, // TODO: Midi versions inbetween 1.0 and 2.0? // TODO: OSC devices? } ``` ## Configuration API: This is the API for the "configuration". The user constructs this configuration in whatever method they choose (from a settings GUI or a config file) and sends it to this crate to be ran. ```rust #[cfg(not(feature = "serde-config"))] #[derive(Debug, Clone, PartialEq)] /// Specifies whether to use a specific configuration or to automatically /// select the best configuration. pub enum AutoOption { /// Use this specific configuration. Use(T), /// Automatically select the best configuration. Auto, } impl Default for AutoOption { fn default() -> Self { AutoOption::Auto } } #[cfg(not(feature = "serde-config"))] #[derive(Debug, Clone, PartialEq)] /// The configuration of audio and MIDI backends and devices. pub struct RustyDawIoConfig { /// The audio backend to use. /// /// Set this to `AutoOption::Auto` to automatically select the best /// backend to use. pub audio_backend: AutoOption, /// The audio device/devices to use. /// /// Set this to `AudioDeviceConfig::Auto` to automatically select the best /// audio device to use. pub audio_device: AudioDeviceConfig, /// The sample rate to use. /// /// Set this to `AutoOption::Auto` to automatically select the best /// sample rate to use. pub sample_rate: AutoOption, /// The block/buffer size to use. /// /// Set this to `AutoOption::Auto` to automatically select the best /// buffer/block size to use. pub block_size: AutoOption, /// The indexes of the audio input ports to use. /// /// The buffers presented in `ProcInfo::audio_in` will appear in this /// exact same order. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of input ports to use. /// /// You may also pass in an empty Vec to have no audio inputs. /// /// This is not relevent when the audio backend is Jack. pub input_channels: AutoOption>, /// The indexes of the audio output ports to use. /// /// The buffers presented in `ProcInfo::audio_out` will appear in this /// exact same order. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of output ports to use. /// /// You may also pass in an empty Vec to have no audio outputs. /// /// This is not relevent when the audio backend is Jack. pub output_channels: AutoOption>, #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] /// When the audio backend is Jack, the names of the audio input ports /// to use. /// /// The buffers presented in `ProcInfo::audio_in` will appear in this /// exact same order. /// /// If a port with the given name does not exist, then an unconnected /// virtual port with that same name will be created. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of input ports to use. /// /// You may also pass in an empty Vec to have no audio inputs. /// /// This is only relevent when the audio backend is Jack. pub jack_input_ports: AutoOption>, #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] /// When the audio backend is Jack, the names of the audio output ports /// to use. /// /// The buffers presented in `ProcInfo::audio_out` will appear in this /// exact same order. /// /// If a port with the given name does not exist, then an unconnected /// virtual port with that same name will be created. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of output ports to use. /// /// You may also pass in an empty Vec to have no audio outputs. /// /// This is only relevent when the audio backend is Jack. pub jack_output_ports: AutoOption>, /// If `true` then it means that the application can request to take /// exclusive access of the device to improve latency. /// /// This is only relevant for WASAPI on Windows. This will always be /// `false` on other backends and platforms. pub take_exclusive_access: bool, #[cfg(feature = "midi")] /// The configuration of MIDI devices. /// /// Set this to `None` to use no MIDI devices. pub midi_config: Option, } impl Default for RustyDawIoConfig { fn default() -> Self { RustyDawIoConfig { audio_backend: AutoOption::Auto, audio_device: AudioDeviceConfig::Auto, sample_rate: AutoOption::Auto, block_size: AutoOption::Auto, input_channels: AutoOption::Use(Vec::new()), output_channels: AutoOption::Auto, #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] jack_input_ports: AutoOption::Use(Vec::new()), #[cfg(any(feature = "jack-linux", feature = "jack-macos", feature = "jack-windows"))] jack_output_ports: AutoOption::Auto, take_exclusive_access: false, #[cfg(feature = "midi")] midi_config: None, } } } #[cfg(not(feature = "serde-config"))] #[derive(Debug, Clone, PartialEq)] /// The configuration of which audio device/devices to use. pub enum AudioDeviceConfig { /// Use a single audio device. These device may be output only, input /// only, or (most commonly) duplex. Single(DeviceID), /// Use an input/output device pair. This is only supported on some /// backends. LinkedInOut { input: DeviceID, output: DeviceID }, /// Automatically select the best configuration. /// /// This should also be used when using the Jack backend. Auto, } impl Default for AudioDeviceConfig { fn default() -> Self { AudioDeviceConfig::Auto } } #[cfg(feature = "midi")] /// The configuration of the MIDI backend and devices. pub struct MidiConfig { /// The MIDI backend to use. /// /// Set this to `AutoOption::Auto` to automatically select the best /// backend to use. pub midi_backend: AutoOption, /// The names of the MIDI input ports to use. /// /// The buffers presented in `ProcInfo::midi_in` will appear in this /// exact same order. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of input ports to use. /// /// You may also pass in an empty Vec to have no MIDI inputs. pub in_device_ports: AutoOption>, /// The names of the MIDI output ports to use. /// /// The buffers presented in `ProcInfo::midi_out` will appear in this /// exact same order. /// /// Set this to `AutoOption::Auto` to automatically select the best /// configuration of output ports to use. /// /// You may also pass in an empty Vec to have no MIDI outputs. pub out_device_ports: AutoOption>, } impl Default for MidiConfig { fn default() -> Self { MidiConfig { midi_backend: AutoOption::Auto, in_device_ports: AutoOption::Auto, out_device_ports: AutoOption::Use(Vec::new()), } } } #[cfg(feature = "midi")] /// The configuration of a MIDI device port pub struct MidiDevicePortConfig { /// The name/ID of the MIDI device to use pub device_id: DeviceID, /// The index of the port on the device pub port_index: usize, /// The control scheme to use for this port pub control_scheme: MidiControlScheme, } ``` ## Running API: The user sends a config to this API to run it. ```rust /// Get the estimated total latency of a particular configuration before running it. /// /// `None` will be returned if the latency is not known at this time or if the /// given config is invalid. pub fn estimated_latency(config: &RustyDawIoConfig) -> Option { ... } /// Get the sample rate of a particular configuration before running it. /// /// `None` will be returned if the sample rate is not known at this time or if the /// given config is invalid. pub fn sample_rate(config: &RustyDawIoConfig) -> Option { ... } /// A processor for a stream. pub trait ProcessHandler: 'static + Send { /// Initialize/allocate any buffers here. This will only be called once on /// creation. fn init(&mut self, stream_info: &StreamInfo); /// This gets called if the user made a change to the configuration that does not /// require restarting the audio thread. fn stream_changed(&mut self, stream_info: &StreamInfo); /// Process the current buffers. This will always be called on a realtime thread. fn process<'a>(&mut self, proc_info: ProcessInfo<'a>); } // See code in the repo for the implementations of `StreamInfo` and `ProcessInfo`. #[derive(Debug, Clone)] /// Additional options for running a stream pub struct RunOptions { /// If `Some`, then the backend will use this name as the /// client name that appears in the audio server. This is only relevent for some /// backends like Jack. /// /// By default this is set to `None`. pub use_application_name: Option, #[cfg(feature = "midi")] /// The maximum number of events a MIDI buffer can hold. /// /// By default this is set to `1024`. pub midi_buffer_size: u32, /// If true, then the backend will mark every input audio buffer that is /// silent (all `0.0`s) before each call to `process()`. /// /// If false, then the backend won't do this check and every buffer will /// be marked as not silent. /// /// By default this is set to `false`. pub check_for_silent_inputs: bool, /// How the system should respond to various errors. pub error_behavior: ErrorBehavior, /// The size of the audio thread to stream handle message buffer. /// /// By default this is set to `512`. pub msg_buffer_size: usize, } // See code in the repo for the implementation of `ErrorBehavior`. /// Run the given configuration in an audio thread. /// /// * `config`: The configuration to use. /// * `options`: Various options for the stream. /// * `process_handler`: An instance of your process handler. /// /// If an error is returned, then it means the config failed to run and no audio /// thread was spawned. pub fn run( config: &Config, options: &RunOptions, process_handler: P, ) -> Result, RunConfigError> { ... } /// The handle to a running audio/midi stream. /// // When this gets dropped, the stream (audio thread) will automatically stop. This /// is the intended method for stopping a stream. pub struct StreamHandle { /// The message channel that recieves notifications from the audio thread /// including any errors that have occurred. pub messages: StreamMsgChannel, ... } impl StreamHandle { /// Returns the actual configuration of the running stream. This may differ /// from the configuration passed into the `run()` method. pub fn stream_info(&self) -> &StreamInfo { ... } /// Change the audio port configuration while the audio thread is still running. /// Support for this will depend on the backend. /// /// If the given config is invalid, an error will be returned with no /// effect on the running audio thread. pub fn change_audio_port_config( &mut self, audio_in_ports: Option>, audio_out_ports: Option>, ) -> Result<(), ChangeAudioPortConfigError> { ... } /// Change the buffer size configuration while the audio thread is still running. /// Support for this will depend on the backend. /// /// If the given config is invalid, an error will be returned with no /// effect on the running audio thread. pub fn change_audio_buffer_size_config( &mut self, config: AudioBufferSizeConfig, ) -> Result<(), ChangeAudioBufferSizeError> { ... } #[cfg(feature = "midi")] /// Change the midi device configuration while the audio thread is still running. /// Support for this will depend on the backend. /// /// If the given config is invalid, an error will be returned with no /// effect on the running audio thread. pub fn change_midi_device_config( &mut self, in_devices: Vec, out_devices: Vec, ) -> Result<(), ChangeMidiDeviceConfigError> { ... } // It may be possible to also add `change_sample_rate_config()` here, but // I'm not sure how useful this would actually be. /// Returns whether or not this backend supports changing the audio bus /// configuration while the audio thread is running. pub fn can_change_audio_port_config(&self) -> bool { ... } // Returns whether or not this backend supports changing the buffer size // configuration while the audio thread is running. pub fn can_change_audio_buffer_size_config(&self) -> bool { ... } #[cfg(feature = "midi")] /// Returns whether or not this backend supports changing the midi device /// config while the audio thread is running. pub fn can_change_midi_device_config(&self) -> bool { ... } } // TODO: Implementations of `RunConfigErrorRunConfigError`, `ChangeAudioPortConfigError`, // and `ChangeBufferSizeConfigError`, and `ChangeMidiDeviceConfigError`. ``` ## Message channel API: After a stream is ran, the user then listens to and responds to events sent to `StreamHandle::messages`. ```rust #[derive(Debug, Clone)] pub enum StreamMsg { /// An audio device was unplugged while the stream was running. Any connected /// ports will input/output silence. AudioDeviceDisconnected(DeviceID), /// An audio device was reconnected while the stream was running. Any connected /// ports will function properly now. /// /// This will only be sent after an `AudioDeviceDisconnected` event. AudioDeviceReconnected(DeviceID), #[cfg(feature = "midi")] /// The MIDI output device was not found. This port will produce no MIDI events. MidiDeviceDisconnected(DeviceID), #[cfg(feature = "midi")] /// A MIDI device was reconnected while the stream was running. Any connected /// ports will function properly now. /// /// This will only be sent after an `MidiDeviceDisconnected` event. MidiDeviceReconnected(DeviceID), /// An error that caused the stream to close. Please discard this Stream Handle /// channel and prepare to start a new stream. Error(StreamError), /// The audio stream was closed gracefully. Please discard this Stream Handle. Closed, } /// The message channel that recieves notifications from the audio thread including /// any errors that have occurred. pub struct StreamMsgChannel { from_audio_thread_rx: Consumer, } impl StreamMsgChannel { pub(crate) fn new(msg_buffer_size: usize) -> (Self, ringbuf::Producer) { let (to_channel_tx, from_audio_thread_rx) = RingBuffer::::new(msg_buffer_size).split(); (Self { from_audio_thread_rx }, to_channel_tx) } /// Returns capacity of the message buffer. /// /// The capacity of the buffer is constant. pub fn capacity(&self) -> usize { self.from_audio_thread_rx.capacity() } /// Checks if the message buffer is empty. /// /// *The result may become irrelevant at any time because of concurring activity of the producer.* pub fn is_empty(&self) -> bool { self.from_audio_thread_rx.is_empty() } /// Removes latest element from the message buffer and returns it. /// Returns `None` if the message buffer is empty. pub fn pop(&mut self) -> Option { self.from_audio_thread_rx.pop() } /// Repeatedly calls the closure `f` passing elements removed from the message buffer to it. /// /// The closure is called until it returns `false` or the message buffer is empty. /// /// The method returns number of elements been removed from the buffer. pub fn pop_each bool>(&mut self, f: F, count: Option) -> usize { self.from_audio_thread_rx.pop_each(f, count) } } // See code in the repo for the implementation of `StreamError`. ``` # Demo Application In addition to the main API, we will also have a full-working demo application with a working settings GUI. This will probably be written in `egui`, but another UI toolkit could be used.