hylarana

A development toolkit for writing low-latency and cross-platform screen casting applications.

Hylarana is a cross-platform screen casting SDK for writing screen casting applications using the Rust programming language. It provides a very high level of abstraction, shields cross-platform details, codec support for software hardware, and supports multiple transmission methods. This project prioritizes latency and performance. Currently, the end-to-end latency is controlled below 250ms, the actual latency performance is a bit off depending on platform differences and transmission methods, and the performance is worse on Linux platform, and only software codecs are supported on Linux platform.

• Transmission method: SRT transmission protocol, UDP multicast.
• Codec support: Open H264, X264, Intel Qsv, Video Toolbox.

Quick Start

hylarana does not need to specifically address cross-platform differences and feature flags:

hylarana = "0.2"

Creating a Sender

The sender is also the initiator of the screen casting, before initiating the screen casting, you need to determine the capture source, here we capture the screen and the system sound as an example.

Get the list of screens:

let video_sources = Capture::get_sources(SourceType::Screen)?;

sources is a list of all the screens on your current device, where is_default with true is your default monitor.

Get the system sound output device:

let audio_sources = Capture::get_sources(SourceType::Audio)?;

The audio output device follows the same rules as the monitor above.

To keep things a bit simpler, we use the first source of the audio source list and the video source list.

let video_source = video_sources.get(0)?;
let audio_source = audio_sources.get(0)?;

Next, create the sender configurations.

Start by creating the encoding configurations for the audio and video sources:

let video_options = HylaranaSenderTrackOptions {
    source: video_source,
    options: VideoOptions {
        codec: VideoEncoderType::X264,
        frame_rate: 30,
        width: 1280,
        height: 720,
        bit_rate: 500 * 1024 * 8,
        key_frame_interval: 21,
    },
}

let audio_options = HylaranaSenderTrackOptions {
    source: audio_source,
    options: AudioOptions {
        sample_rate: 48000,
        bit_rate: 64000,
    },
};

The video encoder uses a software encoder and is fixed at 30 frames per second for 1280x720 video. Audio is fixed at 48khz sample rate.

Then, create the sender.

First, we use UDP multicast as the network transmission method for screen casting:

let transport = TransportOptions {
    strategy: TransportStrategy::Multicast("239.0.0.1:8080".parse()?),
    mtu: 1500,
};

The sender will send the audio and video packets to port 8080 on 239.0.0.1.

Pass these configurations to the create_sender function to create the sender:

let sender = Hylarana::create_sender(
    HylaranaSenderOptions {
        transport,
        media: HylaranaSenderMediaOptions {
            video: video_options,
            audio: audio_options,
        },
    },
    view,
)?;

You may notice that view we haven't created yet, don't worry, we'll go back and create view to display our sender preview screen next.

struct View(Mutex<VideoRender<'a>>);

impl AVFrameStream for View {}

impl AVFrameSink for View {
    fn audio(&self, frame: &AudioFrame) -> bool {
        true
    }

    fn video(&self, frame: &VideoFrame) -> bool {
        self.video.send(frame).is_ok()
    }
}

impl AVFrameObserver for View {
    fn close(&self) {
        println!("view is closed");
    }
}

We implement AVFrameStream for View, which is needed for create_sender. We submit every video frame we receive to the VideoRender, but the audio we don't process because playing native sound will cause an audio loopback (you don't want that).

Let's go back to how to create the VideoRender.

The renderer needs a window to output and display the screen, it is recommended to use winit to create a native window, this library is very easy to use, but instead of showing how to create a window with winit and such, we will assume that a window has been created:

let inner_size = window.inner_size();
let video_render = VideoRender::new(VideoRenderOptions {
    backend: VideoRenderBackend::WebGPU,
    target: window,
    size: Size {
        width: inner_size.width,
        height: inner_size.height,
    },
})?;

Then create View.

let view = View(video_render);

This way, we have a window where we can preview the video screen.

After the creation of the sender is complete, it is important to note that you need to get the unique identifier of the sender via sender.get_id(), which is needed by the receiver to find the sender.

Creating a Receiver

Receiver is used to receive audio and video streams from the sender. The relationship between a receiver and a sender is not one-to-one, and multiple receivers can receive audio and video streams from a sender at the same time.

Creating a receiver is much simpler, and we refer directly to the sender's creation configuration to create the receiver:

let receiver = Hylarana::create_receiver(
    id,
    HylaranaReceiverOptions {
        codec: HylaranaReceiverCodecOptions {
            video: VideoDecoderType::H264,
        },
        transport: TransportOptions {
            strategy: TransportStrategy::Multicast("239.0.0.1:8080".parse()?),
            mtu: 1500,
        },
    },
    view,
)?;

The id comes from the sender, for video decoding we use a software decoder, and the transport layer policy needs to be the same on the receiver side as on the sender side, otherwise the two sides won't be able to communicate with each other using different policies. The creation of the view has already been implemented in the above section on creating the sender, so we won't implement it here. But the receiving end needs to play the sound, you just need to create one more AudioRender and refer to the example above to process the audio frames.

LAN discovery

Considering that if there is no mechanism for LAN discovery, the creation process between the sender and the receiver requires an external server to intervene and synchronize some signaling and configuration information, which is not possible out of the box. So hylarana has a built-in LAN discovery component, where you can register a service with DiscoveryService and pass its description, so that other devices can query the registered service for information.

The DiscoveryService provides interfaces for registration and querying. We will combine the example of creating a sender and a receiver below to create a receiver screen by passing configuration information from the sender to the receiver through LAN discovery.

First, we need to create the sender, but we'll skip that here and refer to the Creating the sender section above and then register a service:

use serde::{Deserialize, Serialize};

#[derive(Serialize, Deserialize)]
struct StreamInfo {
    id: String,
    name: String,
    strategy: TransportStrategy,
}

let service = DiscoveryService::register(3456, &StreamInfo {
    name: "test".to_string(),
    id: sender.get_id().to_string(),
    strategy: TransportStrategy::Multicast("239.0.0.1:8080".parse()?),
})?;

StreamInfo is the descriptive information of our service, any type that implements Serialize and Deserialize can be passed as a service property to the register method. The port number needs to be defined by yourself, I define it as 3456 here. Once registration is complete, DiscoveryService will broadcast the current service from the NIC via the mdns protocol.

Next, we turn to the receiving end side:

let service = DiscoveryService::query(|addrs, info: StreamInfo| {
    if info.name == "test" {
        let receiver = Hylarana::create_receiver(
            info.id,
            HylaranaReceiverOptions {
                codec: HylaranaReceiverCodecOptions {
                    video: VideoDecoderType::H264,
                },
                transport: TransportOptions {
                    strategy: info.strategy,
                    mtu: 1500,
                },
            },
            view,
        ).unwrap();
    }
})?;

On the receiver side, we create the receiver by querying for services that have already been registered, calling DiscoveryService::query will always listen for service registrations, and will trigger a callback if a service has already been registered, or is in the process of being registered.

The first parameter in the callback is the device address of the service, which is a list because there may be multiple network devices. Here, since our transmission is implemented via UDP multicast, we don't need to be concerned with the network address of the sending end.

The second parameter is the properties of the sender's service. We first make sure that the service was created by test, then we get the transport policy and sender ID from the properties and pass it to the sender creation function to create the sender.