created_at2021-03-24 02:09:10.340369
updated_at2023-12-18 22:19:30.514586
descriptionDitto is a peer to peer cross-platform database that allows mobile, web, IoT and server apps to sync with or without an internet connection.
Hamilton Chapman




# Ditto Rust SDK ## Overview Describes the different layers of the Ditto Rust SDK. Ditto's core codebase is written in Rust. Currently, however, Rust does not have a stable ABI suitable for directly linking. Therefore the Rust SDK, like the other Ditto SDKs, is exposed through an external interface that uses the C ABI calling conventions. This core library is compiled as a both a static and dynamic library for a variety of architectures. The `dittolive-ditto-sys` crate contains Rust bindings to this C ABI library. The `build.rs` script will also attempt to identify the proper library binary for the host environment, download it from Ditto, and link to it. The Rust SDK includes this `-sys` crate as a dependency and then exposes an idiomatic Rust interface on top of this library, along with documentation and example apps. ## Building an App with the Rust SDK The following outlines the general process of getting started with a new App based on the Ditto Rust SDK. Rust currently does not support a stable ABI. To work around this, the Ditto Rust SDK is distributed in two parts: an ergonomic, open-source crate and a closed-source pre-compiled library. It is essential to have the correct library for both your development system and ultimate production target. There are also some key terms you will need to know, especially for cross-compiling the RustSDK. * HOST - The host system doing the compiling. This could be a developers MacBook Pro (`x86_64-apple-darwin`) for example. * DITTO_TARGET - The system where the final app is going to run. This will be a Raspberry Pi ZeroW with a CPU that supports ARMv6 32-bit instructions and hard floats: `arm-unknown-linux-gnueabihf`. * DITTO_ROOT - The absolute path for the root directory of the Ditto source tree (where available). * APP_ROOT - The absolute path for the root directory of your app. * LIBDITTO - The binary component of the Ditto SDK which exposes a C FFI wrapper. * RustSDK - The Rust library component of the Ditto SDK which links, via a "-sys crate" to LIBDITTO. ### Using a pre-built binary for your RustSDK-based App The easiest way to get started with the Ditto Rust SDK is to use a pre-built Ditto library for your target architecture, where available. 1. Obtain a Ditto License. 1. Install the `nightly` tool chain for your current development machine using [rustup](https://rustup.rs) 1. Create a new Rust repository using github or `cargo new` 1. In the root of the newly created project directory, edit the `Cargo.toml` file. Add a `dittolive-ditto` as a dependency. This is the crate for the Ditto Rust SDK. 1. Test building the automatically generated Hello World app by running `cargo build`. This will trigger linking against the Ditto SDK for the current *host* system. If Ditto is not present for the current host system, this step will report an error when run with `--verbose` logging. If `curl` is present on the host system and the binary component of the Ditto SDK is absent, the `build.rs` script will attempt to download the appropriate library for the compilation target automatically. This step will also create a `TARGET_DIR` target directory (ie. `target/debug`) in the project root. This is a good location for putting the binary component of the Ditto SDK, especially if the development and production hosts are different. 1. For cross-compilation (ie. development on MacOS for a Linux target), the Rust build script and machinery such as `pkg_config` may not automatically find the correct library for your target platform. In this event the Ditto library should be downloaded manually. Manually prefetching this library may also be desirable for offline CI pipelines and other situations where network access is not desired. 1. The URL for each target's Ditto SDK is `https://software.ditto.live/rust/Ditto////` For example, on an x86_64 MacOS developer machine, one would use `https://software.ditto.live/rust/Ditto/1.0.3-alpha1/x86_64-apple-darwin/release/libdittoffi.dylib`. 1. The environment variable `DITTOFFI_SEARCH_PATH` may be used to manually set a directory to be used to find the binary component of the Ditto SDK. This is especially helpful when cross-compiling for different target architectures. 1. With the binary component now in place, verify that `cargo run` executes successfully and doesn't throw a linker error. 1. On some platforms, similar operations are required for other libraries such as various MacOS Frameworks (ie. CoreFoundation, Security). The `cargo` documentation provides guidance of various ways to provide these libraries to each platforms linker. ### Compiling from libditto from source If you have access to a copy of the full Ditto source code, you can compile the binary component of the Rust SDK, `libdittoffi` yourself. The compilation host machine will need 4GB of RAM (or swap file) for linking purposes. If your target machine doesn't have this, you'll need to cross-compile (see section below). 1. First, clone the Ditto source tree. We will refer to the absolute path to the root of the Ditto source tree as DITTO_ROOT going forward. 2. Select a target system. This can be the same as your host system, or another target triple (cross-compilation). The full list of supported target systems is available [here](https://doc.rust-lang.org/rustc/platform-support.html). Note that you will need a valid C toolchain (eg. `CC`, `ld`) for *both* the host and target system. If you are compiling on Mac OS for Linux, this will require more set up (described in a following section). ```bash export DITTO_TARGET=x86_64-unknown-linux-gnu ``` 3. Build `libdittoffi` for your target platform. This can take a while. ```bash (cd ffi && cargo build --release --target $DITTO_TARGET) ``` 4. Confirm your build is successful. You should see `libdittoffi.a` and `libdittoffi.so` in `$DITTO_ROOT/target/$DITTO_TARGET/release/`. This path is known as your `TARGET_DIR` and is configured by a file `.cargo/config.toml` in DITTO_ROOT. Each target and profile (release, debug) will get its own sub-directory. Note that on Windows and Mac OS, the library file extensions will be different (`.dll` and `.dylib`, respectively). 5. Optionally, you can now install `libdittoffi.so` into a cannonical search location for the *target's linker* to find. When compiling on the execution device, we recommend a symlink to `/usr/local/lib` or `/opt/ditto/lib`. This will make it easy for any apps outside of DITTO_ROOT to find `libdittoffi.so`. ```bash sudo ln -sf "$DITTO_ROOT/target/$DITTO_TARGET/release/libdittoffi.so" /usr/local/bin ``` 6. Alternatively, you can specify your TARGET_DIR as the DITTOFFI_SEARCH_PATH that your app will search for this library. ```bash export DITTOFFI_SEARCH_PATH="$DITTO_ROOT/target/$DITTO_TARGET/release" ``` 7. Create your new Ditto-powered App. We'll refer to the root of this app as APP_ROOT going forward. ```bash cargo new myapp ``` 8. Add the ditto Rust SDK as a dependency by editing your app's Cargo.toml file. This should be the Path to the Rust SDK *not* libdittoffi.so. ```Cargo.toml [dependencies] dittolive-ditto.path = "$DITTO_ROOT/rust" ``` 9. Test building your app from within the your apps root directory. Your app should build `dittolive-ditto` (the Rust SDK) which in turn will build `dittolive-ditto-sys` which links to `libdittoffi.so` or `libdittoffi.a`. ```bash cargo build ``` 10. If you want to force static linking of your app to `libdittoffi.a` you can set the `LIBDITTO_STATIC=1` env var. However, this may result in missing symbol errors on some platforms where shared system libraries are otherwise linked in by default. For example, on Mac OS Darwin you may see missing system framework symbols, because these symbols are *only* available as shared libraries. Your app will need to be configured to tell the linker how to source these symbols. ```bash LIBDITTO_STATIC=1 cargo build ``` 11. Finally, to build a release of your app ```bash cargo build --release --target $DITTO_TARGET ``` 12. Your final executable can be run as follows. This may be copied to another system, but be sure to also provide copies of `libdittoffi.so` and any other dynamically linked libraries. ```bash $APP_ROOT/target/$DITTO_TARGET/release/myapp ``` ### How is the Ditto Library located The `build.rs` script for dittolive-ditto-sys makes a best effort attempt to locate, and if absent download, `libdittoffi` in spite of each target OS having distinct linkers and conventions for managing shared libraries. The absolute location of these files depends on whether the SDK is built from source or downloaded from crates.io as a dependency. The search order is as follows: 1. If `DITTOFFI_SEARCH_PATH` env var is set to a valid directory, use this. This takes priority over all other methods. 1. Look in the CARGO_BUILD_TARGET_DIR (ie. `target/$DITTO_TARGET/{debug,release}`). This is automatic for Cargo. 1. Look in the `deps` folder for the CARGO_BUILD_TARGET (ie. `target/$DITTO_TARGET/{debug,release}/deps`). This is automatic for Cargo. 1. Look in the CARGO_MANIFEST_DIR. 1. The current working directory of `cargo`. 1. Common POSIX paths defined in the Filesystem Hierarchy Standard: `/usr`, `/lib`, `/usr/local/lib`, `/opt`. 1. If on a linux system, try using `pkg_config`. Note that Raspberry Pi OS does not ship with `pkg_config`. 1. If on a windows system, try using `vcpkg`. 1. Download pre-built version from Ditto's S3 bucket The end result of this search is to ensure that two key arguments are passed to the target linker when building your app: * `-ldittoffi` - Your app should be linked against the ditto library * `-L DITTOFFI_SEARCH_PATH` - The directory where the ditto library for the target system can be found You can see this process by building with `cargo build -vv`. If desired, you can explicitly provide these two arguments using `cargo rustc` to build your app, or otherwise configuring your apps build system to provide these values when linking. We recommend explicitly specifying the search path for the ditto library, especially when cross-compiling, to ensure the correct DITTO_TARGET architecture is used. ### Cross-compiling for IOT devices Many IOT devices make poor compilation hosts for Rust, and thus it is better to cross-compile on a developer machine with more resources and then execute on the target IOT device. #### Method 1 - Use a rustembedded/cross Docker and VS Studio Code "devContainer" extension to build from source 1. Install docker on your HOST machine 2. Install Visual Studio Code on your Host Machine 3. Install the VS Code "Remote - Containers" extension. 4. Configure your DITTO_TARGET so that you can ssh onto it and copy files using a utility such as `scp`, `sftp`, or `rsync`. 5. Check out the Ditto source tree in VS Code, and then in the root of the project edit `.devcontainer/devontainer.json`'s `build.Dockerfile` key to point to the docker image for your DITTO_TARGET. In this case `Dockerfile.armv-unknown-linux-gnueabihf`. 6. Use the "Remote - Containers" plug in to "Reload in Container" (the green >< icon in the lower left corner). This will download the image, build it, mount the Ditto repo into `/workspaces/ditto`, which will be our effective DITTO_ROOT going forward. 7. Open a terminal in the container and build libdittoffi. The machining target architecture will be automatically selected. ```bash (cd ffi && cargo build --release) ``` 8. You may now copy the built artifact to your target device (on your host) ```bash scp $DITTO_ROOT/target/$DITTO_TARGET/release/libdittoffi.so pi@raspberrypi:/home/pi ``` 9. SSH onto your Raspberry Pi and Symlink libdittoffi.so into a linker search directory ```bash sudo ln -sf /home/pi/libdittoffi.so /usr/local/lib/libdittoffi.so ``` 10. Repeat steps 1 through 8 for your app. If your app lives outside of the DITTO_ROOT, you will need to set up a distinct VS Code project for it. Alternatively, you can compile your app in place on the Raspberry Pi device. 11. You can verify your app on the target device with the following ```bash ldd ./myapp ``` This should verify the target devices linker can resolve all the shared libraries (including libdittoffi.so) on the target device ## Configuring the RUST SDK The following env vars are commonly used to configure the Ditto SDK. * `RUST_SDK_LOG_LEVEL` - Sets the log level of the Ditto libraries internal logger. Values are error, warn, info, debug, and verbose. Default is "info". * `DITTOFFI_SEARCH_PATH` - Explicitly define where to look for `libdittoffi` for your platform. * `DITTO_DB_PATH` - The absolute path where Ditto should store local copies of documents and attachments. This can also be provided programatically. ### Common config patterns The following are common patterns used to configure test and example apps * `DITTO_LICENSE` - EnvVar containing a valid Ditto license token. * `DITTO_APP_NAME` - The full name of your app. Typically defined in reverse DNS format (ie. "live.ditto.carsapp"). * `DITTO_SITE_ID` - A 64 bit unique identifier of this specific instance of your app. Often associated with a Security subject or Identity. * `DITTO_BINDIP` - The IP and/or Port (ie. where Ditto should listen for TCP or WebSocket Transport traffic.
Commit count: 0

cargo fmt