hdk

The Holochain Development Kit (HDK) provides high and low level functions for writing Holochain applications.

Holochain is built as a client-server architecture. The Conductor, Holochain's runtime, acts as the server. Its Conductor API can be queried by a client to manage hApps and send requests to hApp functions. Read more on Holochain's architecture.

Functions of a hApp are organized into reusable components. In Holochain terminology these components are called "zomes". One or multiple zomes are compiled into WebAssembly (WASM) binaries and bundled into a file referred to as a DNA. All of the DNAs of an application are bundled to a hApp. In short, the structure is hApp -> DNA -> zome -> function.

hApps can be developed using the HDK. See the Holochain Quick Start Guide to get started with hApp development.

Example zomes 🍭

There are numerous example/test WASMs on many aspects of hApp development that can be browsed on Github.

Each example WASM is a minimal demonstration of specific HDK functionality, such as generating random data, creating entries or defining validation callbacks. Some of the examples are very contrived, none are intended as production grade hApp examples, but do highlight key functionality.

Zomes are separated into data model and domain logic

hApps are required to produce and validate data deterministically. There's a data model and a domain logic part to each hApp. In Holochain, the data model is defined in integrity zomes and the domain logic is written in coordinator zomes.

Integrity zomes 📐

Integrity zomes describe a hApp's domain model by defining a set of entry and link types and providing a validation callback function that checks the integrity of any operations that manipulate data of those types. Additionally, a genesis self-check callback can be implemented for basic verification of the data that allows an agent to join a network before they attempt to join it.

The wasm workspace contains examples of integrity zomes like this: https://github.com/holochain/holochain/blob/develop/crates/test_utils/wasm/wasm_workspace/integrity_zome/src/lib.rs

Refer to the HDI crate for more information on the integrity layer.

Coordinator zomes 🐜

Coordinator zomes are the counterpart of integrity zomes in a DNA. They contain the domain logic of how data is read and written. Whereas data is defined and validated in integrity zomes, functions to manipulate data are implemented in coordinator zomes.

An example coordinator zome can be found in the wasm workspace of the Holochain repository: https://github.com/holochain/holochain/blob/develop/crates/test_utils/wasm/wasm_workspace/coordinator_zome/src/lib.rs.

HDK structure 🧱

HDK implements several key features:

• Base HDKT trait for standardisation, mocking, unit testing support: [hdk] module
• Capabilities and function level access control: [capability] module
• [Holochain Deterministic Integrity (HDI)]
• Application data and entry definitions for the source chain and DHT: [entry] module and [entry_types] callback
• Referencing/linking entries on the DHT together into a graph structure: [link] module
• Defining tree-like structures out of links and entries for discoverability and scalability: [hash_path] module
• Create, read, update, delete (CRUD) operations on the above
• Libsodium compatible symmetric/secret (secretbox) and asymmetric/keypair (box) encryption: [x_salsa20_poly1305] module
• Ed25519 signing and verification of data: [ed25519] module
• Exposing information about the current execution context such as zome name: [info] module
• Other utility functions provided by the host such as generating randomness and timestamps that are impossible in WASM: utility module
• Exposing functions to external processes and callbacks to the host: hdk_extern! and map_extern! macros
• Integration with the Rust tracing crate
• Exposing a [prelude] of common types and functions for convenience

Generally these features are structured logically into modules but there are some affordances to the layering of abstractions.

HDK is based on callbacks 👂

The only way to execute logic inside WASM is by having the conductor (host) call a function that is marked as an extern by the zome (guest).

Note: From the perspective of hApp development in WASM, the "guest" is the WASM and the "host" is the running Holochain conductor. The host is not the "host operating system" in this context.

Similarly, the only way for the guest to do anything other than process data and calculations is to call functions the host provides to it at runtime.

Host functions are all defined by the Holochain conductor and implemented by HDK for you, but the guest functions need to all be defined by your application.

Any WASM that does not use the HDK will need to define placeholders for and the interface to the host functions.

All host functions can be called directly as:

use crate::prelude::*;
let _output: HDK.with(|h| h.borrow().host_fn(input));

And every host function defined by Holochain has a convenience wrapper in HDK that does the type juggling for you.

Low-level communication between the conductor and WASM binaries, like typing and serialization of data, is abstracted by the HDK. Using the HDK, hApp developers can focus on their application's logic. Learn more about WASM in Holochain.

External callbacks = Zome functions

To extend a Rust function so that it can be called by the host, add the hdk_extern! attribute.

• The function may take none or one argument that, if provided, must implement serde::Serialize + std::fmt::Debug.
• The function must return an ExternResult where the success value implements serde::Serialize + std::fmt::Debug
• The function must have a unique name across all externs as they share a global namespace in WASM
• Everything inside the function is Rust-as-usual including ? to interact with ExternResult that fails as WasmError
• Use the wasm_error! macro along with the WasmErrorInner::Guest variant for failure conditions that the host or external processes need to be aware of
• Externed functions can be called as normal by other functions inside the same WASM

For example:

use crate::prelude::*;

// This function can be called by any external process that can provide and accept messagepack serialized u32 integers.
#[hdk_extern]
pub fn increment(u: u32) -> ExternResult<u32> {
  Ok(u + 1)
}

// Extern functions can be called as normal by other rust code.
assert_eq!(2, increment(1));

Most externs are simply available to external processes and must be called explicitly e.g. via RPC over websockets. The external process only needs to ensure the input and output data is handled correctly as messagepack.

Internal callbacks

Some externs act as callbacks the host will call at key points in Holochain internal system workflows. These callbacks allow the guest to define how the host proceeds at those decision points. They are defined in zomes like extern callbacks above, but have reserved names listed below.

Callbacks are simply called by name and they are "sparse" in that they are matched incrementally from the most specific name to the least specific name. For example, the validate_{{ create|update|delete }}_{{ agent|entry }} callbacks will all match and all run during validation. All function components with multiple options are optional, e.g. validate will execute and so will validate_create.

Holochain will merge multiple callback results for the same callback in a context sensitive manner. For example, the host will consider initialization failed if any init callback fails.

The callbacks are (see above for examples):

• fn entry_defs() -> ExternResult<EntryDefsCallbackResult>:
  • Typically implemented automatically by macros in the HDK so does NOT require writing the extern for it manually.
  • EntryDefs is a vector defining all entries used by this app.
  • All zomes in a DNA define all their entries at the same time for the host.
  • All entry defs are combined into a single ordered list per zome and exposed to tooling such as DNA generation.
  • Entry defs are referenced by u8 numerical position externally and in DHT actions, and by id/name e.g. "post" in sparse callbacks.
• fn genesis_self_check(_: GenesisSelfCheckData) -> ExternResult<ValidateCallbackResult>:
  • Allows each agent to validate itself before attempting to join the network.
  • Receives GenesisSelfCheckData that includes DNA information, the agent key for the candidate source chain and the membrane proof.
  • Runs before the agent exists on the network so has no ability to use the network and generally only has access to deterministic HDK functions.
• fn init() -> ExternResult<InitCallbackResult>:
  • Allows the guest to pass/fail/retry initialization with InitCallbackResult.
  • Lazy execution - only runs when any zome of the DNA is first called.
  • All zomes in a DNA init at the same time.
  • Any zome failure fails initialization for the DNA, any zome retry (missing dependencies) causes the DNA to retry.
  • Failure overrides retry.
  • See create_cap_grant for an explanation of how to set up capabilities in init.
• fn migrate_agent_{{ open|close }} -> ExternResult<MigrateAgentCallbackResult>:
  • Allows the guest to pass/fail a migration attempt to/from another DNA.
  • Open runs when an agent is starting a new source chain from an old one.
  • Close runs when an agent is deprecating an old source chain in favour of a new one.
  • All zomes in a DNA migrate at the same time.
  • Any failure fails the migration.
• fn post_commit(actions: Vec<SignedActionHashed>):
  • Executes after the WASM call that originated the commits so not bound by the original atomic transaction.
  • Input is all the action hashes that were committed.
  • The zome that originated the commits is called.
• fn validate(op: Op) -> ExternResult<ValidateCallbackResult>:
  • Allows the guest to pass/fail/retry any operation.
  • Only the originating zome is called.
  • Failure overrides retry.

HDK has layers 🧅

HDK is designed in layers so that there is some kind of 80/20 rule. The code is not strictly organised this way but you'll get a feel for it as you write your own hApps.

Roughly speaking, 80% of your apps can be production ready using just 20% of the HDK features and code. These are the 'high level' functions such as [crate::entry::create_entry] and macros like hdk_extern!. Every Holochain function is available with a typed and documented wrapper and there is a set of macros for exposing functions and defining entries.

The 20% of the time that you need to go deeper there is another layer followng its own 80/20 rule. 80% of the time you can fill the gaps from the layer above with host_call or by writing your own entry definition logic. For example you may want to implement generic type interfaces or combinations of structs and enums for entries that isn't handled out of the box.

If you need to go deeper still, the next layer is the holochain_wasmer_guest, holochain_zome_types and holochain_serialization crates. Here you can customise exactly how your externally facing functions are called and how they serialize data and memory. Ideally you never need to go this far but there are rare situations that may require it. For example, you may need to accept data from an external source that cannot be messagepack serialized (e.g. json), or you may want to customise the tracing tooling and error handling.

The lowest layer is the structs and serialization that define how the host and the guest communicate. You cannot change this but you can reimplement it in your language of choice (e.g. Haskell?) by referencing the Rust zome types and extern function signatures.

HDK is atomic on the source chain ⚛

Read up on what the source chain is in Holochain.

All writes to the source chain are atomic within a single extern/callback call.

This means all data will validate and be written together or nothing will.

There are no such guarantees for other side effects. Notably we cannot control anything over the network or outside the Holochain database.

Remote calls will be atomic on the recipients device but could complete successfully while the local agent subsequently errors and rolls back their chain. This means you should not rely on data existing between agents unless you have another source of integrity such as cryptographic countersignatures.

Use a post commit hook and signals or remote calls if you need to notify other agents about completed commits.

HDK should be pinned 📌

The basic functionality of the HDK is to communicate with the Holochain conductor using a specific typed interface.

If any of the following change relative to the conductor your WASM will have bugs:

• Shared types used by the host and guest to communicate
• Serialization logic that generates bytes used by cryptographic algorithms
• Negotiating shared memory between the host and guest
• Functions available to be called by the guest on the host
• Callbacks the guest needs to provide to the host

For this reason we have dedicated crates for serialization and memory handling that rarely change. HDK references these crates with =x.y.z syntax in Cargo.toml to be explicit about this.

HDK itself has a slower release cycle than the Holochain conductor by design to make it easier to pin and track changes.

You should pin your dependency on HDK using the =x.y.z syntax too!

You do not need to pin all your Rust dependencies, just those that take part in defining the host/guest interface.

HDK is integrated with rust tracing for better debugging 🐛

Every extern defined with the hdk_extern! attribute registers a tracing subscriber that works in WASM.

All the basic tracing macros trace!, debug!, warn!, error! are implemented.

However, tracing spans currently do not work, if you attempt to #[instrument], you will likely panic your WASM.

WASM tracing can be filtered at runtime using the WASM_LOG environment variable that works exactly as RUST_LOG does for the Holochain conductor and other Rust binaries.

The most common internal errors, such as invalid deserialization between WASM and external processes, are traced as error! by default.

HDK requires explicit error handling between the guest and host ⚠

All calls to functions provided by the host can fail to execute cleanly, at the least serialization could always fail.

There are many other possibilities for failure, such as a corrupt database or attempting cryptographic operations without a key.

When the host encounters a failure Result, it will serialize the error and pass it back to the WASM guest. The guest must handle this error and either return it back to the host which then rolls back writes (see above), or implement some kind of graceful failure or retry logic.

The Result from the host in the case of host calls indicates whether the execution completed successfully and is in addition to other Result-like enums. For example, a remote call can be Ok from the host's perspective but contain an ZomeCallResponse::Unauthorized "failure" enum variant from the remote agent. Both need to be handled in context.

License: CAL-1.0