--- title: 'carrier network protocol' author: 'Arvid E. Picciani (arvid@devguard.io)' revision: '2' date: 'Revision 2, \today' link-citations: 'true' geometry: 'margin=4cm' bibliography: 'bib.bib' --- # Channels and Messages Channels are peer to peer encrypted udp pairs similar to wireguard[@WIREGUARD]. They are routed by channel id rather than IP address, allowing a stream to continue seamlessly when migrating to a different IP address. A stream of messages is a concept similar to QUIC[@quic-transport-13], except that the message bounderies are kept intact to support soft-realtime streaming similar to MQTT[@MQTT]. Messages will arrive at the receiver authenticated and in the order they where sent. # Message broker a peer on the network can open itself to receiving messages by signing itself into the global services table. ~~~~~ message AnnounceRequest { bytes key = 1; } signature ~~~~~ the static key is used to start 0-RTT encryption to the service. Anyone one the network can send a single message to service, which is already encrypted. This allows for command and control servers to be stateless (for example PHP websites). This is the equivalent of QOS level 0 in MQTT. TODO: IoT systems are usuually vastly less powereful than a server on the internet, making it vulternable to DoS if anyone on the internet could send it messages. To create power-equilibrium, a stake needs to be included by the sender. # Cryptosystem ## Handshake A simple Diffie-Hellman Key exchange is done before messages can be transported, based on the the Noise Protocol Framework [@NOISE]. We build on wireguard[@WIREGUARD]'s idea of not sending responses to unauthenticated packages by using the noise NK pattern. An inititor has to obtain the responders static key from a side channel first before sending the first packet. Unlike wireguard, we do not use a static key for the initiator, because its identity is established using signatures instead. The signatures are sent in the first message, somewhat weakening forward secrecy for when the responders static key is stolen. For this reason, the first message only contains identity signatures. An attacker will only be able to make out which identity connected, but not for what purpose. This is identical to wireguard, which uses the X25519 key as identity instead. If u describes an Ed25519 identity and u(h) a signature over the session hash, The pattern is as follows: ~~~~~ NKSig: <- s ... -> e, es, [u, u(h), t] <- e, ee ~~~~~ The initiator starts by sending its identity and a semi-static x25519 which is generated before boot, and then signed by its actual ed25519 identity. This is done, so that the ed25519 secret can be kept offline. If a semi-static exchange identity is compromised, the offline key can be used to generate a new one with a new serial number. A serial number is any integer that always must go up for the lifetime of the offline identity. The responder keeps track of this number per identity and rejects any handshake that does not increase it. This prevents both replay attacks as well as using stolen keys. The serial does not actually have to be based on a clock, as long as the sender guarantees that it always goes up, even after reboot. The network will keep a distributed append-only log of that number. The epoch should however be synchronized with the network ocassionally, since at some point we might want to purge the log of used identities. Any identity before the cutoff epoch will be invalid. ## Packet Format ### from iniiator to responder: Note that the packet counter is not nessesarily 1. It may be higher if earlier handshake packages are lost. ~~~~~ -------------------------------------------------------- | Vers = 0x09 (1 byte) | Reserved (3 bytes) | -------------------------------------------------------- | Routing Key = 0 (63 bits) | Direction = 0 (1bit) | -------------------------------------------------------- | Packet Counter = 1 (8 bytes unsigned big endian) | -------------------------------------------------------- ------- noise ------- -------------------------------------------------------- | Authentication Tag (16 bytes) | -------------------------------------------------------- | Ephermal (32 bytes) | -------------------------------------------------------- ------ encrypted ---- -------------------------------------------------------- | Header CRC8 (1 Byte) | -------------------------------------------------------- | Flags (1 Byte) | | byte 2 = do not move | | byte 3 = move target | -------------------------------------------------------- | Effective Identity (32 bytes) | -------------------------------------------------------- | Timestamp (8 bytes unsigned big endian) | -------------------------------------------------------- | Library Revision (4 bytes unsigned big endian) | -------------------------------------------------------- | Move Target (32 bytes) | -------------------------------------------------------- | 0x00 Padding to 64 bytes boundary | -------------------------------------------------------- | Handshake Signature (64 bytes) | -------------------------------------------------------- ~~~~~ **Packet Counter** 8 byte big-endian integer that increases for each sent packet. it is never reused and each new packet must always have a packet counter higher than the previous packet. The packet counter is for replay-mitigation based on Appendix C of RFC2401[@RFC2401]. It is also used for measuring packet loss. Packets are never resent, instead reordering is done based on individual stream counters. TODO: quic uses 32bit, this seems rather small. ### from responder to initiator ~~~~~ -------------------------------------------------------- | Vers = 0x08 (1 byte) | Reserved (3 bytes) | -------------------------------------------------------- | Routing Key = (63 bits) | Direction = 1 (1bit) | -------------------------------------------------------- | Packet Counter = 1 (8 bytes unsigned big endian) | -------------------------------------------------------- ------- noise ------- -------------------------------------------------------- | Authentication Tag (16 bytes) | -------------------------------------------------------- | Ephermal (32 bytes) | -------------------------------------------------------- ------ encrypted ---- -------------------------------------------------------- | Header CRC8 (1 Byte) | -------------------------------------------------------- | Flags (1 Byte) | -------------------------------------------------------- | Flags (1 Byte) | | byte 2 = move instruction set | -------------------------------------------------------- | Identity (32 bytes) | -------------------------------------------------------- | Timestamp (8 bytes unsigned big endian) | -------------------------------------------------------- | Move instructions len (2 bytes unsigned big endian) | -------------------------------------------------------- | Move instructions | -------------------------------------------------------- | Library Revision (4 bytes unsigned big endian) | -------------------------------------------------------- | Padding to 64 bytes boundary | -------------------------------------------------------- | Signature (64 bytes) | -------------------------------------------------------- ~~~~~ ### in transport mode ~~~~~ -------------------------------------------------------- | Vers = 0x09 (1 byte) | Reserved (3 bytes) | -------------------------------------------------------- | Routing Key = (8 bytes) | -------------------------------------------------------- | Packet Counter = 2 (8 bytes unsigned big endian) | -------------------------------------------------------- ------- noise ------- -------------------------------------------------------- | Authentication Tag (16 bytes) | -------------------------------------------------------- ------ encrypted ---- -------------------------------------------------------- | Header CRC8 (1 Byte) | -------------------------------------------------------- | Payload len (2 bytes unsigned big endian) | -------------------------------------------------------- | Flags (1 Byte) | -------------------------------------------------------- | Frame 1 | -------------------------------------------------------- | Frame 2 | -------------------------------------------------------- | Frame .. | -------------------------------------------------------- | Padding to 64 bytes boundary | -------------------------------------------------------- ~~~~~ ### RoutingKey the routing key is not shifted after parsing instead the last bit was masked out ``` route[7] &= 0b11111110; ``` so that routing keys are alwas even. that wasn't intended, but is now defacto standard. ## Frame types | Value | name | |-------|------------| | 0x00 | Padding | | 0x01 | Ack | | 0x02 | Ping | | 0x03 | Disconnect | | 0x04 | Open | | 0x05 | Stream | | 0x06 | Close | | 0x07 | Configure | | 0x08 | Fragmented | ### 0x00 Padding Padding in the form of 0x00 bytes can ocure before a frame header, or after a completed frame and must be skipped until there is a value that is not 0x00 (the next useful frame header). ### 0x01 Ack ~~~~~ -------------------------------------------------------- | Frame Type = 0x01 (1 byte) | -------------------------------------------------------- | Ack Delay = delay in ms (2 byte unsigned big endian) | -------------------------------------------------------- | Ack Count (2 byte unsigned big endian) | -------------------------------------------------------- | Ack 1 (8 bytes unsigned big endian) | -------------------------------------------------------- | Ack 2 (8 bytes unsigned big endian) | -------------------------------------------------------- | Ack .. (8 bytes unsigned big endian) | -------------------------------------------------------- ~~~~~ the acks are sorted by largest first ### 0x02 Ping ~~~~~ -------------------------------------------------------- | Frame Type = 0x02 (1 byte) | -------------------------------------------------------- ~~~~~ ### 0x03 Disconnect ~~~~~ -------------------------------------------------------- | Frame Type = 0x03 (1 byte) | -------------------------------------------------------- | Reason (1 byte ) [ since 0x09 ] | | 1 = Application | | 6 = Resource Limit | | 7 = Move | -------------------------------------------------------- | Move len (2 bytes unsigned big endian) [ since 0x09 ]| -------------------------------------------------------- | Move instruction | -------------------------------------------------------- ~~~~~ Tells the other side to stop sending any packets, including retransmissions, since the peer is no longer available. A peer should send close to all streams first and wait for their acks, since an out of order disconnect will terminate all streams, even if they still have packets in flight. ### 0x04 Open ~~~~~ -------------------------------------------------------- | Frame Type = 0x04 (1 byte) | -------------------------------------------------------- | Stream Id (4 bytes unsigned big endian) | -------------------------------------------------------- | Data Size (2 byte unsigned big endian) | -------------------------------------------------------- | Data | -------------------------------------------------------- ~~~~~ Open is followed by stream messages, and then eventually by a close, creating an ordered reliable stream of messages. It is the first packet within an ordered stream, with order id 1, and must be acked. Stream id can be arbitrarily chosen by each peer so that: - 0x00 is never chosen - the initiator of the channel uses ODD-numbered stream ids, - the responder uses EVEN-numbered stream ids. ### 0x05 Stream ~~~~~ -------------------------------------------------------- | Frame Type = 0x05 (1 byte) | -------------------------------------------------------- | Stream Id (4 bytes unsigned big endian) | -------------------------------------------------------- | Order (8 bytes unsigned big endian) | -------------------------------------------------------- | Data Size (2 byte unsigned big endian) | -------------------------------------------------------- | Data | -------------------------------------------------------- ~~~~~ order starts at 2, since 1 is header ### 0x06 Close ~~~~~ -------------------------------------------------------- | Frame Type = 0x06 (1 byte) | -------------------------------------------------------- | Stream Id (4 bytes unsigned big endian) | -------------------------------------------------------- | Order (8 bytes unsigned big endian) | -------------------------------------------------------- | Reason (1 byte ) [ since 0x09 ] | | 1 = Application | | 3 = Application Encoding Error | | 4 = Exceeded Fragmentation Limit | | 6 = Resource Limit | -------------------------------------------------------- ~~~~~ This is the last message in a stream. The sender may still be open to receiving Messages but will not send any new ones. Close Messages must also be reordered before handling so that any preceeding data is handled first. ### 0x07 Config ~~~~~ -------------------------------------------------------- | Frame Type = 0x07 (1 byte) | -------------------------------------------------------- | Configurations (1 bytes) | | 10000 0000 Keepalive | | 01000 0000 Sleeping | -------------------------------------------------------- | Data Len (2 bytes unsigned big endian) | -------------------------------------------------------- ------ data ---- -------------------------------------------------------- | Seconds (4 bytes unsigned big endian) | -------------------------------------------------------- ~~~~~ ### 0x08 Fragmented ~~~~~ -------------------------------------------------------- | Frame Type = 0x05 (1 byte) | -------------------------------------------------------- | Stream Id (4 bytes unsigned big endian) | -------------------------------------------------------- | Order (8 bytes unsigned big endian) | -------------------------------------------------------- | Fragments (4 byte unsigned big endian) | -------------------------------------------------------- | Data | -------------------------------------------------------- Fragmented is sent to indicate a number of following stream frames are actually one big message. The maximum size is application specific, an application should respond with a Close reason 4 if a peer requested a too big buffer. ~~~~~ Configuration can be sent by any peer and must be acked. A parsers must always read 'Data Len' many bytes after the flags field, even if it cannot interpret them due to version incompatbility. Keepalive sets the period after which a ping packet is sent by a peer, if no other packet was sent in the period. This can be used to tune for aggressive firewalls, but it makes most sense in combination with the sleeping flag. Sleeping indicates that within the next Idle period, the sending peer will be unresponsive because it is sleeping. The other side must stop sending any packets except ack until he keepalive period expired, and it must not disconnect the peer due to unresponsiveness. If a peer cannot accept an excessive sleep period, it must respond with Disconnect instead of Ack. # Errata the version and route header is unencrypted, which makes it susceptible to downgrade attacks. version 0x09 adds a crc8 of the entire header in the encrypted payload in version 0x08 the packet header was parsed incorrectly, so that no extension is possible. the parser now ignores the flags and protocol version had to be bumped to 0x09 version 0x08 did not have an encypted flags field and instead sent the cert counter unconditionally in the handshake response in transport there is a previously undocumented encrypted payload len field before frames. its purpose is error correction, since noise can sometimes claim to have correctly decrypt junk. padding was reduced from 255 to 64 bytes, and the parser was fixed to not check padding on recv, since padding should be sender controlled # References