lyquor-network: a modern network abstraction and tooling for building distributed systems. ========================================================================================= Why not libp2p? (and others) ---------------------------- We believe our design is clutter-free, minimalist, and usable at different levels. It does what other libraries like libp2p can do, but with much better abstraction based on the authors' multi-year of cutting-edge research experience in distributed systems and consensus. It is not just a code base that "works", but also meets the demands of a wide spectrum of networked infrastructures, especially those requiring high volume of binary (short or moderately sized) messages. In fact, we build this library to be reused and serve well as the basis for *any* core infrastructure that exchanges messages in a networked manner. Why not Tokio? -------------- We do use tokio to manage some intertwined asynchronous logic in our code, but our core data transmission is done by a carefully crafted engine directly based on mio. Instead of throwing all connections into tokio, by giving them a better abstraction (`Endpoint`) and drive them through our well-engineered `Driver` (which is a pool of `Worker`s), we have established a good separation among the following three aspects of a networking library: the notion of a peer-wise data transmission channel, the actual underlying transportation protocol (could even be SMS, if one implements `Transport` trait) it utilizes, and the execution logic that efficiently drives the asynchronous I/O induced by the transmission. By learning from the past projects and with the help of Rust, we believe our abstraction achieves clarity, simplicity and low overhead at the same time. If not the best in terms of the absolute performance (one can always handcraft a super adhoc solution, say in C, with optimal performance), it still offers a solid basis for any distributed systems with great performance and robustness. Reusable at Different Levels ---------------------------- Our design deliberately divides our abstraction/implementation into different levels so engineers and researchers can easily use them for their own purpose with a lot of freedom of customization. At the highest level, one can either use: - `network` (TBD): will contain the code for self-organizing network, including node discovery, routing, etc. - `hub`: defines and implements a multi-transport "hub" that could automatically connect to other peers given their contacts. This could be useful for a relatively static network setup, such as a quorum-based (or "vote-based", "PoS") consensus protocol, or MPC projects. In `hub`, there are three important abstractions (and their implementations) could be customized: - `Driver`: a pool of `Worker`s, where each worker has an OS-level thread and drives the I/O of multiple endpoints. - `Endpoint`: a user-friendly, bidirectional, virtual channel that transmits data to some remote recipient. An endpoint has a pluggable `Transport` at its core that actually implements the network transmission. - `Transport`: a protocol-specific implementation that follows a unified interface to move data through the endpoint. The existence of this abstraction allows `Endpoint` to be connection-agnostic: the underlying transportation method could either be "connection-based" (such as TCP) or "connectionless" (such as UDP), but the user of endpoints does not need to worry about that. An endpoint may *renew* its underlying transport to be able to resume its data transmission once it is "clogged" (i.e., reconnection) -- all these, are invisible to the endpoint users. What's more, while doing all these, endpoints can be moved among the workers (for load-balancing purposes) on-the-fly, without losing any data. Portability ----------- The vast majority of our code base is also WASM-compatible. Which means one can turn web browsers and web-like environments (such as mobile apps) into a legit node that talks to others. This means one can achieve "true" P2P by running a lightweight node in the client to talk to "servers" (nodes with useful services), without having to invoke a cumbersome RPC protocol like JSON/RPC or web REST calls. License ------- This project is licensed under Apache License 2.0. See `NOTICE` file for more information.