PinBoard: . Implement the Write-ahead-log, which can be replayed or appended, the value type need to be user defined(the new message type). . Segmented log, async file IO, low water mark. . Implement recover logic. 15/Aug/2024 afternoon Complete the signature part. I may need to build another connection part which is a synchronous version. 13/Aug/2024 afternoon I used the ecdsa signature to verify the message and implement the whole logic. Planned to encrypt the data using AES, in fact , I want to refactor the whole messages to encrypted in AES, but the amount of work would be so huge. 10/Aug/2024 evening Going to ECDSA's wiki page https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm, I got a brief understanding of how ECDSA works. I found that in openssl crate, there are a lot of encrtption algorithm including Rsa and ECDSA. So I will transfer the Rsa encryption using rsa crate to that in openssl for simplicity. 05/Aug/2024 night Completed: Implemented the full logic of Rsa commmunication and handle it like a userdefined message. 04/Aug/2024 night Completed: Implemented the short connection version and tested. Completed a big part of rsa signature: logic of getting other node's public key. To be Completed: The full logic of using private key and public key to sending encrypted infomation to each other. 29/Jul/2024 night Seems that there are something wrong in the conn_pool, I still can't implement the long connection. 28/Jul/2024 afternoon I found that there is some problems in the former method of dealing with the user defined message. Because the handle_connection is called by the start_listening method, so it's return value can't be used by the user. It still needs handler registration to accomplish this. I have completed a draft, hope it will work, I'll test it tonight. 27/Jul/2024 afternoon Complete: Support for user defined message. Including write and read. Maintain a conn_pool to ease the message burden. 20/Jul/2024 afternoon Comeplete: Public API for user to send out message of their own. 16/Jul/2024 night These 2 days I reviewed the code of salticidae and find that salticidae use a event loop for user defined logic. So I tried to build my own, I haven't tested it yet and it needs so deep knowledge to fully understand it, which I would try next few days. Completed: AsyncEventQueue draft. 14/Jul/2024 afternoon Completed: completed the rest of segment struct 13/Jul/2024 night Completed: completed the create and open function of segment struct. 12/Jul/2024 night Completed: Ported the the memory map part and some of the segment part. Initiated the Serde serailized version of message, which could save a lot of time for parsing the message. 11/Jul/2024 night Begin to implement write ahead log, the effort and amount of code beyond my expectation. Use unsafe rust, memory reflection and segmented log. Completed: A simple wal. 11/Jul/2024 afternoon Completed: Implemented the error log. 10/Jul/2024 night Completed: Implement the logic of initiate a node with argument. Then I can indicate the id and address with run the node. 09/Jul/2024 night Completed: Implemented the recovery snapshot and stored it through file IO. Implemented the serialization and deserialization of Backup. Implemented backup mechanism. Optimize the overall structure a little. 08/Jul/2024 night Completed: Implmented the 'establish' message and the handle part. This message is mainly for recovering nodes to use. 08/Jul/2024 night Removed all the locks and use Dashmap which is thread safe. Removed connection pool, it is Tcp long connection, there are a lot of stuff there. Every thing is fine now. 07/Jul/2024 night Change the code throughly. 07/Jul/2024 evening Completed: Tried to use docker to hold client and server, succeed ugly. Optimize code to use log to handle some of the error that can be tolerated, such as send hb. Optimize code to use some Rust chained code on iterators. 06/Jul/2024 evening Completed: Message type 'esatblish'. Opetimize the code to avoid dead lock. A method for establishing a connection between two nodes. Optimize the logic of connection, the heartbeat broadcasting operation no longer establish connections to new or dropped nodes. Add some comment. 05/Jul/2024 night Completed: Add some comments. Add signature to heartbeat message and add verification in parsing heartbeart. 05/Jul/2024 afternoon Completed: Use a Tcp connection pool to maintain Tcp connection for those haven't been dropped. Increased performance of asynchronous IO. 03/Jul/2024 night Completed: Add something about crypto, but I don't understand it. 03/Jun/2024 evening node2.rs use Rwlock and dashmap to optimize the performance. But I don't know about Rwlock and dashmap, so I will check them. 03/Jul/2024 noon Completed: The implementation of a node. The node asynchronously handle connections, send heartbeats and check neighbors' activity. Question: How can a node get a neighbor's address. 02/Jul/2024 night Completed: HeartBeat of node. 30/Jun/2024 night Completed: 'apply' method can just apply for specific 4 message types, others need to do there own job. There is a state I have to implement, but I don't know how to decide which field need to be thread-safe which needn't to be. 30/Jun/2024 afternoon Completed: something about config and heartbeat. 28/Jun/2024 evening Completed: A simple demo of ping-pong. (Just cargo run --bin server and client) 27/Jun/2024 night(2) Completed: The Set message. The Unknown message. 27/Jun/2024 night Completed: The Ping message, looks like I don't have to implement heartbeat because it is identical with Ping. 27/Jun/2024 evening Several things to say: 1. The 'Frame' is actually RESP(Redis serialization protocol), a fast and easy to implement serialization protocol. 2. The message type need to be extended later, carrying sender's infomation. 3. The 'apply' method of each message type need to be extended when operations need. 4. Need to implement snapshot, using serde. And maybe referencing to RDB and AOF of redis can help. 5. Need to use snapshot as crash-recovery methods. 27/Jun/2024 afternoon Completed: Implementation of Get message, notice that this is different from Get command in redis because this key is u64 and value is String while in redis is String and bytes. To be completed: other message type. server and client. 26/Jun/2024 evening Completed: The skeleton of msg.rs, there could be message type in the future. By the way, I need to think about the operation by the node self, for example propose in Paxos. To be Completed: I need to read the raft, viewstamp and pbft paper, to get the operation they use. There must be more in the Db and Message. And the specific message operation need to be implemented. 25/Jun/2024 night Completed: Completed the db.rs, think a lot about my project: I can use frameworks like serde, tonic, etc. But those frameworks are too big and I don't need that many features because I just need a little of them. At first I failed to build a demo on my conn and frame mod. Then I began to use tokio_util to complete this. Later I found that there was a mistake on my implemtation of redis protocol. After fixed that, things worked. I can use those frames to build various command and messages. To be completed: heartbeat, broadcast and snapshoot. and I need to implement backup when needed. 25/Jun/2024 afternoon Completed: Do something on the Db. The Db will be responsible for use for handle data which maybe visited repeatly. Such as log and neighbor nodes. To be completed: I need to implement a short version of ping-pong. 24/Jun/2024 night Completed: added some comments for conn.rs and frame.rs. complete the frame.rs. start a db.rs. To be completed: I need to use a db intance for the library because asynchronous operations could lead to data race. 23/Jun/2024 night Completed: next_bytes, next_string, next_int, parse.rs. To be completed: Feaguring out my own cmd list. After some thinking, I need parser libraries to translate dsl to rust code. I need to implement neighbor and broadcast first. 23/Jun/2024 morning Completed: struct Parse, enum ParseError, From for ParseError, impl new and next for Parse. Feagured out what the meaning of cmd and parse. To be completed: The remainning of parse.rs, and feaguring out my own cmd list. 20/Jun/2024 night Completed: get_decimal, get_line, from, try_into, impl others To be completed: implementing ping pong between 2 nodes. 18/Jun/2024 1am Completed: parse, To be completed: to_vec, into, advance, get_line, get_decimal, try_into 17/Jun/2024 afternoon Completed: Feagured out what cursor for. write_decimal, skip, peek_u8, get_u8 To be completed: parse, advance, into, get_line, get_decimal, try_into. 16/Jun/2024 night Completed: Feagured out what frame is. write_frame, write_value, check. To be completed: write_decimal, parse, advance, into, get_u8, get_line, get_decimal, skip, peek_u8, try_into. 16/Jun/2024 1. Frame, Frame::check, Frame::parse. 2. Result. 3. buffer::advance. 4. Any way, check the source code of mini-redis and implement the same way. 15/Jun/2024 1. This could be a library crate for use later, but now I need to implement the Paxos example. 2. Value transferred by network should be seriallized. But now just use u32 to keep simple. 3. Connection and messages sending need to be asynchronously implememted by Tokio. 4. I need several other instances using this code to be proposer, acceptor and learner. I am not sure if I can place tham in a bin folder like in tokio tutorial while keeping use the lib code.