Feature/multi surbs (#1796)

* bunch of wip with focus on serialization

* Being able to send normal data (NO SURBS yet) to yourself again

* Fixed RepliableMessage deserialization

* Recovering data from surb messages

* Extracted common code in sphinx payload construction

* Cleanup within received buffer

* requesting, sending and using additional reply surbs

* Following discussion with @simonwicky, removing sender proof and decreasing size of sender tag

* Made sender tag more easily configurable

* Refactoring of message creation

* Propagating reply surb acks but not retransmitting them yet

* Surb retransmissions

* requesting additional surbs from the retransmission flow

* correctly determining the point of requesting additional surbs

* Ability to use socks5 (and network requester) with surbs

* Improved surbs retranmsission reliability

* naive way of not over-requesting surbs

* wip on tag storage

* Improved error propagation for message construction

* Requesting more surbs for stale entries

* Better controlling the point of having to request additional surbs

* Using pseudorandom sender tag instead of a hardcoded one

* First cleanup round in MessageHandler

* Error cleanup and if simplification

* Assigned a more permanent name to the ReplyController

* Removed PendingReply redundant type

* Made socks5 client less eager to over-send reply surbs

* 'anonymous' field on socks5 client to decide whether to use surbs or attach address

* Dead code and import removal in client-core

* Updating ClientRequest variants

* Adjusted decision threshold for requesting more surbs

* Native client cleanup

* Made socks5 client usage of surbs configurable

* Restored statistics in network requester

* Validator-api compiles once again

* Further improved surb request logic

* boxing the recipient in controller requests

* Removal of hardcoded values in favour of propagating them from the config

* more validation during surb requests

* Fixed ClientRequest::Send deserialization

* Added length checks for request deserialization

* post-merge formatting

* Unit tests once again compile and pass

* controlling retransmission_reply_surb_request_size from config

* More Recipient boxing action

* Requesting additional reply surbs for retransmission BEFORE dipping below the threshold

* Making clippy generally happier

* Wasm client compiles (but might not yet work correctly)

* Feature/use expect instead of panicking (#1797)

* Implementation of 'Debug' on 'RealMessage'

* expect with failed channel name instead of throwing empty panics

* Introduced Debug trait constraint in ProxyRunner

* Derive Debug for socks5_requests::Message

* Fix decrypting stored received msg (#1786)

* Fix decrypting stored received msg

* rustfmt

* Moving binary message recovery to separate function

Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>

* real_traffic_stream: reduce frequency of status print (#1794)

* Properly defined unnamed errors

* Dealing with previously ignored errors

* logging improvements

* Removed old example code

Co-authored-by: Jon Häggblad <jon.haggblad@gmail.com>

validator-api/src/network_monitor/chunker.rs

@@ -2,6 +2,7 @@
 // SPDX-License-Identifier: Apache-2.0
 
 use nymsphinx::forwarding::packet::MixPacket;
+use nymsphinx::message::NymMessage;
 use nymsphinx::{
     acknowledgements::AckKey, addressing::clients::Recipient, preparer::MessagePreparer,
 };
@@ -52,10 +53,9 @@ impl Chunker {
     ) -> Vec<MixPacket> {
         let ack_key: AckKey = AckKey::new(&mut self.rng);
 
-        let (split_message, _reply_keys) = self
+        let split_message = self
             .message_preparer
-            .prepare_and_split_message(message, false, topology)
-            .expect("failed to split the message");
+            .pad_and_split_message(NymMessage::new_plain(message));
 
         let mut mix_packets = Vec::with_capacity(split_message.len());
         for message_chunk in split_message {

validator-api/src/network_monitor/monitor/processor.rs

@@ -1,4 +1,4 @@
-// Copyright 2021 - Nym Technologies SA <contact@nymtech.net>
+// Copyright 2021-2022 - Nym Technologies SA <contact@nymtech.net>
 // SPDX-License-Identifier: Apache-2.0
 
 use crate::network_monitor::gateways_reader::GatewayMessages;
@@ -68,27 +68,30 @@ struct ReceivedProcessorInner {
 }
 
 impl ReceivedProcessorInner {
-    fn on_message(&mut self, message: Vec<u8>) -> Result<(), ProcessingError> {
+    fn on_message(&mut self, mut message: Vec<u8>) -> Result<(), ProcessingError> {
         // if the nonce is none it means the packet was received during the 'waiting' for the
         // next test run
         if self.test_nonce.is_none() {
             return Err(ProcessingError::ReceivedOutsideTestRun);
         }
 
-        let encrypted_bytes = self
+        let plaintext = self
             .message_receiver
-            .recover_plaintext(self.client_encryption_keypair.private_key(), message)
+            .recover_plaintext_from_regular_packet(
+                self.client_encryption_keypair.private_key(),
+                &mut message,
+            )
             .map_err(|_| ProcessingError::MalformedPacketReceived)?;
         let fragment = self
             .message_receiver
-            .recover_fragment(&encrypted_bytes)
+            .recover_fragment(plaintext)
             .map_err(|_| ProcessingError::MalformedPacketReceived)?;
         let (recovered, _) = self
             .message_receiver
             .insert_new_fragment(fragment)
             .map_err(|_| ProcessingError::MalformedPacketReceived)?
             .ok_or(ProcessingError::NonTestPacketReceived)?; // if it's a test packet it MUST BE reconstructed with single fragment
-        let test_packet = TestPacket::try_from_bytes(&recovered.message)
+        let test_packet = TestPacket::try_from_bytes(&recovered.into_inner_data())
             .map_err(|_| ProcessingError::MalformedPacketReceived)?;
 
         // we know nonce is NOT none