mfahampshire
e454d71b78
* tcp conn tracker * make default decay const * first pass connpool * err handling conpool start * added notes for next features * first version working * first pass spin out client_pool * cancel token * logging change * bump default decay time * bugfix: make sure to apply gateway score filtering when choosing initial node * add duplicate packets received to troubleshooting * client_pool.rs mod * client pool example * clippy * client pool example done * added disconnect to client pool * update mod file * add cancel token disconnect fn * comments * comments * add clone * added disconnect thread * update example files tcpproxy * client pool docs * remove comments for future ffi push + lower default pool size from 4 to 2 * comment on ffi * update command help * clone impl * remove clone * fix clippy * fix clippy again * fix test * tweaked text grammar * updated comment in example * future is now * cherry * cherry * fix borked rebase * fix fmt * wasm fix --------- Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>
209 lines
10 KiB
Plaintext
209 lines
10 KiB
Plaintext
import { Callout } from 'nextra/components'
|
||
|
||
# Architecture
|
||
|
||
## Motivations
|
||
The motivation behind the creation of the `TcpProxy` module is to allow developers to interact with the Mixnet in a way that is far more familiar to them: simply setting up a connection with a transport, being returned a socket, and then being able to stream data to/from it, similar to something like the Tor [`arti`](https://gitlab.torproject.org/tpo/core/arti/-/tree/main/crates/arti-client) client.
|
||
|
||
<Callout type="info" emoji="ℹ️">
|
||
This is an initial version of the module which we are releasing to developers to experiment with. If you run into problems or any functionality that is missing, do reach out on Matrix and let us know.
|
||
|
||
Furthermore we will be working on optimisations to the module over time - most of this will occur under the hood (e.g. implementing a configurable connection pool for the `ProxyClient`), but all updates will occur according to SemVer, so don't worry about breaking changes!
|
||
</Callout>
|
||
|
||
## Clients
|
||
Each of the sub-modules exposed by the `TcpProxy` deal with Nym clients in a different way.
|
||
- the `NymProxyClient` relies on the [`Client Pool`](../client-pool) to create clients and keep a certain number of them in reserve. If the amount of incoming TCP connections rises quicker than the Client Pool can create clients, or you have the pool size set to `0`, the `TcpProxyClient` creates an ephemeral client per new TCP connection, which is closed according to the configurable timeout: we map one ephemeral client per TCP connection. This is to deal with multiple simultaneous streams.
|
||
- the `NymProxyServer` has a single Nym client with a persistent identity.
|
||
|
||
## Framing
|
||
We are currently relying on the [`tokio::Bytecodec`](https://docs.rs/tokio-util/latest/tokio_util/codec/struct.BytesCodec.html) and [`framedRead`](https://docs.rs/tokio-util/latest/tokio_util/codec/struct.Framed.html) to frame bytes moving through the `NymProxyClient` and `NymProxyServer`.
|
||
|
||
> For those interested, under the hood the client uses our own [`NymCodec`](https://github.com/nymtech/nym/blob/27ac34522cf0f8bfe1ca265e0b57ee52f2ded0d2/common/nymsphinx/framing/src/codec.rs) to frame message bytes as Sphinx packet payloads.
|
||
|
||
## Sessions & Message Ordering
|
||
We have implemented session management and message ordering, where messages are wrapped in a session ID per connection, with individual messages being given an incrememting message ID. Once all the messages have been sent, the `NymProxyClient` then sends a `Close` message as the last outgoing message. This is to notify the `NymProxyServer` that there are no more outbound messages for this session, and that it can trigger the session timeout.
|
||
|
||
> Session management and message IDs are necessary since *the Mixnet guarantees message delivery but not message ordering*: in the case of trying to e.g. send gRPC protobuf through the Mixnet, ordering is required so that a buffer is not split across Sphinx packet payloads, and that the 2nd half of the frame is not passed upstream to the gRPC parser before the 1st half, even if it is received first.
|
||
|
||
Lets step through a full request/response path between a client process communicating with a remote host via the proxies:
|
||
|
||
### Outgoing Client Request
|
||
The `NymProxyClient` instance, once initialised and running, listens out for incoming TCP connections on its localhost port.
|
||
|
||
On receiving one, it will create a new session ID and packetise the incoming bytes into messages of the following structure:
|
||
|
||
```rust
|
||
pub struct ProxiedMessage {
|
||
message: Payload,
|
||
session_id: Uuid,
|
||
message_id: u16,
|
||
}
|
||
```
|
||
|
||
> This code can be found [here](https://github.com/nymtech/nym/blob/develop/sdk/rust/nym-sdk/src/tcp_proxy/utils.rs#L147C1-L152C2)
|
||
|
||
And then send these to the Nym address of the `NymProxyServer` instance. Not much to see here regarding message ordering, as the potential for reordering only starts once packets are travelling through the Mixnet.
|
||
|
||
```mermaid
|
||
---
|
||
config:
|
||
theme: neo-dark
|
||
layout: elk
|
||
---
|
||
sequenceDiagram
|
||
box Local Machine
|
||
participant Client Process
|
||
participant NymProxyClient
|
||
end
|
||
Client Process->>NymProxyClient: Request bytes
|
||
NymProxyClient->>NymProxyClient: New session
|
||
NymProxyClient->>EntryGateway: Sphinx Packets: Message 1
|
||
EntryGateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>EntryGateway: Sphinx Packets: Message 2
|
||
EntryGateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>EntryGateway: Sphinx Packets: Message 3
|
||
EntryGateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>EntryGateway: Sphinx Packets: Close Message
|
||
NymProxyClient->>NymProxyClient: Start Client Close timeout
|
||
EntryGateway-->>NymProxyClient: Acks
|
||
```
|
||
|
||
### Server Receives Request & Responds
|
||
|
||
Here is a diagrammatic representation of a situation in which the request arrives out of order, and how the message buffer deals with this so as not to pass a malformed request upstream to the process running on the same remote host:
|
||
|
||
```mermaid
|
||
---
|
||
config:
|
||
theme: neo-dark
|
||
layout: elk
|
||
---
|
||
sequenceDiagram
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 2
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 3
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
loop Message Buffer
|
||
NymProxyServer->>NymProxyServer: Wait for Message 1
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 1
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
NymProxyServer->>NymProxyServer: Message Received: trigger upstream send
|
||
end
|
||
Note right of NymProxyServer: Note this happens **per session**
|
||
NymProxyServer->>Upstream Process: Reconstructed request bytes
|
||
Upstream Process->>Upstream Process: Do something with request
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message Close
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
NymProxyServer->>NymProxyServer: Trigger Client timeout start for session
|
||
Upstream Process->>NymProxyServer: Response bytes
|
||
NymProxyServer->>NymProxyServer: Write to provided SURB payloads
|
||
NymProxyServer->>Exit Gateway: Anonymous replies
|
||
|
||
box Remote Host
|
||
participant NymProxyServer
|
||
participant Upstream Process
|
||
end
|
||
```
|
||
|
||
> Note that this is per-session, with a session mapped to a single TCP connection. Both the `NymProxyClient` and `Server` are able to handle multiple concurrent connections.
|
||
|
||
### Client Receives Response
|
||
|
||
The `ProxyClient` deals with incoming traffic in the same way as the `ProxyServer`, with a per-session message queue:
|
||
|
||
```mermaid
|
||
---
|
||
config:
|
||
theme: neo-dark
|
||
layout: elk
|
||
---
|
||
sequenceDiagram
|
||
box Local Machine
|
||
participant Client Process
|
||
participant NymProxyClient
|
||
end
|
||
Entry Gateway--xNymProxyClient: Sphinx Packets: Reply Message 1 dropped: No Ack!
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 2
|
||
NymProxyClient-->Entry Gateway: Ack
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 3
|
||
NymProxyClient-->Entry Gateway: Ack
|
||
Loop Message Buffer:
|
||
NymProxyClient->>NymProxyClient: Wait for Message 1
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Message 1
|
||
NymProxyClient-->>Entry Gateway: Acks
|
||
NymProxyClient->>NymProxyClient: Message Received: trigger send
|
||
NymProxyClient->>Client Process: Response bytes
|
||
end
|
||
Note right of NymProxyClient: Note this happens **per session**
|
||
```
|
||
|
||
After receiving the packets, it can then forward the recoded bytes to the requesting process.
|
||
|
||
### Full Flow Diagram
|
||
```mermaid
|
||
---
|
||
config:
|
||
theme: neo-dark
|
||
layout: elk
|
||
---
|
||
sequenceDiagram
|
||
box Local Machine
|
||
participant Client Process
|
||
participant NymProxyClient
|
||
end
|
||
Client Process->>NymProxyClient: Request bytes
|
||
NymProxyClient->>NymProxyClient: New session
|
||
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 1
|
||
Entry Gateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 2
|
||
Entry Gateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 3
|
||
Entry Gateway-->>NymProxyClient: Acks
|
||
NymProxyClient->>Entry Gateway: Sphinx Packets: Close Message
|
||
Entry Gateway-->>NymProxyClient: Acks
|
||
|
||
Entry Gateway-->>Mix Nodes: All Packets, Acks, etc
|
||
Note right of Mix Nodes: We are omitting the 3 hops etc for brevity here
|
||
Mix Nodes-->> Exit Gateway: All Packets, Acks, etc
|
||
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 2
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 3
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
loop Message Buffer
|
||
NymProxyServer->>NymProxyServer: Wait for Message 1
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 1
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
NymProxyServer->>NymProxyServer: Message Received: trigger upstream send
|
||
end
|
||
Note right of NymProxyServer: Note this happens **per session**
|
||
NymProxyServer->>Upstream Process: Reconstructed request bytes
|
||
Upstream Process->>Upstream Process: Do something with request
|
||
Exit Gateway->>NymProxyServer: Sphinx Packets: Close Message
|
||
NymProxyServer-->>Exit Gateway: Acks
|
||
NymProxyServer->>NymProxyServer: Trigger Client timeout start for session
|
||
Upstream Process->>NymProxyServer: Response bytes
|
||
NymProxyServer->>NymProxyServer: Write to provided SURB payloads
|
||
NymProxyServer->>Exit Gateway: Anonymous replies
|
||
box Remote Host
|
||
participant NymProxyServer
|
||
participant Upstream Process
|
||
end
|
||
|
||
|
||
Entry Gateway--xNymProxyClient: Sphinx Packets: Reply Message 1 dropped: No Ack!
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 2
|
||
NymProxyClient-->Entry Gateway: Ack
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 3
|
||
NymProxyClient-->Entry Gateway: Ack
|
||
Loop Message Buffer:
|
||
NymProxyClient->>NymProxyClient: Wait for Message 1
|
||
Entry Gateway->>NymProxyClient: Sphinx Packets: Message 1
|
||
NymProxyClient-->>Entry Gateway: Acks
|
||
NymProxyClient->>NymProxyClient: Message Received: trigger send
|
||
NymProxyClient->>Client Process: Response bytes
|
||
end
|
||
Note right of NymProxyClient: Note this happens **per session**
|
||
```
|