# Architecture
import { Callout } from 'nextra/components'
There will be a breaking SDK upgrade in the coming months. This upgrade will make the SDK a lot easier to build with.
This upgrade will affect the interface of the SDK dramatically, and will be coupled with a protocol change - stay tuned for information on early access to the new protocol testnet.
It will also be coupled with the documentation of the SDK on [crates.io](https://crates.io/).
## Motivations
The motivation behind the creation of the `TcpProxy` module is to allow developers to interact with the Mixnet in a way that is far more familiar to them: simply setting up a connection with a transport, being returned a socket, and then being able to stream data to/from it, similar to something like the Tor [`arti`](https://gitlab.torproject.org/tpo/core/arti/-/tree/main/crates/arti-client) client.
This is an initial version of the module which we are releasing to developers to experiment with. If you run into problems or any functionality that is missing, do reach out on Matrix and let us know.
Furthermore we will be working on optimisations to the module over time - most of this will occur under the hood (e.g. implementing a configurable connection pool for the `ProxyClient`), but all updates will occur according to SemVer, so don't worry about breaking changes!
## Clients
Each of the sub-modules exposed by the `TcpProxy` deal with Nym clients in a different way.
- the `NymProxyClient` relies on the [`Client Pool`](../client-pool) to create clients and keep a certain number of them in reserve. If the amount of incoming TCP connections rises quicker than the Client Pool can create clients, or you have the pool size set to `0`, the `TcpProxyClient` creates an ephemeral client per new TCP connection, which is closed according to the configurable timeout: we map one ephemeral client per TCP connection. This is to deal with multiple simultaneous streams.
- the `NymProxyServer` has a single Nym client with a persistent identity.
## Framing
We are currently relying on the [`tokio::Bytecodec`](https://docs.rs/tokio-util/latest/tokio_util/codec/struct.BytesCodec.html) and [`framedRead`](https://docs.rs/tokio-util/latest/tokio_util/codec/struct.Framed.html) to frame bytes moving through the `NymProxyClient` and `NymProxyServer`.
> For those interested, under the hood the client uses our own [`NymCodec`](https://github.com/nymtech/nym/blob/27ac34522cf0f8bfe1ca265e0b57ee52f2ded0d2/common/nymsphinx/framing/src/codec.rs) to frame message bytes as Sphinx packet payloads.
## Sessions & Message Ordering
We have implemented session management and message ordering, where messages are wrapped in a session ID per connection, with individual messages being given an incrememting message ID. Once all the messages have been sent, the `NymProxyClient` then sends a `Close` message as the last outgoing message. This is to notify the `NymProxyServer` that there are no more outbound messages for this session, and that it can trigger the session timeout.
> Session management and message IDs are necessary since *the Mixnet guarantees message delivery but not message ordering*: in the case of trying to e.g. send gRPC protobuf through the Mixnet, ordering is required so that a buffer is not split across Sphinx packet payloads, and that the 2nd half of the frame is not passed upstream to the gRPC parser before the 1st half, even if it is received first.
Lets step through a full request/response path between a client process communicating with a remote host via the proxies:
### Outgoing Client Request
The `NymProxyClient` instance, once initialised and running, listens out for incoming TCP connections on its localhost port.
On receiving one, it will create a new session ID and packetise the incoming bytes into messages of the following structure:
```rust
pub struct ProxiedMessage {
message: Payload,
session_id: Uuid,
message_id: u16,
}
```
> This code can be found [here](https://github.com/nymtech/nym/blob/develop/sdk/rust/nym-sdk/src/tcp_proxy/utils.rs#L147C1-L152C2)
And then send these to the Nym address of the `NymProxyServer` instance. Not much to see here regarding message ordering, as the potential for reordering only starts once packets are travelling through the Mixnet.
```mermaid
---
config:
theme: neo-dark
layout: elk
---
sequenceDiagram
box Local Machine
participant Client Process
participant NymProxyClient
end
Client Process->>NymProxyClient: Request bytes
NymProxyClient->>NymProxyClient: New session
NymProxyClient->>EntryGateway: Sphinx Packets: Message 1
EntryGateway-->>NymProxyClient: Acks
NymProxyClient->>EntryGateway: Sphinx Packets: Message 2
EntryGateway-->>NymProxyClient: Acks
NymProxyClient->>EntryGateway: Sphinx Packets: Message 3
EntryGateway-->>NymProxyClient: Acks
NymProxyClient->>EntryGateway: Sphinx Packets: Close Message
NymProxyClient->>NymProxyClient: Start Client Close timeout
EntryGateway-->>NymProxyClient: Acks
```
### Server Receives Request & Responds
Here is a diagrammatic representation of a situation in which the request arrives out of order, and how the message buffer deals with this so as not to pass a malformed request upstream to the process running on the same remote host:
```mermaid
---
config:
theme: neo-dark
layout: elk
---
sequenceDiagram
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 2
NymProxyServer-->>Exit Gateway: Acks
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 3
NymProxyServer-->>Exit Gateway: Acks
loop Message Buffer
NymProxyServer->>NymProxyServer: Wait for Message 1
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 1
NymProxyServer-->>Exit Gateway: Acks
NymProxyServer->>NymProxyServer: Message Received: trigger upstream send
end
Note right of NymProxyServer: Note this happens **per session**
NymProxyServer->>Upstream Process: Reconstructed request bytes
Upstream Process->>Upstream Process: Do something with request
Exit Gateway->>NymProxyServer: Sphinx Packets: Message Close
NymProxyServer-->>Exit Gateway: Acks
NymProxyServer->>NymProxyServer: Trigger Client timeout start for session
Upstream Process->>NymProxyServer: Response bytes
NymProxyServer->>NymProxyServer: Write to provided SURB payloads
NymProxyServer->>Exit Gateway: Anonymous replies
box Remote Host
participant NymProxyServer
participant Upstream Process
end
```
> Note that this is per-session, with a session mapped to a single TCP connection. Both the `NymProxyClient` and `Server` are able to handle multiple concurrent connections.
### Client Receives Response
The `ProxyClient` deals with incoming traffic in the same way as the `ProxyServer`, with a per-session message queue:
```mermaid
---
config:
theme: neo-dark
layout: elk
---
sequenceDiagram
box Local Machine
participant Client Process
participant NymProxyClient
end
Entry Gateway--xNymProxyClient: Sphinx Packets: Reply Message 1 dropped: No Ack!
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 2
NymProxyClient-->Entry Gateway: Ack
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 3
NymProxyClient-->Entry Gateway: Ack
Loop Message Buffer:
NymProxyClient->>NymProxyClient: Wait for Message 1
Entry Gateway->>NymProxyClient: Sphinx Packets: Message 1
NymProxyClient-->>Entry Gateway: Acks
NymProxyClient->>NymProxyClient: Message Received: trigger send
NymProxyClient->>Client Process: Response bytes
end
Note right of NymProxyClient: Note this happens **per session**
```
After receiving the packets, it can then forward the recoded bytes to the requesting process.
### Full Flow Diagram
```mermaid
---
config:
theme: neo-dark
layout: elk
---
sequenceDiagram
box Local Machine
participant Client Process
participant NymProxyClient
end
Client Process->>NymProxyClient: Request bytes
NymProxyClient->>NymProxyClient: New session
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 1
Entry Gateway-->>NymProxyClient: Acks
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 2
Entry Gateway-->>NymProxyClient: Acks
NymProxyClient->>Entry Gateway: Sphinx Packets: Message 3
Entry Gateway-->>NymProxyClient: Acks
NymProxyClient->>Entry Gateway: Sphinx Packets: Close Message
Entry Gateway-->>NymProxyClient: Acks
Entry Gateway-->>Mix Nodes: All Packets, Acks, etc
Note right of Mix Nodes: We are omitting the 3 hops etc for brevity here
Mix Nodes-->> Exit Gateway: All Packets, Acks, etc
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 2
NymProxyServer-->>Exit Gateway: Acks
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 3
NymProxyServer-->>Exit Gateway: Acks
loop Message Buffer
NymProxyServer->>NymProxyServer: Wait for Message 1
Exit Gateway->>NymProxyServer: Sphinx Packets: Message 1
NymProxyServer-->>Exit Gateway: Acks
NymProxyServer->>NymProxyServer: Message Received: trigger upstream send
end
Note right of NymProxyServer: Note this happens **per session**
NymProxyServer->>Upstream Process: Reconstructed request bytes
Upstream Process->>Upstream Process: Do something with request
Exit Gateway->>NymProxyServer: Sphinx Packets: Close Message
NymProxyServer-->>Exit Gateway: Acks
NymProxyServer->>NymProxyServer: Trigger Client timeout start for session
Upstream Process->>NymProxyServer: Response bytes
NymProxyServer->>NymProxyServer: Write to provided SURB payloads
NymProxyServer->>Exit Gateway: Anonymous replies
box Remote Host
participant NymProxyServer
participant Upstream Process
end
Entry Gateway--xNymProxyClient: Sphinx Packets: Reply Message 1 dropped: No Ack!
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 2
NymProxyClient-->Entry Gateway: Ack
Entry Gateway->>NymProxyClient: Sphinx Packets: Reply Message 3
NymProxyClient-->Entry Gateway: Ack
Loop Message Buffer:
NymProxyClient->>NymProxyClient: Wait for Message 1
Entry Gateway->>NymProxyClient: Sphinx Packets: Message 1
NymProxyClient-->>Entry Gateway: Acks
NymProxyClient->>NymProxyClient: Message Received: trigger send
NymProxyClient->>Client Process: Response bytes
end
Note right of NymProxyClient: Note this happens **per session**
```