Files
nym/documentation/docs/pages/developers/rust/stream/architecture.mdx
mfahampshire f648349e82 Max/docs-diataxis-ify (#6494)
* Diatixisify!

* First pass at Typedoc generation for TS SDK

* Remove overview pages

* Fix typos and remove codebase references from docs

Fix typos across network and developer docs: Quorum, available,
cryptosystem, transaction, proportional, Standalone. Remove TODO
placeholder from dVPN protocol page. Strip GitHub source links
from network docs to decouple documentation from repo structure.

* Expand thin landing pages across network and developer docs

- Add intro content to network overview, infrastructure, and reference landing pages
- Expand developer index with "where to start" guide
- Add usage instructions and explanations to all five TS playground pages
- Expand WebSocket client page with setup and message format examples

* Restructure Rust SDK developer docs

- Delete redundant mixnet example, message-helpers, and message-types subpages
- Delete client-pool architecture and example subpages (content folded into landing)
- Delete tcpproxy troubleshooting (folded into landing page)
- Add deprecation notices to TcpProxy pages, pointing to Stream module
- Add stream module docs: landing page, architecture, tutorial, and 4 example pages
- Add mixnet and client-pool tutorials
- Add SDK tour page
- Update navigation and landing pages with docs.rs links

* Restructure TS SDK developer docs

- Merge overview, installation, and getting started into TS SDK landing page
- Fold FAQ content into bundling/troubleshooting section
- Delete redundant overview, installation, start, and FAQ pages
- Update internal links in browsers.mdx and native.mdx
- Update navigation and example page imports

* Flatten and expand APIs section

- Collapse nested API subpages into single pages with inline Redoc embeds
- Rewrite introduction as landing page with decision table
- Add endpoint categories, quick curl examples to each API page
- Mark Explorer API as deprecated
- Move NS API deployment guide to operators/performance-and-testing
- Fix dangling /apis/nym-api/mainnet link in network-components
- Remove sandbox endpoints from all API pages

* Add redirects for moved and deleted pages

- Add 25 redirects covering TS SDK, Rust SDK, APIs, and network sections
- Fix dangling /developers/typescript/start link in operators changelog

* Replace individual example doc pages with GitHub-linked tables, expand tutorials

- replace individual example doc pages with GitHub-linked tables
- expand mixnet tutorial with persistent identity and split_sender sections
- add tcpproxy tutorial
- rename "API Reference" to "TypeDoc Reference" in TS SDK sidebar
- rename "Misc" to "Extras" in developer sidebar, move VPN CLI up
- remove echo server from tools
- update message-queue callout to reference actual modules
- fix mixnet/examples redirect collision

* Add SEO frontmatter, validate encryption standards, clean up URLs

- add title/description/schemaType/section/lastUpdated frontmatter to 48
  pages across developers, network, and APIs sections
- remove network/.archive/ directory (compare against develop instead)
- update nymtech.net → nym.com for website/blog links (keep infra URLs)
- add native proxy "in progress" callout for Rust/C/Go

* API-scraper update (#6598)

* read nodes and locations

* update python-prebuild.sh

* Address PR #6494 review feedback
- Use "mode" consistently instead of "role" on nym-nodes page
- Replace "staking" with "bonding" for NYM token collateral
- Wire up auto-scraped node counts via TimeNow + nodes-count.json
- Fix broken licensing images: download CC icons locally, replace inline HTML
- Fix 9 stale redirects pointing through deleted /network/architecture path

* Fix linkcheck errors
- Fix stale cross-links: /network/concepts/ → /network/mixnet-mode/
- Replace README.md references with globals.md in TypeDoc output
- Add entryFileName: globals to typedoc.json configs to prevent recurrence

* Fix remaining stale /network/architecture links
- zk-nym-overview: architecture/nyx#nym-api → /network/infrastructure/nyx#nym-api
- setup: network/architecture → /network/overview

* Remove accidentally re-included architecture.md file from rebase

* Standardize tutorials, document examples, add llms.txt, apply tone fixes

- Expand Rust SDK tutorials with step-by-step structure; document all SDK examples across mixnet, client-pool, and tcpproxy pages
- Add llms.txt generation script, wire into build and CI workflows
- Apply tone/style fixes: deduplicate callouts, vary sentence structure, standardize voice consistency across changed pages

* Consolidate redundant network overview docs

* Trim dev docs: git-first imports, stream notice, collapse TcpProxy

* Update tutorial

* Refresh auto-generated API and command outputs

* Update network section docs

* Update developer and API docs: reusable components, stream protocol, conventions, tutorial fixes

* Fix Rust SDK tutorial bugs: setup_env, port conflicts, logging,
open_stream race condition

* Update stream.mdx

* Remove docs.rs link from Stream overview for the moment

* add llms.txt and llms-full.txt note to readme

---------

Co-authored-by: import this <97586125+serinko@users.noreply.github.com>
2026-04-09 15:25:31 +00:00

93 lines
4.2 KiB
Plaintext

---
title: "Stream Module Architecture"
description: "Internal architecture of the Nym Stream subsystem: wire protocol, multiplexing, router task, and how concurrent byte channels share a single MixnetClient."
schemaType: "TechArticle"
section: "Developers"
lastUpdated: "2026-03-15"
---
# Stream Architecture
import { Callout } from 'nextra/components'
{/* Canonical source: sdk/rust/nym-sdk/src/mixnet/stream/ARCHITECTURE.md */}
## Overview
The stream subsystem gives each `MixnetClient` the ability to hold many concurrent byte channels (`AsyncRead + AsyncWrite`) to different remote peers, multiplexed over a single client connection.
```mermaid
---
config:
theme: neo-dark
---
flowchart TD
subgraph MixnetClient
SA["MixnetStream A"] -->|writes| CI["Client input channel"]
SB["MixnetStream B"] -->|writes| CI
CI --> MX["── Mixnet ──"]
MX --> RT["Router task"]
RT -->|Open messages| ML["MixnetListener.accept()"]
RT -->|Data messages| SM["Stream routing table"]
SM --> SA
SM --> SB
end
```
## Wire protocol
Every stream message has a fixed 16-byte LP frame header prepended to the payload:
```
[LpFrameKind: 2 bytes LE][StreamId: 8 bytes BE][MsgType: 1 byte][SequenceNum: 4 bytes BE][Reserved: 1 byte][payload ...]
```
- **LpFrameKind:** `3` (SphinxStream). Distinguishes stream traffic from other LP frame types (Opaque, Registration, Forward).
- **StreamId:** random `u64` generated by the opener, used to multiplex streams.
- **MsgType:** `Open` (0) or `Data` (1).
- **SequenceNum:** `u32` counter, incremented per write. Used by the receiver's per-stream reorder buffer to deliver data in the correct order.
- **Reserved:** must be `0x00`.
There is no `Close` message type; see [Known Limitations](#known-limitations) for why.
## Stream mode
Stream mode is activated lazily on the first call to `open_stream()` or `listener()`. This is a **one-way transition**:
1. The client's message receiver is handed off to a background router task
2. `stream_mode` flag is set to `true`
3. Message-based methods (`send_plain_message`, `wait_for_messages`) are disabled and return errors
There is no switching back without disconnecting and creating a new client.
## Opening and accepting streams
**Opening (outbound):**
1. `open_stream(recipient, surbs)` generates a random `StreamId`
2. An `Open` message is sent through the Mixnet to the recipient
3. A `MixnetStream` is returned, ready for writing and reading
**Accepting (inbound):**
1. `listener.accept()` waits for an `Open` message from a remote peer
2. A `MixnetStream` is created with the opener's `sender_tag` for anonymous replies
3. The stream is ready for bidirectional I/O
## Cleanup
- **On `drop`:** the stream deregisters from the routing table. No close message is sent over the wire.
- **Idle timeout:** streams idle for longer than the configured timeout (default: 30 minutes) are automatically cleaned up. Configure with [`MixnetClientBuilder::with_stream_idle_timeout()`](https://docs.rs/nym-sdk/latest/nym_sdk/mixnet/struct.MixnetClientBuilder.html).
## Known limitations
<Callout type="info">
**Sequence-based reordering.** The Mixnet does not guarantee message ordering at the transport level, but each stream write includes a `sequence_num` in the LP frame header. The receiver maintains a per-stream reorder buffer (BTreeMap keyed by sequence number) that buffers out-of-order messages and drains them in sequence. This means protocols that depend on byte ordering (HTTP, TLS, protobuf) work correctly over streams.
- **Buffer cap:** 256 messages per stream. If the buffer fills (e.g. a large gap in sequence numbers), the receiver skips ahead to the lowest buffered sequence.
- **Duplicates:** messages with a sequence number below the next expected are dropped.
- There is no `Close` message type, since a close could race ahead of in-flight data.
</Callout>
## Internal details
For the full implementation details (router task, `StreamMap`, `PollSender` usage, base-client type rationale), see the `ARCHITECTURE.md` file next to the module source code. This will also be available on [docs.rs](https://docs.rs/nym-sdk/latest/nym_sdk/) once crate publication resumes with the Lewes Protocol.