Files
nym/docs/LP_PROTOCOL.md
T
Drazen Urch 8a00ed6071 LP Registration + Telescoping + Gateway Probe Localnet Mode (#6286)
* Add KKT cryptographic primitives

Post-quantum Key Encapsulation Mechanism (KEM) Key Transfer protocol.
Enables efficient distribution of post-quantum KEM public keys.

Squashed from georgio/noise-psq branch.

* Implement LP registration protocol with KKT/PSQ integration

Initial implementation of the Lewes Protocol (LP) for gateway registration:
- Add nym-lp crate with Noise protocol handshake
- Add LP listener to gateway for handling registrations
- Add LP client for registration flow
- Integrate KKT for post-quantum KEM key exchange
- Integrate PSQ for post-quantum PSK derivation
- Add Ed25519 authentication throughout
- Add docker/localnet support for testing

Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>

* Add LP telescoping with nested sessions and subsession support

Extends LP protocol with telescoping architecture for nested sessions:
- Add nested session support with KKpsk0 rekeying
- Add subsession support with collision detection
- Implement unified packet format with outer header
- Refactor gateway handlers for single-packet forwarding
- Add TTL-based state cleanup for stale sessions
- Add outer AEAD encryption layer
- Refactor registration client for packet-per-connection model

* Add gateway-probe localnet mode with WireGuard tunnel support

Adds localnet testing mode to gateway-probe for LP development:
- Add TestMode enum for different probe configurations
- Add --gateway-ip flag for direct gateway testing
- Implement two-hop WireGuard tunnel for localnet
- Add mock ecash support for testing without real credentials
- Add netstack Go bindings for userspace networking
- Restructure probe with mode and common modules
- Update README with localnet mode documentation

* Increase KCP fragment limit from u8 to u16

- Change frg field from u8 to u16 in packet header (25 bytes total)
- Update encode/decode to use get_u16_le/put_u16_le
- Update Segment struct frg field to u16
- Remove truncating cast in session.rs
- Max message size now ~91MB (65,535 fragments × MTU)
- Internal protocol only, no interop concerns

Nym uses KCP for reliability and multiplexing, not standard real-time
use cases. The u8 limit (255 fragments, ~355KB) was insufficient.

Addresses: nym-yih9

* Zeroize Ed25519 key material in to_x25519 conversion

Wrap hash and x25519_bytes in zeroize::Zeroizing to ensure private
key material is cleared from memory after use.

Closes: nym-k55g

* Return Result from KCP session input() for error detection

Change KcpSession::input() to return Result<(), KcpError> so callers
can detect invalid packets instead of silently ignoring them.

- Add ConvMismatch error variant for conversation ID mismatches
- Update driver to propagate errors from session.input()
- Update all test and example callers

Closes: nym-n0kk

* Fix Zeroizing deref in ed25519 to_x25519 conversion

The from_bytes() function expects &[u8], need to deref the Zeroizing
wrapper to get the inner array.

* Add semaphore-based connection limiting for LP packet forwarding

Limits concurrent outbound connections when forwarding LP packets to
prevent file descriptor exhaustion under high load.

Key changes:
- Add max_concurrent_forwards config (default 1000)
- Add forward_semaphore to LpHandlerState
- Acquire semaphore permit before connecting in handle_forward_packet
- Return "Gateway at forward capacity" error when at limit

This provides load signaling so clients can choose another gateway
when the current one is overloaded.

Design note: Connection pooling was considered but provides minimal
benefit since telescope setup is one-time and targets are distributed
across many different gateways. See AIDEV-NOTE in LpHandlerState for
full analysis.

Closes: nym-xi3m

* Return error on session unavailable in handle_subsession_packet

Replace .session().ok() with proper error handling to fail fast when
session is Closed or Processing after state machine processing.

Previously, the code silently continued with outer_key = None, which
could cause protocol errors downstream.

Closes: nym-8de0

* Use explicit bincode Options helper in nested_session

Add bincode_options() helper that returns DefaultOptions with explicit
big_endian and varint_encoding configuration. This future-proofs against
bincode 1.x/2.x default changes and makes serialization format explicit.

Updated all 4 bincode usages in nested_session.rs to use the helper.

* Deduplicate outer_key lookup pattern in nested_session.rs

Extract common state_machine.session().ok().and_then(...) pattern into
two helper methods:
- get_send_key() for encryption (outer_aead_key_for_sending)
- get_recv_key() for decryption (outer_aead_key)

Updated 6 call sites to use the helpers, reducing verbosity.

* Add LpConfig struct and AIDEV-NOTE documentation for KKT+PSQ

- Create config.rs with LpConfig struct (kem_algorithm, psk_ttl, enable_kkt)
- Export LpConfig from lib.rs
- Add AIDEV-NOTE to psk.rs explaining:
  - Why PSQ is embedded in Noise (single round-trip, PSK binding)
  - KEM migration path (X25519 → MlKem768 → XWing)
- Add AIDEV-NOTE to state_machine.rs explaining protocol flow:
  - KKTExchange → Handshaking → Transport state transitions
  - PSK derivation formula (ECDH || PSQ || salt)

* Add forward_timeout to LP client config

Add forward_timeout (30s default) to LpConfig and wrap send_forward_packet's
connect_send_receive call with tokio::time::timeout, matching the pattern
used by register() with registration_timeout.

This prevents indefinite hangs when forwarding packets through entry gateway.

* Add negotiated_version field to LpSession

Add AtomicU8 field to store the protocol version from handshake packet
headers. Includes getter and setter methods for future version negotiation
and compatibility checks.

- negotiated_version() returns current version (defaults to 1)
- set_negotiated_version() allows setting during handshake
- Subsessions inherit version 1 (can be enhanced to inherit parent's)

* Change MessageType from u16 to u32

Breaking wire protocol change: MessageType field increased from 2 bytes
to 4 bytes in LP packets. This future-proofs the message type space and
aligns with other u32 fields.

Changes:
- message.rs: #[repr(u32)], from_u32(), to_u32()
- error.rs: InvalidMessageType(u32)
- codec.rs: All serialization/deserialization updated to 4-byte msg_type
  - Cleartext parsing: inner_bytes[4..8], content at [8..]
  - AEAD parsing: decrypted[4..8], content at [8..]
  - Serialization: 4 bytes for message type

* Various smaller fixes

* Refactor LP to stream-oriented TCP processing

Gateway (handler.rs):
- Add bound_receiver_idx field for session-affine connections
- Convert handle() from single-packet to loop with EOF detection
- Add validate_or_set_binding() for receiver_idx validation
- Set binding in handle_client_hello after collision check
- Centralize emit_lifecycle_metrics in main loop only
- Add is_connection_closed() helper for graceful EOF

Client (client.rs):
- Add stream field for persistent TCP connection
- Add ensure_connected(), send_packet(), receive_packet(), close() methods
- Modify perform_handshake_inner() to use persistent stream
- Modify register_with_credential() to use persistent stream
- Modify send_forward_packet() to use persistent stream
- Keep connect_send_receive() for reference (marked dead_code)

This reduces handshake overhead from ~5 TCP connections to 1.

Drive-by: Fix log::info! -> info! in wireguard peer_controller.rs

* Add persistent exit stream for entry→exit forwarding

Entry gateway now maintains a persistent TCP connection to the exit
gateway per client session, reusing it for all forward requests from
that client. This reduces TCP handshake overhead significantly.

Key changes:
- Add exit_stream: Option<(TcpStream, SocketAddr)> to LpConnectionHandler
- Modify handle_forward_packet() to open on first forward, reuse after
- Clear exit_stream on connection errors (auto-reconnect on next forward)
- Semaphore only acquired for connection opens, not reuse (sequential access)

* Fix code review issues for stream-oriented LP

- Add 30s timeout to exit stream I/O operations (nym-df31)
  Prevents handler from hanging on unresponsive exit gateway

- Return error on forward target address mismatch (nym-zegu)
  Previously warned and proceeded, which could mask bugs

- Close client stream on handshake error paths (nym-scvm)
  Prevents state machine inconsistency on timeout or failure

* Add LP registration idempotency and retry logic

Make LP registration resilient to network failures that could waste
credentials. When registration succeeds on the gateway but the response
is lost (e.g., network drop), clients can retry with the same WG key
and get the cached result instead of spending another credential.

Gateway-side:
- Add check_existing_registration() helper that looks up WG peer and
  returns cached GatewayData if already registered
- Add idempotency check in process_registration() dVPN branch
- Only return cached response if bandwidth > 0 (ensures registration
  was actually completed, not just peer created)
- Track idempotent registrations with lp_registration_dvpn_idempotent metric

Client-side:
- Add register_with_retry() to LpRegistrationClient that acquires
  credential once and retries handshake+registration on failure
- Add handshake_and_register_with_retry() to NestedLpSession for
  exit gateway registration via forwarding
- Add exponential backoff with jitter between retry attempts
- Verify outer session validity before nested session retry

Both retry methods clear state machine before retry to ensure fresh
handshake, and reuse the same credential across all attempts.

* Add no-mix-acks feature flag to nym-sphinx-framing

When enabled, mix nodes skip ack extraction and forwarding entirely.
The full payload (including ack portion) is returned as the message.

Closes: nym-3wrr

* Create nym-lp-speedtest crate scaffold

- Created tools/nym-lp-speedtest/ with Cargo.toml
- Added main.rs with CLI argument parsing
- Created stub modules: client.rs, speedtest.rs, topology.rs
- Added to workspace members
- Verified compilation with cargo check

* Implement topology fetching for nym-lp-speedtest

- Add topology.rs with NymTopology integration
- Fetch mix nodes and gateways from nym-api
- Build GatewayInfo with LP addresses (port 41264)
- Provide random_route_to_gateway() for Sphinx routing
- Add required Cargo.toml dependencies

* Implement LP+Sphinx+KCP client with SURB support

- Add send_data() and send_data_with_surbs() methods for mixnet data
- Integrate KCP reliable delivery with Sphinx packet construction
- Add x25519 encryption keypair for SURB reply mechanism
- Wire up main.rs to test LP handshake and data path
- Add NymRouteProvider support in topology for SURB construction
- Refactor send_data() to delegate to send_data_with_surbs(0) (DRY)

The client can now:
- Perform LP handshake with gateways
- Send data through the mixnet wrapped in KCP + Sphinx packets
- Attach SURBs for bidirectional communication
- Return encryption keys for decrypting replies

* Rename nym-lp-speedtest to nym-lp-client and fix KCP bug

- Rename crate from nym-lp-speedtest to nym-lp-client
- Fix KCP bug: add driver.update() call before fetch_outgoing()
  Without update(), KCP never moves segments from snd_queue to snd_buf
- Update CLI name, about string, and user agent to match new name

* Add LP mixnet mode registration with nym address return

- Extend RegistrationMode::Mixnet to include client_ed25519_pubkey
  and client_x25519_pubkey for nym address construction
- Add LpGatewayData struct containing gateway_identity and
  gateway_sphinx_key for SURB reply routing
- Add lp_gateway_data field to LpRegistrationResponse for mixnet mode
- Implement success_mixnet() constructor for mixnet registrations
- Update gateway registration to insert clients into ActiveClientsStore
  for SURB reply delivery, matching the websocket flow

* Implement LP data handler on UDP:51264

- Add LpDataHandler for UDP data plane (port 51264)
- Decrypt LP layer and forward Sphinx packets to mixnet
- Add outbound_mix_sender to LpHandlerState
- Integrate data handler spawn into LpListener::run()
- Add metrics for data packets received/forwarded/errors

Implements nym-yzzm

* Fix replay protection vulnerability in LP data handler

Use state machine process_input() instead of manual decryption to ensure
proper replay protection:
- Counter check against receiving window
- Counter marking after successful decryption

Also handle subsession actions gracefully (SendPacket ignored on UDP,
clients should use TCP control plane for rekeying).

Security fix for nym-yzzm implementation.

* feat(ipr): add KcpSessionManager for LP client KCP handling

- Add fetch_incoming() and recv() methods to KcpDriver for retrieving
  reassembled messages
- Create KcpSessionManager in ip-packet-router that manages KCP sessions
  keyed by conv_id (first 4 bytes of KCP packet header)
- Store ReplySurbs per session for sending anonymous replies
- Implement session timeout (5 min) and max sessions limit (10000)
- Add comprehensive tests for session lifecycle and KCP roundtrip

* feat(ipr): integrate KcpSessionManager into MixnetListener

- Add KcpSessionManager field to MixnetListener struct
- Add is_kcp_message() helper to detect KCP-wrapped payloads
- Add on_kcp_message() to process LP client KCP messages
- Refactor on_reconstructed_message() to route KCP vs regular IPR
- Add KCP tick timer (100ms) for session updates and cleanup
- Initialize KcpSessionManager in IpPacketRouter::run_service_provider()

KCP messages are detected by checking byte 4 for valid KCP commands
(81-84), which doesn't conflict with IPR protocol version bytes (6-8)
at position 0.

Closes: nym-96zl

* fix(ipr): prevent KCP detection false positives on IPR messages

Add secondary check in is_kcp_message() to exclude messages that match
IPR protocol header pattern (version 6-8 at byte 0, ServiceProviderType
0-2 at byte 1). This prevents false positives where IPR messages with
byte 4 in range 81-84 would be incorrectly routed to KCP processing.

Added 4 unit tests to validate the detection logic.

Closes: nym-6f3x

* fix(ipr): wrap KCP client responses in KCP before SURB reply

- Modify on_kcp_message to handle responses directly instead of returning them
- Add handle_kcp_response method that wraps response in KCP and sends via mixnet
- Ensures KCP clients receive KCP-wrapped responses for proper reassembly

Closes: nym-7oh2

* fix(ipr): send KCP protocol packets in tick instead of just logging

- Add get_sender_tag() and fetch_outgoing_for_conv() to KcpSessionManager
- Change handle_kcp_tick() to actually send ACKs/retransmissions via mixnet
- Reduce KCP tick interval from 100ms to 10ms for better responsiveness

This fixes the KCP reliability protocol which was broken because
protocol packets (ACKs, retransmissions) were generated but never sent.

* feat(lp-client): wrap payload in IpPacketRequest before KCP

- Add nym-ip-packet-requests and bytes dependencies
- Wrap payload in IpPacketRequest::new_data_request() before sending to KCP
- Add LP_DATA_PORT constant (51264) and lp_data_address field to GatewayInfo

This ensures IPR can properly parse incoming messages as DataRequest.
LP framing (wrapping Sphinx in LP before sending) is a separate task.

* feat(lp-client): add LP session management and UDP data plane support

- Add wrap_data() and session_id() to LpRegistrationClient for LP packet
  creation after handshake
- Add init_lp_session() and close_lp_session() to SpeedtestClient for
  managing LP sessions
- Extract prepare_sphinx_fragments() helper to reduce code duplication
  between send_data_with_surbs() and send_data_via_lp()
- Add send_data_via_lp() for sending Sphinx packets through LP's UDP
  data plane (port 51264)

The LP session is kept alive after TCP handshake closes, allowing
subsequent wrap_data() calls for UDP transmission without re-handshaking.

* random formatting

* replaced all instances of bincode::serialize and bincode::deserialize with explicit lp_bincode_serialiser() within the LP

* additional formatting

* removed source of possible panic from nym-kkt

invalid KEM mapping will now return an Err rather than panicking

* integration test for LP entry registration

This includes creation of mocks of various gateway-related components, such as the PeerController

* changed ClientHelloData serialisation

the old variant using bincode did not produce constant-length output in some cases

* Fixed generation of receiver index

removes the possible clash with the boostrap id

* Integration test for nested LP registration

- move `LpTransport` trait definition to shared `nym-lp-transport` crate
- make transport layer within `LpConnectionHandler` generic with respect to the forwarding target. it must, however, use the same type as the incoming client connection
- extracted explicit `LpConnectionHandler::establish_exit_stream` to more easily modify it in the future to fully protect the channel and disallow using untrusted egress points
- fix additional log-string interpolation nits

* resolved clippy issues pointed out by clippy 1.91

* added LP discovery into self-described endpoint:

- removed changes to the node bonding within the contract
- introduced '/api/v1/lewes-protocol' route within nym-node http api
- added 'lewes_protocol' field to 'NymNodeData' inside of NymNodeDescription
- refactored LpConfig to allow separate bind and announce addresses and used more strict typing

* chore: allow unwrap/expect within kkt benchmarking code

* chore: downgraded sha2 dep for cosmwasm compatibility

* clippy

* marking simd calls as unsafe

* fixed calls to '_mm_testz_si128'

* additional clippy fixes

---------

Co-authored-by: Georgio Nicolas <me@georgio.xyz>
Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>
2026-01-14 09:06:02 +00:00

38 KiB

Lewes Protocol (LP) - Technical Specification

Overview

The Lewes Protocol (LP) is a direct TCP-based registration protocol for Nym gateways. It provides an alternative to mixnet-based registration with different trade-offs: lower latency at the cost of revealing client IP to the gateway.

Design Goals:

  • Low latency: Direct TCP connection vs multi-hop mixnet routing
  • High reliability: KCP protocol provides ordered, reliable delivery with ARQ
  • Strong security: Noise XKpsk3 provides mutual authentication and forward secrecy
  • Replay protection: Bitmap-based counter validation prevents replay attacks
  • Observability: Prometheus metrics for production monitoring

Non-Goals:

  • Network-level anonymity (use mixnet registration for that)
  • Persistent connections (LP is registration-only, single-use)
  • Backward compatibility with legacy protocols

Architecture

Protocol Stack

┌─────────────────────────────────────────┐
│  Application Layer                      │
│  - Registration Requests                │
│  - E-cash Credential Verification       │
│  - WireGuard Peer Management            │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│  LP Layer (Lewes Protocol)              │
│  - Noise XKpsk3 Handshake               │
│  - Replay Protection (1024-pkt window)  │
│  - Counter-based Sequencing             │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│  KCP Layer (Reliability)                │
│  - Ordered Delivery                     │
│  - ARQ with Selective ACK               │
│  - Congestion Control                   │
│  - RTT Estimation                       │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│  TCP Layer                              │
│  - Connection Establishment             │
│  - Byte Stream Delivery                 │
└─────────────────────────────────────────┘

Why This Layering?

TCP: Provides connection-oriented byte stream and handles network-level retransmission.

KCP: Adds application-level reliability optimized for low latency:

  • Fast retransmit: Triggered after 2 duplicate ACKs (vs TCP's 3)
  • Selective ACK: Acknowledges specific packets, not just cumulative
  • Configurable RTO: Minimum RTO of 100ms (configurable)
  • No Nagle: Immediate sending for low-latency applications

LP: Provides cryptographic security and session management:

  • Noise XKpsk3: Mutual authentication with pre-shared key
  • Replay protection: Prevents duplicate packet acceptance
  • Session isolation: Each session has unique cryptographic state

Application: Business logic for registration and credential verification.

Protocol Flow

1. Connection Establishment

Client                                    Gateway
  |                                          |
  |--- TCP SYN --------------------------->  |
  |<-- TCP SYN-ACK ------------------------  |
  |--- TCP ACK ----------------------------> |
  |                                          |
  • Control Port: 41264 (default, configurable)
  • Data Port: 51264 (reserved for future use, not currently used)

2. Session Initialization

Client generates session parameters:

// Client-side session setup
let client_lp_keypair = Keypair::generate(); // X25519 keypair
let gateway_lp_public = gateway.lp_public_key; // From gateway descriptor
let salt = [timestamp (8 bytes) || nonce (24 bytes)]; // 32-byte salt

// Derive PSK using ECDH + Blake3 KDF
let shared_secret = ECDH(client_private, gateway_public);
let psk = Blake3_derive_key(
    context = "nym-lp-psk-v1",
    input = shared_secret,
    salt = salt
);

// Calculate session IDs (deterministic from keys)
let lp_id = hash(client_lp_public || 0xCC || gateway_lp_public) & 0xFFFFFFFF;
let kcp_conv_id = hash(client_lp_public || 0xFF || gateway_lp_public) & 0xFFFFFFFF;

Session ID Properties:

  • Deterministic: Same key pair always produces same ID
  • Order-independent: ID(A, B) == ID(B, A) due to sorted hashing
  • Collision-resistant: Uses full hash, truncated to u32
  • Unique per protocol: Different delimiters (0xCC for LP, 0xFF for KCP)

3. Noise Handshake (XKpsk3 Pattern)

Client (Initiator)                        Gateway (Responder)
  |                                          |
  |--- e ----------------------------------> | [1] Client ephemeral
  |                                          |
  |<-- e, ee, s, es ---------------------  | [2] Gateway ephemeral + static
  |                                          |
  |--- s, se, psk ------------------------->  | [3] Client static + PSK mix
  |                                          |
  [Transport mode established]

Message Contents:

[1] Initiator → Responder: e

  • Payload: Client ephemeral public key (32 bytes)
  • Encrypted: No (initial message)

[2] Responder → Initiator: e, ee, s, es

  • e: Responder ephemeral public key
  • ee: Mix ephemeral-ephemeral DH
  • s: Responder static public key (encrypted)
  • es: Mix ephemeral-static DH
  • Encrypted: Yes (with keys from ee)

[3] Initiator → Responder: s, se, psk

  • s: Initiator static public key (encrypted)
  • se: Mix static-ephemeral DH
  • psk: Mix pre-shared key (at position 3)
  • Encrypted: Yes (with keys from ee, es)

Security Properties:

  • Mutual authentication: Both sides prove identity via static keys
  • Forward secrecy: Ephemeral keys provide PFS
  • PSK authentication: Binds session to out-of-band PSK
  • Identity hiding: Static keys encrypted after first message

Handshake Characteristics:

  • Messages: 3 (1.5 round trips)
  • Minimum network RTTs: 1.5
  • Cryptographic operations: ECDH, ChaCha20-Poly1305, SHA-256

4. PSK Derivation Details

Formula:

shared_secret = X25519(client_private_lp, gateway_public_lp)
psk = Blake3_derive_key(
    context = "nym-lp-psk-v1",
    key_material = shared_secret (32 bytes),
    salt = timestamp || nonce (32 bytes)
)

Implementation (from common/nym-lp/src/psk.rs:48):

pub fn derive_psk(
    local_private: &PrivateKey,
    remote_public: &PublicKey,
    salt: &[u8; 32],
) -> [u8; 32] {
    let shared_secret = local_private.diffie_hellman(remote_public);
    nym_crypto::kdf::derive_key_blake3(PSK_CONTEXT, shared_secret.as_bytes(), salt)
}

Why This Design:

  1. Identity-bound: PSK tied to LP keypairs, not ephemeral

    • Prevents MITM without LP private key
    • Links session to long-term identities
  2. Session-specific via salt: Different registrations use different PSKs

    • timestamp: 8-byte Unix timestamp (milliseconds)
    • nonce: 24-byte random value
    • Prevents PSK reuse across sessions
  3. Symmetric derivation: Both sides derive same PSK

    • Client: ECDH(client_priv, gateway_pub)
    • Gateway: ECDH(gateway_priv, client_pub)
    • Mathematical property: ECDH(a, B) == ECDH(b, A)
  4. Blake3 KDF with domain separation:

    • Context string prevents cross-protocol attacks
    • Generates uniform 32-byte output suitable for Noise

Salt Transmission:

  • Included in ClientHello message (cleartext)
  • Gateway extracts salt before deriving PSK
  • Timestamp validation rejects stale salts

5. Replay Protection

Mechanism: Sliding Window with Bitmap (from common/nym-lp/src/replay/validator.rs:32):

const WORD_SIZE: usize = 64;
const N_WORDS: usize = 16;  // 1024 bits total
const N_BITS: usize = WORD_SIZE * N_WORDS;  // 1024

pub struct ReceivingKeyCounterValidator {
    next: u64,              // Next expected counter
    receive_cnt: u64,       // Total packets received
    bitmap: [u64; 16],      // 1024-bit bitmap
}

Algorithm:

For each incoming packet with counter C:
  1. Quick check (branchless):
     - If C >= next: Accept (growing)
     - If C + 1024 < next: Reject (too old, outside window)
     - If bitmap[C % 1024] is set: Reject (duplicate)
     - Else: Accept (out-of-order within window)

  2. After successful processing, mark:
     - Set bitmap[C % 1024] = 1
     - If C >= next: Update next = C + 1
     - Increment receive_cnt

Performance Optimizations:

  1. SIMD-accelerated bitmap operations (from common/nym-lp/src/replay/simd/):

    • AVX2 support (x86_64)
    • SSE2 support (x86_64)
    • NEON support (ARM)
    • Scalar fallback (portable)
  2. Branchless execution (constant-time):

    // No early returns - prevents timing attacks
    let result = if is_growing {
        Some(Ok(()))
    } else if too_far_back {
        Some(Err(ReplayError::OutOfWindow))
    } else if duplicate {
        Some(Err(ReplayError::DuplicateCounter))
    } else {
        Some(Ok(()))
    };
    result.unwrap()
    
  3. Overflow-safe arithmetic:

    let too_far_back = if counter > u64::MAX - 1024 {
        false  // Can't overflow, so not too far back
    } else {
        counter + 1024 < self.next
    };
    

Memory Usage (verified from common/nym-lp/src/replay/validator.rs:738):

// test_memory_usage()
size = size_of::<u64>() * 2 +        // next + receive_cnt = 16 bytes
       size_of::<u64>() * N_WORDS;   // bitmap = 128 bytes
// Total: 144 bytes

6. Registration Request

After handshake completes, client sends encrypted registration request:

pub struct RegistrationRequest {
    pub mode: RegistrationMode,
    pub credential: EcashCredential,
    pub gateway_identity: String,
}

pub enum RegistrationMode {
    Dvpn {
        wg_public_key: [u8; 32],
    },
    Mixnet {
        client_id: String,
        mix_address: Option<String>,
    },
}

Encryption:

  • Encrypted using Noise transport mode
  • Includes 16-byte Poly1305 authentication tag
  • Replay protection via LP counter

7. Credential Verification

Gateway verifies the e-cash credential:

// Gateway-side verification
pub async fn verify_credential(
    &self,
    credential: &EcashCredential,
) -> Result<VerifiedCredential, CredentialError> {
    // 1. Check credential signature (BLS12-381)
    verify_blinded_signature(&credential.signature)?;

    // 2. Check credential not already spent (nullifier check)
    if self.storage.is_nullifier_spent(&credential.nullifier).await? {
        return Err(CredentialError::AlreadySpent);
    }

    // 3. Extract bandwidth allocation
    let bandwidth_bytes = credential.bandwidth_value;

    // 4. Mark nullifier as spent
    self.storage.mark_nullifier_spent(&credential.nullifier).await?;

    Ok(VerifiedCredential {
        bandwidth_bytes,
        expiry: credential.expiry,
    })
}

For dVPN Mode:

let peer_config = WireguardPeerConfig {
    public_key: request.wg_public_key,
    allowed_ips: vec!["10.0.0.0/8"],
    bandwidth_limit: verified.bandwidth_bytes,
};
self.wg_controller.add_peer(peer_config).await?;

8. Registration Response

pub enum RegistrationResponse {
    Success {
        bandwidth_allocated: u64,
        expiry: u64,
        gateway_data: GatewayData,
    },
    Error {
        code: ErrorCode,
        message: String,
    },
}

pub enum ErrorCode {
    InvalidCredential = 1,
    CredentialExpired = 2,
    CredentialAlreadyUsed = 3,
    InsufficientBandwidth = 4,
    WireguardPeerRegistrationFailed = 5,
    InternalError = 99,
}

State Machine and Security Protocol

Protocol Components

The Lewes Protocol combines three cryptographic protocols for secure, post-quantum resistant communication:

  1. KKT (KEM Key Transfer) - Dynamically fetches responder's KEM public key with Ed25519 authentication
  2. PSQ (Post-Quantum Secure PSK) - Derives PSK using KEM-based protocol for HNDL resistance
  3. Noise XKpsk3 - Provides encrypted transport with mutual authentication and forward secrecy

State Machine

The LP state machine orchestrates the complete protocol flow from connection to encrypted transport:

┌─────────────────────────────────────────────────────────────────────┐
│                    LEWES PROTOCOL STATE MACHINE                     │
└─────────────────────────────────────────────────────────────────────┘

                    ┌──────────────────┐
                    │ ReadyToHandshake │
                    │                  │
                    │ • Keys loaded    │
                    │ • Session ID set │
                    └────────┬─────────┘
                             │
                    StartHandshake input
                             │
                             ▼
         ┌───────────────────────────────────────┐
         │          KKTExchange                  │
         │                                       │
         │  Initiator:                          │
         │  1. Send KKT request (signed)        │
         │  2. Receive KKT response             │
         │  3. Validate Ed25519 signature       │
         │  4. Extract KEM public key           │
         │                                       │
         │  Responder:                          │
         │  1. Wait for KKT request             │
         │  2. Validate signature               │
         │  3. Send signed KEM key              │
         └───────────────┬───────────────────────┘
                         │
                  KKT Complete
                         │
                         ▼
         ┌───────────────────────────────────────┐
         │           Handshaking                 │
         │                                       │
         │  PSQ Protocol:                       │
         │  1. Initiator encapsulates PSK       │
         │     (embedded in Noise msg 1)        │
         │  2. Responder decapsulates PSK       │
         │     (sends ctxt_B in Noise msg 2)    │
         │  3. Both derive final PSK:           │
         │     KDF(ECDH || KEM_shared)          │
         │                                       │
         │  Noise XKpsk3 Handshake:             │
         │  → msg 1: e, es, ss + PSQ payload    │
         │  ← msg 2: e, ee, se + ctxt_B         │
         │  → msg 3: s, se (handshake complete) │
         └───────────────┬───────────────────────┘
                         │
                  Handshake Complete
                         │
                         ▼
         ┌───────────────────────────────────────┐
         │            Transport                  │
         │                                       │
         │  • Encrypted data transfer            │
         │  • AEAD with ChaCha20-Poly1305       │
         │  • Replay protection (counters)       │
         │  • Bidirectional communication        │
         └───────────────┬───────────────────────┘
                         │
                    Close input
                         │
                         ▼
                   ┌──────────┐
                   │  Closed  │
                   │          │
                   │ • Reason │
                   └──────────┘

Message Sequence

Complete protocol flow from connection to encrypted transport:

Initiator                                                    Responder
    │                                                            │
    │ ════════════════ KKT EXCHANGE ════════════════            │
    │                                                            │
    │  KKTRequest (signed with Ed25519)                         │
    ├──────────────────────────────────────────────────────────>│
    │                                                            │ Validate
    │                                                            │ signature
    │                        KKTResponse (signed KEM key + hash) │
    │<──────────────────────────────────────────────────────────┤
    │                                                            │
    │ Validate signature                                         │
    │ Extract kem_pk                                             │
    │                                                            │
    │ ══════════════ PSQ + NOISE HANDSHAKE ══════════════       │
    │                                                            │
    │  Noise msg 1: e, es, ss                                   │
    │  + PSQ InitiatorMsg (KEM encapsulation)                   │
    ├──────────────────────────────────────────────────────────>│
    │                                                            │
    │                                                            │ PSQ: Decapsulate
    │                                                            │ Derive PSK
    │                                                            │ Inject into Noise
    │                                  Noise msg 2: e, ee, se    │
    │                                  + ctxt_B (encrypted PSK)  │
    │<──────────────────────────────────────────────────────────┤
    │                                                            │
    │ Extract ctxt_B                                             │
    │ Store for re-registration                                 │
    │ Inject PSK into Noise                                      │
    │                                                            │
    │  Noise msg 3: s, se                                       │
    ├──────────────────────────────────────────────────────────>│
    │                                                            │
    │ Handshake Complete ✓                                       │ Handshake Complete ✓
    │ Transport mode active                                      │ Transport mode active
    │                                                            │
    │ ═══════════════ TRANSPORT MODE ═══════════════            │
    │                                                            │
    │  EncryptedData (AEAD, counter N)                          │
    ├──────────────────────────────────────────────────────────>│
    │                                                            │
    │                                  EncryptedData (counter M) │
    │<──────────────────────────────────────────────────────────┤
    │                                                            │
    │  (bidirectional encrypted communication)                  │
    │◄──────────────────────────────────────────────────────────►
    │                                                            │

KKT (KEM Key Transfer) Protocol

Purpose: Securely obtain responder's KEM public key before PSQ can begin.

Key Features:

  • Ed25519 signatures for authentication (both request and response signed)
  • Optional hash validation for key pinning (future directory service integration)
  • Currently signature-only mode (deployable without infrastructure)
  • Easy upgrade path to hash-based key pinning

Initiator Flow:

1. Generate KKT request with Ed25519 signature
2. Send KKTRequest to responder
3. Receive KKTResponse with signed KEM key
4. Validate Ed25519 signature
5. (Optional) Validate key hash against directory
6. Store KEM key for PSQ encapsulation

Responder Flow:

1. Receive KKTRequest from initiator
2. Validate initiator's Ed25519 signature
3. Generate KKTResponse with:
   - Responder's KEM public key
   - Ed25519 signature over (key || timestamp)
   - Blake3 hash of KEM key
4. Send KKTResponse to initiator

PSQ (Post-Quantum Secure PSK) Protocol

Purpose: Derive a post-quantum secure PSK for Noise protocol.

Security Properties:

  • HNDL resistance: PSK derived from KEM-based protocol
  • Forward secrecy: Ephemeral KEM keypair per session
  • Authentication: Ed25519 signatures prevent MitM
  • Algorithm agility: Easy upgrade from X25519 to ML-KEM

PSK Derivation:

Classical ECDH:
  ecdh_secret = X25519_DH(local_private, remote_public)

KEM Encapsulation (Initiator):
  (kem_shared_secret, ciphertext) = KEM.Encap(responder_kem_pk)

KEM Decapsulation (Responder):
  kem_shared_secret = KEM.Decap(kem_private, ciphertext)

Final PSK:
  combined = ecdh_secret || kem_shared_secret || salt
  psk = Blake3_KDF("nym-lp-psk-psq-v1", combined)

Integration with Noise:

  • PSQ payload embedded in first Noise message (no extra round-trip)
  • Responder sends encrypted PSK handle (ctxt_B) in second Noise message
  • Both sides inject derived PSK before completing Noise handshake
  • Noise validates PSK correctness during handshake

PSK Handle (ctxt_B): The responder's encrypted PSK handle allows future re-registration without repeating PSQ:

  • Encrypted with responder's long-term key
  • Can be presented in future sessions
  • Enables fast re-registration for returning clients

Security Guarantees

Achieved Properties:

  • Mutual authentication: Ed25519 signatures in KKT and PSQ
  • Forward secrecy: Ephemeral keys in Noise handshake
  • Post-quantum PSK: KEM-based PSK derivation
  • HNDL resistance: PSK safe even if private keys compromised later
  • Replay protection: Monotonic counters with sliding window
  • Key confirmation: Noise handshake validates PSK correctness

Implementation Status:

  • 🔄 Key pinning: Hash validation via directory service (signature-only for now)
  • 🔄 ML-KEM support: Easy config upgrade from X25519 to ML-KEM-768
  • 🔄 PSK re-use: ctxt_B handle stored for future re-registration

Algorithm Choices

Current (Testing/Development):

  • KEM: X25519 (DHKEM) - Classical ECDH, widely tested
  • Hash: Blake3 - Fast, secure, parallel
  • Signature: Ed25519 - Fast verification, compact
  • AEAD: ChaCha20-Poly1305 - Fast, constant-time

Future (Production):

  • KEM: ML-KEM-768 - NIST-approved post-quantum KEM
  • Hash: Blake3 - No change needed
  • Signature: Ed25519 - No change needed (or upgrade to ML-DSA)
  • AEAD: ChaCha20-Poly1305 - No change needed

Migration Path:

# Current deployment
[lp.crypto]
kem_algorithm = "x25519"

# Future upgrade (config change only)
[lp.crypto]
kem_algorithm = "ml-kem-768"

Message Types

KKT Messages:

// Message Type 0x0004
struct KKTRequest {
    timestamp: u64,              // Unix timestamp (replay protection)
    initiator_ed25519_pk: [u8; 32], // Initiator's public key
    signature: [u8; 64],         // Ed25519 signature
}

// Message Type 0x0005
struct KKTResponse {
    kem_pk: Vec<u8>,             // Responder's KEM public key
    key_hash: [u8; 32],          // Blake3 hash of KEM key
    timestamp: u64,              // Unix timestamp
    signature: [u8; 64],         // Ed25519 signature
}

PSQ Embedding:

  • PSQ InitiatorMsg embedded in Noise message 1 payload (after 'e, es, ss')
  • PSQ ResponderMsg (ctxt_B) embedded in Noise message 2 payload (after 'e, ee, se')
  • No additional round-trips beyond standard 3-message Noise handshake

KCP Protocol Details

KCP Configuration

From common/nym-kcp/src/session.rs:

pub struct KcpSession {
    conv: u32,           // Conversation ID
    mtu: usize,          // Default: 1400 bytes
    snd_wnd: u16,        // Send window: 128 segments
    rcv_wnd: u16,        // Receive window: 128 segments
    rx_minrto: u32,      // Minimum RTO: 100ms (configurable)
}

KCP Packet Format

┌────────────────────────────────────────────────┐
│ Conv ID (4 bytes) - Conversation identifier    │
├────────────────────────────────────────────────┤
│ Cmd (1 byte) - PSH/ACK/WND/ERR                │
├────────────────────────────────────────────────┤
│ Frg (1 byte) - Fragment number (reverse order) │
├────────────────────────────────────────────────┤
│ Wnd (2 bytes) - Receive window size           │
├────────────────────────────────────────────────┤
│ Timestamp (4 bytes) - Send timestamp           │
├────────────────────────────────────────────────┤
│ Sequence Number (4 bytes) - Packet sequence    │
├────────────────────────────────────────────────┤
│ UNA (4 bytes) - Unacknowledged sequence       │
├────────────────────────────────────────────────┤
│ Length (4 bytes) - Data length                 │
├────────────────────────────────────────────────┤
│ Data (variable) - Payload                      │
└────────────────────────────────────────────────┘

Total header: 24 bytes

KCP Features

Reliability Mechanisms:

  • Sequence Numbers (sn): Track packet ordering
  • Fragment Numbers (frg): Handle message fragmentation
  • UNA (Unacknowledged): Cumulative ACK up to this sequence
  • Selective ACK: Via individual ACK packets
  • Fast Retransmit: Triggered by duplicate ACKs (configurable threshold)
  • RTO Calculation: Smoothed RTT with variance

LP Packet Format

LP Header

┌────────────────────────────────────────────────┐
│ Protocol Version (1 byte) - Currently: 1       │
├────────────────────────────────────────────────┤
│ Session ID (4 bytes) - LP session identifier   │
├────────────────────────────────────────────────┤
│ Counter (8 bytes) - Replay protection counter  │
└────────────────────────────────────────────────┘

Total header: 13 bytes

LP Message Types

pub enum LpMessage {
    Handshake(Vec<u8>),
    EncryptedData(Vec<u8>),
    ClientHello {
        client_lp_public: [u8; 32],
        salt: [u8; 32],
        timestamp: u64,
    },
    Busy,
}

Complete Packet Structure

┌─────────────────────────────────────┐
│  LP Header (13 bytes)               │
│  - Version, Session ID, Counter     │
├─────────────────────────────────────┤
│  LP Message (variable)              │
│  - Type tag (1 byte)                │
│  - Message data                     │
├─────────────────────────────────────┤
│  Trailer (16 bytes)                 │
│  - Reserved for future MAC/tag      │
└─────────────────────────────────────┘

Security Properties

Threat Model

Protected Against:

  • Passive eavesdropping: Noise encryption (ChaCha20-Poly1305)
  • Active MITM: Mutual authentication via static keys + PSK
  • Replay attacks: Counter-based validation with 1024-packet window
  • Packet injection: Poly1305 authentication tags
  • Timestamp replay: 30-second window for ClientHello timestamps (configurable)
  • DoS (connection flood): Connection limit (default: 10,000, configurable)
  • Credential reuse: Nullifier tracking in database

Not Protected Against:

  • Network-level traffic analysis: LP is not anonymous (use mixnet for that)
  • Gateway compromise: Gateway sees client registration data
  • ⚠️ Per-IP DoS: No per-IP rate limiting (global limit only)

Cryptographic Primitives

Component Algorithm Key Size Source
Key Exchange X25519 256 bits RustCrypto
Encryption ChaCha20 256 bits RustCrypto
Authentication Poly1305 256 bits RustCrypto
KDF Blake3 256 bits nym_crypto
Hash (Noise) SHA-256 256 bits snow crate
Signature (E-cash) BLS12-381 381 bits E-cash contract

Forward Secrecy

Noise XKpsk3 provides forward secrecy through ephemeral keys:

  1. Initial handshake: Uses ephemeral + static keys

  2. Key compromise scenario:

    • Compromise of static key: Past sessions remain secure (ephemeral keys destroyed)
    • Compromise of PSK: Attacker needs static key too (two-factor security)
    • Compromise of both: Only future sessions affected, not past
  3. Session key lifetime: Destroyed after single registration completes

Timing Attack Resistance

Constant-time operations:

  • Replay protection check (branchless)
  • Bitmap bit operations (branchless)
  • Noise crypto operations (via snow/RustCrypto)

Variable-time operations:

  • ⚠️ Credential verification (database lookup time varies)
  • ⚠️ WireGuard peer registration (filesystem operations)

Configuration

Gateway Configuration

From gateway/src/node/lp_listener/mod.rs:78:

[lp]
# Enable/disable LP listener
enabled = true

# Bind address
bind_address = "0.0.0.0"

# Control port (for LP handshake and registration)
control_port = 41264

# Data port (reserved for future use)
data_port = 51264

# Maximum concurrent connections
max_connections = 10000

# Timestamp validation window (seconds)
# ClientHello messages older than this are rejected
timestamp_tolerance_secs = 30

# Use mock e-cash verifier (TESTING ONLY!)
use_mock_ecash = false

Firewall Rules

Required inbound rules:

# Allow TCP connections to LP control port
iptables -A INPUT -p tcp --dport 41264 -j ACCEPT

# Optional: Rate limiting
iptables -A INPUT -p tcp --dport 41264 -m state --state NEW \
    -m recent --set --name LP_LIMIT
iptables -A INPUT -p tcp --dport 41264 -m state --state NEW \
    -m recent --update --seconds 60 --hitcount 100 --name LP_LIMIT \
    -j DROP

Metrics

From gateway/src/node/lp_listener/mod.rs:4:

Connection Metrics:

  • active_lp_connections: Gauge tracking current active LP connections
  • lp_connections_total: Counter for total LP connections handled
  • lp_connection_duration_seconds: Histogram of connection durations
  • lp_connections_completed_gracefully: Counter for successful completions
  • lp_connections_completed_with_error: Counter for error terminations

Handshake Metrics:

  • lp_handshakes_success: Counter for successful handshakes
  • lp_handshakes_failed: Counter for failed handshakes
  • lp_handshake_duration_seconds: Histogram of handshake durations
  • lp_client_hello_failed: Counter for ClientHello failures

Registration Metrics:

  • lp_registration_attempts_total: Counter for all registration attempts
  • lp_registration_success_total: Counter for successful registrations
  • lp_registration_failed_total: Counter for failed registrations
  • lp_registration_duration_seconds: Histogram of registration durations

Mode-Specific:

  • lp_registration_dvpn_attempts/success/failed: dVPN mode counters
  • lp_registration_mixnet_attempts/success/failed: Mixnet mode counters

Credential Metrics:

  • lp_credential_verification_attempts/success/failed: Verification counters
  • lp_bandwidth_allocated_bytes_total: Total bandwidth allocated

Error Metrics:

  • lp_errors_handshake: Handshake errors
  • lp_errors_timestamp_too_old/too_far_future: Timestamp validation errors
  • lp_errors_wg_peer_registration: WireGuard peer registration failures

Error Codes

Handshake Errors

Error Description
NOISE_DECRYPT_ERROR Invalid ciphertext or wrong keys
NOISE_PROTOCOL_ERROR Unexpected message or state
REPLAY_DUPLICATE Counter already seen
REPLAY_OUT_OF_WINDOW Counter outside 1024-packet window
TIMESTAMP_TOO_OLD ClientHello > configured tolerance
TIMESTAMP_FUTURE ClientHello from future

Registration Errors

Code Name Description
CREDENTIAL_INVALID Invalid credential Signature verification failed
CREDENTIAL_EXPIRED Credential expired Past expiry timestamp
CREDENTIAL_SPENT Already used Nullifier already in database
INSUFFICIENT_BANDWIDTH Not enough bandwidth Requested > credential value
WIREGUARD_FAILED Peer registration failed Kernel error adding WireGuard peer

Limitations

Current Limitations

  1. No persistent sessions: Each registration is independent
  2. Single registration per session: Connection closes after registration
  3. No streaming: Protocol is request-response only
  4. No gateway discovery: Client must know gateway's LP public key beforehand
  5. No version negotiation: Protocol version fixed at 1
  6. No per-IP rate limiting: Only global connection limit

Testing Gaps

  1. No end-to-end integration tests: Unit tests exist, integration tests pending
  2. No performance benchmarks: Latency/throughput not measured
  3. No load testing: Concurrent connection limits not stress-tested
  4. No security audit: Cryptographic implementation not externally reviewed

References

Specifications

Implementations

  • snow: Rust Noise protocol implementation
  • RustCrypto: Cryptographic primitives (ChaCha20-Poly1305, X25519)
  • tokio: Async runtime for network I/O

Security Audits

  • Noise implementation audit (pending)
  • Replay protection audit (pending)
  • E-cash integration audit (pending)
  • Penetration testing (pending)

Changelog

Version 1.1 (Post-Quantum PSK with KKT)

Implemented:

  • KKTExchange state in state machine for pre-handshake KEM key transfer
  • PSQ (Post-Quantum Secure PSK) protocol integration
  • KKT (KEM Key Transfer) protocol with Ed25519 authentication
  • Optional hash validation for KEM key pinning (signature-only mode active)
  • PSK handle (ctxt_B) storage for future re-registration
  • X25519 DHKEM support (ready for ML-KEM upgrade)
  • Comprehensive state machine tests (7 test cases)
  • generate_fresh_salt() utility for session creation

Security Improvements:

  • Post-quantum PSK derivation (KEM-based)
  • HNDL (Harvest Now, Decrypt Later) resistance
  • Mutual authentication via Ed25519 signatures
  • Easy migration path to ML-KEM-768

Architecture:

  • State flow: ReadyToHandshake → KKTExchange → Handshaking → Transport
  • PSQ embedded in Noise handshake (no extra round-trip)
  • Automatic KKT on StartHandshake (no manual key distribution)

Related Issues:

  • nym-4za: Add KKTExchange state to LpStateMachine

Version 1.0 (Initial Implementation)

Implemented:

  • Noise XKpsk3 handshake
  • KCP reliability layer
  • Replay protection (1024-packet window with SIMD)
  • PSK derivation (ECDH + Blake3)
  • dVPN and Mixnet registration modes
  • E-cash credential verification
  • WireGuard peer management
  • Prometheus metrics
  • DoS protection (connection limits, timestamp validation)

Pending:

  • End-to-end integration tests
  • Performance benchmarks
  • Security audit
  • Client implementation
  • Gateway probe support
  • Per-IP rate limiting