Files
nym/common/nonexhaustive-delayqueue/src/lib.rs
T
Jędrzej Stuczyński 8a93bce32f feat: additional mixnet improvements and metrics (#6874)
* wip

* batch processing of forward packets

* tmp: additional metrics for remote node

* fixed incorrect prometheus metric registration

* unified runtime metrics

* unify mixnet client metrics

* packet forwarding cleanup

* add batching for emptying the delay queue

* cleanup client io loop

* feat(nym-node): reap idle mixnet connections (ingress + egress)

Close mixnet connections that sit with no traffic past a configurable idle period (mixnet.debug.connection_idle_timeout, default 5min, 0 disables) to bound lingering tokio tasks/sockets.

Ingress handle_stream is read-only, so a silently-gone peer (NAT drop, crash without FIN, half-open) never triggers FIN/RST and the task would block on .next() forever; a new idle select arm closes it (the post-loop replay flush still runs, so nothing is stranded). Egress run_io_loop gets the symmetric arm keyed on last_send; on close EvictOnDrop clears the cache entry and the next packet transparently reconnects.

Adds a cumulative nym_node_network_idle_closed_ingress_mixnet_connections counter; egress reaping is observed via the existing active-egress gauge plus an exit_reason=idle_timeout log.

* downgrade sysinfo

* refactor(nym-node): split PacketForwarder into router + delay-queue tasks

Split the single PacketForwarder task into two concurrently-scheduled tasks connected by a bounded handoff channel, so intake and delayed-release no longer block each other.

PacketRouter (router.rs) is the intake task: sole consumer of the ingress channel, it applies the routing filter and either forwards zero/already-elapsed-delay packets directly or hands delayed ones to the delay task. Its per-packet work is sub-µs, so new packets no longer wait behind delayed-release processing (collapses the ForwarderQueue tail).

DelayForwarder (delay.rs) owns the NonExhaustiveDelayQueue exclusively (it can't be shared by reference). Its run loop services BOTH branches on every wakeup - draining pending inserts first to bring the queue current, then flushing everything now due - so the biased select can't let releases and inserts starve each other, and a freshly-arrived-but-already-due packet releases in the same pass (marginally improving DelayQueueOverrun).

The mixnet client is shared as Arc<C>; handoff-channel overflow is dropped as an egress drop rather than blocking, keeping intake decoupled from release.

* feat(nym-node): bound egress flush with a write timeout

Cap how long a single egress batch flush may block on a congested peer socket (mixnet.debug.connection_write_timeout, default 500ms, 0 disables), so a slow peer can no longer back this connection's egress queue up into the multi-second range - the root of the EgressQueue and SocketWrite tails.

A single timeout is treated as transient congestion: the un-fed tail of the batch is abandoned but the connection is retained. This is sound because NoiseStream::poll_write encrypts and buffers each frame synchronously, so a cancelled flush leaves the noise transport nonce-consistent and a later flush resumes the byte stream in order - so a momentary spike costs no re-handshake. Only MAX_CONSECUTIVE_WRITE_TIMEOUTS (3) timeouts in a row, i.e. a persistently congested peer, tears the connection down (it reconnects on the next packet); a successful flush resets the counter.

Buffer-size tuning (maximum_connection_buffer_size) deliberately left for live data.

* revert PacketForwarder split in favour of a single task that clears both channels on wake
2026-06-12 10:31:54 +01:00

127 lines
4.4 KiB
Rust

// Copyright 2021 - Nym Technologies SA <contact@nymtech.net>
// SPDX-License-Identifier: Apache-2.0
use std::pin::Pin;
use std::task::{Context, Poll, Waker};
use std::time::Duration;
use tokio_stream::Stream;
// this is a copy of tokio-util delay_queue with `Sleep` and `Instant` being replaced with
// `wasm_timer` equivalents
#[cfg(not(target_arch = "wasm32"))]
type DelayQueue<T> = tokio_util::time::DelayQueue<T>;
#[cfg(not(target_arch = "wasm32"))]
pub use tokio_util::time::delay_queue::Expired;
#[cfg(not(target_arch = "wasm32"))]
pub type QueueKey = tokio_util::time::delay_queue::Key;
#[cfg(not(target_arch = "wasm32"))]
use tokio::time::Instant;
#[cfg(target_arch = "wasm32")]
type DelayQueue<T> = wasmtimer::tokio_util::DelayQueue<T>;
#[cfg(target_arch = "wasm32")]
pub use wasmtimer::tokio_util::delay_queue::Expired;
#[cfg(target_arch = "wasm32")]
pub type QueueKey = wasmtimer::tokio_util::delay_queue::Key;
#[cfg(target_arch = "wasm32")]
use wasmtimer::std::Instant;
/// A variant of tokio's `DelayQueue`, such that its `Stream` implementation will never return a 'None'.
pub struct NonExhaustiveDelayQueue<T> {
inner: DelayQueue<T>,
waker: Option<Waker>,
}
// more methods of underlying DelayQueue will get exposed as we need them
impl<T> NonExhaustiveDelayQueue<T> {
pub fn new() -> Self {
NonExhaustiveDelayQueue {
inner: DelayQueue::new(),
waker: None,
}
}
pub fn insert(&mut self, value: T, timeout: Duration) -> QueueKey {
let key = self.inner.insert(value, timeout);
if let Some(waker) = self.waker.take() {
// we were waiting for an item - wake the executor!
waker.wake()
}
key
}
pub fn insert_at(&mut self, value: T, when: Instant) -> QueueKey {
let key = self.inner.insert_at(value, when);
if let Some(waker) = self.waker.take() {
// we were waiting for an item - wake the executor!
waker.wake()
}
key
}
// TODO: it seems like this one can cause panic in very rare edge cases, however,
// I can't seem to be able to reproduce it at all.
pub fn remove(&mut self, key: &QueueKey) -> Expired<T> {
self.inner.remove(key)
}
pub fn len(&self) -> usize {
self.inner.len()
}
pub fn is_empty(&self) -> bool {
self.inner.is_empty()
}
/// Pop the next *already-expired* item without awaiting, or `None` if nothing is ready right
/// now (the queue is empty, or its earliest item has not reached its deadline yet). Lets a
/// caller drain a burst of simultaneously-expired items in a tight loop without yielding.
///
/// It polls the inner queue with a **no-op waker**, so a not-yet-due (`None`) result registers
/// no real wakeup. This is therefore sound ONLY when the caller subsequently polls the
/// [`Stream`] impl (`.next().await`) before parking the task - that re-arms the timer against
/// the task's real waker, superseding the no-op one. The intended use is "drain the extra ready
/// items right after `.next()` yielded one, in a loop that returns to `.next().await`". Calling
/// it as the last thing before suspending would drop the wakeup (same caveat as
/// `futures::FutureExt::now_or_never`).
pub fn try_next_expired(&mut self) -> Option<Expired<T>> {
let mut cx = Context::from_waker(Waker::noop());
match Pin::new(&mut self.inner).poll_expired(&mut cx) {
// a ready-expired item, or `None` because the queue is empty
Poll::Ready(maybe_item) => maybe_item,
// queue is non-empty but nothing is due yet
Poll::Pending => None,
}
}
}
impl<T> Default for NonExhaustiveDelayQueue<T> {
fn default() -> Self {
NonExhaustiveDelayQueue::new()
}
}
impl<T> Stream for NonExhaustiveDelayQueue<T> {
type Item = <DelayQueue<T> as Stream>::Item;
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
match Pin::new(&mut self.inner).poll_next(cx) {
Poll::Pending => Poll::Pending,
Poll::Ready(Some(item)) => Poll::Ready(Some(item)),
Poll::Ready(None) => {
// we'll need to keep the waker to notify the executor once we get new item
self.waker = Some(cx.waker().clone());
Poll::Pending
}
}
}
}
// #[cfg(test)]
// mod tests {
// use super::*;
//
//
// }