8a93bce32f
* wip * batch processing of forward packets * tmp: additional metrics for remote node * fixed incorrect prometheus metric registration * unified runtime metrics * unify mixnet client metrics * packet forwarding cleanup * add batching for emptying the delay queue * cleanup client io loop * feat(nym-node): reap idle mixnet connections (ingress + egress) Close mixnet connections that sit with no traffic past a configurable idle period (mixnet.debug.connection_idle_timeout, default 5min, 0 disables) to bound lingering tokio tasks/sockets. Ingress handle_stream is read-only, so a silently-gone peer (NAT drop, crash without FIN, half-open) never triggers FIN/RST and the task would block on .next() forever; a new idle select arm closes it (the post-loop replay flush still runs, so nothing is stranded). Egress run_io_loop gets the symmetric arm keyed on last_send; on close EvictOnDrop clears the cache entry and the next packet transparently reconnects. Adds a cumulative nym_node_network_idle_closed_ingress_mixnet_connections counter; egress reaping is observed via the existing active-egress gauge plus an exit_reason=idle_timeout log. * downgrade sysinfo * refactor(nym-node): split PacketForwarder into router + delay-queue tasks Split the single PacketForwarder task into two concurrently-scheduled tasks connected by a bounded handoff channel, so intake and delayed-release no longer block each other. PacketRouter (router.rs) is the intake task: sole consumer of the ingress channel, it applies the routing filter and either forwards zero/already-elapsed-delay packets directly or hands delayed ones to the delay task. Its per-packet work is sub-µs, so new packets no longer wait behind delayed-release processing (collapses the ForwarderQueue tail). DelayForwarder (delay.rs) owns the NonExhaustiveDelayQueue exclusively (it can't be shared by reference). Its run loop services BOTH branches on every wakeup - draining pending inserts first to bring the queue current, then flushing everything now due - so the biased select can't let releases and inserts starve each other, and a freshly-arrived-but-already-due packet releases in the same pass (marginally improving DelayQueueOverrun). The mixnet client is shared as Arc<C>; handoff-channel overflow is dropped as an egress drop rather than blocking, keeping intake decoupled from release. * feat(nym-node): bound egress flush with a write timeout Cap how long a single egress batch flush may block on a congested peer socket (mixnet.debug.connection_write_timeout, default 500ms, 0 disables), so a slow peer can no longer back this connection's egress queue up into the multi-second range - the root of the EgressQueue and SocketWrite tails. A single timeout is treated as transient congestion: the un-fed tail of the batch is abandoned but the connection is retained. This is sound because NoiseStream::poll_write encrypts and buffers each frame synchronously, so a cancelled flush leaves the noise transport nonce-consistent and a later flush resumes the byte stream in order - so a momentary spike costs no re-handshake. Only MAX_CONSECUTIVE_WRITE_TIMEOUTS (3) timeouts in a row, i.e. a persistently congested peer, tears the connection down (it reconnects on the next packet); a successful flush resets the counter. Buffer-size tuning (maximum_connection_buffer_size) deliberately left for live data. * revert PacketForwarder split in favour of a single task that clears both channels on wake
127 lines
4.4 KiB
Rust
127 lines
4.4 KiB
Rust
// Copyright 2021 - Nym Technologies SA <contact@nymtech.net>
|
|
// SPDX-License-Identifier: Apache-2.0
|
|
|
|
use std::pin::Pin;
|
|
use std::task::{Context, Poll, Waker};
|
|
use std::time::Duration;
|
|
use tokio_stream::Stream;
|
|
|
|
// this is a copy of tokio-util delay_queue with `Sleep` and `Instant` being replaced with
|
|
// `wasm_timer` equivalents
|
|
|
|
#[cfg(not(target_arch = "wasm32"))]
|
|
type DelayQueue<T> = tokio_util::time::DelayQueue<T>;
|
|
#[cfg(not(target_arch = "wasm32"))]
|
|
pub use tokio_util::time::delay_queue::Expired;
|
|
#[cfg(not(target_arch = "wasm32"))]
|
|
pub type QueueKey = tokio_util::time::delay_queue::Key;
|
|
#[cfg(not(target_arch = "wasm32"))]
|
|
use tokio::time::Instant;
|
|
|
|
#[cfg(target_arch = "wasm32")]
|
|
type DelayQueue<T> = wasmtimer::tokio_util::DelayQueue<T>;
|
|
#[cfg(target_arch = "wasm32")]
|
|
pub use wasmtimer::tokio_util::delay_queue::Expired;
|
|
#[cfg(target_arch = "wasm32")]
|
|
pub type QueueKey = wasmtimer::tokio_util::delay_queue::Key;
|
|
#[cfg(target_arch = "wasm32")]
|
|
use wasmtimer::std::Instant;
|
|
|
|
/// A variant of tokio's `DelayQueue`, such that its `Stream` implementation will never return a 'None'.
|
|
pub struct NonExhaustiveDelayQueue<T> {
|
|
inner: DelayQueue<T>,
|
|
waker: Option<Waker>,
|
|
}
|
|
|
|
// more methods of underlying DelayQueue will get exposed as we need them
|
|
impl<T> NonExhaustiveDelayQueue<T> {
|
|
pub fn new() -> Self {
|
|
NonExhaustiveDelayQueue {
|
|
inner: DelayQueue::new(),
|
|
waker: None,
|
|
}
|
|
}
|
|
|
|
pub fn insert(&mut self, value: T, timeout: Duration) -> QueueKey {
|
|
let key = self.inner.insert(value, timeout);
|
|
if let Some(waker) = self.waker.take() {
|
|
// we were waiting for an item - wake the executor!
|
|
waker.wake()
|
|
}
|
|
key
|
|
}
|
|
|
|
pub fn insert_at(&mut self, value: T, when: Instant) -> QueueKey {
|
|
let key = self.inner.insert_at(value, when);
|
|
if let Some(waker) = self.waker.take() {
|
|
// we were waiting for an item - wake the executor!
|
|
waker.wake()
|
|
}
|
|
key
|
|
}
|
|
|
|
// TODO: it seems like this one can cause panic in very rare edge cases, however,
|
|
// I can't seem to be able to reproduce it at all.
|
|
pub fn remove(&mut self, key: &QueueKey) -> Expired<T> {
|
|
self.inner.remove(key)
|
|
}
|
|
|
|
pub fn len(&self) -> usize {
|
|
self.inner.len()
|
|
}
|
|
|
|
pub fn is_empty(&self) -> bool {
|
|
self.inner.is_empty()
|
|
}
|
|
|
|
/// Pop the next *already-expired* item without awaiting, or `None` if nothing is ready right
|
|
/// now (the queue is empty, or its earliest item has not reached its deadline yet). Lets a
|
|
/// caller drain a burst of simultaneously-expired items in a tight loop without yielding.
|
|
///
|
|
/// It polls the inner queue with a **no-op waker**, so a not-yet-due (`None`) result registers
|
|
/// no real wakeup. This is therefore sound ONLY when the caller subsequently polls the
|
|
/// [`Stream`] impl (`.next().await`) before parking the task - that re-arms the timer against
|
|
/// the task's real waker, superseding the no-op one. The intended use is "drain the extra ready
|
|
/// items right after `.next()` yielded one, in a loop that returns to `.next().await`". Calling
|
|
/// it as the last thing before suspending would drop the wakeup (same caveat as
|
|
/// `futures::FutureExt::now_or_never`).
|
|
pub fn try_next_expired(&mut self) -> Option<Expired<T>> {
|
|
let mut cx = Context::from_waker(Waker::noop());
|
|
match Pin::new(&mut self.inner).poll_expired(&mut cx) {
|
|
// a ready-expired item, or `None` because the queue is empty
|
|
Poll::Ready(maybe_item) => maybe_item,
|
|
// queue is non-empty but nothing is due yet
|
|
Poll::Pending => None,
|
|
}
|
|
}
|
|
}
|
|
|
|
impl<T> Default for NonExhaustiveDelayQueue<T> {
|
|
fn default() -> Self {
|
|
NonExhaustiveDelayQueue::new()
|
|
}
|
|
}
|
|
|
|
impl<T> Stream for NonExhaustiveDelayQueue<T> {
|
|
type Item = <DelayQueue<T> as Stream>::Item;
|
|
|
|
fn poll_next(mut self: Pin<&mut Self>, cx: &mut Context<'_>) -> Poll<Option<Self::Item>> {
|
|
match Pin::new(&mut self.inner).poll_next(cx) {
|
|
Poll::Pending => Poll::Pending,
|
|
Poll::Ready(Some(item)) => Poll::Ready(Some(item)),
|
|
Poll::Ready(None) => {
|
|
// we'll need to keep the waker to notify the executor once we get new item
|
|
self.waker = Some(cx.waker().clone());
|
|
Poll::Pending
|
|
}
|
|
}
|
|
}
|
|
}
|
|
|
|
// #[cfg(test)]
|
|
// mod tests {
|
|
// use super::*;
|
|
//
|
|
//
|
|
// }
|