48 Commits

Author SHA1 Message Date
Jędrzej Stuczyński 8a93bce32f feat: additional mixnet improvements and metrics (#6874)
* wip

* batch processing of forward packets

* tmp: additional metrics for remote node

* fixed incorrect prometheus metric registration

* unified runtime metrics

* unify mixnet client metrics

* packet forwarding cleanup

* add batching for emptying the delay queue

* cleanup client io loop

* feat(nym-node): reap idle mixnet connections (ingress + egress)

Close mixnet connections that sit with no traffic past a configurable idle period (mixnet.debug.connection_idle_timeout, default 5min, 0 disables) to bound lingering tokio tasks/sockets.

Ingress handle_stream is read-only, so a silently-gone peer (NAT drop, crash without FIN, half-open) never triggers FIN/RST and the task would block on .next() forever; a new idle select arm closes it (the post-loop replay flush still runs, so nothing is stranded). Egress run_io_loop gets the symmetric arm keyed on last_send; on close EvictOnDrop clears the cache entry and the next packet transparently reconnects.

Adds a cumulative nym_node_network_idle_closed_ingress_mixnet_connections counter; egress reaping is observed via the existing active-egress gauge plus an exit_reason=idle_timeout log.

* downgrade sysinfo

* refactor(nym-node): split PacketForwarder into router + delay-queue tasks

Split the single PacketForwarder task into two concurrently-scheduled tasks connected by a bounded handoff channel, so intake and delayed-release no longer block each other.

PacketRouter (router.rs) is the intake task: sole consumer of the ingress channel, it applies the routing filter and either forwards zero/already-elapsed-delay packets directly or hands delayed ones to the delay task. Its per-packet work is sub-µs, so new packets no longer wait behind delayed-release processing (collapses the ForwarderQueue tail).

DelayForwarder (delay.rs) owns the NonExhaustiveDelayQueue exclusively (it can't be shared by reference). Its run loop services BOTH branches on every wakeup - draining pending inserts first to bring the queue current, then flushing everything now due - so the biased select can't let releases and inserts starve each other, and a freshly-arrived-but-already-due packet releases in the same pass (marginally improving DelayQueueOverrun).

The mixnet client is shared as Arc<C>; handoff-channel overflow is dropped as an egress drop rather than blocking, keeping intake decoupled from release.

* feat(nym-node): bound egress flush with a write timeout

Cap how long a single egress batch flush may block on a congested peer socket (mixnet.debug.connection_write_timeout, default 500ms, 0 disables), so a slow peer can no longer back this connection's egress queue up into the multi-second range - the root of the EgressQueue and SocketWrite tails.

A single timeout is treated as transient congestion: the un-fed tail of the batch is abandoned but the connection is retained. This is sound because NoiseStream::poll_write encrypts and buffers each frame synchronously, so a cancelled flush leaves the noise transport nonce-consistent and a later flush resumes the byte stream in order - so a momentary spike costs no re-handshake. Only MAX_CONSECUTIVE_WRITE_TIMEOUTS (3) timeouts in a row, i.e. a persistently congested peer, tears the connection down (it reconnects on the next packet); a successful flush resets the counter.

Buffer-size tuning (maximum_connection_buffer_size) deliberately left for live data.

* revert PacketForwarder split in favour of a single task that clears both channels on wake
2026-06-12 10:31:54 +01:00
Jędrzej Stuczyński 3730260cf0 feat(nym-node): mixnet packet latency instrumentation (#6852)
- PacketTrace stopwatch + generic Traced<T> carrier threaded receive -> socket-write
- TraceStage enum owns each stage's metric name/help/buckets; observations go straight
  to the global nym-metrics registry under a uniform mixnet_packet_* family
- stages: Unwrap, ReplayCheck (incl. deferral), ForwarderQueue, DelayQueue,
  DelayQueueOverrun (lateness beyond target release), EgressQueue, SocketWrite, Total
- node-side 1-in-N sampling via MixnetDebug.egress_trace_sample_rate (default 100, 0 disables)
2026-06-09 16:22:14 +01:00
Jędrzej Stuczyński b6202b5a6b chore: minor nym-node improvements (#6850)
* set TCP_NODELAY for mixnet connections

* bugfix: correctly compute count deferral threshold

* bugfix: make sure to flush pending packets waiting for bloomfilter check

* implement batch sending into mixnet connection

* adjust default nym-node connection settings
2026-06-08 08:37:41 +01:00
Jędrzej Stuczyński 46c67440bb Mixnode stress testing (#6575)
* Squashing the mix stress testing branch (#6575)

reduced chain watcher per block log severity

update network monitors contract semver to 1.0.0

fix build issues

fix mixnet client dropping initial packet on egress reconnection

adjusted logs for network monitor agent

changed default testing interval to 2h

refresh NM contract information

explicit return type for batch submission

for mixnet listener task to get scheduled before beginning connectivity test

make sure to always use canonical ip for network monitor noise keys

feat: NMv3: make agents decide egress port (#6746)

add config v12->v13 config migration for nym nodes

fix formatting in wallet types

simplified client config creation

remove other swagger redirect

removed swagger redirect on /swagger/ route

log version info on startup

add workflows, contract address, and dockerfile

bugfix: use correct endpoints when setting up orchestrator (#6733)

clippy

adjust DEFAULT_MIN_STRESS_TESTED_NODES ratio

expose route with new performance metrics

fixes and additional docs

use stress testing scores

stub for usage of stress testing scores

stub traits

added new fields to nym-api config controlling usage of stress test data

guard against duplicate packets

prevent usage of chain_authorisation_check_max_attempts with value of 0

make sure duplicate results cant be inserted into the db

submit test results from orchestrator on an interval

docs and fixes

nym-api side of handling result submission

stubs for submitting results

NM orchestrator verifying nym-api result submission permissions

NM orchestrator to update announced key on startup

allow NM orchestrator to announce its identity key to the contract

stubs within nym-api for accepting NMv3 results

added additional metrics

docs

bugfixes + making sure to only assign mixnode testruns

fixed node refresher to only retrieve mixnodes and add additional metrics

topology metrics

defined basic prometheus metrics

authorised endpoint for returning prometheus data

create initial stub for prometheus metrics

post rebasing fixes

adjusted routes

missing implementation for storage getters

a lot of new stubs and db accessors

stubs for results endpoints

update utoipa tags for agent rountes

shared auth between metrics and results

moved stale results eviction into the interval.tick branch

refactor and comments

create background process to evict stale data

include sphinx packet delay as part of the stats

fix mock construction

add median to the calculated latency distribution

remove unused imports

cleanup

performing testrun and submitting the results

assigning testruns to requesting agents

basic stub for http server for the NMv3 orchestrator

chore: rename existing 'NetworkMonitorAgent' to 'NodeStressTester'

make sure to use canonical ips within the noise config

fixed contract tests

cargo fmt

additional comments and unit tests

contract and nym-node support of NM agents being run on the same host

basic unit tests

refactoring

make agents retrieve mix port assignment from the orchestrator

provide sensible defaults to CLI arguments

stub the initial structure for the agent

chore: remove redundant import

missed tick behaviour

removed redundant mutex

removed redundant try_get_client

reuse existing constant for default nymnode port

add node refresher for periodic scraping of bonded nym-node details

- NodeRefresher periodically queries the mixnet contract for all bonded
  nodes and probes each node's HTTP API for host information, sphinx keys,
  noise keys, and key rotation IDs
- Extract NymNodeApiClientRetriever into nym-node-requests with port
  probing, identity verification, and host information signature checking
- Add clone_query_client on NyxdClient so the refresher can hold its own
  query client without locking the signing client
- Batch upsert for nym_node rows (single transaction instead of per-row)
- Reuse the new helpers in nym-api's node_describe_cache

ensure assignment of testrun begins an IMMEDIATE tx

construction of the orchestrator struct

initial set of cli args

make sure to not assign testable nodes too often

very initial database structure and cli

fixed construction of RoutableNetworkMonitors

remove redundant constructor for NoiseNode

forbid 0-nonsense config values

add type safety for test route construction

moved lioness and arrayref to workspace deps

fixed dockerfile build

always use canonical addresses in RoutableNetworkMonitors

fixed old contract formatting issues

removed redundant into() call

network monitor agent fixes

additional logs

config unit tests

more docs

standalone stress testing invocation

further refactoring and changes

refactor testing loop and return valid test result upon completion

initial sending/receiving test loop

generating reusable sphinx headers

additional structure for receiving ingress packets

initial scaffolding for NMv3 agent

added validation of x25519 noise key

removed unstable call to 'is_multiple_of'

remove calls to from_octets as they're unavailable in pre 1.91

additional docs/comments

propagating noise information about NM for mixnet routing

pass full socket address of the agent into the contract storage

feat: store noise keys alongside ip addresses within the contract

removed redundant comment

ensure NM packets can only go to NM

PR review comments

added additional docs

allow NM to replay packets + fix replay prometheus metrics

propagate information about nm agent to connection handler

updated nym-node config migration

feat: introduced nym-node websocket subscription for keeping updated list of NM agents

allow admin to also revoke monitor agents

remove agents upon orchestrator removal

fixed schema generation and regenerated the contract schema

removed rustc restriction on contracts-common

added client methods for interacting with the contract

added unit tests for contract methods

implemented logic of the network monitors contract

create initial structure for network monitors contract

start mix stress testing topic branch

* make nym-node default to the new blockstream rpc/ws node cluster

* reduced mixnet-client log severity

* set network monitors contract address for mainnet
2026-05-22 09:43:20 +01:00
mfahampshire dd6a45f251 Make publication explicit 2026-04-17 09:23:55 +01:00
mfahampshire 924d7d1ccc Enforce ordering of [package] fields in cargo.toml files 2026-04-17 07:49:50 +01:00
Jack Wampler 7b77091fb1 Nym Node spam logging (#6621)
prevent spam logs when downstream node is slow
2026-03-26 11:27:14 -06:00
Tommy Verrall a519f4ccb8 pr feedback
- Moved OTel CLI options into a separate OtelArgs
- Otel is built behind the feature flag otel
- Store timing is in microseconds
- Restore comments to existing files
2026-02-18 10:48:54 +01:00
Tommy Verrall 40a3cd28b7 otel: add tracing 2026-02-16 13:46:17 +01:00
mfahampshire cf3fd00350 Max/crates io prep v2 (#6270)
* - standardise versions for all nym-sdk workspace dependencies
- prepend sqlx-pool-guard with 'nym-'

* Test remove nym-api from deps

* Add oneliner to client_pool doc comments

* Add note to commented out docs.rs link in sdk

* remove nym-api from script

* add publishing file

* bring non-binary / contract / tools into workspace version

* added more info to publishing.md

* make deps workspace version

* remove uploaded sphinx-types crate from script

* remove erroueously included ignore-defaults

* add zeroise to feature

* chore: Release

* add topology to batch

* more cargo versioning

* more cargo versioning - wasm utils

* more cargo versioning - wasm utils

* Add publish=false to manifest for cargo workspaces / crates.io
publishing exclusion

* remove script now switched to manifest based exclusion

* rename import based on rename of contracts-common dep

* Making workspace versions for publication + removing unnecessary crates
from publication

* Remove OOD info from publishing sdk guide

* rename contract imports + remove package

* temp commit: continuing with removal of path from cargo manifest and
replacing with workspace version import for publication

* continuing with cargo.toml updates

* dryrun only erroring on known version problem crates

* remove old published-crates file

* Minor comment change

* remove default features warning

* Additional info on workspace dep comment re publish list

* Add missing description to cargo.toml

* Fix missing feature flags

* Add missing descriptions

* Fix remaining path import

* Add workspace repo / homepage / documentation links to cargo.toml files

* remove workspace version from excluded crate

* Remove todo descriptions

* Minor comment change

* add homepage etc

* move from bls git import to nym_bls_fork crate

* Modify rest of imports from path to workspace import, excluding binaries

* add directory/homepage info

* fix cargo fmt

* add notes to gitignore

* better solution to contracts/ experiment

* wasm -> nym_wasm crate renaming

* fix fatfinger

* add metadata to ecash cargo.toml

* stub publishing guide

* fix misrevolved netlink- version

* Fixes and block publication of rebase re: LP

* first pass @ workflows
2026-01-19 13:19:45 +00:00
Jędrzej Stuczyński fa1d47e941 Bugfix/backwards compat (#5865)
* lowered log severity

* make nodes use legacy encoding for forwarding packets

* note regarding localnet noise
2025-06-19 09:57:46 +01:00
Simon Wicky 8ba58ba11e [Feature] Noise XKpsk3 integration (2025 version) (#5692)
* grand Noise squaheroo

* fix merge conflicts and adapt code for the key rotation changes (#5824)

* remove file that should have been ignored
2025-06-05 11:34:55 +02:00
Jędrzej Stuczyński d8c84cc4d6 feat: key rotation (#5777)
* wip

* wip: wrap node's sphinx key with a manager

* wip: choosing correct key for packet processing

* further propagation of key rotation information

* attaching key rotation information to reply surbs

* added basic key rotation information to mixnet contract

* wip: introducing cached queries for key rotation info from nym api

* unified nym-api contract cache refreshing

* finish packet decoding

* multi api client + retrieving rotation id

* rotating sphinx key files

* logic for migrating config file

* wip: putting new sphinx keys to self described endpoints

* processing loop of KeyRotationController

* fixed sphinx key loading

* rotating bloomfilters

* wired up KeyRotationController

* flushing bloomfilters to disk and loading

* most of nym-node changes

* post rebase fixes

* fixes due to backwards compatible hostkeys

* split http state.rs file

* dont use deprecated fields

* fixed backwards compatible deserialisation of host information

* split up node describe cache

* added a dedicated CacheRefresher listener to perform full refresh outside the set interval

* controlling announced sphinx keys within nym-api

* retrieving rotation id when pulling topology

* split nym-nodes http handlers

* v2 nym-api endpoints to retrieve nodes with additional metadata information

* bug fixes...

* additional bugfixes and guards against stuck epoch

* testnet manager: set first nym-api as the rewarder

* fixed host information deserialisation

* fixed panic during first key rotation

* post rebase fixes

* clippy

* more guards against stuck epochs

* added helper method to reset node's sphinx key

* instantiate mixnet contract with custom key rotation validity

* additional bugfixes and debugging nym-api deadlock

* passing shutdown to nym apis client

* remove dead test

* post rebasing fixes

* missing MixnetQueryClient variants

* remove usage of deprecated methods in sdk example

* fix: incorrect method signature

* post rebasing fixes

* attempt to retrieve key rotation id before doing any config migration work

* ignore tests relying on networking behaviour

* allow networking failures in certain tests
2025-06-03 11:22:51 +01:00
Jędrzej Stuczyński d062524d32 mix throughput tester (#5661)
* wip: sending with single client

* tag packets to measure latency

* constantly logging rates

* concurrency

* adjusting some values

* write results to files upon completion
2025-03-31 15:57:24 +01:00
Jędrzej Stuczyński 7d5e3ef7d3 feature: expand nym-node prometheus metrics (#5298)
* fixed bearer auth for prometheus route

* basic prometheus metrics

* added rates on global values

* improved structure on the prometheus metrics

* added additional metrics for ingress websockets and egress mixnet connections

* some channel business metrics

* fixed metrics registration and added additional variants

* added counter for number of disk persisted packets

* counter for pending egress packets

* counter for pending egress forward packets

* clippy
2024-12-20 10:32:56 +00:00
Jędrzej Stuczyński 5a07b73375 feature: hopefully final steps of the smoosh™️ (#5201)
* removed mnemonic from gateway config struct

scaffolding for common mixnet listener

running verloc unconditionally in a nym-node

remove filtering by mixnode

extracted verloc to separate crate

integrated nym-node-http-server more tightly with the binary

most logic for handling forward packets

running all mixnode-related tasks natively inside nymnode

removed gateway storage trait in favour of the only concrete implementation

most logic for handling final hop packets

using nym-node owned socket listener for gateways

utility for sending plain message through mixnet + gateway fix

using common packet forwarding in both modes

nifying nym-node metrics

reproduce behaviour of the console logger

cleaned up cli args

redesigned gateway tasks startup procedure

removing dead code

scaffolding for old config v6

config migration

implemented MixnetMetricsCleaner

* clippy

* require entry/exit for wireguard

* removed dead code in migration code

* updated config template

* use custom user agent for verloc queries

* fixed premature shutdown of gateway tasks

* hidden nym-api flag to allow illegal node ips

* experiment: final hop handing with wireguard

* added additional startup logs

* typo

* fixed legacy stats endpoint data

* additional logs

* apply review comments

* fixed local testnet manager
2024-12-05 17:21:36 +00:00
Jon Häggblad f4dd9a915d Add some license entries to Cargo.toml files (#4250)
* Add Apache 2.0 to bunch of common crates

* Allow some basic licenses

* Add more licenses
2023-12-19 09:21:33 +01:00
Jędrzej Stuczyński bac0f24cf7 Feature/issued credentials api (#4207)
* split up coconut module a bit

* internal tool for watching dkg state and updating group contract

* debug dkg state

* display past dealer data

* improved EpochState Display impl

* display contract errors + advance epoch state

* check admin

* panic handler

* simplify app.rs

* split action enum

* added new tab with logger information

* new dealing display

* sort by index

* [fixedup] wip: updating epoch issued credentials - OG 92ade10384a6d7b6c6c222d2e29d69d3b3446a4c

* storing and signing partial blinded credentials

* starting cleanup

* fixed coconut tests + clippy

* fixed nym-api tests

* removed dkg-manager tool

it was moved to a different branch

* implemented remaining endpoints

* unit tests + bug fixes

* clippy

* added persistent identity keys to nym-api

theyre not yet announced - this will be in another PR

* cargo fmt

* clippy

* fixed loading of old configs without storage paths set

* added additional logs for blind-sign endpoint

* fixed up licenses

* lowercasing error variants

* changed 'issued_credentials' to a post

* added minimal client support

* fixed the unit test
2023-12-18 12:51:34 +01:00
Jędrzej Stuczyński 114d92f93f Feature/gateway inbuilt nr (#3877)
* changed NymConfigTemplate trait to call 'template' by reference

* network requester lib

* introduced generic parameter to 'MixTrafficController' to allow non-remote gateways

* allowing for custom gateway sender

* types cleanup

* minor gateway cmds refactor + initial NR work

* wip

* running a NR inside gateway

note: this NR isnt tied to the gateway yet

* rebase fixes

* propagating same shutdown handle

* wip

* starting NR with appropriate local transceiver

* fixed premature shutdown

* wiring up PacketRouter

* both ends wired together

* actually working

so much cleanup to do now

* started removing dead code

* wip

* temp: hardcode gateway

* further cleanup

* fixed build of other binaries

* setup-network-requester subcmd

* overriding NR config in gateway init/run

* wip making it wasm-compatible [again]

* refactored 'GatewaySetup'

* clippy and friends

* removed debug code

* rust 1.72 lints

* ensuring local gateway is available + some comments

* correctly putting network requester data in the same underlying details struct

* improved gateway errors

* changed 'network_requester_config' deserialization

* missing clap annotation for 'enabled' flag in 'setup-network-requester' command

* saving config file after 'setup-network-requester'

* removed dead code

* review comments

* make embedded NR wait for gateway to come online (for at most 70min)

* fixed shutdown on successful gateway wait

* updated NR config override
2023-09-20 15:47:05 +01:00
Jędrzej Stuczyński d13c8bde57 Feature/wasm tendermint rpc client (#3744)
* wasm-compatible reqwest-based rpc client

* better constructors for the reqwest based client

* fixed usages of the client

* introduced /network/details endpoint to nym-api to return used network information (#3758)

* introduced /network/details endpoint to nym-api to return used network information

* introduced endpoints for nym contract information
2023-08-17 10:56:10 +01:00
Drazen Urch b5c8b69547 Outfox integration (#3331)
* Experiment with serde

* Framed encoding serde POC

* Outfox framing

* Outfox rest compat (#3333)

* Outfox forwarding compat

* Tidy up interface

* PacketSize compat

* Address PR comments

* Rebase on develop

commit 342883fcbe
Author: durch <durch@users.noreply.github.com>
Date:   Thu Apr 27 09:17:18 2023 +0200

    Put back PacketType 1

commit 61a0ee5a19
Author: Tommy Verrall <tommyvez@protonmail.com>
Date:   Wed Apr 26 16:37:29 2023 +0100

    change output for cpu-cycle management logs

commit 3956109c7e
Author: Tommy Verrall <tommy@nymtech.net>
Date:   Wed Apr 26 12:13:22 2023 +0100

    change the workflow file to build with cpucycles

commit 8d725b13c5
Author: durch <durch@users.noreply.github.com>
Date:   Mon Apr 24 13:14:58 2023 +0200

    Outfox client compat

commit 4d166c389b
Author: durch <durch@users.noreply.github.com>
Date:   Fri Apr 21 00:30:46 2023 +0200

    Address PR comments

commit 145c3c1223
Author: durch <durch@users.noreply.github.com>
Date:   Fri Apr 21 00:12:35 2023 +0200

    Rename PacketMode

commit cbd654d6fd
Author: Drazen Urch <drazen@urch.eu>
Date:   Thu Apr 20 23:59:40 2023 +0200

    Outfox rest compat (#3333)

    * Outfox forwarding compat

    * Tidy up interface

    * PacketSize compat

commit e7be91a94c
Author: durch <durch@users.noreply.github.com>
Date:   Wed Apr 19 16:36:48 2023 +0200

    Remove serde cruft

commit 582e7d566a
Author: durch <durch@users.noreply.github.com>
Date:   Wed Apr 19 16:24:09 2023 +0200

    Outfox framing

commit 6464da5f01
Author: durch <durch@users.noreply.github.com>
Date:   Tue Apr 18 22:23:02 2023 +0200

    Framing compat

commit d5e77e499b
Author: durch <durch@users.noreply.github.com>
Date:   Tue Apr 18 18:18:54 2023 +0200

    Framed encoding serde POC

commit f086f9c35a
Author: durch <durch@users.noreply.github.com>
Date:   Tue Apr 18 16:54:21 2023 +0200

    Experiment with serde

* Client tweaks

* Speed up from_plaintext

* SurbAcks

* More work on the reciever end, and outfox format

* Cleanup and fmt

* Wrap up rebase

* Happy clippy

* Fix lock files

* Final cleanup
2023-05-29 10:05:11 +02:00
Jon Häggblad dc5c765ecb Last set of crates to add the nym- prefix to (#3286)
* Add nym- prefix to mixnode-common

* Add nym- prefix to mixnet-client

* Add nym-client- prefix to websocket-requests

* Makefile: add check target

* Rename to nym-credential-client

* rustfmt

* update to nym-credential-client in github workflow
2023-04-07 21:25:28 +02:00
Jon Häggblad 46e2c74a98 Rename crate to nym-task (#3064)
* Rename to nym-task

* Update imports to use nym_task

* Update to nym_task in explicit module prefix

* Touchups and rustfmt

* rustfmt

* crate metadata

* fix nym-connect-android
2023-02-20 21:55:29 +01:00
Jon Häggblad c75c5e0903 Rename to crate name to nym-sphinx (#3060)
* nymsphinx: rename to nym- prefix in Cargo.toml files

* regex use nymsphinx to use nym_sphinx

* all: updated explict crate name in a few places
2023-02-20 16:28:08 +01:00
Jon Häggblad e548d6f1f8 cargo: try out inheriting workspace metadata (#2853)
* cargo: add log as a workspace dependency

* cargo: add authors to top-level workspace

* cargo: add a few more entried to workspace package
2023-01-18 10:13:11 +01:00
Jędrzej Stuczyński bbb46ebd90 chore: upgrade tokio to 1.24.1 (+ tokio-util and tokio-stream) (#2843)
* updated 'tokio' to 1.24.1 in the whole codebase

* ibid for 'tokio-stream'

* ibid for 'tokio-util'

* Removed lock file from verify-signature example
2023-01-13 14:46:10 +00:00
Jędrzej Stuczyński b473aeb3be Feature/gateway graceful shutdown (#2834)
* task dependency

* unifying some startup code and passing TaskClient around

* graceful shutdown handling for mix socket listener

* graceful shutdown handling for clients listener

* graceful shutdown handling for packet forwarding

* unified waiting for interrupt across binaries

* made 'validate_bech32_address_or_exit' into a function that returns proper Result

* printing formatted message on main error

* fixed failing test

* removed duplicate code that should have been gone ages ago

* ibid

* removed biased selection for authenticated handler

* minor refactoring to 'ensure_config_version_compatibility'
2023-01-13 11:56:37 +00:00
Jędrzej Stuczyński 97b01db23e Chore/more error macros (#2686)
* cleaned up MixProcessingError

* Added Error impl to (hopefully) all error enums in the codebase

* Replaced all occurences of error("{0}") with error(transparent)

* Changelog entry
2022-12-13 17:42:11 +00:00
Jędrzej Stuczyński 49ce56c367 Jedrzej/feature/version field in framed sphinx packets (#1723)
* introduced PacketVersion into FramedSphinxPacket

* Using legacy mode by default in mixnodes and gateways

* fixed unit tests
2022-10-31 16:56:37 +00:00
Jędrzej Stuczyński a43d183b4f Feature/wasm client updates (#1673)
* Compiles but runtime time fails

* wip

* Beginning of clean-up - creation of config to keep things together

* Removed unused module

* Removed hardcoded constants

* Easier way of sending binary messages

* WIP cleanup before machine switch

* Upgrade wasm-bindgen to 0.2.83

* Fixed compilation warnings for wasm client

* all clients compiling without warnings

* disabling topology refresh in wasm

* Added a config option to disable loop cover traffic stream

* config changes

* Make webassembly work in a web worker
- `wasm-timer` modified to work in web worker
- add worker target to webpack
- add client to call from HTML
- update README to build WASM for bundling (this does not build ES modules)

* Restored topology refreshing

* correctly polling items in the wasm delay_queue

* Allow client to read up to 8 messages at once from gateway connection (#1669)

* Allow client to read up to 8 messages at once from gateway connection

* Importing tokio::select in wasm32 target

* Updated changelog

* missing imports

* Introduced disable_main_poisson_packet_distribution to force real_traffic_stream to disable poisson sending (#1664)

* Introduced disable_main_poisson_packet_distribution to force real_traffic_stream to disable poisson sending

* Updated changelog

* Adjusting default settings

* Introduced a client-configurable option to force it to use extended packet size

* local adjustments

* Removed warning associated with receiving extended packets

* Minimal v2-required changes

* Updated changelog

* explicitly allowing clippy drop_non_drop

Co-authored-by: Mark Sinclair <mmsinclair@gmail.com>
2022-10-10 16:27:51 +01:00
Jon Häggblad e44ddc419c clippy: fix warnings in beta toolchain (#1639) 2022-09-21 09:25:22 +02:00
vnpmid d8fed178aa Upgrade to tokio 1.19.1, tokio-util 0.7.3, tokio-stream 0.1.9, tokio-test 0.4.2 (#1305)
* Upgrade to tokio 1.19.1, tokio-util 0.7.3, tokio-stream 0.1.9, tokio-test 0.4.2

* Tokio-util 0.7.3 handle_done_delaying, handle_expired_ack_timer

* Upgrade to tokio 1.19.1, tokio-util 0.7.3, tokio-stream 0.1.9, tokio-test 0.4.2
2022-06-06 14:38:01 +01:00
Jędrzej Stuczyński 0f6f47c5ac Fixed overflow in determining reconnection backoff (#1260)
* Fixed overflow in determining connection backoff

* Updated changelog
2022-05-09 13:23:14 +01:00
Jon Häggblad c6e41ca3f3 Update to rust edition 2021 everywhere (#1086)
* chore: update to rust edition 2021 everywhere

* validator-api: simplify into_iter call in rust 2021
2022-02-01 13:28:02 +01:00
Jon Häggblad 1368ab133e Add trait to mock client dependency in DelayForwarder (#1073)
* client-libs/mixnode: add trait for client sending

* mixnode: instantiate and test DelayForwarder

* mixnode: remove commented out line
2022-01-27 15:34:32 +01:00
Jędrzej Stuczyński e5afd54ce0 Feature/node status api (#680)
* Basic storage stub

* New models for new node status api

* Route handling

* Mounting new routes

* Missing selective commit

* Moved network monitor related files to separate package

* Starting to see some sqlx action

* Schema updates

* Log statement upon finished migration

* Removed old diesel related imports

* Converted mixnode cache initialisation into a fairing

* Moved cache related functionalities to separate package

Also defined staging there

* Created run method for validator cache + removed unwrap

* Removed old node-status-api types and left bunch of todo placeholders in their place

* Fixed managing validatorcache

* Status reports are starting to get constructed

* Submitting some dummy results to the database

* Removing duplicate code for generating reports

* Removed statuses older than 48h

* Initial attempt at trying to obtain reports for all active nodes

* Removed duplicates from the full report

* Grabbing uptime history

* Updating historical uptimes of active nodes

* Updated sqlx-data.json

* Removed all placeholder foomp owner values

* Changed Layer serde behaviour for easier usage

* Extended validator api config

* Initial (seems working !) integration with network monitor

* Added database path configuration to config

* Using ValidatorCache in NetworkMonitor

* Flag indicating whether validator cache has been initialised

* Introduced a locla-only route for reward script to perform daily chores

* Flag to save config to a file

* Moved spawning of receiving future to run method rather than new

* Removed arguments that dont make sense to be configured via CLI

* Removed dead code from config file

* More dead code removal

* Added validator API to CI

* Corrected manifest-path arguments

* Constructing network monitor by passing config

* Combined validator API CI with the main CI file

* Using query_as for NodeStatus

* Checking if historical uptimes were already calculated on particular day

* Making id field NOT NULL

* More query_as! action

* Updated sqlx-data.json

* Removed unused chrono feature

* Renamed the migration file

* Changed default validator endpoint to point to local validator

* Removing unnecessary clone

* More appropriate naming

* Removed dead code

* Lock file updates

* Updated network monitor address in contract code

* Don't stage node status api if network monitor is disabled

* cargo fmt

* Updated all license notices to SPDX
2021-07-19 14:02:47 +01:00
Jędrzej Stuczyński a521bc0f54 Logging adjustment (#585)
* Reduced severity of common harmless warnings

* Changed the same ip node warning

* More explicit error on network address binding failure
2021-04-28 09:11:04 -04:00
Jędrzej Stuczyński 596bc76cc6 Chore/dependency updates (#549)
* Updated all non-breaking dependencies

* Updated common/crypto dependencies

* Updated all tokio [and associated] dependencies to most recent version

* Bumped version of rand_distr

* Fixed api changes in tests

* Made clippy happier about the acronym

* Fixed the type while trying to make clippy even happier

* nightly cargo fmt
2021-03-29 15:32:34 +01:00
Jędrzej Stuczyński 2d3b4f4b91 Feature/GitHub actions and clippy cleanup (#493)
* Added github actions templates

* removed travis .yml file

* initial clippy cleanup pass

* fixed the rest of clippy warnings

* Made github badges more fancy and consistent with the ones in sphinx

* Updated local rustc version and removed compilation warningns

* ... and fresh clippy warnings

* formatting

* beta clippy specific warnings fixed

* Fixed all nightly clippy warnings

* Fixed trying to unwrap a ()

* Actually running all tests

* Correctly passing the --all flag

* Hopefullly third time's a charm in fixing argument passing
2021-01-18 11:50:29 +00:00
Jędrzej Stuczyński 210b5466bb Minor mixnet client code simplification and optimization (#472)
* Minor code simplification and optimization

* Made the connection establishing call non-blocking
2020-11-26 13:06:05 +00:00
Jędrzej Stuczyński 33d5effbdc Putting initial packet onto the queue when establishing connection (#471) 2020-11-25 15:09:40 +00:00
Jędrzej Stuczyński cb59153422 Feature/nonblocking mix send (#464)
* No backoff

* reconnections

* Increased default network monitor default sending rate

* Comment for the future

* Maximum connection buffer size as part of config
2020-11-23 17:27:14 +00:00
Jędrzej Stuczyński 89f03e63c7 Feature/delay queue mixnodes (#462)
* Initial idea for the delayer

* Initial rought integration into mixnode

* Removed lock requirement from 'received' metrics

* 'sent' metrics

* Comment fix
2020-11-20 14:49:22 +00:00
Jędrzej Stuczyński 72496ad2aa Feature/max retry (#409)
* Maximum number of allowed reconnection attempts

* Requiring initial internode connection to be successful

* Decreased logging level for failing to establish initial connection
2020-11-03 14:55:10 +00:00
Jędrzej Stuczyński 4f6b2aea19 Feature/instant sending (#359)
* Ability to set client in vpn mode

* Connection handler for mixnode

* Initial vpn mode for mixes

* Updated SphinxCodec to contain more metadata

* Renaming

* Removed handle from mixnet client and introduced forwarder

* Mixnode using new forwarder

* Mixnode common module containing shared packet processing

* ibid. incorporated inside mixnode

* New processing for gateway

* Type cleanup

* Wasm fix

* Fixed client config

* Fixed mixnode runtime issues

* Formatting

* Client re-using secret on 'normal' packets

* Using the same key for acks

* WIP

* vpn key manager cleanup

* wasm fix

* VPN_KEY_REUSE_LIMIT moved to config

* Moved AckDelayQueue to separate common crate

* Key cache invalidator

* Updated dashmap used in gateway

* Old typo

* Additional comment

* Cargo fmt

* Fixed tests

* Sphinx update

* cache ttl as config option

* Cargo fmt
2020-09-30 17:30:17 +01:00
Jędrzej Stuczyński d9d549fd0f Feature/reply surbs (#299)
* Changed identity keypair to use ed25519

* Encryption key is now x25519 based + compatibiltiy with sphinx

* Pathing and import fixes

* Moved all asymmetric keys to sub-module in crypto

* Extracted aes to separate module

* kdf module in crypto

* Ability to perform diffie hellman on encryption keys

* ecdsa on identity keys

* Extremely rough and incomplete registration handshake

* Authentication primitives

* Creating new random authenticationIV

* Wrapper type for the derived shared key

* Removed AuthToken in favour of using SharedKey for authentication

* Gateway identity keys

* Registration handshake without error mapping

* Gateway address in client config

* Added extra key for gateway presence

* Updated pemstore to work on borrows instead

* Gateway client trying to perform the handshake

* Gateway changes to allow for handshake and shared key

* Debug trait on sharedkey

* native client using updated gateway client

* Slightly updated gateway API

* Minor cleanup

* Fixed pemstore to correctly save multiple keypairs

* Gateway actually deriving shared key during handshake

* Gateway sending correct mid-handshake message

* Missing quotation mark in client config template

* Fixed template for correct shared key serialization

* Fixed gateway authentication

* Fixed tests

* Using correct gateway key when converting to sphinx node

* "get_all_clients" takes them from gateways as opposed to providers now

* cargo fmt

* Renamed pemstore methods

* Unused import

* Encryption of forward requests between client and gateway

* Updated sphinx dependency to use public revision

* Sending 'error' on handshake processing error

* Removed some dead code

* Dead code I forgot to remove before

* Extracted AckAes128Key into a struct

* Slight pemstore revamp allowing for symmetric key store

* ibid.

* PemStorableKey for SharedKey

* Introduced single location responsible for key management for client

* WIP

* Sphinx version update

* Stop using NodeAddressBytes for two distinct and confusing purposes

* Abstracting away SocketAddr from sphinx forwarding

* Passing the bool for reply surbs

* Attack plan for replies + encryption

* Comment + removed variable binding

* ReplySURB usage

* Topology import in nymsphinx

* Sphinx version update

* Changed 'Recipient' to contain client's encryption key

* Message preparation taking shape!

* reply surb also containing the encryption key

* Very initial message receiver

* Sphinx version update

* A possibly working way of receiving surbs

* Fixed incorrect field name in client config template

* camel casing all request arguments

* Renamed and moved `MessageMode` to more appropriate file

* Restored reconstruction tests

* Removed dead code from chunking

* Made rust examples compilable

* reply SURB key storage

* Replies as an InputMessage

* Forgotten commented code

* No retransmission processing for cover or replies

* Received reply processing

* Renamed client pathfinder to something more appropriate

* Made HasherOutputSize public

* Added key store path to config

* Reply surb attaching key digest when used

* Changes due to previous renaming

* Removed comment

* Fixed insert_encryption_key

* Assigning initial value of key store path

* Computing key digest with correct algorithm

* Initial and presumably temporary request serialization

* hacky way of introducing 'FragmentIdentifier' for replies

* Moved responsibility of reply encryption, padding, etc, to message preparer

* Optional recipient in try_get_valid_topology_ref

* Handling new reply surbs with acks and padding

* Updated go and python examples to include replies in text and binary cases

* Updated rust examples + binaryserverresponse

* Helpers in rust examples

* And updated JS example

* Moved shared key generation function to crypto crate

* Cover traffic encryption!

* hmac computation in crypto

* Updated aes imports due to new dependencies

* hkdf made more generic

* crypto cleanup + algorithms in params

* Clippy cleanup pass

* Generating encryption+mac shared keys between client and gateway

* MACs attached to forward requests to gateway

* Gateway messages encrypted and mac'd

* Lowered logging level

* compiler warning cleanup

* Some minor cleanup

* Generic stream cipher

* Generic shared key derivation + algorithm definitions

* Project-wide AES clean-up

* Comment fix

* Removed commented imports

* Updated comments

* Fixed topology test fixture
2020-08-07 18:16:54 +01:00
Jędrzej Stuczyński 2a29bbd5a2 Bugfix/remove packet buffering (#300)
* Removed pending packets buffer

* Corrected reconnection backoff
2020-08-07 17:17:15 +01:00
Jędrzej Stuczyński 860a69b246 Feature/sphinx socket packet encoder (#245)
* Ability to send sphinx packets of different sizes + more efficient decoding

* Closing connection on connection corruption

* Missing semicolons

* Missing license notices

* Default for packetsize
2020-05-27 10:08:08 +01:00