* Squashing the mix stress testing branch (#6575)
reduced chain watcher per block log severity
update network monitors contract semver to 1.0.0
fix build issues
fix mixnet client dropping initial packet on egress reconnection
adjusted logs for network monitor agent
changed default testing interval to 2h
refresh NM contract information
explicit return type for batch submission
for mixnet listener task to get scheduled before beginning connectivity test
make sure to always use canonical ip for network monitor noise keys
feat: NMv3: make agents decide egress port (#6746)
add config v12->v13 config migration for nym nodes
fix formatting in wallet types
simplified client config creation
remove other swagger redirect
removed swagger redirect on /swagger/ route
log version info on startup
add workflows, contract address, and dockerfile
bugfix: use correct endpoints when setting up orchestrator (#6733)
clippy
adjust DEFAULT_MIN_STRESS_TESTED_NODES ratio
expose route with new performance metrics
fixes and additional docs
use stress testing scores
stub for usage of stress testing scores
stub traits
added new fields to nym-api config controlling usage of stress test data
guard against duplicate packets
prevent usage of chain_authorisation_check_max_attempts with value of 0
make sure duplicate results cant be inserted into the db
submit test results from orchestrator on an interval
docs and fixes
nym-api side of handling result submission
stubs for submitting results
NM orchestrator verifying nym-api result submission permissions
NM orchestrator to update announced key on startup
allow NM orchestrator to announce its identity key to the contract
stubs within nym-api for accepting NMv3 results
added additional metrics
docs
bugfixes + making sure to only assign mixnode testruns
fixed node refresher to only retrieve mixnodes and add additional metrics
topology metrics
defined basic prometheus metrics
authorised endpoint for returning prometheus data
create initial stub for prometheus metrics
post rebasing fixes
adjusted routes
missing implementation for storage getters
a lot of new stubs and db accessors
stubs for results endpoints
update utoipa tags for agent rountes
shared auth between metrics and results
moved stale results eviction into the interval.tick branch
refactor and comments
create background process to evict stale data
include sphinx packet delay as part of the stats
fix mock construction
add median to the calculated latency distribution
remove unused imports
cleanup
performing testrun and submitting the results
assigning testruns to requesting agents
basic stub for http server for the NMv3 orchestrator
chore: rename existing 'NetworkMonitorAgent' to 'NodeStressTester'
make sure to use canonical ips within the noise config
fixed contract tests
cargo fmt
additional comments and unit tests
contract and nym-node support of NM agents being run on the same host
basic unit tests
refactoring
make agents retrieve mix port assignment from the orchestrator
provide sensible defaults to CLI arguments
stub the initial structure for the agent
chore: remove redundant import
missed tick behaviour
removed redundant mutex
removed redundant try_get_client
reuse existing constant for default nymnode port
add node refresher for periodic scraping of bonded nym-node details
- NodeRefresher periodically queries the mixnet contract for all bonded
nodes and probes each node's HTTP API for host information, sphinx keys,
noise keys, and key rotation IDs
- Extract NymNodeApiClientRetriever into nym-node-requests with port
probing, identity verification, and host information signature checking
- Add clone_query_client on NyxdClient so the refresher can hold its own
query client without locking the signing client
- Batch upsert for nym_node rows (single transaction instead of per-row)
- Reuse the new helpers in nym-api's node_describe_cache
ensure assignment of testrun begins an IMMEDIATE tx
construction of the orchestrator struct
initial set of cli args
make sure to not assign testable nodes too often
very initial database structure and cli
fixed construction of RoutableNetworkMonitors
remove redundant constructor for NoiseNode
forbid 0-nonsense config values
add type safety for test route construction
moved lioness and arrayref to workspace deps
fixed dockerfile build
always use canonical addresses in RoutableNetworkMonitors
fixed old contract formatting issues
removed redundant into() call
network monitor agent fixes
additional logs
config unit tests
more docs
standalone stress testing invocation
further refactoring and changes
refactor testing loop and return valid test result upon completion
initial sending/receiving test loop
generating reusable sphinx headers
additional structure for receiving ingress packets
initial scaffolding for NMv3 agent
added validation of x25519 noise key
removed unstable call to 'is_multiple_of'
remove calls to from_octets as they're unavailable in pre 1.91
additional docs/comments
propagating noise information about NM for mixnet routing
pass full socket address of the agent into the contract storage
feat: store noise keys alongside ip addresses within the contract
removed redundant comment
ensure NM packets can only go to NM
PR review comments
added additional docs
allow NM to replay packets + fix replay prometheus metrics
propagate information about nm agent to connection handler
updated nym-node config migration
feat: introduced nym-node websocket subscription for keeping updated list of NM agents
allow admin to also revoke monitor agents
remove agents upon orchestrator removal
fixed schema generation and regenerated the contract schema
removed rustc restriction on contracts-common
added client methods for interacting with the contract
added unit tests for contract methods
implemented logic of the network monitors contract
create initial structure for network monitors contract
start mix stress testing topic branch
* make nym-node default to the new blockstream rpc/ws node cluster
* reduced mixnet-client log severity
* set network monitors contract address for mainnet
* squashing localnet-v2 commits (again)
cargo fmt
fixes to localnet purge
provide path in the error message
output args
log failed exec
print based on tty
check-prerequisites cmd
checked iptables update
basic kernel features check
enable ipv6 rules
add forwarding rules
squashing localnet-v2 commits
additional changes
propagate custom-dns flag to all run containers
remove is_mock from EcashManager
another localnet squash
unused import
chore: remove redundant testnet manager
missing impl
additional linux fixes
command to rebuild container image
wait for at least 2 blocks
additional node startup fixes
added --custom-dns flag to nym node setup
add gateway probe + wait for DKG magic file
fixed localnet down on linux
container ls
re-enable state resync
additional feature locking
macos adjustments
working nyxd startup on linux
wip linux box
wip
separating network inspect betweewn macos and linux
initial linux feature locking
moved all container commands into a single location
finally working initial node performance
squashing orchestrator commits
cleanup
fixed condition for naive rearrangement
added cache of cosmwasm contracts for speed up on subsequent runs
'down' command
refreshing described cache after nodes are bonded
nym nodes setup + wip on nym api refresh
nodes setup WIP
first pass cleanup
placeholder for nym-node setup
bypassing the dkg
further progress on nym-api setup
wip: api setup
up/down/purge placeholders
persisting contract setup data
fix contract upload by forcing amd64 container platform
wip: contracts setup4
wip: contracts setup3
wip: contracts setup2
wip: contracts setup
include network setup
init and spawn nyxd
build nyxd image in dedicated orchestrator
build nyxd image
squashed cherry-picked lp changes
Bits and bobs to make everything work
Title
MacOS setup instructions
Docker/Container localnet
* clippy
* fixes on non-unix targets
---------
Co-authored-by: durch <durch@users.noreply.github.com>
This includes:
- mixnet contract cache
- described nodes cache
- nodes annotations cache (performance)
those changes include taking some code developed for the purposes of #6277
* announce KEM key hashes and use generated value within LpStateMachine
* added digest of remote KEM key into LpSession
* changed constructor to LpSession to take explicit key materials for local and remote
this makes it easier to change keys required by each party without having to change all the interfaces everywhere again
* extended the changes to LpStateMachine constructor
* modify the interface to LpRegistrationHandler and LpListener
* gateway probe fixes
* temp nym-lp-client fixes
* review nits
* remove network test
* introduced v2/nym-nodes/described endpoint for returning nodes description alongside LP data
* missed V1 -> V2 description replacements
* removed deprecated call within mix-fetch
* use old v1 call in network stats
* one commit to rule them all
* remove too aggressive copy pasting
* update details when outdated
* typo and serde alias
* no hostname option and fixes
* fix wasm client?
* non fallback fixed
* improve gateway details update
* better ws addresses
* PR review fixes
* improve type safety on update_gateway_published_data
* fix client gateway storage migration
dont build netstack in CI
additional rust 2024 fixes
fixes
removed temp.rs
first round of cleanup
removed duplicated NS types
moved gateway probe to the monorepo
* remove [most of] legacy data from nym-api endpoints
* chore: removed contamination with legacy nodes data
* added /v1/nym-nodes/stake-saturation/{node_id}
* added /v1/legacy/mixnodes and /v1/legacy/gateways
* removed scraping of legacy mixnodes in NS api
* remove export of removed types
* huge warnings on attempting to use removed commands in the wallet
* fixed reference to removed type in tests
* squashing work on using cancellation in nym crates
making nym-task wasm compilable
removed sending of status messages
replaced TaskManager with ShutdownManager in the validator rewarder
additional helpers for ShutdownManager
simplified ShutdownToken by removing the name field
TaskClient => ShutdownToken within all client tasks
wip: remove TaskHandle
* track all long-living client tasks
* add task tracking for most top level tasks within nym-node
* improved default builder
* split up cancellation module
* module documentation and unit tests
* nym node fixes and naming consistency
* wasm fixes
* assert_eq => assert
* wasm fixes and made 'run_until_shutdown' take reference instead of ownership
* linux-specific fixes to IpPacketRouter
* post rebasing fixes for signing monitor
* add ShutdownManager constructor to build it from an external token
* applying PR review suggestions
* make nym-api use ShutdownToken instead of TaskClient
* ignore public-api tests if env is not set
* removed default features to avoid pulling in openssl
* wip: changes to surb logic + stronger db typing
* surb invalidation logic
* chore: remove unused deps
* resolving todos
* a lot of additional bugfixes
* 1.88 clippy
* wasm fixes
* wasm clippy
* wallet clippy
* wait for epoch end when setting up new network
* split ReplyController into Sender and Receiver for easier reasoning
* additional reply surbs improvements
includes, but is not limited to: unconditionally reseting sender tag on restart, limiting number of surb re-requests, resetting stale surbs on load
* fixed calculation of number of removed surbs
* add additional calculated field to key rotation info
* DBG: 'request_reply_surbs_for_queue_clearing' temp logs
* fixes for silly mistakes
* conditionally reduce log severity
* renamed nym-api config fields
* decouple rewarder startup from network monitor
* additional sections in nym-api config
* removed vesting queries in circulating supply calculator
* added memoized field for last submitted performance measurement
* wip: performance contract refresher
* cleaned up various contract caches
* modified cache refresher to allow passing update fn
* implement performance cache refreshing
* updated lefthook.yml to run cargo fmt
* impl NodePerformanceProvider trait
* dynamically using specific performance provider
* pre warm up performance contract cache and forbid the mode if its empty
* clippy
* introduce fallback setting for performance contract if value for given epoch is not available
* move some functions around
* wip
* wip: wrap node's sphinx key with a manager
* wip: choosing correct key for packet processing
* further propagation of key rotation information
* attaching key rotation information to reply surbs
* added basic key rotation information to mixnet contract
* wip: introducing cached queries for key rotation info from nym api
* unified nym-api contract cache refreshing
* finish packet decoding
* multi api client + retrieving rotation id
* rotating sphinx key files
* logic for migrating config file
* wip: putting new sphinx keys to self described endpoints
* processing loop of KeyRotationController
* fixed sphinx key loading
* rotating bloomfilters
* wired up KeyRotationController
* flushing bloomfilters to disk and loading
* most of nym-node changes
* post rebase fixes
* fixes due to backwards compatible hostkeys
* split http state.rs file
* dont use deprecated fields
* fixed backwards compatible deserialisation of host information
* split up node describe cache
* added a dedicated CacheRefresher listener to perform full refresh outside the set interval
* controlling announced sphinx keys within nym-api
* retrieving rotation id when pulling topology
* split nym-nodes http handlers
* v2 nym-api endpoints to retrieve nodes with additional metadata information
* bug fixes...
* additional bugfixes and guards against stuck epoch
* testnet manager: set first nym-api as the rewarder
* fixed host information deserialisation
* fixed panic during first key rotation
* post rebase fixes
* clippy
* more guards against stuck epochs
* added helper method to reset node's sphinx key
* instantiate mixnet contract with custom key rotation validity
* additional bugfixes and debugging nym-api deadlock
* passing shutdown to nym apis client
* remove dead test
* post rebasing fixes
* missing MixnetQueryClient variants
* remove usage of deprecated methods in sdk example
* fix: incorrect method signature
* post rebasing fixes
* attempt to retrieve key rotation id before doing any config migration work
* ignore tests relying on networking behaviour
* allow networking failures in certain tests
* removed support for legacy packet types from NymCodec
I think nodes had plenty of time to upgrade given versioned variant was introduced in 2022
* temp: use local sphinx packet for development
* introduce new messages that use more efficient reply surbs encoding
* checks for incorrect encoding
* generate correct message depending on config value
* fixed current packet version
* made packet type selection configurable
* updated sphinx packet crate to the published version
* fixed wasm build
* fixes in outfox due to sphinx api changes
* additional tests
* clippy
* fixed log/tracing import
* ignore legacy nodes for test route selection and bias selection with existing score
* feature: dont keep persistent GatewayClient inside NMv1 (#5211)
* removed overly complex logic for requesting mutex permits for packet processing
* dont keep persistent gateway connections. instead make them on demand
* clippy
* added statement logging to nym-api db
* adding additional pool options
* dont blow up upon failing to submit network monitor results
* store in-memory cache of node database ids
WIP; rebasing
Another branch squash
Squashing the v3 branch
changing min pledge amounts
logic for adding new nymnode into the contract
converting mixnode/gateway bonding into nym-node bonding
logic for migrating gateways into nymnodes
ibid for mixnodes
further nym-node work + fixed most existing unit tests
forbid nymnode migration with pending cost params changes
preassign nodeid for gateways
changing role assignment and epoch progression
changing role assignment and epoch progression
optional custom http port
logic for unbonding a nym-node
updating Delegation struct
logic for increasing pledge of either mixnode or nymnode
logic for decreasing pledge of either mixnode or a nym node
logic for changing cost params of either mixnode or a nym node
wip
initialise nymnodes storage
fixing transaction tests
fixed naive family tests
reward-compatibility related works
resolving delegation events
introduced rewarded set metadata
another iteration of restoring old tests
updated rewarding part of nym-api
parking the branch
unparking the branch
wip
purged families
added 'ExitGateway' role
passing explicit work factor for rewarding function
remove legacy layers storage
wip: node description queries
added announced ports to self-described api
step1 in gruelling journey of adding node_id to gateways
ensure epoch work never goes above 1.0
changed active set to contain role distribution
[theoretically] sending rewarding messages for the new rewarded set
[theoretically] assigning new rewarded set
reimplementing more nym-api features
remove legacy types
re-implement legacy network monitor
restoring further routes + minor refactor of NodeStatusCache
skimmed routes now return legacy nodes alongside nym-nodes
seemingly restored all functionalities in nym-api
removing more legacy things from the contract
initial contract cleanup
added nym-api endpoints to return generic annotations regardless of type
updated simulator to use new rewarding parameters
more contract cleanup
made existing mixnet contract tests compile
extra validation of nym-node bonding parameters
fixed additional compilation issues
fixed nym-api v3 database migration failure
added additional nym-node contract queries
updated the schema
made additional delegation/rewards queries compatible with both legacy mixnodes and nym-nodes
fixing existing unit tests in mixnet contract
wip
resolved first batch of 500 compiler errors
re-deprecating routes
making wallet's rust backend compile
fixed non-determinism in contract + nym-api build
fixes to the build
populating cotracts-cache with nym-nodes data
more missing nymnodes queries
temp mixnet contract methods + restored result submission in nym-api
allow deprecated routes
submitting correct results for mixnode results
removed deprecated re-export of AxumAppState and removed smurf naming
moved axum modules into support::http
cleaning up nym-api warnings
determine entry gateways before exits
exposed transaction to update nym-node config
missing memo for updating node config
new routes
added routes to swagger and fixed relative paths
fixed some macro derivations
added nym-node commands to nym-cli
* Change mixnode, gateway, network and nym-api binary licenses to GPLv3
* Add license to cargo files
* Revert model license to match crate
* Add license to nym-node
* Revert model license to match crate
* Fix formatting
---------
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
* changed NymConfigTemplate trait to call 'template' by reference
* network requester lib
* introduced generic parameter to 'MixTrafficController' to allow non-remote gateways
* allowing for custom gateway sender
* types cleanup
* minor gateway cmds refactor + initial NR work
* wip
* running a NR inside gateway
note: this NR isnt tied to the gateway yet
* rebase fixes
* propagating same shutdown handle
* wip
* starting NR with appropriate local transceiver
* fixed premature shutdown
* wiring up PacketRouter
* both ends wired together
* actually working
so much cleanup to do now
* started removing dead code
* wip
* temp: hardcode gateway
* further cleanup
* fixed build of other binaries
* setup-network-requester subcmd
* overriding NR config in gateway init/run
* wip making it wasm-compatible [again]
* refactored 'GatewaySetup'
* clippy and friends
* removed debug code
* rust 1.72 lints
* ensuring local gateway is available + some comments
* correctly putting network requester data in the same underlying details struct
* improved gateway errors
* changed 'network_requester_config' deserialization
* missing clap annotation for 'enabled' flag in 'setup-network-requester' command
* saving config file after 'setup-network-requester'
* removed dead code
* review comments
* make embedded NR wait for gateway to come online (for at most 70min)
* fixed shutdown on successful gateway wait
* updated NR config override
Jędrzej Stuczyński