Compare commits

...

6 Commits

Author SHA1 Message Date
durch ba27d1cdcd Bump Node status API version 2025-07-07 14:36:23 +02:00
durch 401f8d1fd2 Bump Node status API version 2025-07-07 14:32:53 +02:00
durch 038234e5de feat(nym-node-status): implement primary/secondary server architecture
- Agent now requests testruns only from primary server (first in list)
- Results are submitted to all configured servers in parallel
- Secondary servers accept external testruns via new v2 endpoint
- Added auto-creation of gateway and testrun records on secondary servers
- New database queries: get_or_create_gateway, insert_external_testrun
- Client library enhanced with submit_results_with_context method
2025-07-04 21:59:16 +02:00
durch be403c6ee8 feat(nym-node-status-agent): add multi-API support with random selection
Agents can now connect to multiple APIs and randomly select one for each testrun:
- Accept multiple --server arguments in format "address:port:auth_key"
- Randomly shuffle server list before attempting connections
- Try each server until a testrun is obtained
- Submit results back only to the API that provided the testrun
- Continue to next server if one is down or has no testruns available
2025-07-04 14:51:42 +02:00
durch 0b51d98e2b feat(nym-node-status-api): add PostgreSQL database support via feature flags
Implement dual database support for SQLite and PostgreSQL through Cargo feature flags.
The implementation uses a query wrapper that automatically converts SQLite-style ?
placeholders to PostgreSQL-style $1, $2, ... placeholders at runtime.

Key changes:
- Add query wrapper functions that handle placeholder conversion
- Convert all sqlx::query\! macros to use wrapper functions
- Handle type conversions between databases (i64 vs i32)
- Add feature-gated implementations for database-specific SQL syntax
- Update Makefile with clippy targets for both database features
- Document database support in README
2025-07-04 00:26:04 +02:00
durch 76d749baaf feat(db): add SQL query wrapper for PostgreSQL placeholder conversion
- Created query_wrapper module with functions to automatically convert
  SQLite ? placeholders to PostgreSQL $1, $2, ... format
- Updated build.rs to handle mutually exclusive feature flags
- Modified one query in mixnodes.rs as proof of concept
- Added type conversions for PostgreSQL compatibility (u32->i64, u16->i32)

This is a checkpoint commit before converting all queries to use the wrapper.
2025-07-03 22:52:46 +02:00
96 changed files with 3487 additions and 2570 deletions
+686
View File
@@ -0,0 +1,686 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Nym is a privacy platform that uses mixnet technology to protect against metadata surveillance. The platform consists of several key components:
- Mixnet nodes (mixnodes) for packet mixing
- Gateways (entry/exit points for the network)
- Clients for interacting with the network
- Network monitoring tools
- Validators for network consensus
- Various service providers and integrations
## Build Commands
### Rust Components
```bash
# Default build (debug)
cargo build
# Release build
cargo build --release
# Build a specific package
cargo build -p <package-name>
# Build main components
make build
# Build release versions of main binaries and contracts
make build-release
# Build specific binaries
make build-nym-cli
cargo build -p nym-node --release
cargo build -p nym-api --release
```
### Testing
```bash
# Run clippy, unit tests, and formatting
make test
# Run all tests including slow tests
make test-all
# Run clippy on all workspaces
make clippy
# Run unit tests for a specific package
cargo test -p <package-name>
# Run only expensive/ignored tests
cargo test --workspace -- --ignored
# Run API tests
dotenv -f envs/sandbox.env -- cargo test --test public-api-tests
# Run tests with specific log level
RUST_LOG=debug cargo test -p <package-name>
# Run specific test scripts
./nym-node/tests/test_apis.sh
./scripts/wireguard-exit-policy/exit-policy-tests.sh
```
### Linting and Formatting
```bash
# Run rustfmt on all code
make fmt
# Check formatting without modifying
cargo fmt --all -- --check
# Run clippy with all targets
cargo clippy --workspace --all-targets -- -D warnings
# TypeScript linting
yarn lint
yarn lint:fix
yarn types:lint:fix
# Check dependencies for security/licensing issues
cargo deny check
```
### WASM Components
```bash
# Build all WASM components
make sdk-wasm-build
# Build TypeScript SDK
yarn build:sdk
npx lerna run --scope @nymproject/sdk build --stream
# Build and test WASM components
make sdk-wasm
# Build specific WASM packages
cd wasm/client && make
cd wasm/mix-fetch && make
cd wasm/node-tester && make
```
### Contract Development
```bash
# Build all contracts
make contracts
# Build contracts in release mode
make build-release-contracts
# Generate contract schemas
make contract-schema
# Run wasm-opt on contracts
make wasm-opt-contracts
# Check contracts with cosmwasm-check
make cosmwasm-check-contracts
```
### Running Components
```bash
# Run nym-node as a mixnode
cargo run -p nym-node -- run --mode mixnode
# Run nym-node as a gateway
cargo run -p nym-node -- run --mode gateway
# Run the network monitor
cargo run -p nym-network-monitor
# Run the API server
cargo run -p nym-api
# Run with specific environment
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Start a local network
./scripts/localnet_start.sh
```
## Architecture
The Nym platform consists of various components organized as a monorepo:
1. **Core Mixnet Infrastructure**:
- `nym-node`: Core binary supporting mixnode and gateway modes
- `common/nymsphinx`: Implementation of the Sphinx packet format
- `common/topology`: Network topology management
- `common/types`: Shared data types across components
2. **Network Monitoring**:
- `nym-network-monitor`: Monitors the network's reliability and performance
- `nym-api`: API server for network stats and monitoring data
- Metrics tracking for nodes, routes, and overall network health
3. **Client Implementations**:
- `clients/native`: Native Rust client implementation
- `clients/socks5`: SOCKS5 proxy client for standard applications
- `wasm`: WebAssembly client implementations (for browsers)
- `nym-connect`: Desktop and mobile clients
4. **Blockchain & Smart Contracts**:
- `common/cosmwasm-smart-contracts`: Smart contract implementations
- `contracts`: CosmWasm contracts for the Nym network
- `common/ledger`: Blockchain integration
5. **Utilities & Tools**:
- `tools`: Various CLI tools and utilities
- `sdk`: SDKs for different languages and platforms
- `documentation`: Documentation generation and management
## Packet System
Nym uses a modified Sphinx packet format for its mixnet:
1. **Message Chunking**:
- Messages are divided into "sets" and "fragments"
- Each fragment fits in a single Sphinx packet
- The `common/nymsphinx/chunking` module handles message fragmentation
2. **Routing**:
- Packets traverse through 3 layers of mixnodes
- Routing information is encrypted in layers (onion routing)
- The final gateway receives and processes the messages
3. **Monitoring**:
- Monitoring system tracks packet delivery through the network
- Routes are analyzed for reliability statistics
- Node performance metrics are collected
## Network Protocol
Nym implements the Loopix mixnet design with several key privacy features:
1. **Continuous-time Mixing**:
- Each mixnode delays messages independently with an exponential distribution
- This creates random reordering of packets, destroying timing correlations
- Offers better anonymity properties than batch mixing approaches
2. **Cover Traffic**:
- Clients and nodes generate dummy "loop" packets that circulate through the network
- These packets are indistinguishable from real traffic
- Creates a baseline level of traffic that hides actual communication patterns
- Provides unobservability (hiding when and how much real traffic is being sent)
3. **Stratified Network Architecture**:
- Traffic flows through Entry Gateway → 3 Mixnode Layers → Exit Gateway
- Path selection is independent per-message (unlike Tor)
- Each node connects only to adjacent layers
4. **Anonymous Replies**:
- Single-Use Reply Blocks (SURBs) allow receiving messages without revealing identity
- Enables bidirectional communication while maintaining privacy
## Network Monitoring Architecture
The network monitoring system is a core component that measures mixnet reliability:
1. The `nym-network-monitor` sends test packets through the network
2. These packets follow predefined routes through multiple mixnodes
3. Metrics are collected about:
- Successful and failed packet deliveries
- Node reliability (percentage of successful packet handling)
- Route reliability (which specific route combinations work best)
4. Results are stored in the database and used by `nym-api` to:
- Present node performance statistics
- Determine network rewards
- Provide route selection guidance to clients
In the current branch, metrics collection is being enhanced with a fanout approach to submit to multiple API endpoints.
## Development Environment
### Required Dependencies
- Rust toolchain (stable, 1.80+)
- Node.js (v20+) and yarn for TypeScript components
- SQLite for local database development
- PostgreSQL for API database (optional, for full API functionality)
- CosmWasm tools for contract development
- For building contracts: `wasm-opt` tool from `binaryen`
- Python 3.8+ for some scripts
- Docker (optional, for containerized development)
- protoc (Protocol Buffers compiler) for some components
### Environment Configurations
The `envs/` directory contains pre-configured environments:
#### Available Environments
- **`local.env`**: Local development environment
- Points to local services (localhost)
- Uses test mnemonics and keys
- Ideal for testing without external dependencies
- **`sandbox.env`**: Sandbox test network
- Public test network with real nodes
- Test tokens available from faucet
- Contract addresses for sandbox deployment
- API: https://sandbox-nym-api1.nymtech.net
- **`mainnet.env`**: Production mainnet
- Real network with real tokens
- Production contract addresses
- API: https://validator.nymtech.net
- Use with caution!
- **`canary.env`**: Canary deployment
- Pre-release testing environment
- Tests new features before mainnet
- **`mainnet-local-api.env`**: Hybrid environment
- Uses mainnet contracts but local API
- Useful for API development against mainnet data
#### Key Environment Variables
```bash
# Network configuration
NETWORK_NAME=sandbox # Network identifier
BECH32_PREFIX=n # Address prefix (n for sandbox, n for mainnet)
NYM_API=https://sandbox-nym-api1.nymtech.net/api
NYXD=https://rpc.sandbox.nymtech.net
NYM_API_NETWORK=sandbox
# Contract addresses (network-specific)
MIXNET_CONTRACT_ADDRESS=n1xr3rq8yvd7qplsw5yx90ftsr2zdhg4e9z60h5duusgxpv72hud3sjkxkav
VESTING_CONTRACT_ADDRESS=n1unyuj8qnmygvzuex3dwmg9yzt9alhvyeat0uu0jedg2wj33efl5qackslz
# ... other contract addresses
# Mnemonic for testing (NEVER use in production)
MNEMONIC="clutch captain shoe salt awake harvest setup primary inmate ugly among become"
# API Keys and tokens
IPINFO_API_TOKEN=your_token_here
AUTHENTICATOR_PASSWORD=password_here
# Logging
RUST_LOG=info # Options: error, warn, info, debug, trace
RUST_BACKTRACE=1 # Enable backtraces
# Database
DATABASE_URL=postgresql://user:pass@localhost/nym_api
```
#### Using Environment Files
```bash
# Load environment and run command
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Export to shell
source envs/sandbox.env
# Use with make targets
dotenv -f envs/sandbox.env -- make run-api-tests
```
## Initial Setup
### First Time Setup
1. **Install Prerequisites**
```bash
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install Node.js and yarn
# Via nvm (recommended):
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
nvm install 20
npm install -g yarn
# Install build tools
# Ubuntu/Debian:
sudo apt-get install build-essential pkg-config libssl-dev protobuf-compiler libpq-dev
# macOS:
brew install protobuf postgresql
# Install wasm-opt for contract builds
npm install -g wasm-opt
# Add wasm target for Rust
rustup target add wasm32-unknown-unknown
```
2. **Clone and Setup Repository**
```bash
git clone https://github.com/nymtech/nym.git
cd nym/nym
# Install JavaScript dependencies
yarn install
# Build the project
make build
```
3. **Database Setup (Optional, for API development)**
```bash
# Install PostgreSQL
# Create database
createdb nym_api
# Run migrations (from nym-api directory)
cd nym-api
sqlx migrate run
```
### Quick Start
```bash
# Run a mixnode locally
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode mixnode --id my-mixnode
# Run a gateway locally
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode gateway --id my-gateway
# Run the API server
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Run a client
cargo run -p nym-client -- init --id my-client
cargo run -p nym-client -- run --id my-client
```
## CI/CD Pipeline
The project uses GitHub Actions for CI/CD with several key workflows:
1. **Build and Test**:
- `ci-build.yml`: Main build workflow for Rust components
- Tests are run on multiple platforms (Linux, Windows, macOS)
- Includes formatting check (rustfmt) and linting (clippy)
2. **Release Process**:
- Binary artifacts are published on release tags
- Multiple platform builds are created
3. **Documentation**:
- Documentation is automatically built and deployed
## Database Structure
The system uses SQLite databases with tables like:
- `mixnode_status`: Status information about mixnodes
- `gateway_status`: Status information about gateways
- `routes`: Route performance information (success/failure of specific paths)
- `monitor_run`: Information about monitoring test runs
## Development Workflows
### Running a Node
To run the mixnode or gateway:
```bash
# Run nym-node as a mixnode with specified identity
cargo run -p nym-node -- run --mode mixnode --id my-mixnode
# Run nym-node as a gateway
cargo run -p nym-node -- run --mode gateway --id my-gateway
```
### Configuration
Nodes can be configured with files in various locations:
- Command-line arguments
- Environment variables
- `.env` files specified with `--config-env-file`
### Monitoring
To monitor the health of your node:
- View logs for real-time information
- Use the node's HTTP API for status information
- Check the explorer for public node statistics
## Common Libraries
- `common/types`: Shared data types across all components
- `common/crypto`: Cryptographic primitives and wrappers
- `common/client-core`: Core client functionality
- `common/gateway-client`: Client-gateway communication
- `common/task`: Task management and concurrency utilities
- `common/nymsphinx`: Sphinx packet implementation for mixnet
- `common/topology`: Network topology management
- `common/credentials`: Credential system for privacy-preserving authentication
- `common/bandwidth-controller`: Bandwidth management and accounting
## Code Conventions
- Error handling: Use anyhow/thiserror for structured error handling
- Logging: Use the tracing framework for logging and diagnostics
- State management: Generally use Tokio/futures for async code
- Configuration: Use the config crate and env vars with defaults
- Database: Use sqlx for type-safe database queries
- Follow clippy recommendations and rustfmt formatting
- Use semantic commit messages: feat, fix, docs, refactor, test, chore
## When Making Changes
- Run `make test` before submitting PRs
- Follow Rust naming conventions
- Use `clippy` to check for common issues
- Update SQLx query caches when modifying DB queries: `cargo sqlx prepare`
- Consider backward compatibility for protocol changes
- Use lefthook pre-commit hooks for TypeScript formatting
- Run `cargo deny check` to verify dependency compliance
- Test against both sandbox and local environments when possible
- Update relevant documentation and CHANGELOG.md
## Development Tools
### Useful Cargo Commands
```bash
# Check for outdated dependencies
cargo outdated
# Analyze binary size
cargo bloat --release -p nym-node
# Generate dependency graph
cargo tree -p nym-api
# Run with instrumentation
cargo run --features profiling -p nym-node
# Check for security advisories
cargo audit
```
### Database Tools
```bash
# SQLx CLI for migrations
cargo install sqlx-cli
# Create new migration
cd nym-api && sqlx migrate add <migration_name>
# Prepare query metadata for offline compilation
cargo sqlx prepare --workspace
# View database schema
./nym-api/enter_db.sh
```
### Development Scripts
- `scripts/build_topology.py`: Generate network topology files
- `scripts/node_api_check.py`: Verify node API endpoints
- `scripts/network_tunnel_manager.sh`: Manage network tunnels
- `scripts/localnet_start.sh`: Start a local test network
- Various deployment scripts in `deployment/` for different environments
## Debugging
- Enable more verbose logging with the RUST_LOG environment variable:
```
RUST_LOG=debug,nym_node=trace cargo run -p nym-node -- run --mode mixnode
```
- Use the HTTP API endpoints for status information
- Check monitoring data in the database for network performance metrics
- For complex issues, use tracing tools to follow packet flow
- Enable backtraces: `RUST_BACKTRACE=full`
- For WASM debugging: Use browser developer tools with source maps
## Deployment and Advanced Configurations
### Deployment Structure
The `deployment/` directory contains Ansible playbooks and configurations for various deployment scenarios:
- **`aws/`**: AWS-specific deployment configurations
- **`mixnode/`**: Mixnode deployment playbooks
- **`gateway/`**: Gateway deployment playbooks
- **`validator/`**: Validator node deployment
- **`sandbox-v2/`**: Complete sandbox environment setup
- **`big-dipper-2/`**: Block explorer deployment
### Sandbox V2 Deployment
The sandbox-v2 deployment (`deployment/sandbox-v2/`) provides a complete test environment:
```bash
# Key playbooks:
- deploy.yaml # Main deployment orchestrator
- deploy-mixnodes.yaml # Deploy mixnodes
- deploy-gateways.yaml # Deploy gateways
- deploy-validators.yaml # Deploy validator nodes
- deploy-nym-api.yaml # Deploy API services
```
### Custom Environment Setup
To create a custom environment:
1. Copy an existing env file: `cp envs/sandbox.env envs/custom.env`
2. Modify the network endpoints and contract addresses
3. Update the `NETWORK_NAME` to your identifier
4. Set appropriate mnemonics and keys (use fresh ones for production!)
### Contract Addresses
Contract addresses are network-specific and defined in environment files:
- Mixnet contract: Manages mixnode/gateway registry
- Vesting contract: Handles token vesting schedules
- Coconut contracts: Privacy-preserving credentials
- Name service: Human-readable address mapping
- Ecash contract: Electronic cash functionality
### Local Network Setup
For a completely local network:
```bash
# Start local chain
./scripts/localnet_start.sh
# Deploy contracts
cd contracts
make deploy-local
# Start nodes with local config
dotenv -f envs/local.env -- cargo run -p nym-node -- run --mode mixnode
```
## Common Issues and Troubleshooting
### Database Issues
- When modifying database queries, you must update SQLx query caches:
```bash
cargo sqlx prepare
```
- If you see SQLx errors about missing query files, this is likely the cause
- For "database is locked" errors with SQLite, ensure only one process accesses the DB
- For PostgreSQL connection issues, verify DATABASE_URL and that the server is running
### API Connection Issues
- Check the environment variables pointing to the APIs (NYM_API, NYXD)
- Verify network connectivity and API health endpoints
- For authentication issues, check node keys and credentials
- Common endpoints to verify:
- API health: `$NYM_API/health`
- Chain status: `$NYXD/status`
- Contract info: `$NYXD/cosmwasm/wasm/v1/contract/$CONTRACT_ADDRESS`
### Build Problems
- Clean dependencies with `cargo clean` for a fresh build
- Check for compatible Rust version (1.80+ recommended)
- For smart contract builds, ensure wasm-opt is installed: `npm install -g wasm-opt`
- For cross-compilation issues, check target-specific dependencies
- WASM build issues: Ensure wasm32-unknown-unknown target is installed:
```bash
rustup target add wasm32-unknown-unknown
```
- For "cannot find -lpq" errors, install PostgreSQL development files:
```bash
# Ubuntu/Debian
sudo apt-get install libpq-dev
# macOS
brew install postgresql
```
### Environment Issues
- Contract address mismatches: Ensure you're using the correct environment file
- "Account sequence mismatch": The account nonce is out of sync, wait and retry
- Token decimal issues: Sandbox uses different decimal places than mainnet
- API version mismatches: Ensure your local API version matches the network
- "Insufficient funds": Get test tokens from faucet (sandbox) or check balance
- Gateway/mixnode bonding issues: Verify minimum stake requirements
## Working with Routes and Monitoring
1. Route monitoring metrics are stored in a `routes` table with:
- Layer node IDs (layer1, layer2, layer3, gw)
- Success flag (boolean)
- Timestamp
2. To analyze routes:
- Check `NetworkAccount` and `AccountingRoute` in `nym-network-monitor/src/accounting.rs`
- View monitoring logic in `common/nymsphinx/chunking/monitoring.rs`
- Observe how routes are submitted to the database in the `submit_accounting_routes_to_db` function
## Performance Optimization
### Profiling and Benchmarking
```bash
# Run benchmarks
cargo bench -p nym-node
# Profile with perf (Linux)
cargo build --release --features profiling
perf record --call-graph=dwarf ./target/release/nym-node run --mode mixnode
perf report
# Generate flamegraph
cargo install flamegraph
cargo flamegraph --bin nym-node -- run --mode mixnode
```
### Common Performance Considerations
- Use bounded channels for backpressure
- Batch database operations where possible
- Monitor memory usage with `RUST_LOG=nym_node::metrics=debug`
- Use connection pooling for database connections
- Consider using `jemalloc` for better memory allocation performance
Generated
+841 -932
View File
File diff suppressed because it is too large Load Diff
+58
View File
@@ -4,3 +4,61 @@ The Node Status API serves information about individual `nym-nodes` in the Mixne
We recommend that developers building applications such as explorers or analytics interfaces about the Mixnet run their own instance of the API, in order to promote a robust network of downstream services, and spread the load of API calls amongst as many endpoints as possible.
You can find build and operation instructions in the [docs](https://nym.com/docs/apis/ns-api).
## Database Support
The Node Status API supports both SQLite and PostgreSQL databases through Cargo feature flags:
- **SQLite** (default): Lightweight, file-based database suitable for development and small deployments
- **PostgreSQL**: Full-featured database recommended for production deployments
### Building with Different Database Backends
```bash
# Build with SQLite (default)
cargo build --features sqlite --no-default-features
# Build with PostgreSQL
cargo build --features pg --no-default-features
```
### Running Tests
```bash
# Test with SQLite
cargo test --features sqlite --no-default-features
# Test with PostgreSQL
make test-db # This sets up a test PostgreSQL instance
```
### Development Commands
The project includes a Makefile with helpful commands for both database backends:
```bash
# Check code compilation
make check-sqlite # Check with SQLite
make check-pg # Check with PostgreSQL
# Run clippy linter
make clippy-sqlite # Lint with SQLite
make clippy-pg # Lint with PostgreSQL
make clippy # Run both
# PostgreSQL development
make dev-db # Start a PostgreSQL instance for development
make prepare-pg # Prepare SQLx offline cache for PostgreSQL
```
### Implementation Details
The database abstraction is implemented using a query wrapper that automatically converts SQLite-style `?` placeholders to PostgreSQL-style `$1, $2, ...` placeholders at runtime. This allows writing queries once using SQLite syntax while maintaining compatibility with both databases.
Key differences handled:
- Placeholder syntax (`?` vs `$1, $2, ...`)
- Type conversions (SQLite uses i64, PostgreSQL uses i32 for many fields)
- SQL dialect differences (e.g., `INSERT OR IGNORE` vs `ON CONFLICT DO NOTHING`)
- RETURNING clause behavior
For more details on PostgreSQL setup, see [README_PG.md](nym-node-status-api/README_PG.md).
@@ -14,13 +14,22 @@ rust-version.workspace = true
readme.workspace = true
[dependencies]
anyhow = { workspace = true}
anyhow = { workspace = true }
clap = { workspace = true, features = ["derive", "env"] }
nym-bin-common = { path = "../../common/bin-common", features = ["models"]}
futures = { workspace = true }
nym-bin-common = { path = "../../common/bin-common", features = ["models"] }
nym-node-status-client = { path = "../nym-node-status-client" }
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "rand"] }
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"asymmetric",
"rand",
] }
rand = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "process", "fs"] }
tokio = { workspace = true, features = [
"macros",
"rt-multi-thread",
"process",
"fs",
] }
tracing = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter"] }
@@ -1,17 +1,45 @@
use crate::probe::GwProbe;
use clap::{Parser, Subcommand};
use nym_bin_common::bin_info;
use nym_crypto::asymmetric::ed25519::PrivateKey;
use std::sync::OnceLock;
pub(crate) mod generate_keypair;
pub(crate) mod run_probe;
#[derive(Debug)]
pub(crate) struct ServerConfig {
pub(crate) address: String,
pub(crate) port: u16,
pub(crate) auth_key: PrivateKey,
}
// Helper for passing LONG_VERSION to clap
fn pretty_build_info_static() -> &'static str {
static PRETTY_BUILD_INFORMATION: OnceLock<String> = OnceLock::new();
PRETTY_BUILD_INFORMATION.get_or_init(|| bin_info!().pretty_print())
}
fn parse_server_config(s: &str) -> Result<ServerConfig, String> {
let parts: Vec<&str> = s.split(':').collect();
if parts.len() != 3 {
return Err("Server config must be in format 'address:port:auth_key'".to_string());
}
let address = parts[0].to_string();
let port = parts[1]
.parse::<u16>()
.map_err(|_| "Invalid port number".to_string())?;
let auth_key =
PrivateKey::from_base58_string(parts[2]).map_err(|_| "Invalid auth key".to_string())?;
Ok(ServerConfig {
address,
port,
auth_key,
})
}
#[derive(Parser, Debug)]
#[clap(author = "Nymtech", version, long_version = pretty_build_info_static(), about)]
pub(crate) struct Args {
@@ -22,15 +50,10 @@ pub(crate) struct Args {
#[derive(Subcommand, Debug)]
pub(crate) enum Command {
RunProbe {
#[arg(short, long, env = "NODE_STATUS_AGENT_SERVER_ADDRESS")]
server_address: String,
#[arg(short = 'p', long, env = "NODE_STATUS_AGENT_SERVER_PORT")]
server_port: u16,
/// base58-encoded private key
#[arg(long, env = "NODE_STATUS_AGENT_AUTH_KEY")]
ns_api_auth_key: String,
/// Server configurations in format "address:port:auth_key"
/// Can be specified multiple times for multiple servers
#[arg(short, long, required = true)]
server: Vec<String>,
/// path of binary to run
#[arg(long, env = "NODE_STATUS_AGENT_PROBE_PATH")]
@@ -54,22 +77,28 @@ impl Args {
pub(crate) async fn execute(&self) -> anyhow::Result<()> {
match &self.command {
Command::RunProbe {
server_address,
server_port,
ns_api_auth_key,
server,
probe_path,
probe_extra_args,
} => run_probe::run_probe(
server_address,
server_port.to_owned(),
ns_api_auth_key,
probe_path,
probe_extra_args,
)
.await
.inspect_err(|err| {
tracing::error!("{err}");
})?,
} => {
// Parse server configs
let mut servers = Vec::new();
for s in server {
match parse_server_config(s) {
Ok(config) => servers.push(config),
Err(e) => {
tracing::error!("Invalid server config '{}': {}", s, e);
anyhow::bail!("Invalid server config '{}': {}", s, e);
}
}
}
run_probe::run_probe(&servers, probe_path, probe_extra_args)
.await
.inspect_err(|err| {
tracing::error!("{err}");
})?
}
Command::GenerateKeypair { path } => {
let path = path
.to_owned()
@@ -1,33 +1,117 @@
use crate::cli::GwProbe;
use anyhow::Context;
use nym_crypto::asymmetric::ed25519::PrivateKey;
use crate::cli::{GwProbe, ServerConfig};
pub(crate) async fn run_probe(
server_ip: &str,
server_port: u16,
ns_api_auth_key: &str,
servers: &[ServerConfig],
probe_path: &str,
probe_extra_args: &Vec<String>,
) -> anyhow::Result<()> {
let auth_key = PrivateKey::from_base58_string(ns_api_auth_key)
.context("Couldn't parse auth key, exiting")?;
let ns_api_client = nym_node_status_client::NsApiClient::new(server_ip, server_port, auth_key);
if servers.is_empty() {
anyhow::bail!("No servers configured");
}
let probe = GwProbe::new(probe_path.to_string());
let version = probe.version().await;
tracing::info!("Probe version:\n{}", version);
if let Some(testrun) = ns_api_client.request_testrun().await? {
let log = probe.run_and_get_log(&Some(testrun.gateway_identity_key), probe_extra_args);
// Always use first server as primary
let primary_server = &servers[0];
tracing::info!(
"Requesting testrun from primary server: {}:{}",
primary_server.address,
primary_server.port
);
ns_api_client
.submit_results(testrun.testrun_id, log, testrun.assigned_at_utc)
.await?;
} else {
tracing::info!("No testruns available, exiting")
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
&primary_server.auth_key.to_bytes(),
)
.expect("Failed to clone auth key");
let ns_api_client = nym_node_status_client::NsApiClient::new(
&primary_server.address,
primary_server.port,
auth_key,
);
match ns_api_client.request_testrun().await {
Ok(Some(testrun)) => {
tracing::info!(
"Received testrun {} for gateway {} from primary",
testrun.testrun_id,
testrun.gateway_identity_key
);
// Run the probe
let log = probe.run_and_get_log(
&Some(testrun.gateway_identity_key.clone()),
probe_extra_args,
);
// Submit to ALL servers in parallel
let handles = servers
.iter()
.enumerate()
.map(|(idx, server)| {
let testrun = testrun.clone();
let log = log.clone();
async move {
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
&server.auth_key.to_bytes(),
)
.expect("Failed to clone auth key");
let client = nym_node_status_client::NsApiClient::new(
&server.address,
server.port,
auth_key,
);
let result = client
.submit_results_with_context(
testrun.testrun_id,
log,
testrun.assigned_at_utc,
testrun.gateway_identity_key,
)
.await;
(idx, server.address.clone(), server.port, result)
}
})
.collect::<Vec<_>>();
let results = futures::future::join_all(handles).await;
for result in results {
match result.3 {
Ok(()) => {
tracing::info!(
"✅ Successfully submitted to server[{}] {}:{}",
result.0,
result.1,
result.2
);
}
Err(e) => {
tracing::warn!(
"❌ Failed to submit to server[{}] {}:{} - {}",
result.0,
result.1,
result.2,
e
);
}
}
}
Ok(())
}
Ok(None) => {
tracing::info!("No testruns available from primary server");
Ok(())
}
Err(e) => {
tracing::error!("Failed to contact primary server: {}", e);
Err(e)
}
}
Ok(())
}
@@ -0,0 +1,32 @@
# Example environment variables for nym-node-status-api
# Database configuration
# For SQLite:
# DATABASE_URL=sqlite://nym-node-status-api.sqlite
# For PostgreSQL:
# DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# Network configuration
NETWORK_NAME=sandbox
NYM_API=https://sandbox-nym-api1.nymtech.net/api
NYXD=https://rpc.sandbox.nymtech.net
# API configuration
NYM_NODE_STATUS_API_HTTP_PORT=8000
NYM_API_CLIENT_TIMEOUT=15
SQLX_BUSY_TIMEOUT_S=5
# Monitoring intervals
NODE_STATUS_API_MONITOR_REFRESH_INTERVAL=300
NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL=300
NODE_STATUS_API_GEODATA_TTL=86400
# Agent keys (comma-separated list)
NODE_STATUS_API_AGENT_KEY_LIST=
# External service tokens
IPINFO_API_TOKEN=your_token_here
# MixNodes (Optional)
DATA_PROVIDER_DELEGATION_LIST=
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_daily_stats (\n mix_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,\n packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,\n packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "01ee4a30bc3104712e5bc371a45d614a89d88adf02358800433e06100c13c548"
}
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT mix_id as node_id, host, http_api_port\n FROM mixnodes\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "host",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "http_api_port",
"ordinal": 2,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false
]
},
"hash": "021c6c65d1ed806d8430bef7883906b42a7e4b280c8efb32db15d7c6a51d7a27"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_description (\n mix_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (mix_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "06065394c157927e4002ddd5c7c1af626ae15728d615f539470cd7c189312385"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n gateway_identity_key\n FROM\n gateways\n WHERE\n id = ?",
"describe": {
"columns": [
{
"name": "gateway_identity_key",
"ordinal": 0,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "06b17d1e5f61201a1b7542896ba55c69cd5c1a7e7d87073c94600c783a0a3984"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n testruns\n SET\n status = ?\n WHERE\n status = ?\n AND\n last_assigned_utc < ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "0cf0e4d4f30e90caecffd6255ef85dab12730e538be194438f19ed7f198bd50e"
}
@@ -1,43 +1,43 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "\n SELECT\n date_utc as \"date_utc!\",\n SUM(total_stake) as \"total_stake!: i64\",\n SUM(packets_received) as \"total_packets_received!: i64\",\n SUM(packets_sent) as \"total_packets_sent!: i64\",\n SUM(packets_dropped) as \"total_packets_dropped!: i64\"\n FROM (\n SELECT\n date_utc,\n n.total_stake,\n n.packets_received,\n n.packets_sent,\n n.packets_dropped\n FROM nym_node_daily_mixing_stats n\n UNION ALL\n SELECT\n m.date_utc,\n m.total_stake,\n m.packets_received,\n m.packets_sent,\n m.packets_dropped\n FROM mixnode_daily_stats m\n LEFT JOIN nym_node_daily_mixing_stats ON m.mix_id = nym_node_daily_mixing_stats.node_id\n WHERE nym_node_daily_mixing_stats.node_id IS NULL\n )\n GROUP BY date_utc\n ORDER BY date_utc ASC\n ",
"describe": {
"columns": [
{
"name": "date_utc!",
"ordinal": 0,
"type_info": "Text"
"name": "date_utc!",
"type_info": "Varchar"
},
{
"name": "total_stake!: i64",
"ordinal": 1,
"type_info": "Integer"
"name": "total_stake!: i64",
"type_info": "Numeric"
},
{
"name": "total_packets_received!: i64",
"ordinal": 2,
"type_info": "Integer"
"name": "total_packets_received!: i64",
"type_info": "Int8"
},
{
"name": "total_packets_sent!: i64",
"ordinal": 3,
"type_info": "Integer"
"name": "total_packets_sent!: i64",
"type_info": "Int8"
},
{
"name": "total_packets_dropped!: i64",
"ordinal": 4,
"type_info": "Integer"
"name": "total_packets_dropped!: i64",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
false,
true,
true,
true
null,
null,
null,
null,
null
]
},
"hash": "124d45b9604439584650f401607c46bdbd162c7c689f74fe9ddfdfd48f5ddc07"
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n key as \"key!\",\n value_json as \"value_json!\",\n last_updated_utc as \"last_updated_utc!\"\n FROM summary",
"describe": {
"columns": [
{
"name": "key!",
"ordinal": 0,
"type_info": "Text"
},
{
"name": "value_json!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "last_updated_utc!",
"ordinal": 2,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
true,
true,
false
]
},
"hash": "1327b5118f9144dddbcf8edb11f7dc549cf503409fd6dfedcdc02dbcd61d5454"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_packet_stats_raw (\n mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "21e44766729777756f6eb04bf3b81df3e591008a1e3fd664ed83ca86ac51bd8c"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n id,\n gateway_identity_key\n FROM gateways\n WHERE id = ?\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_identity_key",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false
]
},
"hash": "2236299f9f691376db54cbd58ec5ceb89b9925cba46efcf4ed79ef0759a01129"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n AND\n self_described IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true
]
},
"hash": "227539374e7473f6f9642289c5b5d1bcd636315ab23537cb5f6d2f82a2bcb7bf"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT gateway_identity_key\n FROM gateways\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "gateway_identity_key",
"ordinal": 0,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false
]
},
"hash": "25300e435780101fa207c8e26ef2f49ba5db84d63e89440bb494e8327fe73686"
}
@@ -1,86 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n AND\n bond_info IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "ed25519_identity_pubkey",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "total_stake",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "ip_addresses!: serde_json::Value",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "mix_port",
"ordinal": 4,
"type_info": "Integer"
},
{
"name": "x25519_sphinx_pubkey",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "node_role: serde_json::Value",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "supported_roles: serde_json::Value",
"ordinal": 7,
"type_info": "Text"
},
{
"name": "entry: serde_json::Value",
"ordinal": 8,
"type_info": "Text"
},
{
"name": "performance",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 11,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
true,
true
]
},
"hash": "283f49a65c7d70bf271702ff6a5c7ad6e68c81932d295ff18ed198c54706a57c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT OR IGNORE INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
"describe": {
"columns": [],
"parameters": {
"Right": 9
},
"nullable": []
},
"hash": "3243cf5646255a9430d1e6710970505d0dbcc62703f40e090e80ff48c77723c4"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO mixnodes\n (mix_id, identity_key, bonded, total_stake,\n host, http_api_port, full_details,\n self_described, last_updated_utc, is_dp_delegatee)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id) DO UPDATE SET\n bonded=excluded.bonded,\n total_stake=excluded.total_stake, host=excluded.host,\n http_api_port=excluded.http_api_port,\n full_details=excluded.full_details,self_described=excluded.self_described,\n last_updated_utc=excluded.last_updated_utc,\n is_dp_delegatee = excluded.is_dp_delegatee;",
"describe": {
"columns": [],
"parameters": {
"Right": 10
},
"nullable": []
},
"hash": "3cd5cb4bfca4243925da4ddbccd811e842090e98982e1032670df77961870b32"
}
@@ -1,38 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_identity_key as \"gateway_identity_key!\",\n self_described as \"self_described?\",\n explorer_pretty_bond as \"explorer_pretty_bond?\"\n FROM gateways\n WHERE gateway_identity_key = ?\n AND bonded = true\n ORDER BY gateway_identity_key\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_identity_key!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "self_described?",
"ordinal": 2,
"type_info": "Text"
},
{
"name": "explorer_pretty_bond?",
"ordinal": 3,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
true,
false,
false,
true
]
},
"hash": "3e7e987780937873cdb393b157d7708c9f01047b0689eb0d4f7a973b328c609d"
}
@@ -1,92 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n gw.gateway_identity_key as \"gateway_identity_key!\",\n gw.bonded as \"bonded: bool\",\n gw.performance as \"performance!\",\n gw.self_described as \"self_described?\",\n gw.explorer_pretty_bond as \"explorer_pretty_bond?\",\n gw.last_probe_result as \"last_probe_result?\",\n gw.last_probe_log as \"last_probe_log?\",\n gw.last_testrun_utc as \"last_testrun_utc?\",\n gw.last_updated_utc as \"last_updated_utc!\",\n COALESCE(gd.moniker, \"NA\") as \"moniker!\",\n COALESCE(gd.website, \"NA\") as \"website!\",\n COALESCE(gd.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(gd.details, \"NA\") as \"details!\"\n FROM gateways gw\n LEFT JOIN gateway_description gd\n ON gw.gateway_identity_key = gd.gateway_identity_key\n ORDER BY gw.gateway_identity_key",
"describe": {
"columns": [
{
"name": "gateway_identity_key!",
"ordinal": 0,
"type_info": "Text"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "performance!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "self_described?",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "explorer_pretty_bond?",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "last_probe_result?",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_probe_log?",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "last_testrun_utc?",
"ordinal": 7,
"type_info": "Integer"
},
{
"name": "last_updated_utc!",
"ordinal": 8,
"type_info": "Integer"
},
{
"name": "moniker!",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "website!",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "security_contact!",
"ordinal": 11,
"type_info": "Text"
},
{
"name": "details!",
"ordinal": 12,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
true,
true,
true,
true,
false,
false,
false,
false,
false
]
},
"hash": "3eb1d8491bda3c1d6e071b6eb364b9a979f4bdb11ea81b2d0f022555bab51ecb"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n total_stake\n FROM mixnodes\n WHERE mix_id = ?\n ",
"describe": {
"columns": [
{
"name": "total_stake",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "3fc2baabf194b147b20be2a49401cc0c100a1d7a7c347393adde2410fa6f4dfe"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE testruns SET status = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "418944f2eccb838cb3882f34469203c8569f03fdd39ce09d7b74177896e52a8c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_probe_log = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "4afcc6673890f795c2793f1e2f8570ee787fc7daf00fcb916f18d1cb7d6c8f08"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "4d865e873c9cb133883da94db72dcdebd4969e1f240def9fb1bf946f4a1f342f"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)\n ON CONFLICT DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Int8",
"Date",
"Int8",
"Int8",
"Varchar",
"Varchar",
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "4fca38abbb416d9457c34a8ba4faf481a837eda4f3e1bee1d430a4eb102a5b3d"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_node_daily_mixing_stats (\n node_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,\n packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,\n packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "5912ea335a957d217f5e2b3a63a25b31715c2098310fe7a9db688bc2fd36aad4"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE testruns\n SET\n status = ?,\n last_assigned_utc = ?\n WHERE rowid =\n (\n SELECT rowid\n FROM testruns\n WHERE status = ?\n ORDER BY created_utc asc\n LIMIT 1\n )\n RETURNING\n id as \"id!\",\n gateway_id\n ",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id",
"ordinal": 1,
"type_info": "Integer"
}
],
"parameters": {
"Right": 3
},
"nullable": [
false,
false
]
},
"hash": "5e9cbb39f5fb53774803270f422989e199aac4d4a71913c7074359b4bd676b02"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO nym_nodes\n (node_id, ed25519_identity_pubkey,\n total_stake,\n ip_addresses, mix_port,\n x25519_sphinx_pubkey, node_role,\n supported_roles, entry,\n self_described,\n bond_info,\n performance, last_updated_utc\n )\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id) DO UPDATE SET\n ed25519_identity_pubkey=excluded.ed25519_identity_pubkey,\n ip_addresses=excluded.ip_addresses,\n mix_port=excluded.mix_port,\n x25519_sphinx_pubkey=excluded.x25519_sphinx_pubkey,\n node_role=excluded.node_role,\n supported_roles=excluded.supported_roles,\n entry=excluded.entry,\n self_described=excluded.self_described,\n bond_info=excluded.bond_info,\n performance=excluded.performance,\n last_updated_utc=excluded.last_updated_utc\n ;",
"describe": {
"columns": [],
"parameters": {
"Right": 13
},
"nullable": []
},
"hash": "664e059ac2c58e1115fe214376a6b326b31c93298f20019772cce2e277a194f8"
}
@@ -1,19 +1,19 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT count(id) FROM mixnodes",
"describe": {
"columns": [
{
"name": "count(id)",
"ordinal": 0,
"type_info": "Integer"
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false
null
]
},
"hash": "670b7ed7d57a6986181b24be24ca667e8cacdf677ccb906415b3fe92be0c436b"
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_node_descriptions (\n node_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (node_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "6a9780aad1f2f0c8ef780e51fe856679d5e28f95143f4e2a2b409009dc0f55ba"
}
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Int4",
"Int4",
"Varchar",
"Int8",
"Varchar"
]
},
"nullable": [
false
]
},
"hash": "6de15de62c0caa545910a17877a3ac5ebe44a398b199ab0120207a5569f54d0b"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_probe_result = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "6ef3efde571d46961244cd90420f3de5949a5ff2083453cb879af8a1689efe2f"
}
@@ -1,66 +1,66 @@
{
"db_name": "SQLite",
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n COALESCE(md.moniker, \"NA\") as \"moniker!\",\n COALESCE(md.website, \"NA\") as \"website!\",\n COALESCE(md.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(md.details, \"NA\") as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
"db_name": "PostgreSQL",
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n md.moniker as \"moniker!\",\n md.website as \"website!\",\n md.security_contact as \"security_contact!\",\n md.details as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
"describe": {
"columns": [
{
"name": "mix_id!",
"ordinal": 0,
"type_info": "Integer"
"name": "mix_id!",
"type_info": "Int8"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
"name": "bonded: bool",
"type_info": "Bool"
},
{
"name": "is_dp_delegatee: bool",
"ordinal": 2,
"type_info": "Integer"
"name": "is_dp_delegatee: bool",
"type_info": "Bool"
},
{
"name": "total_stake!",
"ordinal": 3,
"type_info": "Integer"
"name": "total_stake!",
"type_info": "Int8"
},
{
"name": "full_details!",
"ordinal": 4,
"type_info": "Text"
"name": "full_details!",
"type_info": "Varchar"
},
{
"name": "self_described",
"ordinal": 5,
"type_info": "Text"
"name": "self_described",
"type_info": "Varchar"
},
{
"name": "last_updated_utc!",
"ordinal": 6,
"type_info": "Integer"
"name": "last_updated_utc!",
"type_info": "Int8"
},
{
"name": "moniker!",
"ordinal": 7,
"type_info": "Text"
"name": "moniker!",
"type_info": "Varchar"
},
{
"name": "website!",
"ordinal": 8,
"type_info": "Text"
"name": "website!",
"type_info": "Varchar"
},
{
"name": "security_contact!",
"ordinal": 9,
"type_info": "Text"
"name": "security_contact!",
"type_info": "Varchar"
},
{
"name": "details!",
"ordinal": 10,
"type_info": "Text"
"name": "details!",
"type_info": "Varchar"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
@@ -70,11 +70,11 @@
true,
true,
false,
false,
false,
false,
false
true,
true,
true,
true
]
},
"hash": "f75af377da33db1455c6e0f612e0fa9583888f343b8b59faf37fc6799b244379"
"hash": "74b76cd0d94c1afc51c21c14c12236a2b964b981c8cbcc282117fe9bc38338dd"
}
@@ -1,38 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n date as \"date!\",\n timestamp_utc as \"timestamp_utc!\",\n value_json as \"value_json!\"\n FROM summary_history\n ORDER BY date DESC\n LIMIT 30",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "date!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "timestamp_utc!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "value_json!",
"ordinal": 3,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
true,
false,
false,
true
]
},
"hash": "7600823da7ce80b8ffda933608603a2752e28df775d1af8fd943a5fc8d7dc00d"
}
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM mixnode_packet_stats_raw\n WHERE mix_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
"describe": {
"columns": [
{
"name": "packets_received!: _",
"ordinal": 0,
"type_info": "Null"
},
{
"name": "packets_sent!: _",
"ordinal": 1,
"type_info": "Null"
},
{
"name": "packets_dropped!: _",
"ordinal": 2,
"type_info": "Null"
}
],
"parameters": {
"Right": 1
},
"nullable": [
null,
null,
null
]
},
"hash": "786b6a5d954e38b204cebf322711c74c8cf1c08e5e2896a1d6d5b85c91991753"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO summary_history\n (date, timestamp_utc, value_json)\n VALUES (?, ?, ?)\n ON CONFLICT(date) DO UPDATE SET\n timestamp_utc=excluded.timestamp_utc,\n value_json=excluded.value_json;",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "788515c34588aec352773df4b6e6c5e41f3c0bb56a27648b5e25466b8634a578"
}
@@ -1,19 +1,19 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT count(id) FROM gateways",
"describe": {
"columns": [
{
"name": "count(id)",
"ordinal": 0,
"type_info": "Integer"
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false
null
]
},
"hash": "86ff64db477a1d6235179b0b88d86b86d1b9be62336c9eac0eef44987a5451b5"
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM nym_nodes_packet_stats_raw\n WHERE node_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
"describe": {
"columns": [
{
"name": "packets_received!: _",
"ordinal": 0,
"type_info": "Null"
},
{
"name": "packets_sent!: _",
"ordinal": 1,
"type_info": "Null"
},
{
"name": "packets_dropped!: _",
"ordinal": 2,
"type_info": "Null"
}
],
"parameters": {
"Right": 1
},
"nullable": [
null,
null,
null
]
},
"hash": "8bdf85a61e443fa5f4835bffd0bffc8ed1011f56714fde6007e50951e569854b"
}
@@ -1,21 +1,21 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT\n gateway_identity_key as \"gateway_identity_key!\",\n bonded as \"bonded: bool\"\n FROM gateways\n ORDER BY last_testrun_utc",
"describe": {
"columns": [
{
"name": "gateway_identity_key!",
"ordinal": 0,
"type_info": "Text"
"name": "gateway_identity_key!",
"type_info": "Varchar"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
"name": "bonded: bool",
"type_info": "Bool"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
@@ -1,86 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "ed25519_identity_pubkey",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "total_stake",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "ip_addresses!: serde_json::Value",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "mix_port",
"ordinal": 4,
"type_info": "Integer"
},
{
"name": "x25519_sphinx_pubkey",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "node_role: serde_json::Value",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "supported_roles: serde_json::Value",
"ordinal": 7,
"type_info": "Text"
},
{
"name": "entry: serde_json::Value",
"ordinal": 8,
"type_info": "Text"
},
{
"name": "performance",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 11,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
true,
true
]
},
"hash": "9334f0c91252fcd7ec72558a271222615bb282e5334665700709ae475a5daea2"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO gateways\n (gateway_identity_key, bonded,\n self_described, explorer_pretty_bond,\n last_updated_utc, performance)\n VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(gateway_identity_key) DO UPDATE SET\n bonded=excluded.bonded,\n self_described=excluded.self_described,\n explorer_pretty_bond=excluded.explorer_pretty_bond,\n last_updated_utc=excluded.last_updated_utc,\n performance = excluded.performance;",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "9796d354ae075eab4cbd3438839c39da94025494395ec7b093aefef696f2d0c5"
}
@@ -1,56 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE gateway_id = ? AND status != 2\n ORDER BY id DESC\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id!",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "status!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "created_utc!",
"ordinal": 3,
"type_info": "Integer"
},
{
"name": "ip_address!",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "log!",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_assigned_utc",
"ordinal": 6,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false,
false,
false,
false,
true
]
},
"hash": "a1d9eb816acd1a91ed0975c801c9295c01a78861a2a0597ad28b1579a14bf008"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n COUNT(id) as \"count: i64\"\n FROM testruns\n WHERE\n status = ?\n ",
"describe": {
"columns": [
{
"name": "count: i64",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "a79a61b87325f3f1d9c5a4fb386ccd585be0641e5878acb6283b879f22ed2b4c"
}
@@ -1,44 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n nd.node_id,\n moniker,\n website,\n security_contact,\n details\n FROM\n nym_node_descriptions nd\n INNER JOIN\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "moniker",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "website",
"ordinal": 2,
"type_info": "Text"
},
{
"name": "security_contact",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "details",
"ordinal": 4,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true,
true,
true,
true
]
},
"hash": "aa602604c0e7b6eef41ea3cd83c16610e15be2d7ee3f6c4d3debf23f95fb9c2e"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE nym_nodes\n SET\n self_described = NULL,\n bond_info = NULL",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "b68796d1d8d2384b30f1aace06269682c4ae96f774261f5c298264d3c12e5b67"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "c214c001acbbf79fa499816f36ec586c4c29c03efb4cf0c40b73a5c76159cf5c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n mixnodes\n SET\n bonded = false\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "c42917c9542c1d720d92035863064741aefc9f7a7d1630f6b863ebd8174b6684"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n self_described as \"self_described: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n ORDER BY\n node_id\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true
]
},
"hash": "c7656b2b1b4328415772ce69d0568bd5438d6c8496ca9cbdcfb70bb5375b345e"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT mix_id\n FROM mixnodes\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "mix_id",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false
]
},
"hash": "c910788edefe64bbb34379702bcbde9ec6159c9fa03b13652e1f620dcd92125e"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "DELETE FROM gateway_session_stats WHERE day <= ?",
"describe": {
"columns": [],
"parameters": {
"Right": 1
},
"nullable": []
},
"hash": "c9e61180ec35dfab8623333fafa3b216b93440d0fddc0a37dd1b6c1813741f6a"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n total_stake\n FROM nym_nodes\n WHERE node_id = ?\n ",
"describe": {
"columns": [
{
"name": "total_stake",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "d2e07d44594ca5b44a6100482ff432c39d761f2a0ac1d6515cf73416f2eb6c61"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO summary\n (key, value_json, last_updated_utc)\n VALUES (?, ?, ?)\n ON CONFLICT(key) DO UPDATE SET\n value_json=excluded.value_json,\n last_updated_utc=excluded.last_updated_utc;",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "e0c76a959276e3b0f44c720af9c74a5bf4912ee73468e62e7d0d96b1d9074cbe"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO gateway_description (\n gateway_identity_key,\n moniker,\n website,\n security_contact,\n details,\n last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (gateway_identity_key) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "e9790b63ebe4bff5172bb8cb7bfc288364855003cf0e4d63e95047e7b502c650"
}
@@ -1,56 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE\n id = ?\n AND\n status = ?\n ORDER BY created_utc\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id!",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "status!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "created_utc!",
"ordinal": 3,
"type_info": "Integer"
},
{
"name": "ip_address!",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "log!",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_assigned_utc",
"ordinal": 6,
"type_info": "Integer"
}
],
"parameters": {
"Right": 2
},
"nullable": [
false,
false,
false,
false,
false,
false,
true
]
},
"hash": "f0c64794cbaed87a1d3166251d8e6720c9a9fc929144188460849be85d915004"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n gateways\n SET\n bonded = false\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "f7e3fa31d68c028bf39cc95389f29f8758ec922dd2e7ea064a1e537e580c9ee5"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_nodes_packet_stats_raw (\n node_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "fcb1698d9e0e3a14524c92e7c99a811588c2bbc50d4975487a0464321a1b18c9"
}
@@ -3,7 +3,7 @@
[package]
name = "nym-node-status-api"
version = "3.1.1"
version = "3.1.3"
authors.workspace = true
repository.workspace = true
homepage.workspace = true
@@ -28,11 +28,15 @@ moka = { workspace = true, features = ["future"] }
# Nym API: revert after Cheddar is out
nym-contracts-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-mixnet-contract-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["openapi"]}
nym-node-status-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"openapi",
] }
nym-node-status-client = { path = "../nym-node-status-client" }
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["middleware"]}
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"middleware",
] }
nym-network-defaults = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-serde-helpers = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-statistics-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
@@ -62,7 +66,7 @@ serde_json = { workspace = true }
serde_json_path = { workspace = true }
strum = { workspace = true }
strum_macros = { workspace = true }
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "sqlite", "time"] }
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "time"] }
thiserror = { workspace = true }
time = { workspace = true, features = ["formatting"] }
tokio = { workspace = true, features = ["rt-multi-thread"] }
@@ -77,13 +81,16 @@ utoipauto = { workspace = true }
nym-node-metrics = { path = "../../nym-node/nym-node-metrics" }
[features]
default = ["pg"]
sqlite = ["sqlx/sqlite"]
pg = ["sqlx/postgres"]
[build-dependencies]
anyhow = { workspace = true }
tokio = { workspace = true, features = ["macros"] }
sqlx = { workspace = true, features = [
"runtime-tokio-rustls",
"sqlite",
"macros",
"migrate",
] }
@@ -0,0 +1,117 @@
# Makefile for nym-node-status-api database management
# --- Configuration ---
TEST_DATABASE_URL := postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# Docker compose service names
DB_SERVICE_NAME := postgres-test
DB_CONTAINER_NAME := nym_node_status_api_postgres_test
# Default target
.PHONY: default
default: help
# --- Main Targets ---
.PHONY: prepare-pg
prepare-pg: test-db-up test-db-wait test-db-migrate test-db-prepare test-db-down ## Setup PostgreSQL and prepare SQLx offline cache
.PHONY: test-db
test-db: test-db-up test-db-wait test-db-migrate test-db-run test-db-down ## Run tests with PostgreSQL database
.PHONY: dev-db
dev-db: test-db-up test-db-wait test-db-migrate ## Start PostgreSQL for development (keeps running)
@echo "PostgreSQL is running on port 5433"
@echo "Connection string: $(TEST_DATABASE_URL)"
# --- Docker Compose Targets ---
.PHONY: test-db-up
test-db-up: ## Start the PostgreSQL test database in the background
@echo "Starting PostgreSQL test database..."
docker compose up -d $(DB_SERVICE_NAME)
.PHONY: test-db-wait
test-db-wait: ## Wait for the PostgreSQL database to be healthy
@echo "Waiting for PostgreSQL database..."
@while ! docker inspect --format='{{.State.Health.Status}}' $(DB_CONTAINER_NAME) 2>/dev/null | grep -q 'healthy'; do \
echo -n "."; \
sleep 1; \
done; \
echo " Database is healthy!"
.PHONY: test-db-down
test-db-down: ## Stop and remove the test database
@echo "Stopping PostgreSQL test database..."
docker compose down
# --- SQLx Targets ---
.PHONY: test-db-migrate
test-db-migrate: ## Run database migrations against PostgreSQL
@echo "Running PostgreSQL migrations..."
DATABASE_URL="$(TEST_DATABASE_URL)" sqlx migrate run --source migrations_pg
.PHONY: test-db-prepare
test-db-prepare: ## Run sqlx prepare for compile-time query verification
@echo "Running sqlx prepare for PostgreSQL..."
DATABASE_URL="$(TEST_DATABASE_URL)" cargo sqlx prepare -- --features pg
# --- Build and Test Targets ---
.PHONY: test-db-run
test-db-run: ## Run tests with PostgreSQL feature
@echo "Running tests with PostgreSQL..."
DATABASE_URL="$(TEST_DATABASE_URL)" cargo test --features pg --no-default-features
.PHONY: build-pg
build-pg: ## Build with PostgreSQL feature
@echo "Building with PostgreSQL feature..."
cargo build --features pg --no-default-features
.PHONY: build-sqlite
build-sqlite: ## Build with SQLite feature (default)
@echo "Building with SQLite feature..."
cargo build --features sqlite --no-default-features
.PHONY: check-pg
check-pg: ## Check code with PostgreSQL feature
@echo "Checking code with PostgreSQL feature..."
cargo check --features pg --no-default-features
.PHONY: check-sqlite
check-sqlite: ## Check code with SQLite feature
@echo "Checking code with SQLite feature..."
cargo check --features sqlite --no-default-features
.PHONY: clippy
clippy: clippy-pg clippy-sqlite
.PHONY: clippy-pg
clippy-pg: ## Run clippy with PostgreSQL feature
@echo "Running clippy with PostgreSQL feature..."
cargo clippy --features pg --no-default-features -- -D warnings
.PHONY: clippy-sqlite
clippy-sqlite: ## Run clippy with SQLite feature (default)
@echo "Running clippy with SQLite feature..."
cargo clippy --features sqlite --no-default-features -- -D warnings
# --- Cleanup Targets ---
.PHONY: clean
clean: ## Clean build artifacts and SQLx cache
cargo clean
rm -rf .sqlx
.PHONY: clean-db
clean-db: test-db-down ## Stop database and clean volumes
docker volume rm -f nym-node-status-api_postgres_test_data 2>/dev/null || true
# --- Utility Targets ---
.PHONY: sqlx-cli
sqlx-cli: ## Install sqlx-cli if not already installed
@command -v sqlx >/dev/null 2>&1 || cargo install sqlx-cli --features postgres,sqlite
.PHONY: psql
psql: ## Connect to the running PostgreSQL database with psql
@docker exec -it $(DB_CONTAINER_NAME) psql -U testuser -d nym_node_status_api_test
.PHONY: help
help: ## Show help for Makefile targets
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
@@ -0,0 +1,104 @@
# PostgreSQL Support for nym-node-status-api
This project now supports both SQLite (default) and PostgreSQL databases.
## Quick Start with PostgreSQL
### 1. Install Prerequisites
```bash
# Install sqlx-cli if not already installed
make sqlx-cli
```
### 2. Prepare PostgreSQL for Development
```bash
# This will:
# - Start PostgreSQL in Docker
# - Run migrations
# - Generate SQLx offline query cache
# - Stop the database
make prepare-pg
```
### 3. Build with PostgreSQL
```bash
# Build with PostgreSQL feature
make build-pg
# Or manually:
cargo build --features pg --no-default-features
```
### 4. Run with PostgreSQL
```bash
# Start PostgreSQL for development (keeps running)
make dev-db
# In another terminal, run the application
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test \
cargo run --features pg --no-default-features
```
## Database Features
- `sqlite` (default): Uses SQLite database
- `pg`: Uses PostgreSQL database
Only one database feature can be active at a time.
## Migration Differences
SQLite migrations are in `migrations/`, PostgreSQL migrations are in `migrations_pg/`.
Key differences:
- **AUTOINCREMENT** → **SERIAL**
- **INTEGER CHECK (0,1)** → **BOOLEAN**
- **REAL** → **DOUBLE PRECISION**
- No table recreation needed for constraint changes in PostgreSQL
## Makefile Targets
```bash
make help # Show all available targets
make prepare-pg # Setup PostgreSQL and prepare SQLx cache
make dev-db # Start PostgreSQL for development
make test-db # Run tests with PostgreSQL
make build-pg # Build with PostgreSQL
make build-sqlite # Build with SQLite
make psql # Connect to running PostgreSQL
make clean # Clean build artifacts
make clean-db # Stop database and clean volumes
```
## Environment Variables
See `.env.example` for all configuration options. Key variable:
```bash
# For PostgreSQL:
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# For SQLite:
DATABASE_URL=sqlite://nym-node-status-api.sqlite
```
## Troubleshooting
### SQLx Offline Mode
If you see "no cached data for this query" errors:
1. Ensure PostgreSQL is running: `make dev-db`
2. Run: `make test-db-prepare`
### Connection Refused
If you see "Connection refused" errors:
1. Check Docker is running: `docker ps`
2. Check PostgreSQL container: `docker ps | grep nym_node_status_api_postgres_test`
3. Restart database: `make test-db-down && make dev-db`
@@ -1,17 +1,20 @@
use anyhow::{anyhow, Result};
use anyhow::Result;
#[cfg(feature = "sqlite")]
use sqlx::{Connection, SqliteConnection};
#[cfg(feature = "sqlite")]
#[cfg(target_family = "unix")]
use std::fs::Permissions;
#[cfg(feature = "sqlite")]
#[cfg(target_family = "unix")]
use std::os::unix::fs::PermissionsExt;
#[cfg(feature = "sqlite")]
use tokio::{fs::File, io::AsyncWriteExt};
#[cfg(feature = "sqlite")]
const SQLITE_DB_FILENAME: &str = "nym-node-status-api.sqlite";
/// If you need to re-run migrations or reset the db, just run
/// cargo clean -p nym-node-status-api
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()> {
#[cfg(feature = "sqlite")]
async fn init_db() -> Result<()> {
let out_dir = read_env_var("OUT_DIR")?;
let database_path = format!("{out_dir}/{SQLITE_DB_FILENAME}?mode=rwc");
@@ -30,11 +33,30 @@ async fn main() -> Result<()> {
Ok(())
}
#[cfg(all(feature = "pg", not(feature = "sqlite")))]
async fn init_db() -> Result<()> {
// PostgreSQL doesn't need build-time initialization
// Just ensure DATABASE_URL is available for runtime
if let Ok(database_url) = std::env::var("DATABASE_URL") {
println!("cargo::rustc-env=DATABASE_URL={database_url}");
}
Ok(())
}
/// If you need to re-run migrations or reset the db, just run
/// cargo clean -p nym-node-status-api
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()> {
init_db().await
}
#[cfg(feature = "sqlite")]
fn read_env_var(var: &str) -> Result<String> {
std::env::var(var).map_err(|_| anyhow!("You need to set {} env var", var))
std::env::var(var).map_err(|_| anyhow::anyhow!("You need to set {} env var", var))
}
/// use `./enter_db.sh` to inspect DB
#[cfg(feature = "sqlite")]
async fn write_db_path_to_file(out_dir: &str, db_filename: &str) -> anyhow::Result<()> {
let mut file = File::create("settings.sql").await?;
let settings = ".mode columns
@@ -0,0 +1,21 @@
services:
postgres-test:
image: postgres:16-alpine
container_name: nym_node_status_api_postgres_test
environment:
POSTGRES_DB: nym_node_status_api_test
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
ports:
- '5433:5432' # Map to 5433 to avoid conflicts with default PostgreSQL
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U testuser -d nym_node_status_api_test']
interval: 5s
timeout: 5s
retries: 5
# Optional: Add volume for persistent data during development
# volumes:
# - postgres_test_data:/var/lib/postgresql/data
# volumes:
# postgres_test_data:
@@ -0,0 +1,113 @@
CREATE TABLE gateways
(
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR NOT NULL UNIQUE,
self_described VARCHAR NOT NULL,
explorer_pretty_bond VARCHAR,
last_probe_result VARCHAR,
last_probe_log VARCHAR,
config_score INTEGER NOT NULL DEFAULT 0,
config_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
config_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
routing_score INTEGER NOT NULL DEFAULT 0,
routing_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
routing_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
test_run_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
last_testrun_utc BIGINT,
last_updated_utc BIGINT NOT NULL,
bonded BOOLEAN NOT NULL DEFAULT FALSE,
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
performance INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_gateway_description_gateway_identity_key ON gateways (gateway_identity_key);
CREATE TABLE mixnodes (
id SERIAL PRIMARY KEY,
identity_key VARCHAR NOT NULL UNIQUE,
mix_id BIGINT NOT NULL UNIQUE,
bonded BOOLEAN NOT NULL DEFAULT FALSE,
total_stake BIGINT NOT NULL,
host VARCHAR NOT NULL,
http_api_port BIGINT NOT NULL,
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
full_details VARCHAR,
self_described VARCHAR,
last_updated_utc BIGINT NOT NULL,
is_dp_delegatee BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE INDEX idx_mixnodes_mix_id ON mixnodes (mix_id);
CREATE INDEX idx_mixnodes_identity_key ON mixnodes (identity_key);
CREATE TABLE mixnode_description (
id SERIAL PRIMARY KEY,
mix_id BIGINT UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc BIGINT NOT NULL,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
);
-- Indexes for description table
CREATE INDEX idx_mixnode_description_mix_id ON mixnode_description (mix_id);
CREATE TABLE summary
(
key VARCHAR PRIMARY KEY,
value_json VARCHAR,
last_updated_utc BIGINT NOT NULL
);
CREATE TABLE summary_history
(
id SERIAL PRIMARY KEY,
date VARCHAR UNIQUE NOT NULL,
timestamp_utc BIGINT NOT NULL,
value_json VARCHAR
);
CREATE INDEX idx_summary_history_timestamp_utc ON summary_history (timestamp_utc);
CREATE INDEX idx_summary_history_date ON summary_history (date);
CREATE TABLE gateway_description (
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc BIGINT NOT NULL,
FOREIGN KEY (gateway_identity_key) REFERENCES gateways (gateway_identity_key)
);
CREATE TABLE mixnode_daily_stats (
id SERIAL PRIMARY KEY,
mix_id BIGINT NOT NULL,
total_stake BIGINT NOT NULL,
date_utc VARCHAR NOT NULL,
packets_received INTEGER DEFAULT 0,
packets_sent INTEGER DEFAULT 0,
packets_dropped INTEGER DEFAULT 0,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id),
UNIQUE (mix_id, date_utc) -- This constraint automatically creates an index
);
CREATE TABLE testruns
(
id SERIAL PRIMARY KEY,
gateway_id INTEGER NOT NULL,
status INTEGER NOT NULL, -- 0=pending, 1=in-progress, 2=complete
timestamp_utc BIGINT NOT NULL,
ip_address VARCHAR NOT NULL,
log VARCHAR NOT NULL,
FOREIGN KEY (gateway_id) REFERENCES gateways (id)
);
@@ -0,0 +1,5 @@
ALTER TABLE testruns
RENAME COLUMN timestamp_utc TO created_utc;
ALTER TABLE testruns
ADD COLUMN last_assigned_utc BIGINT;
@@ -0,0 +1,16 @@
CREATE TABLE gateway_session_stats (
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR NOT NULL,
node_id BIGINT NOT NULL,
day DATE NOT NULL,
unique_active_clients BIGINT NOT NULL,
session_started BIGINT NOT NULL,
users_hashes VARCHAR,
vpn_sessions VARCHAR,
mixnet_sessions VARCHAR,
unknown_sessions VARCHAR,
UNIQUE (node_id, day) -- This constraint automatically creates an index
);
CREATE INDEX idx_gateway_session_stats_identity_key ON gateway_session_stats (gateway_identity_key);
CREATE INDEX idx_gateway_session_stats_day ON gateway_session_stats (day);
@@ -0,0 +1,11 @@
CREATE TABLE mixnode_packet_stats_raw (
id SERIAL PRIMARY KEY,
mix_id BIGINT NOT NULL,
timestamp_utc BIGINT NOT NULL,
packets_received INTEGER,
packets_sent INTEGER,
packets_dropped INTEGER,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
);
CREATE INDEX idx_mixnode_packet_stats_raw_mix_id_timestamp_utc ON mixnode_packet_stats_raw (mix_id, timestamp_utc);
@@ -0,0 +1,54 @@
ALTER TABLE mixnodes DROP COLUMN blacklisted;
ALTER TABLE gateways DROP COLUMN blacklisted;
CREATE TABLE nym_nodes (
node_id INTEGER PRIMARY KEY,
ed25519_identity_pubkey VARCHAR NOT NULL UNIQUE,
total_stake BIGINT NOT NULL,
ip_addresses TEXT NOT NULL,
mix_port INTEGER NOT NULL,
x25519_sphinx_pubkey VARCHAR NOT NULL UNIQUE,
node_role TEXT NOT NULL,
supported_roles TEXT NOT NULL,
performance VARCHAR NOT NULL,
entry TEXT,
last_updated_utc INTEGER NOT NULL
);
CREATE INDEX idx_nym_nodes_node_id ON nym_nodes (node_id);
CREATE INDEX idx_nym_nodes_ed25519_identity_pubkey ON nym_nodes (ed25519_identity_pubkey);
CREATE TABLE nym_node_descriptions (
id SERIAL PRIMARY KEY,
node_id INTEGER UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc INTEGER NOT NULL,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
);
CREATE TABLE nym_nodes_packet_stats_raw (
id SERIAL PRIMARY KEY,
node_id INTEGER NOT NULL,
timestamp_utc INTEGER NOT NULL,
packets_received INTEGER,
packets_sent INTEGER,
packets_dropped INTEGER,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
);
CREATE INDEX idx_nym_nodes_packet_stats_raw_node_id_timestamp_utc ON nym_nodes_packet_stats_raw (node_id, timestamp_utc);
CREATE TABLE nym_node_daily_mixing_stats (
id SERIAL PRIMARY KEY,
node_id INTEGER NOT NULL,
total_stake BIGINT NOT NULL,
date_utc VARCHAR NOT NULL,
packets_received INTEGER DEFAULT 0,
packets_sent INTEGER DEFAULT 0,
packets_dropped INTEGER DEFAULT 0,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id),
UNIQUE (node_id, date_utc) -- This constraint automatically creates an index
);
@@ -0,0 +1,23 @@
ALTER TABLE nym_nodes ADD COLUMN self_described TEXT;
ALTER TABLE nym_nodes ADD COLUMN bond_info TEXT;
-- PostgreSQL doesn't need table recreation for adding ON DELETE CASCADE
-- We can drop and recreate the foreign key constraints directly
-- Drop existing foreign key constraints
ALTER TABLE nym_node_descriptions DROP CONSTRAINT IF EXISTS nym_node_descriptions_node_id_fkey;
ALTER TABLE nym_nodes_packet_stats_raw DROP CONSTRAINT IF EXISTS nym_nodes_packet_stats_raw_node_id_fkey;
ALTER TABLE nym_node_daily_mixing_stats DROP CONSTRAINT IF EXISTS nym_node_daily_mixing_stats_node_id_fkey;
-- Add foreign key constraints with ON DELETE CASCADE
ALTER TABLE nym_node_descriptions
ADD CONSTRAINT nym_node_descriptions_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
ALTER TABLE nym_nodes_packet_stats_raw
ADD CONSTRAINT nym_nodes_packet_stats_raw_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
ALTER TABLE nym_node_daily_mixing_stats
ADD CONSTRAINT nym_node_daily_mixing_stats_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
@@ -0,0 +1,9 @@
-- Removing UNIQUE constraints on nym_nodes
-- In PostgreSQL, we can drop constraints directly without recreating the table
-- Drop the unique constraints
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_ed25519_identity_pubkey_key;
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_x25519_sphinx_pubkey_key;
-- The columns and indexes remain, only the unique constraints are removed
-- The existing indexes idx_nym_nodes_node_id and idx_nym_nodes_ed25519_identity_pubkey remain unchanged
@@ -0,0 +1,112 @@
-- for a couple of days after migrating chrono -> time, we stored dates as
-- 2025-June-DD instead of 2025-06-DD. This migration fixes those entries.
--
-- Because of a UNIQUE constraint on (node_id, date_utc), we can't just rename in-place.
-- - merge (add) node stats back to the original table where conflict (node_id, date_utc) would exist
-- - delete invalid records from original table (those stats were merged into correct rows above)
-- - insert rows that did not have a conflicting (node_Id, date_utc) combo
-- Conflicts affect only the date which has both kinds of entries,
-- e.g. 2025-06-05 and 2025-June-05 (date when this change was deployed)
--
-- This applies to both affected tables.
-- ----------------------------------------
-- mixnode_daily_stats
-- ----------------------------------------
-- First, copy over rows with invalid date to a temp table (in the correct date format)
CREATE TEMP TABLE tmp_mix AS
SELECT
mix_id,
REPLACE(date_utc,'June','06') AS new_date,
SUM(total_stake) AS total_stake_sum,
SUM(packets_received) AS packets_received_sum,
SUM(packets_sent) AS packets_sent_sum,
SUM(packets_dropped) AS packets_dropped_sum
FROM mixnode_daily_stats
WHERE date_utc LIKE '%June%'
GROUP BY mix_id, new_date;
UPDATE mixnode_daily_stats AS m
SET
total_stake = m.total_stake,
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc)
WHERE EXISTS (
SELECT 1 FROM tmp_mix
WHERE mix_id = m.mix_id
AND new_date = m.date_utc
);
DELETE FROM mixnode_daily_stats
WHERE date_utc LIKE '%June%';
INSERT INTO mixnode_daily_stats
(mix_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
SELECT
mix_id,
new_date,
total_stake_sum,
packets_received_sum,
packets_sent_sum,
packets_dropped_sum
FROM tmp_mix AS t
-- only those whose new_date did _not_ already exist
WHERE NOT EXISTS (
SELECT 1 FROM mixnode_daily_stats AS m
WHERE m.mix_id = t.mix_id
AND m.date_utc = t.new_date
);
DROP TABLE tmp_mix;
-- ----------------------------------------
-- nym_node_daily_mixing_stats
-- ----------------------------------------
CREATE TEMP TABLE tmp_nym_node_stats AS
SELECT
node_id,
REPLACE(date_utc,'June','06') AS new_date,
SUM(total_stake) AS total_stake_sum,
SUM(packets_received) AS packets_received_sum,
SUM(packets_sent) AS packets_sent_sum,
SUM(packets_dropped) AS packets_dropped_sum
FROM nym_node_daily_mixing_stats
WHERE date_utc LIKE '%June%'
GROUP BY node_id, new_date;
UPDATE nym_node_daily_mixing_stats AS m
SET
total_stake = m.total_stake,
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc)
WHERE EXISTS (
SELECT 1 FROM tmp_nym_node_stats
WHERE node_id = m.node_id
AND new_date = m.date_utc
);
DELETE FROM nym_node_daily_mixing_stats
WHERE date_utc LIKE '%June%';
INSERT INTO nym_node_daily_mixing_stats
(node_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
SELECT
node_id,
new_date,
total_stake_sum,
packets_received_sum,
packets_sent_sum,
packets_dropped_sum
FROM tmp_nym_node_stats AS t
WHERE NOT EXISTS (
SELECT 1 FROM nym_node_daily_mixing_stats AS m
WHERE m.node_id = t.node_id
AND m.date_utc = t.new_date
);
DROP TABLE tmp_nym_node_stats;
@@ -1,24 +1,49 @@
use anyhow::{anyhow, Result};
use sqlx::{
migrate::Migrator,
query,
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
ConnectOptions, SqlitePool,
};
use std::{str::FromStr, time::Duration};
pub(crate) mod models;
pub(crate) mod queries;
pub(crate) mod query_wrapper;
// Re-export the query wrapper functions for easier access
pub(crate) use query_wrapper::query;
#[allow(unused_imports)]
pub(crate) use query_wrapper::{query_as, query_scalar};
#[cfg(feature = "sqlite")]
use sqlx::{
migrate::Migrator,
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
ConnectOptions, SqlitePool,
};
#[cfg(feature = "pg")]
use sqlx::{migrate::Migrator, postgres::PgConnectOptions, ConnectOptions, PgPool};
#[cfg(feature = "sqlite")]
static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
#[cfg(feature = "pg")]
static MIGRATOR: Migrator = sqlx::migrate!("./migrations_pg");
#[cfg(feature = "sqlite")]
pub(crate) type DbPool = SqlitePool;
#[cfg(feature = "pg")]
pub(crate) type DbPool = PgPool;
#[cfg(feature = "sqlite")]
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Sqlite>;
#[cfg(feature = "pg")]
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Postgres>;
pub(crate) struct Storage {
pool: DbPool,
}
impl Storage {
#[cfg(feature = "sqlite")]
pub async fn init(connection_url: String, busy_timeout: Duration) -> Result<Self> {
let connect_options = SqliteConnectOptions::from_str(&connection_url)?
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal)
@@ -41,16 +66,31 @@ impl Storage {
Ok(Storage { pool })
}
#[cfg(feature = "pg")]
pub async fn init(connection_url: String, _busy_timeout: Duration) -> Result<Self> {
let connect_options =
PgConnectOptions::from_str(&connection_url)?.disable_statement_logging();
let pool = sqlx::PgPool::connect_with(connect_options)
.await
.map_err(|err| anyhow!("Failed to connect to {}: {}", &connection_url, err))?;
MIGRATOR.run(&pool).await?;
Ok(Storage { pool })
}
/// Cloning pool is cheap, it's the same underlying set of connections
pub fn pool_owned(&self) -> DbPool {
self.pool.clone()
}
#[cfg(feature = "sqlite")]
async fn assert_busy_timeout(pool: DbPool, expected_busy_timeout_s: i64) -> Result<()> {
let mut conn = pool.acquire().await?;
// Sqlite stores this value as miliseconds
// https://www.sqlite.org/pragma.html#pragma_busy_timeout
let busy_timeout_db = query!("PRAGMA busy_timeout;")
let busy_timeout_db = sqlx::query!("PRAGMA busy_timeout;")
.fetch_one(conn.as_mut())
.await?;
@@ -38,7 +38,7 @@ pub(crate) struct GatewayInsertRecord {
pub(crate) performance: u8,
}
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub(crate) struct GatewayDto {
pub(crate) gateway_identity_key: String,
pub(crate) bonded: bool,
@@ -121,7 +121,7 @@ pub(crate) struct MixnodeRecord {
pub(crate) is_dp_delegatee: bool,
}
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub(crate) struct MixnodeDto {
pub(crate) mix_id: i64,
pub(crate) bonded: bool,
@@ -183,14 +183,14 @@ pub(crate) struct BondedStatusDto {
}
#[allow(unused)]
#[derive(Debug, Clone, Default)]
#[derive(Debug, Clone, Default, FromRow)]
pub(crate) struct SummaryDto {
pub(crate) key: String,
pub(crate) value_json: String,
pub(crate) last_updated_utc: i64,
}
#[derive(Debug, Clone, Default)]
#[derive(Debug, Clone, Default, FromRow)]
pub(crate) struct SummaryHistoryDto {
#[allow(dead_code)]
pub id: i64,
@@ -287,7 +287,7 @@ pub(crate) mod gateway {
}
#[allow(dead_code)] // not dead code, this is SQL data model
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub struct TestRunDto {
pub id: i64,
pub gateway_id: i64,
@@ -313,7 +313,7 @@ pub struct GatewayIdentityDto {
}
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub struct GatewayInfoDto {
pub id: i64,
pub gateway_identity_key: String,
@@ -362,7 +362,7 @@ impl TryFrom<GatewaySessionsRecord> for http::models::SessionStats {
}
}
#[derive(strum_macros::Display)]
#[derive(strum_macros::Display, Clone)]
pub(crate) enum ScrapeNodeKind {
LegacyMixnode { mix_id: i64 },
MixingNymNode { node_id: i64 },
@@ -379,6 +379,7 @@ impl ScrapeNodeKind {
}
}
#[derive(Clone)]
pub(crate) struct ScraperNodeInfo {
pub node_kind: ScrapeNodeKind,
pub hosts: Vec<String>,
@@ -514,7 +515,7 @@ impl TryFrom<NymNodeDto> for SkimmedNode {
}
}
#[derive(Debug, Serialize, Deserialize, sqlx::Decode)]
#[derive(Debug, Serialize, Deserialize, sqlx::Decode, FromRow)]
pub struct NodeStats {
pub packets_received: i64,
pub packets_sent: i64,
@@ -3,32 +3,32 @@ use std::collections::HashSet;
use crate::{
db::{
models::{GatewayDto, GatewayInsertRecord},
DbPool,
DbConnection, DbPool,
},
http::models::Gateway,
mixnet_scraper::helpers::NodeDescriptionResponse,
};
use futures_util::TryStreamExt;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use tracing::error;
pub(crate) async fn select_gateway_identity(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
gateway_pk: i64,
) -> anyhow::Result<String> {
let record = sqlx::query!(
let record = crate::db::query(
r#"SELECT
gateway_identity_key
FROM
gateways
WHERE
id = ?"#,
gateway_pk
)
.bind(gateway_pk)
.fetch_one(conn.as_mut())
.await?;
Ok(record.gateway_identity_key)
Ok(record.try_get("gateway_identity_key")?)
}
pub(crate) async fn update_bonded_gateways(
@@ -37,7 +37,7 @@ pub(crate) async fn update_bonded_gateways(
) -> anyhow::Result<()> {
let mut tx = pool.begin().await?;
sqlx::query!(
crate::db::query(
r#"UPDATE
gateways
SET
@@ -48,7 +48,7 @@ pub(crate) async fn update_bonded_gateways(
.await?;
for record in gateways {
sqlx::query!(
crate::db::query(
"INSERT INTO gateways
(gateway_identity_key, bonded,
self_described, explorer_pretty_bond,
@@ -60,13 +60,13 @@ pub(crate) async fn update_bonded_gateways(
explorer_pretty_bond=excluded.explorer_pretty_bond,
last_updated_utc=excluded.last_updated_utc,
performance = excluded.performance;",
record.identity_key,
record.bonded,
record.self_described,
record.explorer_pretty_bond,
record.last_updated_utc,
record.performance
)
.bind(record.identity_key)
.bind(record.bonded)
.bind(record.self_described)
.bind(record.explorer_pretty_bond)
.bind(record.last_updated_utc)
.bind(record.performance as i8)
.execute(&mut *tx)
.await?;
}
@@ -78,22 +78,21 @@ pub(crate) async fn update_bonded_gateways(
pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gateway>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as!(
GatewayDto,
let items = crate::db::query_as::<GatewayDto>(
r#"SELECT
gw.gateway_identity_key as "gateway_identity_key!",
gw.bonded as "bonded: bool",
gw.performance as "performance!",
gw.self_described as "self_described?",
gw.explorer_pretty_bond as "explorer_pretty_bond?",
gw.last_probe_result as "last_probe_result?",
gw.last_probe_log as "last_probe_log?",
gw.last_testrun_utc as "last_testrun_utc?",
gw.last_updated_utc as "last_updated_utc!",
COALESCE(gd.moniker, "NA") as "moniker!",
COALESCE(gd.website, "NA") as "website!",
COALESCE(gd.security_contact, "NA") as "security_contact!",
COALESCE(gd.details, "NA") as "details!"
gw.gateway_identity_key,
gw.bonded,
gw.performance,
gw.self_described,
gw.explorer_pretty_bond,
gw.last_probe_result,
gw.last_probe_log,
gw.last_testrun_utc,
gw.last_updated_utc,
COALESCE(gd.moniker, 'NA') as moniker,
COALESCE(gd.website, 'NA') as website,
COALESCE(gd.security_contact, 'NA') as security_contact,
COALESCE(gd.details, 'NA') as details
FROM gateways gw
LEFT JOIN gateway_description gd
ON gw.gateway_identity_key = gd.gateway_identity_key
@@ -114,29 +113,29 @@ pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gatewa
pub(crate) async fn get_bonded_gateway_id_keys(pool: &DbPool) -> anyhow::Result<HashSet<String>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query!(
let items = crate::db::query(
r#"
SELECT gateway_identity_key
FROM gateways
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?
.into_iter()
.map(|record| record.gateway_identity_key)
.map(|record| record.try_get::<String, _>("gateway_identity_key").unwrap())
.collect::<HashSet<_>>();
Ok(items)
}
pub(crate) async fn insert_gateway_description(
conn: &mut PoolConnection<Sqlite>,
identity_key: &str,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
identity_key: String,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO gateway_description (
gateway_identity_key,
@@ -153,15 +152,54 @@ pub(crate) async fn insert_gateway_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
identity_key,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(identity_key)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn get_or_create_gateway(
conn: &mut DbConnection,
gateway_identity_key: &str,
) -> anyhow::Result<i64> {
// Try to find existing gateway
let existing = crate::db::query("SELECT id FROM gateways WHERE gateway_identity_key = ?")
.bind(gateway_identity_key.to_string())
.fetch_optional(conn.as_mut())
.await?;
if let Some(row) = existing {
return Ok(row.try_get("id")?);
}
// Create new gateway
tracing::info!("Creating new gateway record for {}", gateway_identity_key);
let now = crate::utils::now_utc().unix_timestamp();
let result = crate::db::query(
r#"INSERT INTO gateways (
gateway_identity_key,
bonded,
performance,
self_described,
last_updated_utc
) VALUES (?, ?, ?, ?, ?)
RETURNING id"#,
)
.bind(gateway_identity_key.to_string())
.bind(true) // Assume bonded since being tested
.bind(0) // Initial performance
.bind("null")
.bind(now)
.fetch_one(conn.as_mut())
.await?;
Ok(result.try_get("id")?)
}
@@ -6,6 +6,7 @@ use futures_util::TryStreamExt;
use time::Date;
use tracing::error;
#[cfg(feature = "sqlite")]
pub(crate) async fn insert_session_records(
pool: &DbPool,
records: Vec<GatewaySessionsRecord>,
@@ -36,6 +37,38 @@ pub(crate) async fn insert_session_records(
Ok(())
}
#[cfg(feature = "pg")]
pub(crate) async fn insert_session_records(
pool: &DbPool,
records: Vec<GatewaySessionsRecord>,
) -> anyhow::Result<()> {
let mut tx = pool.begin().await?;
for record in records {
sqlx::query!(
"INSERT INTO gateway_session_stats
(gateway_identity_key, node_id, day,
unique_active_clients, session_started, users_hashes,
vpn_sessions, mixnet_sessions, unknown_sessions)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT DO NOTHING",
record.gateway_identity_key,
record.node_id,
record.day,
record.unique_active_clients,
record.session_started,
record.users_hashes,
record.vpn_sessions,
record.mixnet_sessions,
record.unknown_sessions,
)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
Ok(())
}
pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<SessionStats>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as(
@@ -68,7 +101,8 @@ pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<Sess
pub(crate) async fn delete_old_records(pool: &DbPool, cut_off: Date) -> anyhow::Result<()> {
let mut conn = pool.acquire().await?;
sqlx::query!("DELETE FROM gateway_session_stats WHERE day <= ?", cut_off)
crate::db::query("DELETE FROM gateway_session_stats WHERE day <= ?")
.bind(cut_off)
.execute(&mut *conn)
.await?;
Ok(())
@@ -6,8 +6,8 @@ use crate::db::{models::NetworkSummary, DbPool};
/// `daily_summary`
pub(crate) async fn insert_summaries(
pool: &DbPool,
summaries: &[(&str, usize)],
summary: &NetworkSummary,
summaries: Vec<(String, usize)>,
summary: NetworkSummary,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
insert_summary(pool, summaries, last_updated).await?;
@@ -19,7 +19,7 @@ pub(crate) async fn insert_summaries(
async fn insert_summary(
pool: &DbPool,
summaries: &[(&str, usize)],
summaries: Vec<(String, usize)>,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
let timestamp = last_updated.unix_timestamp();
@@ -27,17 +27,17 @@ async fn insert_summary(
for (kind, value) in summaries {
let value = value.to_string();
sqlx::query!(
crate::db::query(
"INSERT INTO summary
(key, value_json, last_updated_utc)
VALUES (?, ?, ?)
ON CONFLICT(key) DO UPDATE SET
value_json=excluded.value_json,
last_updated_utc=excluded.last_updated_utc;",
kind,
value,
timestamp
)
.bind(kind.clone())
.bind(value)
.bind(timestamp)
.execute(&mut *tx)
.await
.map_err(|err| {
@@ -60,7 +60,7 @@ async fn insert_summary(
/// This is not aggregate data, it's a set of latest data points
async fn insert_summary_history(
pool: &DbPool,
summary: &NetworkSummary,
summary: NetworkSummary,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
let mut conn = pool.acquire().await?;
@@ -70,17 +70,17 @@ async fn insert_summary_history(
let date = datetime_to_only_date_str(last_updated);
sqlx::query!(
crate::db::query(
"INSERT INTO summary_history
(date, timestamp_utc, value_json)
VALUES (?, ?, ?)
ON CONFLICT(date) DO UPDATE SET
timestamp_utc=excluded.timestamp_utc,
value_json=excluded.value_json;",
date,
timestamp,
value_json
)
.bind(date)
.bind(timestamp)
.bind(value_json)
.execute(&mut *conn)
.await?;
@@ -1,13 +1,13 @@
use std::collections::HashSet;
use futures_util::TryStreamExt;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use tracing::error;
use crate::{
db::{
models::{MixnodeDto, MixnodeRecord},
DbPool,
DbConnection, DbPool,
},
http::models::{DailyStats, Mixnode},
mixnet_scraper::helpers::NodeDescriptionResponse,
@@ -20,7 +20,7 @@ pub(crate) async fn update_mixnodes(
let mut tx = pool.begin().await?;
// mark all as unbonded
sqlx::query!(
crate::db::query(
r#"UPDATE
mixnodes
SET
@@ -31,9 +31,9 @@ pub(crate) async fn update_mixnodes(
.await?;
// existing nodes get updated on insert
for record in mixnodes.iter() {
for record in mixnodes.into_iter() {
// https://www.sqlite.org/lang_upsert.html
sqlx::query!(
crate::db::query(
"INSERT INTO mixnodes
(mix_id, identity_key, bonded, total_stake,
host, http_api_port, full_details,
@@ -46,17 +46,17 @@ pub(crate) async fn update_mixnodes(
full_details=excluded.full_details,self_described=excluded.self_described,
last_updated_utc=excluded.last_updated_utc,
is_dp_delegatee = excluded.is_dp_delegatee;",
record.mix_id,
record.identity_key,
record.bonded,
record.total_stake,
record.host,
record.http_port,
record.full_details,
record.self_described,
record.last_updated_utc,
record.is_dp_delegatee
)
.bind(record.mix_id as i64)
.bind(record.identity_key)
.bind(record.bonded)
.bind(record.total_stake)
.bind(record.host)
.bind(record.http_port as i32)
.bind(record.full_details)
.bind(record.self_described)
.bind(record.last_updated_utc)
.bind(record.is_dp_delegatee)
.execute(&mut *tx)
.await?;
}
@@ -78,10 +78,10 @@ pub(crate) async fn get_all_mixnodes(pool: &DbPool) -> anyhow::Result<Vec<Mixnod
mn.full_details as "full_details!",
mn.self_described as "self_described",
mn.last_updated_utc as "last_updated_utc!",
COALESCE(md.moniker, "NA") as "moniker!",
COALESCE(md.website, "NA") as "website!",
COALESCE(md.security_contact, "NA") as "security_contact!",
COALESCE(md.details, "NA") as "details!"
md.moniker as "moniker!",
md.website as "website!",
md.security_contact as "security_contact!",
md.details as "details!"
FROM mixnodes mn
LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id
ORDER BY mn.mix_id"#
@@ -144,29 +144,29 @@ pub(crate) async fn get_daily_stats(pool: &DbPool) -> anyhow::Result<Vec<DailySt
pub(crate) async fn get_bonded_mix_ids(pool: &DbPool) -> anyhow::Result<HashSet<i64>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query!(
let items = crate::db::query(
r#"
SELECT mix_id
FROM mixnodes
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?
.into_iter()
.map(|record| record.mix_id)
.map(|record| record.try_get::<i64, _>("mix_id").unwrap())
.collect::<HashSet<_>>();
Ok(items)
}
pub(crate) async fn insert_mixnode_description(
conn: &mut PoolConnection<Sqlite>,
mix_id: &i64,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
mix_id: i64,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO mixnode_description (
mix_id, moniker, website, security_contact, details, last_updated_utc
@@ -178,13 +178,13 @@ pub(crate) async fn insert_mixnode_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
mix_id,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(mix_id)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
@@ -9,7 +9,8 @@ mod summary;
pub(crate) mod testruns;
pub(crate) use gateways::{
get_all_gateways, get_bonded_gateway_id_keys, select_gateway_identity, update_bonded_gateways,
get_all_gateways, get_bonded_gateway_id_keys, get_or_create_gateway, select_gateway_identity,
update_bonded_gateways,
};
pub(crate) use gateways_stats::{delete_old_records, get_sessions_stats, insert_session_records};
pub(crate) use misc::insert_summaries;
@@ -4,14 +4,14 @@ use nym_validator_client::{
client::{NodeId, NymNodeDetails},
models::NymNodeDescription,
};
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use std::collections::HashMap;
use tracing::instrument;
use crate::{
db::{
models::{NymNodeDto, NymNodeInsertRecord},
DbPool,
DbConnection, DbPool,
},
mixnet_scraper::helpers::NodeDescriptionResponse,
};
@@ -19,21 +19,20 @@ use crate::{
pub(crate) async fn get_all_nym_nodes(pool: &DbPool) -> anyhow::Result<Vec<NymNodeDto>> {
let mut conn = pool.acquire().await?;
sqlx::query_as!(
NymNodeDto,
crate::db::query_as::<NymNodeDto>(
r#"SELECT
node_id,
ed25519_identity_pubkey,
total_stake,
ip_addresses as "ip_addresses!: serde_json::Value",
ip_addresses,
mix_port,
x25519_sphinx_pubkey,
node_role as "node_role: serde_json::Value",
supported_roles as "supported_roles: serde_json::Value",
entry as "entry: serde_json::Value",
node_role,
supported_roles,
entry,
performance,
self_described as "self_described: serde_json::Value",
bond_info as "bond_info: serde_json::Value"
self_described,
bond_info
FROM
nym_nodes
ORDER BY
@@ -56,21 +55,20 @@ pub(crate) async fn get_described_bonded_nym_nodes(
) -> anyhow::Result<Vec<NymNodeDto>> {
let mut conn = pool.acquire().await?;
sqlx::query_as!(
NymNodeDto,
crate::db::query_as::<NymNodeDto>(
r#"SELECT
node_id,
ed25519_identity_pubkey,
total_stake,
ip_addresses as "ip_addresses!: serde_json::Value",
ip_addresses,
mix_port,
x25519_sphinx_pubkey,
node_role as "node_role: serde_json::Value",
supported_roles as "supported_roles: serde_json::Value",
entry as "entry: serde_json::Value",
node_role,
supported_roles,
entry,
performance,
self_described as "self_described: serde_json::Value",
bond_info as "bond_info: serde_json::Value"
self_described,
bond_info
FROM
nym_nodes
WHERE
@@ -92,7 +90,7 @@ pub(crate) async fn update_nym_nodes(
) -> anyhow::Result<usize> {
let mut tx = pool.begin().await?;
sqlx::query!(
crate::db::query(
"UPDATE nym_nodes
SET
self_described = NULL,
@@ -104,7 +102,7 @@ pub(crate) async fn update_nym_nodes(
let inserted = node_records.len();
for record in node_records {
// https://www.sqlite.org/lang_upsert.html
sqlx::query!(
crate::db::query(
"INSERT INTO nym_nodes
(node_id, ed25519_identity_pubkey,
total_stake,
@@ -129,20 +127,20 @@ pub(crate) async fn update_nym_nodes(
performance=excluded.performance,
last_updated_utc=excluded.last_updated_utc
;",
record.node_id,
record.ed25519_identity_pubkey,
record.total_stake,
record.ip_addresses,
record.mix_port,
record.x25519_sphinx_pubkey,
record.node_role,
record.supported_roles,
record.entry,
record.self_described,
record.bond_info,
record.performance,
record.last_updated_utc,
)
.bind(record.node_id)
.bind(record.ed25519_identity_pubkey)
.bind(record.total_stake)
.bind(record.ip_addresses)
.bind(record.mix_port as i32)
.bind(record.x25519_sphinx_pubkey)
.bind(record.node_role)
.bind(record.supported_roles)
.bind(record.entry)
.bind(record.self_described)
.bind(record.bond_info)
.bind(record.performance)
.bind(record.last_updated_utc)
.execute(&mut *tx)
.await
.map_err(|e| anyhow::anyhow!("Failed to INSERT node_id={}: {}", record.node_id, e))?;
@@ -158,10 +156,10 @@ pub(crate) async fn get_described_node_bond_info(
) -> anyhow::Result<HashMap<NodeId, NymNodeDetails>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
node_id,
bond_info as "bond_info: serde_json::Value"
bond_info
FROM
nym_nodes
WHERE
@@ -176,11 +174,11 @@ pub(crate) async fn get_described_node_bond_info(
records
.into_iter()
.filter_map(|record| {
record
.bond_info
// only return details for nodes which have details stored
.and_then(|bond_info| serde_json::from_value::<NymNodeDetails>(bond_info).ok())
.map(|res| (record.node_id as NodeId, res))
let node_id: i64 = record.try_get("node_id").ok()?;
let bond_info: serde_json::Value = record.try_get("bond_info").ok()?;
serde_json::from_value::<NymNodeDetails>(bond_info)
.ok()
.map(|res| (node_id as NodeId, res))
})
.collect::<HashMap<_, _>>()
})
@@ -192,10 +190,10 @@ pub(crate) async fn get_node_self_description(
) -> anyhow::Result<HashMap<NodeId, NymNodeDescription>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
node_id,
self_described as "self_described: serde_json::Value"
self_described
FROM
nym_nodes
WHERE
@@ -210,13 +208,11 @@ pub(crate) async fn get_node_self_description(
records
.into_iter()
.filter_map(|record| {
record
.self_described
// only return details for nodes which have details stored
.and_then(|description| {
serde_json::from_value::<NymNodeDescription>(description).ok()
})
.map(|res| (record.node_id as NodeId, res))
let node_id: i64 = record.try_get("node_id").ok()?;
let self_described: serde_json::Value = record.try_get("self_described").ok()?;
serde_json::from_value::<NymNodeDescription>(self_described)
.ok()
.map(|res| (node_id as NodeId, res))
})
.collect::<HashMap<_, _>>()
})
@@ -228,7 +224,7 @@ pub(crate) async fn get_bonded_node_description(
) -> anyhow::Result<HashMap<NodeId, NodeDescription>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
nd.node_id,
moniker,
@@ -249,14 +245,15 @@ pub(crate) async fn get_bonded_node_description(
records
.into_iter()
.map(|elem| {
let node_id: NodeId = elem.node_id.try_into().unwrap_or_default();
let node_id: i64 = elem.try_get("node_id").unwrap_or(0);
let node_id: NodeId = node_id.try_into().unwrap_or_default();
(
node_id,
NodeDescription {
moniker: elem.moniker.unwrap_or_default(),
website: elem.website.unwrap_or_default(),
security_contact: elem.security_contact.unwrap_or_default(),
details: elem.details.unwrap_or_default(),
moniker: elem.try_get("moniker").unwrap_or_default(),
website: elem.try_get("website").unwrap_or_default(),
security_contact: elem.try_get("security_contact").unwrap_or_default(),
details: elem.try_get("details").unwrap_or_default(),
},
)
})
@@ -266,12 +263,12 @@ pub(crate) async fn get_bonded_node_description(
}
pub(crate) async fn insert_nym_node_description(
conn: &mut PoolConnection<Sqlite>,
node_id: &i64,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
node_id: i64,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO nym_node_descriptions (
node_id, moniker, website, security_contact, details, last_updated_utc
@@ -283,13 +280,13 @@ pub(crate) async fn insert_nym_node_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
node_id,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(node_id)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
@@ -6,7 +6,7 @@ use anyhow::Result;
pub(crate) async fn insert_node_packet_stats(
pool: &DbPool,
node_kind: &ScrapeNodeKind,
node_kind: ScrapeNodeKind,
stats: &NodeStats,
timestamp_utc: i64,
) -> Result<()> {
@@ -14,35 +14,35 @@ pub(crate) async fn insert_node_packet_stats(
match node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO mixnode_packet_stats_raw (
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES (?, ?, ?, ?, ?)
"#,
mix_id,
timestamp_utc,
stats.packets_received,
stats.packets_sent,
stats.packets_dropped,
)
.bind(mix_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(&mut *conn)
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO nym_nodes_packet_stats_raw (
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES (?, ?, ?, ?, ?)
"#,
node_id,
timestamp_utc,
stats.packets_received,
stats.packets_sent,
stats.packets_dropped,
)
.bind(node_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(&mut *conn)
.await?;
}
@@ -53,7 +53,7 @@ pub(crate) async fn insert_node_packet_stats(
pub(crate) async fn get_raw_node_stats(
pool: &DbPool,
node: &ScraperNodeInfo,
node: ScraperNodeInfo,
) -> Result<Option<NodeStats>> {
let mut conn = pool.acquire().await?;
@@ -61,39 +61,37 @@ pub(crate) async fn get_raw_node_stats(
// if no packets are found, it's fine to assume 0 because that's also
// SQL default value if none provided
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query_as!(
NodeStats,
crate::db::query_as::<NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as "packets_received!: _",
COALESCE(packets_sent, 0) as "packets_sent!: _",
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM mixnode_packet_stats_raw
WHERE mix_id = ?
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
mix_id
)
.bind(mix_id)
.fetch_optional(&mut *conn)
.await?
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query_as!(
NodeStats,
crate::db::query_as::<NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as "packets_received!: _",
COALESCE(packets_sent, 0) as "packets_sent!: _",
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM nym_nodes_packet_stats_raw
WHERE node_id = ?
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
node_id
)
.bind(node_id)
.fetch_optional(&mut *conn)
.await?
}
@@ -104,27 +102,27 @@ pub(crate) async fn get_raw_node_stats(
pub(crate) async fn insert_daily_node_stats(
pool: &DbPool,
node: &ScraperNodeInfo,
date_utc: &str,
node: ScraperNodeInfo,
date_utc: String,
packets: NodeStats,
) -> Result<()> {
let mut conn = pool.acquire().await?;
match node.node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
let total_stake = sqlx::query_scalar!(
let total_stake = crate::db::query_scalar::<i64>(
r#"
SELECT
total_stake
FROM mixnodes
WHERE mix_id = ?
"#,
mix_id
)
.bind(mix_id)
.fetch_one(&mut *conn)
.await?;
sqlx::query!(
crate::db::query(
r#"
INSERT INTO mixnode_daily_stats (
mix_id, date_utc,
@@ -137,31 +135,31 @@ pub(crate) async fn insert_daily_node_stats(
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
"#,
mix_id,
date_utc,
total_stake,
packets.packets_received,
packets.packets_sent,
packets.packets_dropped,
)
.bind(mix_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(&mut *conn)
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
let total_stake = sqlx::query_scalar!(
let total_stake = crate::db::query_scalar::<i64>(
r#"
SELECT
total_stake
FROM nym_nodes
WHERE node_id = ?
"#,
node_id
)
.bind(node_id)
.fetch_one(&mut *conn)
.await?;
sqlx::query!(
crate::db::query(
r#"
INSERT INTO nym_node_daily_mixing_stats (
node_id, date_utc,
@@ -174,13 +172,13 @@ pub(crate) async fn insert_daily_node_stats(
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
"#,
node_id,
date_utc,
total_stake,
packets.packets_received,
packets.packets_sent,
packets.packets_dropped,
)
.bind(node_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(&mut *conn)
.await?;
}
@@ -12,6 +12,7 @@ use crate::{
};
use anyhow::Result;
use nym_validator_client::nym_api::SkimmedNode;
use sqlx::Row;
pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperNodeInfo>> {
let mut nodes_to_scrape = Vec::new();
@@ -68,12 +69,12 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
tracing::debug!("Fetched {} 🚪 entry/exit nodes", entry_exit_nodes);
let mut conn = pool.acquire().await?;
let mixnodes = sqlx::query!(
let mixnodes = crate::db::query(
r#"
SELECT mix_id as node_id, host, http_api_port
FROM mixnodes
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?;
@@ -85,18 +86,20 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
let mut legacy_not_in_nym_node_list = 0;
let total_legacy_mixnodes = mixnodes.len();
for mixnode in mixnodes {
let node_id: i64 = mixnode.try_get("node_id")?;
let host: String = mixnode.try_get("host")?;
let http_api_port: i64 = mixnode.try_get("http_api_port")?;
if nodes_to_scrape
.iter()
.all(|node| node.node_id() != &mixnode.node_id)
.all(|node| node.node_id() != &node_id)
{
// in case polyfilling on Nym API gets removed, this part ensures
// mixnodes are added to the final list of nodes to scrape
nodes_to_scrape.push(ScraperNodeInfo {
node_kind: ScrapeNodeKind::LegacyMixnode {
mix_id: mixnode.node_id,
},
hosts: vec![mixnode.host],
http_api_port: mixnode.http_api_port,
node_kind: ScrapeNodeKind::LegacyMixnode { mix_id: node_id },
hosts: vec![host],
http_api_port,
});
legacy_not_in_nym_node_list += 1;
@@ -121,8 +124,8 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
pub(crate) async fn insert_scraped_node_description(
pool: &DbPool,
node_kind: &ScrapeNodeKind,
description: &NodeDescriptionResponse,
node_kind: ScrapeNodeKind,
description: NodeDescriptionResponse,
) -> Result<()> {
let timestamp = now_utc().unix_timestamp();
let mut conn = pool.acquire().await?;
@@ -138,7 +141,7 @@ pub(crate) async fn insert_scraped_node_description(
node_id,
identity_key,
} => {
insert_nym_node_description(&mut conn, node_id, description, timestamp).await?;
insert_nym_node_description(&mut conn, node_id, description.clone(), timestamp).await?;
// for historic reasons (/gateways API), store this info into gateways table as well
insert_gateway_description(&mut conn, identity_key, description, timestamp).await?;
}
@@ -23,13 +23,12 @@ use crate::{
pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<SummaryHistory>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as!(
SummaryHistoryDto,
let items = crate::db::query_as::<SummaryHistoryDto>(
r#"SELECT
id as "id!",
date as "date!",
timestamp_utc as "timestamp_utc!",
value_json as "value_json!"
id,
date,
timestamp_utc,
value_json
FROM summary_history
ORDER BY date DESC
LIMIT 30"#,
@@ -51,13 +50,12 @@ pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<Sum
async fn get_summary_dto(pool: &DbPool) -> anyhow::Result<Vec<SummaryDto>> {
let mut conn = pool.acquire().await?;
Ok(sqlx::query_as!(
SummaryDto,
Ok(crate::db::query_as::<SummaryDto>(
r#"SELECT
key as "key!",
value_json as "value_json!",
last_updated_utc as "last_updated_utc!"
FROM summary"#
key,
value_json,
last_updated_utc
FROM summary"#,
)
.fetch(&mut *conn)
.try_collect::<Vec<_>>()
@@ -1,40 +1,38 @@
use crate::db::models::{TestRunDto, TestRunStatus};
use crate::db::DbConnection;
use crate::db::DbPool;
use crate::http::models::TestrunAssignment;
use crate::utils::now_utc;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use time::Duration;
pub(crate) async fn count_testruns_in_progress(
conn: &mut PoolConnection<Sqlite>,
) -> anyhow::Result<i64> {
sqlx::query_scalar!(
r#"SELECT
COUNT(id) as "count: i64"
FROM testruns
WHERE
status = ?
"#,
TestRunStatus::InProgress as i64,
)
.fetch_one(conn.as_mut())
.await
.map_err(anyhow::Error::from)
pub(crate) async fn count_testruns_in_progress(conn: &mut DbConnection) -> anyhow::Result<i64> {
#[cfg(feature = "sqlite")]
let sql = "SELECT COUNT(id) FROM testruns WHERE status = ?";
#[cfg(feature = "pg")]
let sql = "SELECT COUNT(id) FROM testruns WHERE status = $1";
let count: i64 = sqlx::query_scalar(sql)
.bind(TestRunStatus::InProgress as i64)
.fetch_one(conn.as_mut())
.await?;
Ok(count)
}
pub(crate) async fn get_in_progress_testrun_by_id(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
testrun_id: i64,
) -> anyhow::Result<TestRunDto> {
sqlx::query_as!(
TestRunDto,
crate::db::query_as::<TestRunDto>(
r#"SELECT
id as "id!",
gateway_id as "gateway_id!",
status as "status!",
created_utc as "created_utc!",
ip_address as "ip_address!",
log as "log!",
id,
gateway_id,
status,
created_utc,
ip_address,
log,
last_assigned_utc
FROM testruns
WHERE
@@ -43,9 +41,9 @@ pub(crate) async fn get_in_progress_testrun_by_id(
status = ?
ORDER BY created_utc
LIMIT 1"#,
testrun_id,
TestRunStatus::InProgress as i64,
)
.bind(testrun_id)
.bind(TestRunStatus::InProgress as i64)
.fetch_one(conn.as_mut())
.await
.map_err(|e| anyhow::anyhow!("Couldn't retrieve testrun {testrun_id}: {e}"))
@@ -59,7 +57,7 @@ pub(crate) async fn update_testruns_assigned_before(
let previous_run = now_utc() - max_age;
let cutoff_timestamp = previous_run.unix_timestamp();
let res = sqlx::query!(
let res = crate::db::query(
r#"UPDATE
testruns
SET
@@ -69,10 +67,10 @@ pub(crate) async fn update_testruns_assigned_before(
AND
last_assigned_utc < ?
"#,
TestRunStatus::Queued as i64,
TestRunStatus::InProgress as i64,
cutoff_timestamp
)
.bind(TestRunStatus::Queued as i64)
.bind(TestRunStatus::InProgress as i64)
.bind(cutoff_timestamp)
.execute(conn.as_mut())
.await?;
@@ -89,11 +87,11 @@ pub(crate) async fn update_testruns_assigned_before(
}
pub(crate) async fn assign_oldest_testrun(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
) -> anyhow::Result<Option<TestrunAssignment>> {
let now = now_utc().unix_timestamp();
// find & mark as "In progress" in the same transaction to avoid race conditions
let returning = sqlx::query!(
let returning = crate::db::query(
r#"UPDATE testruns
SET
status = ?,
@@ -107,18 +105,18 @@ pub(crate) async fn assign_oldest_testrun(
LIMIT 1
)
RETURNING
id as "id!",
id,
gateway_id
"#,
TestRunStatus::InProgress as i64,
now,
TestRunStatus::Queued as i64,
)
.bind(TestRunStatus::InProgress as i64)
.bind(now)
.bind(TestRunStatus::Queued as i64)
.fetch_optional(conn.as_mut())
.await?;
if let Some(testrun) = returning {
let gw_identity = sqlx::query!(
let gw_identity = crate::db::query(
r#"
SELECT
id,
@@ -126,14 +124,14 @@ pub(crate) async fn assign_oldest_testrun(
FROM gateways
WHERE id = ?
LIMIT 1"#,
testrun.gateway_id
)
.bind(testrun.try_get::<i64, _>("gateway_id")?)
.fetch_one(conn.as_mut())
.await?;
Ok(Some(TestrunAssignment {
testrun_id: testrun.id,
gateway_identity_key: gw_identity.gateway_identity_key,
testrun_id: testrun.try_get("id")?,
gateway_identity_key: gw_identity.try_get("gateway_identity_key")?,
assigned_at_utc: now,
}))
} else {
@@ -142,67 +140,134 @@ pub(crate) async fn assign_oldest_testrun(
}
pub(crate) async fn update_testrun_status(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
testrun_id: i64,
status: TestRunStatus,
) -> anyhow::Result<()> {
let status = status as u32;
sqlx::query!(
"UPDATE testruns SET status = ? WHERE id = ?",
status,
testrun_id
)
.execute(conn.as_mut())
.await?;
let status = status as i32;
crate::db::query("UPDATE testruns SET status = ? WHERE id = ?")
.bind(status)
.bind(testrun_id)
.execute(conn.as_mut())
.await?;
Ok(())
}
pub(crate) async fn update_gateway_last_probe_log(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
gateway_pk: i64,
log: &str,
log: String,
) -> anyhow::Result<()> {
sqlx::query!(
"UPDATE gateways SET last_probe_log = ? WHERE id = ?",
log,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_probe_log = ? WHERE id = ?")
.bind(log)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn update_gateway_last_probe_result(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
gateway_pk: i64,
result: &str,
result: String,
) -> anyhow::Result<()> {
sqlx::query!(
"UPDATE gateways SET last_probe_result = ? WHERE id = ?",
result,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_probe_result = ? WHERE id = ?")
.bind(result)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn update_gateway_score(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
gateway_pk: i64,
) -> anyhow::Result<()> {
let now = now_utc().unix_timestamp();
sqlx::query!(
"UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
now,
now,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?")
.bind(now)
.bind(now)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn get_testrun_by_id(
conn: &mut DbConnection,
testrun_id: i64,
) -> anyhow::Result<TestRunDto> {
crate::db::query_as::<TestRunDto>(
r#"SELECT
id,
gateway_id,
status,
created_utc,
ip_address,
log,
last_assigned_utc
FROM testruns
WHERE id = ?"#,
)
.bind(testrun_id)
.fetch_one(conn.as_mut())
.await
.map_err(|e| anyhow::anyhow!("Testrun {} not found: {}", testrun_id, e))
}
pub(crate) async fn insert_external_testrun(
conn: &mut DbConnection,
testrun_id: i64,
gateway_id: i64,
assigned_at_utc: i64,
) -> anyhow::Result<()> {
let now = crate::utils::now_utc().unix_timestamp();
crate::db::query(
r#"INSERT INTO testruns (
id,
gateway_id,
status,
created_utc,
last_assigned_utc,
ip_address,
log
) VALUES (?, ?, ?, ?, ?, ?, ?)"#,
)
.bind(testrun_id)
.bind(gateway_id)
.bind(TestRunStatus::InProgress as i64)
.bind(now)
.bind(assigned_at_utc)
.bind("external") // Marker for external origin
.bind("") // Empty initial log
.execute(conn.as_mut())
.await?;
tracing::debug!(
"Created external testrun {} for gateway {}",
testrun_id,
gateway_id
);
Ok(())
}
pub(crate) async fn update_testrun_status_by_gateway(
conn: &mut DbConnection,
gateway_id: i64,
status: TestRunStatus,
) -> anyhow::Result<()> {
let status = status as i32;
crate::db::query("UPDATE testruns SET status = ? WHERE gateway_id = ? AND status = ?")
.bind(status)
.bind(gateway_id)
.bind(TestRunStatus::InProgress as i32)
.execute(conn.as_mut())
.await?;
Ok(())
}
@@ -0,0 +1,144 @@
use sqlx::Database;
/// Converts SQLite-style ? placeholders to PostgreSQL $N format
#[cfg(feature = "pg")]
fn convert_placeholders(query: &str) -> String {
let mut result = String::with_capacity(query.len() + 10);
let mut placeholder_count = 0;
let chars = query.chars().peekable();
let mut in_string = false;
let mut escape_next = false;
for ch in chars {
if escape_next {
result.push(ch);
escape_next = false;
continue;
}
match ch {
'\\' => {
result.push(ch);
escape_next = true;
}
'\'' => {
result.push(ch);
in_string = !in_string;
}
'?' if !in_string => {
placeholder_count += 1;
result.push_str(&format!("${placeholder_count}"));
}
_ => {
result.push(ch);
}
}
}
result
}
/// Creates a query that automatically handles placeholder conversion
#[cfg(feature = "sqlite")]
pub fn query(
sql: &str,
) -> sqlx::query::Query<'_, sqlx::Sqlite, <sqlx::Sqlite as Database>::Arguments<'_>> {
sqlx::query(sql)
}
#[cfg(feature = "pg")]
pub fn query(
sql: &str,
) -> sqlx::query::Query<'static, sqlx::Postgres, <sqlx::Postgres as Database>::Arguments<'static>> {
let converted = convert_placeholders(sql);
sqlx::query(Box::leak(converted.into_boxed_str()))
}
/// Creates a query_as that automatically handles placeholder conversion
#[cfg(feature = "sqlite")]
pub fn query_as<O>(
sql: &str,
) -> sqlx::query::QueryAs<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
where
O: for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
{
sqlx::query_as(sql)
}
#[cfg(feature = "pg")]
pub fn query_as<O>(
sql: &str,
) -> sqlx::query::QueryAs<
'static,
sqlx::Postgres,
O,
<sqlx::Postgres as Database>::Arguments<'static>,
>
where
O: for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
{
let converted = convert_placeholders(sql);
sqlx::query_as(Box::leak(converted.into_boxed_str()))
}
/// Creates a query_scalar that automatically handles placeholder conversion
#[cfg(feature = "sqlite")]
pub fn query_scalar<O>(
sql: &str,
) -> sqlx::query::QueryScalar<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
where
(O,): for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
{
sqlx::query_scalar(sql)
}
#[cfg(feature = "pg")]
pub fn query_scalar<O>(
sql: &str,
) -> sqlx::query::QueryScalar<
'static,
sqlx::Postgres,
O,
<sqlx::Postgres as Database>::Arguments<'static>,
>
where
(O,): for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
{
let converted = convert_placeholders(sql);
sqlx::query_scalar(Box::leak(converted.into_boxed_str()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
#[cfg(feature = "pg")]
fn test_convert_placeholders() {
assert_eq!(
convert_placeholders("SELECT * FROM table WHERE id = ?"),
"SELECT * FROM table WHERE id = $1"
);
assert_eq!(
convert_placeholders("INSERT INTO table (a, b, c) VALUES (?, ?, ?)"),
"INSERT INTO table (a, b, c) VALUES ($1, $2, $3)"
);
assert_eq!(
convert_placeholders("SELECT * FROM table WHERE name = 'test?' AND id = ?"),
"SELECT * FROM table WHERE name = 'test?' AND id = $1"
);
assert_eq!(
convert_placeholders("UPDATE table SET a = ?, b = ? WHERE id = ?"),
"UPDATE table SET a = $1, b = $2 WHERE id = $3"
);
// Test with 10 placeholders (like in update_mixnodes)
assert_eq!(
convert_placeholders("INSERT INTO t VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"),
"INSERT INTO t VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)"
);
}
}
@@ -1,5 +1,6 @@
use crate::db::models::TestRunStatus;
use crate::db::models::{TestRunDto, TestRunStatus};
use crate::db::queries;
use crate::db::DbConnection;
use crate::utils::{now_utc, unix_timestamp_to_utc_rfc3339};
use crate::{
db,
@@ -17,7 +18,7 @@ use axum::{
};
use nym_node_status_client::{
auth::VerifiableRequest,
models::{get_testrun, submit_results},
models::{get_testrun, submit_results, submit_results_v2},
};
use reqwest::StatusCode;
use tracing::warn;
@@ -29,6 +30,7 @@ pub(crate) fn routes() -> Router<AppState> {
Router::new()
.route("/", axum::routing::get(request_testrun))
.route("/:testrun_id", axum::routing::post(submit_testrun))
.route("/:testrun_id/v2", axum::routing::post(submit_testrun_v2))
.layer(DefaultBodyLimit::max(1024 * 1024 * 5))
}
@@ -138,7 +140,7 @@ async fn submit_testrun(
queries::testruns::update_gateway_last_probe_log(
&mut conn,
assigned_testrun.gateway_id,
&submitted_result.payload.probe_result,
submitted_result.payload.probe_result.clone(),
)
.await
.map_err(HttpError::internal_with_logging)?;
@@ -146,7 +148,7 @@ async fn submit_testrun(
queries::testruns::update_gateway_last_probe_result(
&mut conn,
assigned_testrun.gateway_id,
&result,
result,
)
.await
.map_err(HttpError::internal_with_logging)?;
@@ -170,6 +172,72 @@ async fn submit_testrun(
Ok(StatusCode::CREATED)
}
#[tracing::instrument(level = "debug", skip_all)]
async fn submit_testrun_v2(
Path(submitted_testrun_id): Path<i64>,
State(state): State<AppState>,
Json(submission): Json<submit_results_v2::SubmitResultsV2>,
) -> HttpResult<StatusCode> {
authenticate(&submission, &state)?;
is_fresh(&submission.payload.assigned_at_utc)?;
let db = state.db_pool();
let mut conn = db
.acquire()
.await
.map_err(HttpError::internal_with_logging)?;
// Try to find existing testrun
match queries::testruns::get_testrun_by_id(&mut conn, submitted_testrun_id).await {
Ok(testrun) => {
// Validate it matches the submission
let gw_identity = queries::select_gateway_identity(&mut conn, testrun.gateway_id)
.await
.map_err(HttpError::internal_with_logging)?;
if gw_identity != submission.payload.gateway_identity_key {
tracing::warn!(
"Gateway mismatch for testrun {}: expected {}, got {}",
submitted_testrun_id,
gw_identity,
submission.payload.gateway_identity_key
);
return Err(HttpError::invalid_input("Gateway identity mismatch"));
}
// Process normally using existing testrun
process_testrun_submission(testrun, submission.payload, &mut conn).await
}
Err(_) => {
// External testrun - create records
tracing::info!(
"Creating external testrun {} for gateway {}",
submitted_testrun_id,
submission.payload.gateway_identity_key
);
// Get or create gateway
let gateway_id =
queries::get_or_create_gateway(&mut conn, &submission.payload.gateway_identity_key)
.await
.map_err(HttpError::internal_with_logging)?;
// Create testrun
queries::testruns::insert_external_testrun(
&mut conn,
submitted_testrun_id,
gateway_id,
submission.payload.assigned_at_utc,
)
.await
.map_err(HttpError::internal_with_logging)?;
// Process submission
process_testrun_submission_by_gateway(gateway_id, submission.payload, &mut conn).await
}
}
}
// TODO dz this should be middleware
#[tracing::instrument(level = "debug", skip_all)]
fn authenticate(request: &impl VerifiableRequest, state: &AppState) -> HttpResult<()> {
@@ -212,3 +280,70 @@ fn get_result_from_log(log: &str) -> String {
}
"".to_string()
}
async fn process_testrun_submission(
testrun: TestRunDto,
payload: submit_results_v2::Payload,
conn: &mut DbConnection,
) -> HttpResult<StatusCode> {
// Validate timestamp matches
if Some(payload.assigned_at_utc) != testrun.last_assigned_utc {
tracing::warn!(
"Submitted testrun timestamp mismatch: {} != {:?}, rejecting",
payload.assigned_at_utc,
testrun.last_assigned_utc
);
return Err(HttpError::invalid_input("Invalid testrun submitted"));
}
// Process the submission
process_testrun_submission_by_gateway(testrun.gateway_id, payload, conn).await
}
async fn process_testrun_submission_by_gateway(
gateway_id: i64,
payload: submit_results_v2::Payload,
conn: &mut DbConnection,
) -> HttpResult<StatusCode> {
let gw_identity = &payload.gateway_identity_key;
tracing::debug!(
"Processing testrun submission for gateway {} ({} bytes)",
gw_identity,
payload.probe_result.len(),
);
// Update testrun status to complete
queries::testruns::update_testrun_status_by_gateway(conn, gateway_id, TestRunStatus::Complete)
.await
.map_err(HttpError::internal_with_logging)?;
// Update gateway with results
queries::testruns::update_gateway_last_probe_log(
conn,
gateway_id,
payload.probe_result.clone(),
)
.await
.map_err(HttpError::internal_with_logging)?;
let result = get_result_from_log(&payload.probe_result);
queries::testruns::update_gateway_last_probe_result(conn, gateway_id, result)
.await
.map_err(HttpError::internal_with_logging)?;
queries::testruns::update_gateway_score(conn, gateway_id)
.await
.map_err(HttpError::internal_with_logging)?;
let assigned_at = unix_timestamp_to_utc_rfc3339(payload.assigned_at_utc);
let now = now_utc();
tracing::info!(
"✅ Testrun for gateway {} complete (assigned at {}, current time {})",
gw_identity,
assigned_at,
now
);
Ok(StatusCode::CREATED)
}
@@ -5,6 +5,12 @@ use nym_task::signal::wait_for_signal;
use nym_validator_client::nyxd::NyxdClient;
use std::sync::Arc;
#[cfg(all(feature = "sqlite", feature = "pg"))]
compile_error!("Features 'sqlite' and 'pg' are mutually exclusive");
#[cfg(not(any(feature = "sqlite", feature = "pg")))]
compile_error!("Either 'sqlite' or 'pg' feature must be enabled");
mod cli;
mod db;
mod http;
@@ -5,6 +5,7 @@ use crate::{
get_raw_node_stats, insert_daily_node_stats, insert_node_packet_stats,
insert_scraped_node_description,
},
DbPool,
},
utils::{generate_node_name, now_utc},
};
@@ -12,11 +13,10 @@ use ammonia::Builder;
use anyhow::{anyhow, Result};
use reqwest;
use serde::{Deserialize, Serialize};
use sqlx::SqlitePool;
use std::time::Duration;
use time::UtcDateTime;
#[derive(Debug, Serialize, Deserialize)]
#[derive(Debug, Serialize, Deserialize, Clone)]
pub struct NodeDescriptionResponse {
pub moniker: Option<String>,
pub website: Option<String>,
@@ -116,7 +116,7 @@ pub fn sanitize_description(
}
}
pub async fn scrape_and_store_description(pool: &SqlitePool, node: &ScraperNodeInfo) -> Result<()> {
pub async fn scrape_and_store_description(pool: &DbPool, node: ScraperNodeInfo) -> Result<()> {
let client = build_client()?;
let urls = node.contact_addresses();
@@ -151,15 +151,12 @@ pub async fn scrape_and_store_description(pool: &SqlitePool, node: &ScraperNodeI
})?;
let sanitized_description = sanitize_description(description, *node.node_id());
insert_scraped_node_description(pool, &node.node_kind, &sanitized_description).await?;
insert_scraped_node_description(pool, node.node_kind.clone(), sanitized_description).await?;
Ok(())
}
pub async fn scrape_and_store_packet_stats(
pool: &SqlitePool,
node: &ScraperNodeInfo,
) -> Result<()> {
pub async fn scrape_and_store_packet_stats(pool: &DbPool, node: ScraperNodeInfo) -> Result<()> {
let client = build_client()?;
let urls = node.contact_addresses();
@@ -189,7 +186,7 @@ pub async fn scrape_and_store_packet_stats(
let timestamp = now_utc();
let timestamp_utc = timestamp.unix_timestamp();
insert_node_packet_stats(pool, &node.node_kind, &stats, timestamp_utc).await?;
insert_node_packet_stats(pool, node.node_kind.clone(), &stats, timestamp_utc).await?;
// Update daily stats
update_daily_stats(pool, node, timestamp, &stats).await?;
@@ -198,8 +195,8 @@ pub async fn scrape_and_store_packet_stats(
}
pub async fn update_daily_stats(
pool: &SqlitePool,
node: &ScraperNodeInfo,
pool: &DbPool,
node: ScraperNodeInfo,
timestamp: UtcDateTime,
current_stats: &NodeStats,
) -> Result<()> {
@@ -211,7 +208,7 @@ pub async fn update_daily_stats(
);
// Get previous stats
let previous_stats = get_raw_node_stats(pool, node).await?;
let previous_stats = get_raw_node_stats(pool, node.clone()).await?;
let (diff_received, diff_sent, diff_dropped) = if let Some(prev) = previous_stats {
(
@@ -226,7 +223,7 @@ pub async fn update_daily_stats(
insert_daily_node_stats(
pool,
node,
&date_utc,
date_utc,
NodeStats {
packets_received: diff_received,
packets_sent: diff_sent,
@@ -2,9 +2,9 @@ use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
pub mod helpers;
use crate::db::DbPool;
use anyhow::Result;
use helpers::{scrape_and_store_description, scrape_and_store_packet_stats};
use sqlx::SqlitePool;
use tracing::{debug, error, instrument, warn};
use crate::db::models::ScraperNodeInfo;
@@ -19,13 +19,13 @@ static TASK_COUNTER: AtomicUsize = AtomicUsize::new(0);
static TASK_ID_COUNTER: AtomicUsize = AtomicUsize::new(0);
pub struct Scraper {
pool: SqlitePool,
pool: DbPool,
description_queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
packet_queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
}
impl Scraper {
pub fn new(pool: SqlitePool) -> Self {
pub fn new(pool: DbPool) -> Self {
Self {
pool,
description_queue: Arc::new(Mutex::new(Vec::new())),
@@ -71,7 +71,7 @@ impl Scraper {
#[instrument(level = "info", name = "description_scraper", skip_all)]
async fn run_description_scraper(
pool: &SqlitePool,
pool: &DbPool,
queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
) -> Result<()> {
let nodes = get_nodes_for_scraping(pool).await?;
@@ -88,7 +88,7 @@ impl Scraper {
#[instrument(level = "info", name = "packet_scraper", skip_all)]
async fn run_packet_scraper(
pool: &SqlitePool,
pool: &DbPool,
queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
) -> Result<()> {
let nodes = get_nodes_for_scraping(pool).await?;
@@ -104,7 +104,7 @@ impl Scraper {
Ok(())
}
async fn process_description_queue(pool: &SqlitePool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
async fn process_description_queue(pool: &DbPool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
loop {
let running_tasks = TASK_COUNTER.load(Ordering::Relaxed);
@@ -127,7 +127,7 @@ impl Scraper {
let pool = pool.clone();
tokio::spawn(async move {
match scrape_and_store_description(&pool, &node).await {
match scrape_and_store_description(&pool, node.clone()).await {
Ok(_) => debug!(
"📝 ✅ Description task #{} for node {} complete",
task_id,
@@ -149,7 +149,7 @@ impl Scraper {
}
}
async fn process_packet_queue(pool: &SqlitePool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
async fn process_packet_queue(pool: &DbPool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
loop {
let running_tasks = TASK_COUNTER.load(Ordering::Relaxed);
@@ -172,7 +172,7 @@ impl Scraper {
let pool = pool.clone();
tokio::spawn(async move {
match scrape_and_store_packet_stats(&pool, &node).await {
match scrape_and_store_packet_stats(&pool, node.clone()).await {
Ok(_) => debug!(
"📊 ✅ Packet stats task #{} for node {} complete",
task_id,
@@ -252,17 +252,23 @@ impl Monitor {
//
let nodes_summary = vec![
(NYMNODE_COUNT, nym_node_count),
(ASSIGNED_MIXING_COUNT, assigned_mixing_count),
(MIXNODES_LEGACY_COUNT, count_legacy_mixnodes),
(NYMNODES_DESCRIBED_COUNT, described_nodes.len()),
(GATEWAYS_BONDED_COUNT, count_bonded_gateways),
(ASSIGNED_ENTRY_COUNT, assigned_entry_count),
(ASSIGNED_EXIT_COUNT, assigned_exit_count),
(NYMNODE_COUNT.to_string(), nym_node_count),
(ASSIGNED_MIXING_COUNT.to_string(), assigned_mixing_count),
(MIXNODES_LEGACY_COUNT.to_string(), count_legacy_mixnodes),
(NYMNODES_DESCRIBED_COUNT.to_string(), described_nodes.len()),
(GATEWAYS_BONDED_COUNT.to_string(), count_bonded_gateways),
(ASSIGNED_ENTRY_COUNT.to_string(), assigned_entry_count),
(ASSIGNED_EXIT_COUNT.to_string(), assigned_exit_count),
// TODO dz doesn't make sense, could make sense with historical Nym
// Nodes if we really need this data
(MIXNODES_HISTORICAL_COUNT, all_historical_mixnodes),
(GATEWAYS_HISTORICAL_COUNT, all_historical_gateways),
(
MIXNODES_HISTORICAL_COUNT.to_string(),
all_historical_mixnodes,
),
(
GATEWAYS_HISTORICAL_COUNT.to_string(),
all_historical_gateways,
),
];
let last_updated = now_utc();
@@ -295,7 +301,8 @@ impl Monitor {
},
};
queries::insert_summaries(&pool, &nodes_summary, &network_summary, last_updated).await?;
queries::insert_summaries(&pool, nodes_summary.clone(), network_summary, last_updated)
.await?;
let mut log_lines: Vec<String> = vec![];
for (key, value) in nodes_summary.iter() {
@@ -495,15 +502,31 @@ impl Monitor {
async fn historical_count(pool: &DbPool) -> anyhow::Result<(usize, usize)> {
let mut conn = pool.acquire().await?;
#[cfg(feature = "sqlite")]
let all_historical_gateways = sqlx::query_scalar!(r#"SELECT count(id) FROM gateways"#)
.fetch_one(&mut *conn)
.await?
.cast_checked()?;
#[cfg(feature = "pg")]
let all_historical_gateways = sqlx::query_scalar!(r#"SELECT count(id) FROM gateways"#)
.fetch_one(&mut *conn)
.await?
.unwrap_or(0)
.cast_checked()?;
#[cfg(feature = "sqlite")]
let all_historical_mixnodes = sqlx::query_scalar!(r#"SELECT count(id) FROM mixnodes"#)
.fetch_one(&mut *conn)
.await?
.cast_checked()?;
#[cfg(feature = "pg")]
let all_historical_mixnodes = sqlx::query_scalar!(r#"SELECT count(id) FROM mixnodes"#)
.fetch_one(&mut *conn)
.await?
.unwrap_or(0)
.cast_checked()?;
Ok((all_historical_gateways, all_historical_mixnodes))
}
@@ -9,7 +9,7 @@ pub struct GatewayIdentityDto {
#[derive(Debug, Clone, Deserialize, Serialize, utoipa::ToSchema)]
pub struct TestRun {
pub id: u32,
pub id: i32,
pub identity_key: String,
pub status: String,
pub log: String,
@@ -1,13 +1,12 @@
use crate::db::models::{GatewayInfoDto, TestRunDto, TestRunStatus};
use crate::db::DbConnection;
use crate::testruns::models::TestRun;
use crate::utils::now_utc;
use anyhow::anyhow;
use futures_util::TryStreamExt;
use sqlx::pool::PoolConnection;
use sqlx::Sqlite;
pub(crate) async fn try_queue_testrun(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
identity_key: String,
ip_address: String,
) -> anyhow::Result<TestRun> {
@@ -15,20 +14,19 @@ pub(crate) async fn try_queue_testrun(
let timestamp = now.unix_timestamp();
let timestamp_pretty = now.to_string();
let items = sqlx::query_as!(
GatewayInfoDto,
let items = crate::db::query_as::<GatewayInfoDto>(
r#"SELECT
id as "id!",
gateway_identity_key as "gateway_identity_key!",
self_described as "self_described?",
explorer_pretty_bond as "explorer_pretty_bond?"
id,
gateway_identity_key,
self_described,
explorer_pretty_bond
FROM gateways
WHERE gateway_identity_key = ?
AND bonded = true
ORDER BY gateway_identity_key
LIMIT 1"#,
identity_key,
)
.bind(identity_key.clone())
// TODO dz should call .fetch_one
// TODO dz replace this in other queries as well
.fetch(conn.as_mut())
@@ -48,22 +46,21 @@ pub(crate) async fn try_queue_testrun(
//
// check if there is already a test run for this gateway
//
let items = sqlx::query_as!(
TestRunDto,
let items = crate::db::query_as::<TestRunDto>(
r#"SELECT
id as "id!",
gateway_id as "gateway_id!",
status as "status!",
created_utc as "created_utc!",
ip_address as "ip_address!",
log as "log!",
id,
gateway_id,
status,
created_utc,
ip_address,
log,
last_assigned_utc
FROM testruns
WHERE gateway_id = ? AND status != 2
ORDER BY id DESC
LIMIT 1"#,
gateway_id,
)
.bind(gateway_id)
.fetch(conn.as_mut())
.try_collect::<Vec<_>>()
.await?;
@@ -71,8 +68,8 @@ pub(crate) async fn try_queue_testrun(
if !items.is_empty() {
let testrun = items.first().unwrap();
return Ok(TestRun {
id: testrun.id as u32,
identity_key,
id: testrun.id as i32,
identity_key: identity_key.clone(),
status: format!(
"{}",
TestRunStatus::from_repr(testrun.status as u8).unwrap()
@@ -84,24 +81,43 @@ pub(crate) async fn try_queue_testrun(
//
// save test run
//
let status = TestRunStatus::Queued as u32;
let status = TestRunStatus::Queued as i32;
let log = format!("Test for {identity_key} requested at {timestamp_pretty} UTC\n\n");
let id = sqlx::query!(
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
gateway_id,
status,
ip_address,
timestamp,
log,
)
#[cfg(feature = "sqlite")]
let id = {
sqlx::query!(
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
gateway_id,
status,
ip_address,
timestamp,
log,
)
.execute(conn.as_mut())
.await?
.last_insert_rowid();
.last_insert_rowid()
};
#[cfg(feature = "pg")]
let id = {
let record = sqlx::query!(
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
gateway_id as i32,
status,
ip_address,
timestamp,
log,
)
.fetch_one(conn.as_mut())
.await?;
record.id
};
Ok(TestRun {
id: id as u32,
identity_key,
#[allow(clippy::useless_conversion)]
id: id.try_into().unwrap(),
identity_key: identity_key.clone(),
status: format!("{}", TestRunStatus::Queued),
log,
})
@@ -16,8 +16,11 @@ readme.workspace = true
[dependencies]
anyhow = { workspace = true }
bincode = { workspace = true }
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "serde"] }
nym-http-api-client = { path = "../../common/http-api-client" }
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"asymmetric",
"serde",
] }
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
reqwest = { workspace = true }
serde = { workspace = true, features = ["derive"] }
serde_json = { workspace = true }
@@ -15,4 +15,11 @@ impl ApiPaths {
pub(super) fn submit_results(&self, testrun_id: impl Display) -> String {
format!("{}/internal/testruns/{}", self.server_address, testrun_id)
}
pub(super) fn submit_results_v2(&self, testrun_id: impl Display) -> String {
format!(
"{}/internal/testruns/{}/v2",
self.server_address, testrun_id
)
}
}
@@ -1,4 +1,4 @@
use crate::models::{get_testrun, submit_results, TestrunAssignment};
use crate::models::{get_testrun, submit_results, submit_results_v2, TestrunAssignment};
use anyhow::bail;
use api::ApiPaths;
use nym_crypto::asymmetric::ed25519::{PrivateKey, Signature};
@@ -94,6 +94,37 @@ impl NsApiClient {
Ok(())
}
#[instrument(level = "debug", skip(self, probe_result))]
pub async fn submit_results_with_context(
&self,
testrun_id: i64,
probe_result: String,
assigned_at_utc: i64,
gateway_identity_key: String,
) -> anyhow::Result<()> {
let target_url = self.api.submit_results_v2(testrun_id);
let payload = submit_results_v2::Payload {
probe_result,
agent_public_key: self.auth_key.public_key(),
assigned_at_utc,
gateway_identity_key,
};
let signature = self.sign_message(&payload)?;
let submit_results = submit_results_v2::SubmitResultsV2 { payload, signature };
let res = self
.client
.post(target_url)
.json(&submit_results)
.send()
.await
.and_then(|response| response.error_for_status())?;
tracing::debug!("Submitted results with context: {})", res.status());
Ok(())
}
fn sign_message<T>(&self, message: &T) -> anyhow::Result<Signature>
where
T: serde::Serialize,
@@ -74,3 +74,38 @@ pub mod submit_results {
}
}
}
pub mod submit_results_v2 {
use crate::auth::SignedRequest;
use super::*;
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct Payload {
pub probe_result: String,
pub agent_public_key: PublicKey,
pub assigned_at_utc: i64,
pub gateway_identity_key: String,
}
#[derive(Debug, Clone, Deserialize, Serialize)]
pub struct SubmitResultsV2 {
pub payload: Payload,
pub signature: Signature,
}
impl SignedRequest for SubmitResultsV2 {
type Payload = Payload;
fn public_key(&self) -> &PublicKey {
&self.payload.agent_public_key
}
fn signature(&self) -> &Signature {
&self.signature
}
fn payload(&self) -> &Self::Payload {
&self.payload
}
}
}