Allow PG database backend (#5880)

* feat(db): add SQL query wrapper for PostgreSQL placeholder conversion

- Created query_wrapper module with functions to automatically convert
  SQLite ? placeholders to PostgreSQL $1, $2, ... format
- Updated build.rs to handle mutually exclusive feature flags
- Modified one query in mixnodes.rs as proof of concept
- Added type conversions for PostgreSQL compatibility (u32->i64, u16->i32)

This is a checkpoint commit before converting all queries to use the wrapper.

* feat(nym-node-status-api): add PostgreSQL database support via feature flags

Implement dual database support for SQLite and PostgreSQL through Cargo feature flags.
The implementation uses a query wrapper that automatically converts SQLite-style ?
placeholders to PostgreSQL-style $1, $2, ... placeholders at runtime.

Key changes:
- Add query wrapper functions that handle placeholder conversion
- Convert all sqlx::query\! macros to use wrapper functions
- Handle type conversions between databases (i64 vs i32)
- Add feature-gated implementations for database-specific SQL syntax
- Update Makefile with clippy targets for both database features
- Document database support in README

* feat(nym-node-status-agent): add multi-API support with random selection

Agents can now connect to multiple APIs and randomly select one for each testrun:
- Accept multiple --server arguments in format "address:port:auth_key"
- Randomly shuffle server list before attempting connections
- Try each server until a testrun is obtained
- Submit results back only to the API that provided the testrun
- Continue to next server if one is down or has no testruns available

* feat(nym-node-status): implement primary/secondary server architecture

- Agent now requests testruns only from primary server (first in list)
- Results are submitted to all configured servers in parallel
- Secondary servers accept external testruns via new v2 endpoint
- Added auto-creation of gateway and testrun records on secondary servers
- New database queries: get_or_create_gateway, insert_external_testrun
- Client library enhanced with submit_results_with_context method

* Bump Node status API version

* Fix build workdir

* Bump to 3.1.4

* Fix types and queries

* 3.1.6

* Fix gateway perf, bump 3.1.7

* NodeId -> i32, 3.1.8

* Bump agent version

* i64 -> i32

* Use image yq

* Migration and more types

* Update remaining JSONB columns

* Simplify server config

* Update build path

* Change delimiter

* bump agent

* Split up pg and sqlite builds

* More typing fixes, build-and-push script

* Fix Dockerfile-pg

* Bump node-status-api

* TYping

* Agent build script

* More logging around testruns

* Fail loudly on read errors

* Cleanup

* Debug get gateways query

* Fix get_gateways query

* Use pg cert, 3.1.16

* Submit regular results to primary server

* Bump freshenss cutoff

* Update Cargo.lock

* fix: resolve rebase conflicts and compilation errors

After rebasing onto develop, fixed several issues:
- Fixed borrowed data escapes error by using sqlx::query directly in transaction functions
- Removed unused imports and cleaned up code
- Maintained database-specific implementations for transaction functions

* fmt

* Make PG default to make lives easier

* Performance improvements for Explorer v2

* Fix sqlite build

* Fix PG migration

* Tests round 1

* DB tests

* More tests

* And some more tests

* And some more, more tests

* cargo fmt

* Fix some failing lints

* Fix lioness version problems

* Clippy in tests

---------

Co-authored-by: dynco-nym <173912580+dynco-nym@users.noreply.github.com>
This commit is contained in:
Drazen Urch
2025-07-22 15:25:43 +02:00
committed by GitHub
parent a9ae2017f5
commit af9f6e5ca0
129 changed files with 6752 additions and 2761 deletions
@@ -38,10 +38,9 @@ jobs:
git config --global user.name "Lawrence Stalder"
- name: Get version from cargo.toml
uses: mikefarah/yq@v4.45.4
id: get_version
with:
cmd: yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
run: |
yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
- name: cleanup-gateway-probe-ref
id: cleanup_gateway_probe_ref
+2 -3
View File
@@ -32,10 +32,9 @@ jobs:
git config --global user.name "Lawrence Stalder"
- name: Get version from cargo.toml
uses: mikefarah/yq@v4.45.4
id: get_version
with:
cmd: yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
run: |
yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
- name: Set GIT_TAG variable
run: echo "GIT_TAG=${{ env.WORKING_DIRECTORY }}-${{ steps.get_version.outputs.result }}" >> $GITHUB_ENV
+686
View File
@@ -0,0 +1,686 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
Nym is a privacy platform that uses mixnet technology to protect against metadata surveillance. The platform consists of several key components:
- Mixnet nodes (mixnodes) for packet mixing
- Gateways (entry/exit points for the network)
- Clients for interacting with the network
- Network monitoring tools
- Validators for network consensus
- Various service providers and integrations
## Build Commands
### Rust Components
```bash
# Default build (debug)
cargo build
# Release build
cargo build --release
# Build a specific package
cargo build -p <package-name>
# Build main components
make build
# Build release versions of main binaries and contracts
make build-release
# Build specific binaries
make build-nym-cli
cargo build -p nym-node --release
cargo build -p nym-api --release
```
### Testing
```bash
# Run clippy, unit tests, and formatting
make test
# Run all tests including slow tests
make test-all
# Run clippy on all workspaces
make clippy
# Run unit tests for a specific package
cargo test -p <package-name>
# Run only expensive/ignored tests
cargo test --workspace -- --ignored
# Run API tests
dotenv -f envs/sandbox.env -- cargo test --test public-api-tests
# Run tests with specific log level
RUST_LOG=debug cargo test -p <package-name>
# Run specific test scripts
./nym-node/tests/test_apis.sh
./scripts/wireguard-exit-policy/exit-policy-tests.sh
```
### Linting and Formatting
```bash
# Run rustfmt on all code
make fmt
# Check formatting without modifying
cargo fmt --all -- --check
# Run clippy with all targets
cargo clippy --workspace --all-targets -- -D warnings
# TypeScript linting
yarn lint
yarn lint:fix
yarn types:lint:fix
# Check dependencies for security/licensing issues
cargo deny check
```
### WASM Components
```bash
# Build all WASM components
make sdk-wasm-build
# Build TypeScript SDK
yarn build:sdk
npx lerna run --scope @nymproject/sdk build --stream
# Build and test WASM components
make sdk-wasm
# Build specific WASM packages
cd wasm/client && make
cd wasm/mix-fetch && make
cd wasm/node-tester && make
```
### Contract Development
```bash
# Build all contracts
make contracts
# Build contracts in release mode
make build-release-contracts
# Generate contract schemas
make contract-schema
# Run wasm-opt on contracts
make wasm-opt-contracts
# Check contracts with cosmwasm-check
make cosmwasm-check-contracts
```
### Running Components
```bash
# Run nym-node as a mixnode
cargo run -p nym-node -- run --mode mixnode
# Run nym-node as a gateway
cargo run -p nym-node -- run --mode gateway
# Run the network monitor
cargo run -p nym-network-monitor
# Run the API server
cargo run -p nym-api
# Run with specific environment
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Start a local network
./scripts/localnet_start.sh
```
## Architecture
The Nym platform consists of various components organized as a monorepo:
1. **Core Mixnet Infrastructure**:
- `nym-node`: Core binary supporting mixnode and gateway modes
- `common/nymsphinx`: Implementation of the Sphinx packet format
- `common/topology`: Network topology management
- `common/types`: Shared data types across components
2. **Network Monitoring**:
- `nym-network-monitor`: Monitors the network's reliability and performance
- `nym-api`: API server for network stats and monitoring data
- Metrics tracking for nodes, routes, and overall network health
3. **Client Implementations**:
- `clients/native`: Native Rust client implementation
- `clients/socks5`: SOCKS5 proxy client for standard applications
- `wasm`: WebAssembly client implementations (for browsers)
- `nym-connect`: Desktop and mobile clients
4. **Blockchain & Smart Contracts**:
- `common/cosmwasm-smart-contracts`: Smart contract implementations
- `contracts`: CosmWasm contracts for the Nym network
- `common/ledger`: Blockchain integration
5. **Utilities & Tools**:
- `tools`: Various CLI tools and utilities
- `sdk`: SDKs for different languages and platforms
- `documentation`: Documentation generation and management
## Packet System
Nym uses a modified Sphinx packet format for its mixnet:
1. **Message Chunking**:
- Messages are divided into "sets" and "fragments"
- Each fragment fits in a single Sphinx packet
- The `common/nymsphinx/chunking` module handles message fragmentation
2. **Routing**:
- Packets traverse through 3 layers of mixnodes
- Routing information is encrypted in layers (onion routing)
- The final gateway receives and processes the messages
3. **Monitoring**:
- Monitoring system tracks packet delivery through the network
- Routes are analyzed for reliability statistics
- Node performance metrics are collected
## Network Protocol
Nym implements the Loopix mixnet design with several key privacy features:
1. **Continuous-time Mixing**:
- Each mixnode delays messages independently with an exponential distribution
- This creates random reordering of packets, destroying timing correlations
- Offers better anonymity properties than batch mixing approaches
2. **Cover Traffic**:
- Clients and nodes generate dummy "loop" packets that circulate through the network
- These packets are indistinguishable from real traffic
- Creates a baseline level of traffic that hides actual communication patterns
- Provides unobservability (hiding when and how much real traffic is being sent)
3. **Stratified Network Architecture**:
- Traffic flows through Entry Gateway → 3 Mixnode Layers → Exit Gateway
- Path selection is independent per-message (unlike Tor)
- Each node connects only to adjacent layers
4. **Anonymous Replies**:
- Single-Use Reply Blocks (SURBs) allow receiving messages without revealing identity
- Enables bidirectional communication while maintaining privacy
## Network Monitoring Architecture
The network monitoring system is a core component that measures mixnet reliability:
1. The `nym-network-monitor` sends test packets through the network
2. These packets follow predefined routes through multiple mixnodes
3. Metrics are collected about:
- Successful and failed packet deliveries
- Node reliability (percentage of successful packet handling)
- Route reliability (which specific route combinations work best)
4. Results are stored in the database and used by `nym-api` to:
- Present node performance statistics
- Determine network rewards
- Provide route selection guidance to clients
In the current branch, metrics collection is being enhanced with a fanout approach to submit to multiple API endpoints.
## Development Environment
### Required Dependencies
- Rust toolchain (stable, 1.80+)
- Node.js (v20+) and yarn for TypeScript components
- SQLite for local database development
- PostgreSQL for API database (optional, for full API functionality)
- CosmWasm tools for contract development
- For building contracts: `wasm-opt` tool from `binaryen`
- Python 3.8+ for some scripts
- Docker (optional, for containerized development)
- protoc (Protocol Buffers compiler) for some components
### Environment Configurations
The `envs/` directory contains pre-configured environments:
#### Available Environments
- **`local.env`**: Local development environment
- Points to local services (localhost)
- Uses test mnemonics and keys
- Ideal for testing without external dependencies
- **`sandbox.env`**: Sandbox test network
- Public test network with real nodes
- Test tokens available from faucet
- Contract addresses for sandbox deployment
- API: https://sandbox-nym-api1.nymtech.net
- **`mainnet.env`**: Production mainnet
- Real network with real tokens
- Production contract addresses
- API: https://validator.nymtech.net
- Use with caution!
- **`canary.env`**: Canary deployment
- Pre-release testing environment
- Tests new features before mainnet
- **`mainnet-local-api.env`**: Hybrid environment
- Uses mainnet contracts but local API
- Useful for API development against mainnet data
#### Key Environment Variables
```bash
# Network configuration
NETWORK_NAME=sandbox # Network identifier
BECH32_PREFIX=n # Address prefix (n for sandbox, n for mainnet)
NYM_API=https://sandbox-nym-api1.nymtech.net/api
NYXD=https://rpc.sandbox.nymtech.net
NYM_API_NETWORK=sandbox
# Contract addresses (network-specific)
MIXNET_CONTRACT_ADDRESS=n1xr3rq8yvd7qplsw5yx90ftsr2zdhg4e9z60h5duusgxpv72hud3sjkxkav
VESTING_CONTRACT_ADDRESS=n1unyuj8qnmygvzuex3dwmg9yzt9alhvyeat0uu0jedg2wj33efl5qackslz
# ... other contract addresses
# Mnemonic for testing (NEVER use in production)
MNEMONIC="clutch captain shoe salt awake harvest setup primary inmate ugly among become"
# API Keys and tokens
IPINFO_API_TOKEN=your_token_here
AUTHENTICATOR_PASSWORD=password_here
# Logging
RUST_LOG=info # Options: error, warn, info, debug, trace
RUST_BACKTRACE=1 # Enable backtraces
# Database
DATABASE_URL=postgresql://user:pass@localhost/nym_api
```
#### Using Environment Files
```bash
# Load environment and run command
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Export to shell
source envs/sandbox.env
# Use with make targets
dotenv -f envs/sandbox.env -- make run-api-tests
```
## Initial Setup
### First Time Setup
1. **Install Prerequisites**
```bash
# Install Rust
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Install Node.js and yarn
# Via nvm (recommended):
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
nvm install 20
npm install -g yarn
# Install build tools
# Ubuntu/Debian:
sudo apt-get install build-essential pkg-config libssl-dev protobuf-compiler libpq-dev
# macOS:
brew install protobuf postgresql
# Install wasm-opt for contract builds
npm install -g wasm-opt
# Add wasm target for Rust
rustup target add wasm32-unknown-unknown
```
2. **Clone and Setup Repository**
```bash
git clone https://github.com/nymtech/nym.git
cd nym/nym
# Install JavaScript dependencies
yarn install
# Build the project
make build
```
3. **Database Setup (Optional, for API development)**
```bash
# Install PostgreSQL
# Create database
createdb nym_api
# Run migrations (from nym-api directory)
cd nym-api
sqlx migrate run
```
### Quick Start
```bash
# Run a mixnode locally
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode mixnode --id my-mixnode
# Run a gateway locally
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode gateway --id my-gateway
# Run the API server
dotenv -f envs/sandbox.env -- cargo run -p nym-api
# Run a client
cargo run -p nym-client -- init --id my-client
cargo run -p nym-client -- run --id my-client
```
## CI/CD Pipeline
The project uses GitHub Actions for CI/CD with several key workflows:
1. **Build and Test**:
- `ci-build.yml`: Main build workflow for Rust components
- Tests are run on multiple platforms (Linux, Windows, macOS)
- Includes formatting check (rustfmt) and linting (clippy)
2. **Release Process**:
- Binary artifacts are published on release tags
- Multiple platform builds are created
3. **Documentation**:
- Documentation is automatically built and deployed
## Database Structure
The system uses SQLite databases with tables like:
- `mixnode_status`: Status information about mixnodes
- `gateway_status`: Status information about gateways
- `routes`: Route performance information (success/failure of specific paths)
- `monitor_run`: Information about monitoring test runs
## Development Workflows
### Running a Node
To run the mixnode or gateway:
```bash
# Run nym-node as a mixnode with specified identity
cargo run -p nym-node -- run --mode mixnode --id my-mixnode
# Run nym-node as a gateway
cargo run -p nym-node -- run --mode gateway --id my-gateway
```
### Configuration
Nodes can be configured with files in various locations:
- Command-line arguments
- Environment variables
- `.env` files specified with `--config-env-file`
### Monitoring
To monitor the health of your node:
- View logs for real-time information
- Use the node's HTTP API for status information
- Check the explorer for public node statistics
## Common Libraries
- `common/types`: Shared data types across all components
- `common/crypto`: Cryptographic primitives and wrappers
- `common/client-core`: Core client functionality
- `common/gateway-client`: Client-gateway communication
- `common/task`: Task management and concurrency utilities
- `common/nymsphinx`: Sphinx packet implementation for mixnet
- `common/topology`: Network topology management
- `common/credentials`: Credential system for privacy-preserving authentication
- `common/bandwidth-controller`: Bandwidth management and accounting
## Code Conventions
- Error handling: Use anyhow/thiserror for structured error handling
- Logging: Use the tracing framework for logging and diagnostics
- State management: Generally use Tokio/futures for async code
- Configuration: Use the config crate and env vars with defaults
- Database: Use sqlx for type-safe database queries
- Follow clippy recommendations and rustfmt formatting
- Use semantic commit messages: feat, fix, docs, refactor, test, chore
## When Making Changes
- Run `make test` before submitting PRs
- Follow Rust naming conventions
- Use `clippy` to check for common issues
- Update SQLx query caches when modifying DB queries: `cargo sqlx prepare`
- Consider backward compatibility for protocol changes
- Use lefthook pre-commit hooks for TypeScript formatting
- Run `cargo deny check` to verify dependency compliance
- Test against both sandbox and local environments when possible
- Update relevant documentation and CHANGELOG.md
## Development Tools
### Useful Cargo Commands
```bash
# Check for outdated dependencies
cargo outdated
# Analyze binary size
cargo bloat --release -p nym-node
# Generate dependency graph
cargo tree -p nym-api
# Run with instrumentation
cargo run --features profiling -p nym-node
# Check for security advisories
cargo audit
```
### Database Tools
```bash
# SQLx CLI for migrations
cargo install sqlx-cli
# Create new migration
cd nym-api && sqlx migrate add <migration_name>
# Prepare query metadata for offline compilation
cargo sqlx prepare --workspace
# View database schema
./nym-api/enter_db.sh
```
### Development Scripts
- `scripts/build_topology.py`: Generate network topology files
- `scripts/node_api_check.py`: Verify node API endpoints
- `scripts/network_tunnel_manager.sh`: Manage network tunnels
- `scripts/localnet_start.sh`: Start a local test network
- Various deployment scripts in `deployment/` for different environments
## Debugging
- Enable more verbose logging with the RUST_LOG environment variable:
```
RUST_LOG=debug,nym_node=trace cargo run -p nym-node -- run --mode mixnode
```
- Use the HTTP API endpoints for status information
- Check monitoring data in the database for network performance metrics
- For complex issues, use tracing tools to follow packet flow
- Enable backtraces: `RUST_BACKTRACE=full`
- For WASM debugging: Use browser developer tools with source maps
## Deployment and Advanced Configurations
### Deployment Structure
The `deployment/` directory contains Ansible playbooks and configurations for various deployment scenarios:
- **`aws/`**: AWS-specific deployment configurations
- **`mixnode/`**: Mixnode deployment playbooks
- **`gateway/`**: Gateway deployment playbooks
- **`validator/`**: Validator node deployment
- **`sandbox-v2/`**: Complete sandbox environment setup
- **`big-dipper-2/`**: Block explorer deployment
### Sandbox V2 Deployment
The sandbox-v2 deployment (`deployment/sandbox-v2/`) provides a complete test environment:
```bash
# Key playbooks:
- deploy.yaml # Main deployment orchestrator
- deploy-mixnodes.yaml # Deploy mixnodes
- deploy-gateways.yaml # Deploy gateways
- deploy-validators.yaml # Deploy validator nodes
- deploy-nym-api.yaml # Deploy API services
```
### Custom Environment Setup
To create a custom environment:
1. Copy an existing env file: `cp envs/sandbox.env envs/custom.env`
2. Modify the network endpoints and contract addresses
3. Update the `NETWORK_NAME` to your identifier
4. Set appropriate mnemonics and keys (use fresh ones for production!)
### Contract Addresses
Contract addresses are network-specific and defined in environment files:
- Mixnet contract: Manages mixnode/gateway registry
- Vesting contract: Handles token vesting schedules
- Coconut contracts: Privacy-preserving credentials
- Name service: Human-readable address mapping
- Ecash contract: Electronic cash functionality
### Local Network Setup
For a completely local network:
```bash
# Start local chain
./scripts/localnet_start.sh
# Deploy contracts
cd contracts
make deploy-local
# Start nodes with local config
dotenv -f envs/local.env -- cargo run -p nym-node -- run --mode mixnode
```
## Common Issues and Troubleshooting
### Database Issues
- When modifying database queries, you must update SQLx query caches:
```bash
cargo sqlx prepare
```
- If you see SQLx errors about missing query files, this is likely the cause
- For "database is locked" errors with SQLite, ensure only one process accesses the DB
- For PostgreSQL connection issues, verify DATABASE_URL and that the server is running
### API Connection Issues
- Check the environment variables pointing to the APIs (NYM_API, NYXD)
- Verify network connectivity and API health endpoints
- For authentication issues, check node keys and credentials
- Common endpoints to verify:
- API health: `$NYM_API/health`
- Chain status: `$NYXD/status`
- Contract info: `$NYXD/cosmwasm/wasm/v1/contract/$CONTRACT_ADDRESS`
### Build Problems
- Clean dependencies with `cargo clean` for a fresh build
- Check for compatible Rust version (1.80+ recommended)
- For smart contract builds, ensure wasm-opt is installed: `npm install -g wasm-opt`
- For cross-compilation issues, check target-specific dependencies
- WASM build issues: Ensure wasm32-unknown-unknown target is installed:
```bash
rustup target add wasm32-unknown-unknown
```
- For "cannot find -lpq" errors, install PostgreSQL development files:
```bash
# Ubuntu/Debian
sudo apt-get install libpq-dev
# macOS
brew install postgresql
```
### Environment Issues
- Contract address mismatches: Ensure you're using the correct environment file
- "Account sequence mismatch": The account nonce is out of sync, wait and retry
- Token decimal issues: Sandbox uses different decimal places than mainnet
- API version mismatches: Ensure your local API version matches the network
- "Insufficient funds": Get test tokens from faucet (sandbox) or check balance
- Gateway/mixnode bonding issues: Verify minimum stake requirements
## Working with Routes and Monitoring
1. Route monitoring metrics are stored in a `routes` table with:
- Layer node IDs (layer1, layer2, layer3, gw)
- Success flag (boolean)
- Timestamp
2. To analyze routes:
- Check `NetworkAccount` and `AccountingRoute` in `nym-network-monitor/src/accounting.rs`
- View monitoring logic in `common/nymsphinx/chunking/monitoring.rs`
- Observe how routes are submitted to the database in the `submit_accounting_routes_to_db` function
## Performance Optimization
### Profiling and Benchmarking
```bash
# Run benchmarks
cargo bench -p nym-node
# Profile with perf (Linux)
cargo build --release --features profiling
perf record --call-graph=dwarf ./target/release/nym-node run --mode mixnode
perf report
# Generate flamegraph
cargo install flamegraph
cargo flamegraph --bin nym-node -- run --mode mixnode
```
### Common Performance Considerations
- Use bounded channels for backpressure
- Batch database operations where possible
- Monitor memory usage with `RUST_LOG=nym_node::metrics=debug`
- Use connection pooling for database connections
- Consider using `jemalloc` for better memory allocation performance
Generated
+1005 -949
View File
File diff suppressed because it is too large Load Diff
+58
View File
@@ -4,3 +4,61 @@ The Node Status API serves information about individual `nym-nodes` in the Mixne
We recommend that developers building applications such as explorers or analytics interfaces about the Mixnet run their own instance of the API, in order to promote a robust network of downstream services, and spread the load of API calls amongst as many endpoints as possible.
You can find build and operation instructions in the [docs](https://nym.com/docs/apis/ns-api).
## Database Support
The Node Status API supports both SQLite and PostgreSQL databases through Cargo feature flags:
- **SQLite** (default): Lightweight, file-based database suitable for development and small deployments
- **PostgreSQL**: Full-featured database recommended for production deployments
### Building with Different Database Backends
```bash
# Build with SQLite (default)
cargo build --features sqlite --no-default-features
# Build with PostgreSQL
cargo build --features pg --no-default-features
```
### Running Tests
```bash
# Test with SQLite
cargo test --features sqlite --no-default-features
# Test with PostgreSQL
make test-db # This sets up a test PostgreSQL instance
```
### Development Commands
The project includes a Makefile with helpful commands for both database backends:
```bash
# Check code compilation
make check-sqlite # Check with SQLite
make check-pg # Check with PostgreSQL
# Run clippy linter
make clippy-sqlite # Lint with SQLite
make clippy-pg # Lint with PostgreSQL
make clippy # Run both
# PostgreSQL development
make dev-db # Start a PostgreSQL instance for development
make prepare-pg # Prepare SQLx offline cache for PostgreSQL
```
### Implementation Details
The database abstraction is implemented using a query wrapper that automatically converts SQLite-style `?` placeholders to PostgreSQL-style `$1, $2, ...` placeholders at runtime. This allows writing queries once using SQLite syntax while maintaining compatibility with both databases.
Key differences handled:
- Placeholder syntax (`?` vs `$1, $2, ...`)
- Type conversions (SQLite uses i64, PostgreSQL uses i32 for many fields)
- SQL dialect differences (e.g., `INSERT OR IGNORE` vs `ON CONFLICT DO NOTHING`)
- RETURNING clause behavior
For more details on PostgreSQL setup, see [README_PG.md](nym-node-status-api/README_PG.md).
@@ -3,7 +3,7 @@
[package]
name = "nym-node-status-agent"
version = "1.0.0"
version = "1.0.4"
authors.workspace = true
repository.workspace = true
homepage.workspace = true
@@ -14,13 +14,25 @@ rust-version.workspace = true
readme.workspace = true
[dependencies]
anyhow = { workspace = true}
anyhow = { workspace = true }
clap = { workspace = true, features = ["derive", "env"] }
nym-bin-common = { path = "../../common/bin-common", features = ["models"]}
futures = { workspace = true }
# nym-bin-common = { path = "../../common/bin-common", features = ["models"] }
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"models",
] }
nym-node-status-client = { path = "../nym-node-status-client" }
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "rand"] }
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"asymmetric",
"rand",
] }
rand = { workspace = true }
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "process", "fs"] }
tokio = { workspace = true, features = [
"macros",
"rt-multi-thread",
"process",
"fs",
] }
tracing = { workspace = true }
tracing-subscriber = { workspace = true, features = ["env-filter"] }
@@ -16,7 +16,7 @@ WORKDIR /usr/src/nym-vpn-client/nym-vpn-core
RUN cargo build --release --package nym-gateway-probe
COPY ./ /usr/src/nym
WORKDIR /usr/src/nym/nym-node-status-agent
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-agent
RUN cargo build --release
#-------------------------------------------------------------------
@@ -0,0 +1,71 @@
#!/bin/bash
# Build and push Node Status Agent container to harbor.nymte.ch
set -e
# Configuration
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
WORKING_DIRECTORY="${SCRIPT_DIR}"
CONTAINER_NAME="node-status-agent"
REGISTRY="harbor.nymte.ch"
NAMESPACE="nym"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Function to display usage
usage() {
echo "Usage: $0 <gateway-probe-git-ref>"
echo " gateway-probe-git-ref - Git reference (branch/tag/commit) for gateway probe"
echo ""
echo "Example: $0 main"
echo "Example: $0 release/2025.11-cheddar"
echo "Example: $0 v1.2.3"
exit 1
}
# Parse arguments
if [ $# -ne 1 ]; then
usage
fi
GATEWAY_PROBE_GIT_REF="$1"
# Get version from Cargo.toml
VERSION=$(grep "^version = " "${WORKING_DIRECTORY}/Cargo.toml" | sed -E 's/version = "(.*)"/\1/')
if [ -z "$VERSION" ]; then
echo -e "${RED}Error: Could not extract version from Cargo.toml${NC}"
exit 1
fi
# Clean up git ref for use in tag (replace / with -)
GIT_REF_SLUG="${GATEWAY_PROBE_GIT_REF//\//-}"
echo -e "${YELLOW}Building Node Status Agent${NC}"
echo -e "${YELLOW}Version: ${VERSION}${NC}"
echo -e "${YELLOW}Gateway Probe Git Ref: ${GATEWAY_PROBE_GIT_REF} (slug: ${GIT_REF_SLUG})${NC}"
# Login to Harbor
echo -e "${GREEN}Logging into Harbor...${NC}"
docker login "${REGISTRY}"
# Build the container
echo -e "${GREEN}Building container with gateway probe from ${GATEWAY_PROBE_GIT_REF}...${NC}"
# Build from repository root (two levels up from script location)
docker build \
--build-arg GIT_REF="${GATEWAY_PROBE_GIT_REF}" \
-f "${WORKING_DIRECTORY}/Dockerfile" \
"${SCRIPT_DIR}/../.." \
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}" \
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${GIT_REF_SLUG}"
# Push to Harbor
echo -e "${GREEN}Pushing container to Harbor...${NC}"
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}"
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${GIT_REF_SLUG}"
echo -e "${GREEN}Successfully built and pushed ${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}${NC}"
@@ -1,17 +1,45 @@
use crate::probe::GwProbe;
use clap::{Parser, Subcommand};
use nym_bin_common::bin_info;
use std::sync::OnceLock;
use nym_crypto::asymmetric::ed25519::PrivateKey;
use std::{env, sync::OnceLock};
pub(crate) mod generate_keypair;
pub(crate) mod run_probe;
#[derive(Debug)]
pub(crate) struct ServerConfig {
pub(crate) address: String,
pub(crate) port: u16,
pub(crate) auth_key: PrivateKey,
}
// Helper for passing LONG_VERSION to clap
fn pretty_build_info_static() -> &'static str {
static PRETTY_BUILD_INFORMATION: OnceLock<String> = OnceLock::new();
PRETTY_BUILD_INFORMATION.get_or_init(|| bin_info!().pretty_print())
}
fn parse_server_config(s: &str) -> Result<ServerConfig, String> {
let parts: Vec<&str> = s.split('|').collect();
if parts.len() != 2 {
return Err("Server config must be in format 'address|port'".to_string());
}
let address = parts[0].to_string();
let port = parts[1]
.parse::<u16>()
.map_err(|_| "Invalid port number".to_string())?;
let auth_key =
PrivateKey::from_base58_string(env::var("NODE_STATUS_AGENT_AUTH_KEY").unwrap()).unwrap();
Ok(ServerConfig {
address,
port,
auth_key,
})
}
#[derive(Parser, Debug)]
#[clap(author = "Nymtech", version, long_version = pretty_build_info_static(), about)]
pub(crate) struct Args {
@@ -22,15 +50,10 @@ pub(crate) struct Args {
#[derive(Subcommand, Debug)]
pub(crate) enum Command {
RunProbe {
#[arg(short, long, env = "NODE_STATUS_AGENT_SERVER_ADDRESS")]
server_address: String,
#[arg(short = 'p', long, env = "NODE_STATUS_AGENT_SERVER_PORT")]
server_port: u16,
/// base58-encoded private key
#[arg(long, env = "NODE_STATUS_AGENT_AUTH_KEY")]
ns_api_auth_key: String,
/// Server configurations in format "address:port:auth_key"
/// Can be specified multiple times for multiple servers
#[arg(short, long, required = true)]
server: Vec<String>,
/// path of binary to run
#[arg(long, env = "NODE_STATUS_AGENT_PROBE_PATH")]
@@ -58,24 +81,29 @@ impl Args {
pub(crate) async fn execute(&self) -> anyhow::Result<()> {
match &self.command {
Command::RunProbe {
server_address,
server_port,
ns_api_auth_key,
server,
probe_path,
mnemonic,
probe_extra_args,
} => run_probe::run_probe(
server_address,
server_port.to_owned(),
ns_api_auth_key,
probe_path,
mnemonic,
probe_extra_args,
)
.await
.inspect_err(|err| {
tracing::error!("{err}");
})?,
} => {
// Parse server configs
let mut servers = Vec::new();
for s in server {
match parse_server_config(s) {
Ok(config) => servers.push(config),
Err(e) => {
tracing::error!("Invalid server config '{}': {}", s, e);
anyhow::bail!("Invalid server config '{}': {}", s, e);
}
}
}
run_probe::run_probe(&servers, probe_path, mnemonic, probe_extra_args)
.await
.inspect_err(|err| {
tracing::error!("{err}");
})?
}
Command::GenerateKeypair { path } => {
let path = path
.to_owned()
@@ -1,38 +1,143 @@
use crate::cli::GwProbe;
use anyhow::Context;
use nym_crypto::asymmetric::ed25519::PrivateKey;
use crate::cli::{GwProbe, ServerConfig};
pub(crate) async fn run_probe(
server_ip: &str,
server_port: u16,
ns_api_auth_key: &str,
servers: &[ServerConfig],
probe_path: &str,
mnemonic: &str,
probe_extra_args: &Vec<String>,
) -> anyhow::Result<()> {
let auth_key = PrivateKey::from_base58_string(ns_api_auth_key)
.context("Couldn't parse auth key, exiting")?;
let ns_api_client = nym_node_status_client::NsApiClient::new(server_ip, server_port, auth_key);
if servers.is_empty() {
anyhow::bail!("No servers configured");
}
let probe = GwProbe::new(probe_path.to_string());
let version = probe.version().await;
tracing::info!("Probe version:\n{}", version);
if let Some(testrun) = ns_api_client.request_testrun().await? {
let log = probe.run_and_get_log(
&Some(testrun.gateway_identity_key),
mnemonic,
probe_extra_args,
);
// Always use first server as primary
let primary_server = &servers[0];
tracing::info!(
"Requesting testrun from primary server: {}:{}",
primary_server.address,
primary_server.port
);
ns_api_client
.submit_results(testrun.testrun_id, log, testrun.assigned_at_utc)
.await?;
} else {
tracing::info!("No testruns available, exiting")
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
&primary_server.auth_key.to_bytes(),
)
.expect("Failed to clone auth key");
let ns_api_client = nym_node_status_client::NsApiClient::new(
&primary_server.address,
primary_server.port,
auth_key,
);
match ns_api_client.request_testrun().await {
Ok(Some(testrun)) => {
tracing::info!(
"Received testrun {} for gateway {} from primary",
testrun.testrun_id,
testrun.gateway_identity_key
);
// Run the probe
let log = probe.run_and_get_log(
&Some(testrun.gateway_identity_key.clone()),
mnemonic,
probe_extra_args,
);
// Submit to ALL servers in parallel
let handles = servers
.iter()
.enumerate()
.map(|(idx, server)| {
let testrun = testrun.clone();
let log = log.clone();
async move {
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
&server.auth_key.to_bytes(),
)
.expect("Failed to clone auth key");
let client = nym_node_status_client::NsApiClient::new(
&server.address,
server.port,
auth_key,
);
let result = if idx == 0 {
// Primary server: submit regular results without context
client
.submit_results(
testrun.testrun_id as i64,
log,
testrun.assigned_at_utc,
)
.await
} else {
// Other servers: submit results with context
client
.submit_results_with_context(
testrun.testrun_id,
log,
testrun.assigned_at_utc,
testrun.gateway_identity_key,
)
.await
};
(idx, server.address.clone(), server.port, result)
}
})
.collect::<Vec<_>>();
let results = futures::future::join_all(handles).await;
for result in results {
match result.3 {
Ok(()) => {
let method = if result.0 == 0 {
"regular"
} else {
"with context"
};
tracing::info!(
"✅ Successfully submitted {} to server[{}] {}:{}",
method,
result.0,
result.1,
result.2
);
}
Err(e) => {
let method = if result.0 == 0 {
"regular"
} else {
"with context"
};
tracing::warn!(
"❌ Failed to submit {} to server[{}] {}:{} - {}",
method,
result.0,
result.1,
result.2,
e
);
}
}
}
Ok(())
}
Ok(None) => {
tracing::info!("No testruns available from primary server");
Ok(())
}
Err(e) => {
tracing::error!("Failed to contact primary server: {}", e);
Err(e)
}
}
Ok(())
}
@@ -0,0 +1,32 @@
# Example environment variables for nym-node-status-api
# Database configuration
# For SQLite:
# DATABASE_URL=sqlite://nym-node-status-api.sqlite
# For PostgreSQL:
# DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# Network configuration
NETWORK_NAME=sandbox
NYM_API=https://sandbox-nym-api1.nymtech.net/api
NYXD=https://rpc.sandbox.nymtech.net
# API configuration
NYM_NODE_STATUS_API_HTTP_PORT=8000
NYM_API_CLIENT_TIMEOUT=15
SQLX_BUSY_TIMEOUT_S=5
# Monitoring intervals
NODE_STATUS_API_MONITOR_REFRESH_INTERVAL=300
NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL=300
NODE_STATUS_API_GEODATA_TTL=86400
# Agent keys (comma-separated list)
NODE_STATUS_API_AGENT_KEY_LIST=
# External service tokens
IPINFO_API_TOKEN=your_token_here
# MixNodes (Optional)
DATA_PROVIDER_DELEGATION_LIST=
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_daily_stats (\n mix_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,\n packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,\n packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "01ee4a30bc3104712e5bc371a45d614a89d88adf02358800433e06100c13c548"
}
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT mix_id as node_id, host, http_api_port\n FROM mixnodes\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "host",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "http_api_port",
"ordinal": 2,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false
]
},
"hash": "021c6c65d1ed806d8430bef7883906b42a7e4b280c8efb32db15d7c6a51d7a27"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_description (\n mix_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (mix_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "06065394c157927e4002ddd5c7c1af626ae15728d615f539470cd7c189312385"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n gateway_identity_key\n FROM\n gateways\n WHERE\n id = ?",
"describe": {
"columns": [
{
"name": "gateway_identity_key",
"ordinal": 0,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "06b17d1e5f61201a1b7542896ba55c69cd5c1a7e7d87073c94600c783a0a3984"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n testruns\n SET\n status = ?\n WHERE\n status = ?\n AND\n last_assigned_utc < ?\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "0cf0e4d4f30e90caecffd6255ef85dab12730e538be194438f19ed7f198bd50e"
}
@@ -1,43 +1,43 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "\n SELECT\n date_utc as \"date_utc!\",\n SUM(total_stake) as \"total_stake!: i64\",\n SUM(packets_received) as \"total_packets_received!: i64\",\n SUM(packets_sent) as \"total_packets_sent!: i64\",\n SUM(packets_dropped) as \"total_packets_dropped!: i64\"\n FROM (\n SELECT\n date_utc,\n n.total_stake,\n n.packets_received,\n n.packets_sent,\n n.packets_dropped\n FROM nym_node_daily_mixing_stats n\n UNION ALL\n SELECT\n m.date_utc,\n m.total_stake,\n m.packets_received,\n m.packets_sent,\n m.packets_dropped\n FROM mixnode_daily_stats m\n LEFT JOIN nym_node_daily_mixing_stats ON m.mix_id = nym_node_daily_mixing_stats.node_id\n WHERE nym_node_daily_mixing_stats.node_id IS NULL\n )\n GROUP BY date_utc\n ORDER BY date_utc ASC\n ",
"describe": {
"columns": [
{
"name": "date_utc!",
"ordinal": 0,
"type_info": "Text"
"name": "date_utc!",
"type_info": "Varchar"
},
{
"name": "total_stake!: i64",
"ordinal": 1,
"type_info": "Integer"
"name": "total_stake!: i64",
"type_info": "Numeric"
},
{
"name": "total_packets_received!: i64",
"ordinal": 2,
"type_info": "Integer"
"name": "total_packets_received!: i64",
"type_info": "Int8"
},
{
"name": "total_packets_sent!: i64",
"ordinal": 3,
"type_info": "Integer"
"name": "total_packets_sent!: i64",
"type_info": "Int8"
},
{
"name": "total_packets_dropped!: i64",
"ordinal": 4,
"type_info": "Integer"
"name": "total_packets_dropped!: i64",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
false,
true,
true,
true
null,
null,
null,
null,
null
]
},
"hash": "124d45b9604439584650f401607c46bdbd162c7c689f74fe9ddfdfd48f5ddc07"
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n key as \"key!\",\n value_json as \"value_json!\",\n last_updated_utc as \"last_updated_utc!\"\n FROM summary",
"describe": {
"columns": [
{
"name": "key!",
"ordinal": 0,
"type_info": "Text"
},
{
"name": "value_json!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "last_updated_utc!",
"ordinal": 2,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
true,
true,
false
]
},
"hash": "1327b5118f9144dddbcf8edb11f7dc549cf503409fd6dfedcdc02dbcd61d5454"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO mixnode_packet_stats_raw (\n mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "21e44766729777756f6eb04bf3b81df3e591008a1e3fd664ed83ca86ac51bd8c"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n id,\n gateway_identity_key\n FROM gateways\n WHERE id = ?\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_identity_key",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false
]
},
"hash": "2236299f9f691376db54cbd58ec5ceb89b9925cba46efcf4ed79ef0759a01129"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n AND\n self_described IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true
]
},
"hash": "227539374e7473f6f9642289c5b5d1bcd636315ab23537cb5f6d2f82a2bcb7bf"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT gateway_identity_key\n FROM gateways\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "gateway_identity_key",
"ordinal": 0,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false
]
},
"hash": "25300e435780101fa207c8e26ef2f49ba5db84d63e89440bb494e8327fe73686"
}
@@ -1,86 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n AND\n bond_info IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "ed25519_identity_pubkey",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "total_stake",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "ip_addresses!: serde_json::Value",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "mix_port",
"ordinal": 4,
"type_info": "Integer"
},
{
"name": "x25519_sphinx_pubkey",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "node_role: serde_json::Value",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "supported_roles: serde_json::Value",
"ordinal": 7,
"type_info": "Text"
},
{
"name": "entry: serde_json::Value",
"ordinal": 8,
"type_info": "Text"
},
{
"name": "performance",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 11,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
true,
true
]
},
"hash": "283f49a65c7d70bf271702ff6a5c7ad6e68c81932d295ff18ed198c54706a57c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT OR IGNORE INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
"describe": {
"columns": [],
"parameters": {
"Right": 9
},
"nullable": []
},
"hash": "3243cf5646255a9430d1e6710970505d0dbcc62703f40e090e80ff48c77723c4"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO mixnodes\n (mix_id, identity_key, bonded, total_stake,\n host, http_api_port, full_details,\n self_described, last_updated_utc, is_dp_delegatee)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id) DO UPDATE SET\n bonded=excluded.bonded,\n total_stake=excluded.total_stake, host=excluded.host,\n http_api_port=excluded.http_api_port,\n full_details=excluded.full_details,self_described=excluded.self_described,\n last_updated_utc=excluded.last_updated_utc,\n is_dp_delegatee = excluded.is_dp_delegatee;",
"describe": {
"columns": [],
"parameters": {
"Right": 10
},
"nullable": []
},
"hash": "3cd5cb4bfca4243925da4ddbccd811e842090e98982e1032670df77961870b32"
}
@@ -1,38 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_identity_key as \"gateway_identity_key!\",\n self_described as \"self_described?\",\n explorer_pretty_bond as \"explorer_pretty_bond?\"\n FROM gateways\n WHERE gateway_identity_key = ?\n AND bonded = true\n ORDER BY gateway_identity_key\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_identity_key!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "self_described?",
"ordinal": 2,
"type_info": "Text"
},
{
"name": "explorer_pretty_bond?",
"ordinal": 3,
"type_info": "Text"
}
],
"parameters": {
"Right": 1
},
"nullable": [
true,
false,
false,
true
]
},
"hash": "3e7e987780937873cdb393b157d7708c9f01047b0689eb0d4f7a973b328c609d"
}
@@ -1,92 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n gw.gateway_identity_key as \"gateway_identity_key!\",\n gw.bonded as \"bonded: bool\",\n gw.performance as \"performance!\",\n gw.self_described as \"self_described?\",\n gw.explorer_pretty_bond as \"explorer_pretty_bond?\",\n gw.last_probe_result as \"last_probe_result?\",\n gw.last_probe_log as \"last_probe_log?\",\n gw.last_testrun_utc as \"last_testrun_utc?\",\n gw.last_updated_utc as \"last_updated_utc!\",\n COALESCE(gd.moniker, \"NA\") as \"moniker!\",\n COALESCE(gd.website, \"NA\") as \"website!\",\n COALESCE(gd.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(gd.details, \"NA\") as \"details!\"\n FROM gateways gw\n LEFT JOIN gateway_description gd\n ON gw.gateway_identity_key = gd.gateway_identity_key\n ORDER BY gw.gateway_identity_key",
"describe": {
"columns": [
{
"name": "gateway_identity_key!",
"ordinal": 0,
"type_info": "Text"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "performance!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "self_described?",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "explorer_pretty_bond?",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "last_probe_result?",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_probe_log?",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "last_testrun_utc?",
"ordinal": 7,
"type_info": "Integer"
},
{
"name": "last_updated_utc!",
"ordinal": 8,
"type_info": "Integer"
},
{
"name": "moniker!",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "website!",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "security_contact!",
"ordinal": 11,
"type_info": "Text"
},
{
"name": "details!",
"ordinal": 12,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
true,
true,
true,
true,
false,
false,
false,
false,
false
]
},
"hash": "3eb1d8491bda3c1d6e071b6eb364b9a979f4bdb11ea81b2d0f022555bab51ecb"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n total_stake\n FROM mixnodes\n WHERE mix_id = ?\n ",
"describe": {
"columns": [
{
"name": "total_stake",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "3fc2baabf194b147b20be2a49401cc0c100a1d7a7c347393adde2410fa6f4dfe"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE testruns SET status = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "418944f2eccb838cb3882f34469203c8569f03fdd39ce09d7b74177896e52a8c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_probe_log = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "4afcc6673890f795c2793f1e2f8570ee787fc7daf00fcb916f18d1cb7d6c8f08"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "4d865e873c9cb133883da94db72dcdebd4969e1f240def9fb1bf946f4a1f342f"
}
@@ -0,0 +1,22 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)\n ON CONFLICT DO NOTHING",
"describe": {
"columns": [],
"parameters": {
"Left": [
"Varchar",
"Int8",
"Date",
"Int8",
"Int8",
"Varchar",
"Varchar",
"Varchar",
"Varchar"
]
},
"nullable": []
},
"hash": "4fca38abbb416d9457c34a8ba4faf481a837eda4f3e1bee1d430a4eb102a5b3d"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_node_daily_mixing_stats (\n node_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,\n packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,\n packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "5912ea335a957d217f5e2b3a63a25b31715c2098310fe7a9db688bc2fd36aad4"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE testruns\n SET\n status = ?,\n last_assigned_utc = ?\n WHERE rowid =\n (\n SELECT rowid\n FROM testruns\n WHERE status = ?\n ORDER BY created_utc asc\n LIMIT 1\n )\n RETURNING\n id as \"id!\",\n gateway_id\n ",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id",
"ordinal": 1,
"type_info": "Integer"
}
],
"parameters": {
"Right": 3
},
"nullable": [
false,
false
]
},
"hash": "5e9cbb39f5fb53774803270f422989e199aac4d4a71913c7074359b4bd676b02"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO nym_nodes\n (node_id, ed25519_identity_pubkey,\n total_stake,\n ip_addresses, mix_port,\n x25519_sphinx_pubkey, node_role,\n supported_roles, entry,\n self_described,\n bond_info,\n performance, last_updated_utc\n )\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id) DO UPDATE SET\n ed25519_identity_pubkey=excluded.ed25519_identity_pubkey,\n ip_addresses=excluded.ip_addresses,\n mix_port=excluded.mix_port,\n x25519_sphinx_pubkey=excluded.x25519_sphinx_pubkey,\n node_role=excluded.node_role,\n supported_roles=excluded.supported_roles,\n entry=excluded.entry,\n self_described=excluded.self_described,\n bond_info=excluded.bond_info,\n performance=excluded.performance,\n last_updated_utc=excluded.last_updated_utc\n ;",
"describe": {
"columns": [],
"parameters": {
"Right": 13
},
"nullable": []
},
"hash": "664e059ac2c58e1115fe214376a6b326b31c93298f20019772cce2e277a194f8"
}
@@ -1,19 +1,19 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT count(id) FROM mixnodes",
"describe": {
"columns": [
{
"name": "count(id)",
"ordinal": 0,
"type_info": "Integer"
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false
null
]
},
"hash": "670b7ed7d57a6986181b24be24ca667e8cacdf677ccb906415b3fe92be0c436b"
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_node_descriptions (\n node_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (node_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "6a9780aad1f2f0c8ef780e51fe856679d5e28f95143f4e2a2b409009dc0f55ba"
}
@@ -0,0 +1,26 @@
{
"db_name": "PostgreSQL",
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
"describe": {
"columns": [
{
"ordinal": 0,
"name": "id",
"type_info": "Int4"
}
],
"parameters": {
"Left": [
"Int4",
"Int4",
"Varchar",
"Int8",
"Varchar"
]
},
"nullable": [
false
]
},
"hash": "6de15de62c0caa545910a17877a3ac5ebe44a398b199ab0120207a5569f54d0b"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_probe_result = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 2
},
"nullable": []
},
"hash": "6ef3efde571d46961244cd90420f3de5949a5ff2083453cb879af8a1689efe2f"
}
@@ -1,66 +1,66 @@
{
"db_name": "SQLite",
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n COALESCE(md.moniker, \"NA\") as \"moniker!\",\n COALESCE(md.website, \"NA\") as \"website!\",\n COALESCE(md.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(md.details, \"NA\") as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
"db_name": "PostgreSQL",
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n md.moniker as \"moniker!\",\n md.website as \"website!\",\n md.security_contact as \"security_contact!\",\n md.details as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
"describe": {
"columns": [
{
"name": "mix_id!",
"ordinal": 0,
"type_info": "Integer"
"name": "mix_id!",
"type_info": "Int8"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
"name": "bonded: bool",
"type_info": "Bool"
},
{
"name": "is_dp_delegatee: bool",
"ordinal": 2,
"type_info": "Integer"
"name": "is_dp_delegatee: bool",
"type_info": "Bool"
},
{
"name": "total_stake!",
"ordinal": 3,
"type_info": "Integer"
"name": "total_stake!",
"type_info": "Int8"
},
{
"name": "full_details!",
"ordinal": 4,
"type_info": "Text"
"name": "full_details!",
"type_info": "Varchar"
},
{
"name": "self_described",
"ordinal": 5,
"type_info": "Text"
"name": "self_described",
"type_info": "Varchar"
},
{
"name": "last_updated_utc!",
"ordinal": 6,
"type_info": "Integer"
"name": "last_updated_utc!",
"type_info": "Int8"
},
{
"name": "moniker!",
"ordinal": 7,
"type_info": "Text"
"name": "moniker!",
"type_info": "Varchar"
},
{
"name": "website!",
"ordinal": 8,
"type_info": "Text"
"name": "website!",
"type_info": "Varchar"
},
{
"name": "security_contact!",
"ordinal": 9,
"type_info": "Text"
"name": "security_contact!",
"type_info": "Varchar"
},
{
"name": "details!",
"ordinal": 10,
"type_info": "Text"
"name": "details!",
"type_info": "Varchar"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
@@ -70,11 +70,11 @@
true,
true,
false,
false,
false,
false,
false
true,
true,
true,
true
]
},
"hash": "f75af377da33db1455c6e0f612e0fa9583888f343b8b59faf37fc6799b244379"
"hash": "74b76cd0d94c1afc51c21c14c12236a2b964b981c8cbcc282117fe9bc38338dd"
}
@@ -1,38 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n date as \"date!\",\n timestamp_utc as \"timestamp_utc!\",\n value_json as \"value_json!\"\n FROM summary_history\n ORDER BY date DESC\n LIMIT 30",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "date!",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "timestamp_utc!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "value_json!",
"ordinal": 3,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
true,
false,
false,
true
]
},
"hash": "7600823da7ce80b8ffda933608603a2752e28df775d1af8fd943a5fc8d7dc00d"
}
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM mixnode_packet_stats_raw\n WHERE mix_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
"describe": {
"columns": [
{
"name": "packets_received!: _",
"ordinal": 0,
"type_info": "Null"
},
{
"name": "packets_sent!: _",
"ordinal": 1,
"type_info": "Null"
},
{
"name": "packets_dropped!: _",
"ordinal": 2,
"type_info": "Null"
}
],
"parameters": {
"Right": 1
},
"nullable": [
null,
null,
null
]
},
"hash": "786b6a5d954e38b204cebf322711c74c8cf1c08e5e2896a1d6d5b85c91991753"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO summary_history\n (date, timestamp_utc, value_json)\n VALUES (?, ?, ?)\n ON CONFLICT(date) DO UPDATE SET\n timestamp_utc=excluded.timestamp_utc,\n value_json=excluded.value_json;",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "788515c34588aec352773df4b6e6c5e41f3c0bb56a27648b5e25466b8634a578"
}
@@ -1,19 +1,19 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT count(id) FROM gateways",
"describe": {
"columns": [
{
"name": "count(id)",
"ordinal": 0,
"type_info": "Integer"
"name": "count",
"type_info": "Int8"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false
null
]
},
"hash": "86ff64db477a1d6235179b0b88d86b86d1b9be62336c9eac0eef44987a5451b5"
@@ -1,32 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM nym_nodes_packet_stats_raw\n WHERE node_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
"describe": {
"columns": [
{
"name": "packets_received!: _",
"ordinal": 0,
"type_info": "Null"
},
{
"name": "packets_sent!: _",
"ordinal": 1,
"type_info": "Null"
},
{
"name": "packets_dropped!: _",
"ordinal": 2,
"type_info": "Null"
}
],
"parameters": {
"Right": 1
},
"nullable": [
null,
null,
null
]
},
"hash": "8bdf85a61e443fa5f4835bffd0bffc8ed1011f56714fde6007e50951e569854b"
}
@@ -1,21 +1,21 @@
{
"db_name": "SQLite",
"db_name": "PostgreSQL",
"query": "SELECT\n gateway_identity_key as \"gateway_identity_key!\",\n bonded as \"bonded: bool\"\n FROM gateways\n ORDER BY last_testrun_utc",
"describe": {
"columns": [
{
"name": "gateway_identity_key!",
"ordinal": 0,
"type_info": "Text"
"name": "gateway_identity_key!",
"type_info": "Varchar"
},
{
"name": "bonded: bool",
"ordinal": 1,
"type_info": "Integer"
"name": "bonded: bool",
"type_info": "Bool"
}
],
"parameters": {
"Right": 0
"Left": []
},
"nullable": [
false,
@@ -1,86 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "ed25519_identity_pubkey",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "total_stake",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "ip_addresses!: serde_json::Value",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "mix_port",
"ordinal": 4,
"type_info": "Integer"
},
{
"name": "x25519_sphinx_pubkey",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "node_role: serde_json::Value",
"ordinal": 6,
"type_info": "Text"
},
{
"name": "supported_roles: serde_json::Value",
"ordinal": 7,
"type_info": "Text"
},
{
"name": "entry: serde_json::Value",
"ordinal": 8,
"type_info": "Text"
},
{
"name": "performance",
"ordinal": 9,
"type_info": "Text"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 10,
"type_info": "Text"
},
{
"name": "bond_info: serde_json::Value",
"ordinal": 11,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
false,
false,
false,
false,
false,
false,
false,
true,
false,
true,
true
]
},
"hash": "9334f0c91252fcd7ec72558a271222615bb282e5334665700709ae475a5daea2"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO gateways\n (gateway_identity_key, bonded,\n self_described, explorer_pretty_bond,\n last_updated_utc, performance)\n VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(gateway_identity_key) DO UPDATE SET\n bonded=excluded.bonded,\n self_described=excluded.self_described,\n explorer_pretty_bond=excluded.explorer_pretty_bond,\n last_updated_utc=excluded.last_updated_utc,\n performance = excluded.performance;",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "9796d354ae075eab4cbd3438839c39da94025494395ec7b093aefef696f2d0c5"
}
@@ -1,56 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE gateway_id = ? AND status != 2\n ORDER BY id DESC\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id!",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "status!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "created_utc!",
"ordinal": 3,
"type_info": "Integer"
},
{
"name": "ip_address!",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "log!",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_assigned_utc",
"ordinal": 6,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false,
false,
false,
false,
false,
false,
true
]
},
"hash": "a1d9eb816acd1a91ed0975c801c9295c01a78861a2a0597ad28b1579a14bf008"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n COUNT(id) as \"count: i64\"\n FROM testruns\n WHERE\n status = ?\n ",
"describe": {
"columns": [
{
"name": "count: i64",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "a79a61b87325f3f1d9c5a4fb386ccd585be0641e5878acb6283b879f22ed2b4c"
}
@@ -1,44 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n nd.node_id,\n moniker,\n website,\n security_contact,\n details\n FROM\n nym_node_descriptions nd\n INNER JOIN\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "moniker",
"ordinal": 1,
"type_info": "Text"
},
{
"name": "website",
"ordinal": 2,
"type_info": "Text"
},
{
"name": "security_contact",
"ordinal": 3,
"type_info": "Text"
},
{
"name": "details",
"ordinal": 4,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true,
true,
true,
true
]
},
"hash": "aa602604c0e7b6eef41ea3cd83c16610e15be2d7ee3f6c4d3debf23f95fb9c2e"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE nym_nodes\n SET\n self_described = NULL,\n bond_info = NULL",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "b68796d1d8d2384b30f1aace06269682c4ae96f774261f5c298264d3c12e5b67"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "c214c001acbbf79fa499816f36ec586c4c29c03efb4cf0c40b73a5c76159cf5c"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n mixnodes\n SET\n bonded = false\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "c42917c9542c1d720d92035863064741aefc9f7a7d1630f6b863ebd8174b6684"
}
@@ -1,26 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n node_id,\n self_described as \"self_described: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n ORDER BY\n node_id\n ",
"describe": {
"columns": [
{
"name": "node_id",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "self_described: serde_json::Value",
"ordinal": 1,
"type_info": "Text"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false,
true
]
},
"hash": "c7656b2b1b4328415772ce69d0568bd5438d6c8496ca9cbdcfb70bb5375b345e"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT mix_id\n FROM mixnodes\n WHERE bonded = true\n ",
"describe": {
"columns": [
{
"name": "mix_id",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 0
},
"nullable": [
false
]
},
"hash": "c910788edefe64bbb34379702bcbde9ec6159c9fa03b13652e1f620dcd92125e"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "DELETE FROM gateway_session_stats WHERE day <= ?",
"describe": {
"columns": [],
"parameters": {
"Right": 1
},
"nullable": []
},
"hash": "c9e61180ec35dfab8623333fafa3b216b93440d0fddc0a37dd1b6c1813741f6a"
}
@@ -1,20 +0,0 @@
{
"db_name": "SQLite",
"query": "\n SELECT\n total_stake\n FROM nym_nodes\n WHERE node_id = ?\n ",
"describe": {
"columns": [
{
"name": "total_stake",
"ordinal": 0,
"type_info": "Integer"
}
],
"parameters": {
"Right": 1
},
"nullable": [
false
]
},
"hash": "d2e07d44594ca5b44a6100482ff432c39d761f2a0ac1d6515cf73416f2eb6c61"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "INSERT INTO summary\n (key, value_json, last_updated_utc)\n VALUES (?, ?, ?)\n ON CONFLICT(key) DO UPDATE SET\n value_json=excluded.value_json,\n last_updated_utc=excluded.last_updated_utc;",
"describe": {
"columns": [],
"parameters": {
"Right": 3
},
"nullable": []
},
"hash": "e0c76a959276e3b0f44c720af9c74a5bf4912ee73468e62e7d0d96b1d9074cbe"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO gateway_description (\n gateway_identity_key,\n moniker,\n website,\n security_contact,\n details,\n last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (gateway_identity_key) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 6
},
"nullable": []
},
"hash": "e9790b63ebe4bff5172bb8cb7bfc288364855003cf0e4d63e95047e7b502c650"
}
@@ -1,56 +0,0 @@
{
"db_name": "SQLite",
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE\n id = ?\n AND\n status = ?\n ORDER BY created_utc\n LIMIT 1",
"describe": {
"columns": [
{
"name": "id!",
"ordinal": 0,
"type_info": "Integer"
},
{
"name": "gateway_id!",
"ordinal": 1,
"type_info": "Integer"
},
{
"name": "status!",
"ordinal": 2,
"type_info": "Integer"
},
{
"name": "created_utc!",
"ordinal": 3,
"type_info": "Integer"
},
{
"name": "ip_address!",
"ordinal": 4,
"type_info": "Text"
},
{
"name": "log!",
"ordinal": 5,
"type_info": "Text"
},
{
"name": "last_assigned_utc",
"ordinal": 6,
"type_info": "Integer"
}
],
"parameters": {
"Right": 2
},
"nullable": [
false,
false,
false,
false,
false,
false,
true
]
},
"hash": "f0c64794cbaed87a1d3166251d8e6720c9a9fc929144188460849be85d915004"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "UPDATE\n gateways\n SET\n bonded = false\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 0
},
"nullable": []
},
"hash": "f7e3fa31d68c028bf39cc95389f29f8758ec922dd2e7ea064a1e537e580c9ee5"
}
@@ -1,12 +0,0 @@
{
"db_name": "SQLite",
"query": "\n INSERT INTO nym_nodes_packet_stats_raw (\n node_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
"describe": {
"columns": [],
"parameters": {
"Right": 5
},
"nullable": []
},
"hash": "fcb1698d9e0e3a14524c92e7c99a811588c2bbc50d4975487a0464321a1b18c9"
}
@@ -3,7 +3,7 @@
[package]
name = "nym-node-status-api"
version = "3.1.2"
version = "3.2.2"
authors.workspace = true
repository.workspace = true
homepage.workspace = true
@@ -28,11 +28,15 @@ moka = { workspace = true, features = ["future"] }
# Nym API: revert after Cheddar is out
nym-contracts-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-mixnet-contract-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["openapi"]}
nym-node-status-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"openapi",
] }
nym-node-status-client = { path = "../nym-node-status-client" }
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["middleware"]}
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
"middleware",
] }
nym-network-defaults = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-serde-helpers = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
nym-statistics-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
@@ -62,7 +66,7 @@ serde_json = { workspace = true }
serde_json_path = { workspace = true }
strum = { workspace = true }
strum_macros = { workspace = true }
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "sqlite", "time"] }
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "time"] }
thiserror = { workspace = true }
time = { workspace = true, features = ["formatting"] }
tokio = { workspace = true, features = ["rt-multi-thread"] }
@@ -77,16 +81,20 @@ utoipauto = { workspace = true }
nym-node-metrics = { path = "../../nym-node/nym-node-metrics" }
[features]
default = ["pg"]
sqlite = ["sqlx/sqlite"]
pg = ["sqlx/postgres"]
[build-dependencies]
anyhow = { workspace = true }
tokio = { workspace = true, features = ["macros"] }
sqlx = { workspace = true, features = [
"runtime-tokio-rustls",
"sqlite",
"macros",
"migrate",
] }
[dev-dependencies]
axum-test = "17.3.0"
time = { workspace = true, features = ["macros"] }
@@ -2,9 +2,9 @@
FROM harbor.nymte.ch/dockerhub/rust:latest AS builder
COPY ./ /usr/src/nym
WORKDIR /usr/src/nym/nym-node-status-api
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-api/
RUN cargo build --release
RUN cargo build --release --features pg
#-------------------------------------------------------------------
@@ -0,0 +1,37 @@
# this will only work with VPN, otherwise remove the harbor part
FROM harbor.nymte.ch/dockerhub/rust:latest AS builder
COPY ./ /usr/src/nym
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-api/
RUN cargo build --release --features sqlite --no-default-features
#-------------------------------------------------------------------
# The following environment variables are required at runtime:
#
# EXPLORER_API
# NYXD
# NYM_API
# DATABASE_URL
#
# And optionally:
#
# NYM_NODE_STATUS_API_NYM_HTTP_CACHE_TTL
# NYM_NODE_STATUS_API_HTTP_PORT
# NYM_API_CLIENT_TIMEOUT
# EXPLORER_CLIENT_TIMEOUT
# NODE_STATUS_API_MONITOR_REFRESH_INTERVAL
# NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL
#
# see https://github.com/nymtech/nym/blob/develop/nym-node-status-api/src/cli.rs for details
#-------------------------------------------------------------------
FROM harbor.nymte.ch/dockerhub/ubuntu:24.04
RUN apt-get update && apt-get install -y ca-certificates
WORKDIR /nym
COPY --from=builder /usr/src/nym/target/release/nym-node-status-api ./
ENTRYPOINT [ "/nym/nym-node-status-api" ]
@@ -0,0 +1,117 @@
# Makefile for nym-node-status-api database management
# --- Configuration ---
TEST_DATABASE_URL := postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# Docker compose service names
DB_SERVICE_NAME := postgres-test
DB_CONTAINER_NAME := nym_node_status_api_postgres_test
# Default target
.PHONY: default
default: help
# --- Main Targets ---
.PHONY: prepare-pg
prepare-pg: test-db-up test-db-wait test-db-migrate test-db-prepare test-db-down ## Setup PostgreSQL and prepare SQLx offline cache
.PHONY: test-db
test-db: test-db-up test-db-wait test-db-migrate test-db-run test-db-down ## Run tests with PostgreSQL database
.PHONY: dev-db
dev-db: test-db-up test-db-wait test-db-migrate ## Start PostgreSQL for development (keeps running)
@echo "PostgreSQL is running on port 5433"
@echo "Connection string: $(TEST_DATABASE_URL)"
# --- Docker Compose Targets ---
.PHONY: test-db-up
test-db-up: ## Start the PostgreSQL test database in the background
@echo "Starting PostgreSQL test database..."
docker compose up -d $(DB_SERVICE_NAME)
.PHONY: test-db-wait
test-db-wait: ## Wait for the PostgreSQL database to be healthy
@echo "Waiting for PostgreSQL database..."
@while ! docker inspect --format='{{.State.Health.Status}}' $(DB_CONTAINER_NAME) 2>/dev/null | grep -q 'healthy'; do \
echo -n "."; \
sleep 1; \
done; \
echo " Database is healthy!"
.PHONY: test-db-down
test-db-down: ## Stop and remove the test database
@echo "Stopping PostgreSQL test database..."
docker compose down
# --- SQLx Targets ---
.PHONY: test-db-migrate
test-db-migrate: ## Run database migrations against PostgreSQL
@echo "Running PostgreSQL migrations..."
DATABASE_URL="$(TEST_DATABASE_URL)" sqlx migrate run --source migrations_pg
.PHONY: test-db-prepare
test-db-prepare: ## Run sqlx prepare for compile-time query verification
@echo "Running sqlx prepare for PostgreSQL..."
DATABASE_URL="$(TEST_DATABASE_URL)" cargo sqlx prepare -- --features pg
# --- Build and Test Targets ---
.PHONY: test-db-run
test-db-run: ## Run tests with PostgreSQL feature
@echo "Running tests with PostgreSQL..."
DATABASE_URL="$(TEST_DATABASE_URL)" cargo test --features pg --no-default-features
.PHONY: build-pg
build-pg: ## Build with PostgreSQL feature
@echo "Building with PostgreSQL feature..."
cargo build --features pg --no-default-features
.PHONY: build-sqlite
build-sqlite: ## Build with SQLite feature (default)
@echo "Building with SQLite feature..."
cargo build --features sqlite --no-default-features
.PHONY: check-pg
check-pg: ## Check code with PostgreSQL feature
@echo "Checking code with PostgreSQL feature..."
cargo check --features pg --no-default-features
.PHONY: check-sqlite
check-sqlite: ## Check code with SQLite feature
@echo "Checking code with SQLite feature..."
cargo check --features sqlite --no-default-features
.PHONY: clippy
clippy: clippy-pg clippy-sqlite
.PHONY: clippy-pg
clippy-pg: ## Run clippy with PostgreSQL feature
@echo "Running clippy with PostgreSQL feature..."
cargo clippy --features pg --no-default-features -- -D warnings
.PHONY: clippy-sqlite
clippy-sqlite: ## Run clippy with SQLite feature (default)
@echo "Running clippy with SQLite feature..."
cargo clippy --features sqlite --no-default-features -- -D warnings
# --- Cleanup Targets ---
.PHONY: clean
clean: ## Clean build artifacts and SQLx cache
cargo clean
rm -rf .sqlx
.PHONY: clean-db
clean-db: test-db-down ## Stop database and clean volumes
docker volume rm -f nym-node-status-api_postgres_test_data 2>/dev/null || true
# --- Utility Targets ---
.PHONY: sqlx-cli
sqlx-cli: ## Install sqlx-cli if not already installed
@command -v sqlx >/dev/null 2>&1 || cargo install sqlx-cli --features postgres,sqlite
.PHONY: psql
psql: ## Connect to the running PostgreSQL database with psql
@docker exec -it $(DB_CONTAINER_NAME) psql -U testuser -d nym_node_status_api_test
.PHONY: help
help: ## Show help for Makefile targets
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
@@ -0,0 +1,104 @@
# PostgreSQL Support for nym-node-status-api
This project now supports both SQLite (default) and PostgreSQL databases.
## Quick Start with PostgreSQL
### 1. Install Prerequisites
```bash
# Install sqlx-cli if not already installed
make sqlx-cli
```
### 2. Prepare PostgreSQL for Development
```bash
# This will:
# - Start PostgreSQL in Docker
# - Run migrations
# - Generate SQLx offline query cache
# - Stop the database
make prepare-pg
```
### 3. Build with PostgreSQL
```bash
# Build with PostgreSQL feature
make build-pg
# Or manually:
cargo build --features pg --no-default-features
```
### 4. Run with PostgreSQL
```bash
# Start PostgreSQL for development (keeps running)
make dev-db
# In another terminal, run the application
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test \
cargo run --features pg --no-default-features
```
## Database Features
- `sqlite` (default): Uses SQLite database
- `pg`: Uses PostgreSQL database
Only one database feature can be active at a time.
## Migration Differences
SQLite migrations are in `migrations/`, PostgreSQL migrations are in `migrations_pg/`.
Key differences:
- **AUTOINCREMENT** → **SERIAL**
- **INTEGER CHECK (0,1)** → **BOOLEAN**
- **REAL** → **DOUBLE PRECISION**
- No table recreation needed for constraint changes in PostgreSQL
## Makefile Targets
```bash
make help # Show all available targets
make prepare-pg # Setup PostgreSQL and prepare SQLx cache
make dev-db # Start PostgreSQL for development
make test-db # Run tests with PostgreSQL
make build-pg # Build with PostgreSQL
make build-sqlite # Build with SQLite
make psql # Connect to running PostgreSQL
make clean # Clean build artifacts
make clean-db # Stop database and clean volumes
```
## Environment Variables
See `.env.example` for all configuration options. Key variable:
```bash
# For PostgreSQL:
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
# For SQLite:
DATABASE_URL=sqlite://nym-node-status-api.sqlite
```
## Troubleshooting
### SQLx Offline Mode
If you see "no cached data for this query" errors:
1. Ensure PostgreSQL is running: `make dev-db`
2. Run: `make test-db-prepare`
### Connection Refused
If you see "Connection refused" errors:
1. Check Docker is running: `docker ps`
2. Check PostgreSQL container: `docker ps | grep nym_node_status_api_postgres_test`
3. Restart database: `make test-db-down && make dev-db`
@@ -0,0 +1,81 @@
#!/bin/bash
# Build and push Node Status API container to harbor.nymte.ch
set -e
# Configuration
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
WORKING_DIRECTORY="${SCRIPT_DIR}"
CONTAINER_NAME="node-status-api"
REGISTRY="harbor.nymte.ch"
NAMESPACE="nym"
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Function to display usage
usage() {
echo "Usage: $0 [pg|sqlite|both]"
echo " pg - Build and push PostgreSQL version"
echo " sqlite - Build and push SQLite version"
echo " both - Build and push both versions (default)"
exit 1
}
# Parse arguments
DB_TYPE="${1:-both}"
if [[ ! "$DB_TYPE" =~ ^(pg|sqlite|both)$ ]]; then
usage
fi
# Get version from Cargo.toml
VERSION=$(grep "^version = " "${WORKING_DIRECTORY}/Cargo.toml" | sed -E 's/version = "(.*)"/\1/')
if [ -z "$VERSION" ]; then
echo -e "${RED}Error: Could not extract version from Cargo.toml${NC}"
exit 1
fi
echo -e "${YELLOW}Version: ${VERSION}${NC}"
# Login to Harbor
echo -e "${GREEN}Logging into Harbor...${NC}"
docker login "${REGISTRY}"
# Function to build and push
build_and_push() {
local db_type=$1
local dockerfile="Dockerfile-${db_type}"
echo -e "${GREEN}Building ${db_type} container...${NC}"
# Build from repository root (two levels up from script location)
docker build -f "${WORKING_DIRECTORY}/${dockerfile}" "${SCRIPT_DIR}/../.." \
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${db_type}" \
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${db_type}"
echo -e "${GREEN}Pushing ${db_type} container to Harbor...${NC}"
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${db_type}"
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${db_type}"
echo -e "${GREEN}Successfully built and pushed ${CONTAINER_NAME}:${VERSION}-${db_type}${NC}"
}
# Build based on selection
case "$DB_TYPE" in
pg)
build_and_push "pg"
;;
sqlite)
build_and_push "sqlite"
;;
both)
build_and_push "pg"
echo ""
build_and_push "sqlite"
;;
esac
echo -e "${GREEN}All builds completed successfully!${NC}"
@@ -1,17 +1,20 @@
use anyhow::{anyhow, Result};
use anyhow::Result;
#[cfg(feature = "sqlite")]
use sqlx::{Connection, SqliteConnection};
#[cfg(feature = "sqlite")]
#[cfg(target_family = "unix")]
use std::fs::Permissions;
#[cfg(feature = "sqlite")]
#[cfg(target_family = "unix")]
use std::os::unix::fs::PermissionsExt;
#[cfg(feature = "sqlite")]
use tokio::{fs::File, io::AsyncWriteExt};
#[cfg(feature = "sqlite")]
const SQLITE_DB_FILENAME: &str = "nym-node-status-api.sqlite";
/// If you need to re-run migrations or reset the db, just run
/// cargo clean -p nym-node-status-api
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()> {
#[cfg(feature = "sqlite")]
async fn init_db() -> Result<()> {
let out_dir = read_env_var("OUT_DIR")?;
let database_path = format!("{out_dir}/{SQLITE_DB_FILENAME}?mode=rwc");
@@ -30,11 +33,22 @@ async fn main() -> Result<()> {
Ok(())
}
/// If you need to re-run migrations or reset the db, just run
/// cargo clean -p nym-node-status-api
#[tokio::main(flavor = "current_thread")]
async fn main() -> Result<()> {
#[cfg(feature = "sqlite")]
init_db().await?;
Ok(())
}
#[cfg(feature = "sqlite")]
fn read_env_var(var: &str) -> Result<String> {
std::env::var(var).map_err(|_| anyhow!("You need to set {} env var", var))
std::env::var(var).map_err(|_| anyhow::anyhow!("You need to set {} env var", var))
}
/// use `./enter_db.sh` to inspect DB
#[cfg(feature = "sqlite")]
async fn write_db_path_to_file(out_dir: &str, db_filename: &str) -> anyhow::Result<()> {
let mut file = File::create("settings.sql").await?;
let settings = ".mode columns
@@ -0,0 +1,21 @@
services:
postgres-test:
image: postgres:16-alpine
container_name: nym_node_status_api_postgres_test
environment:
POSTGRES_DB: nym_node_status_api_test
POSTGRES_USER: testuser
POSTGRES_PASSWORD: testpass
ports:
- '5433:5432' # Map to 5433 to avoid conflicts with default PostgreSQL
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U testuser -d nym_node_status_api_test']
interval: 5s
timeout: 5s
retries: 5
# Optional: Add volume for persistent data during development
# volumes:
# - postgres_test_data:/var/lib/postgresql/data
# volumes:
# postgres_test_data:
@@ -0,0 +1,16 @@
-- Add partial indexes for NOT NULL filtering to improve performance of /explorer/v3/nodes endpoint
-- Index for queries filtering on self_described IS NOT NULL
CREATE INDEX IF NOT EXISTS idx_nym_nodes_self_described_not_null
ON nym_nodes(node_id)
WHERE self_described IS NOT NULL;
-- Index for queries filtering on bond_info IS NOT NULL
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_info_not_null
ON nym_nodes(node_id)
WHERE bond_info IS NOT NULL;
-- Composite index for queries filtering on both bond_info AND self_described
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_self_described
ON nym_nodes(node_id)
WHERE bond_info IS NOT NULL AND self_described IS NOT NULL;
@@ -0,0 +1,113 @@
CREATE TABLE gateways
(
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR NOT NULL UNIQUE,
self_described VARCHAR NOT NULL,
explorer_pretty_bond VARCHAR,
last_probe_result VARCHAR,
last_probe_log VARCHAR,
config_score INTEGER NOT NULL DEFAULT 0,
config_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
config_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
routing_score INTEGER NOT NULL DEFAULT 0,
routing_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
routing_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
test_run_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
last_testrun_utc BIGINT,
last_updated_utc BIGINT NOT NULL,
bonded BOOLEAN NOT NULL DEFAULT FALSE,
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
performance INTEGER NOT NULL DEFAULT 0
);
CREATE INDEX idx_gateway_description_gateway_identity_key ON gateways (gateway_identity_key);
CREATE TABLE mixnodes (
id SERIAL PRIMARY KEY,
identity_key VARCHAR NOT NULL UNIQUE,
mix_id BIGINT NOT NULL UNIQUE,
bonded BOOLEAN NOT NULL DEFAULT FALSE,
total_stake BIGINT NOT NULL,
host VARCHAR NOT NULL,
http_api_port BIGINT NOT NULL,
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
full_details VARCHAR,
self_described VARCHAR,
last_updated_utc BIGINT NOT NULL,
is_dp_delegatee BOOLEAN NOT NULL DEFAULT FALSE
);
CREATE INDEX idx_mixnodes_mix_id ON mixnodes (mix_id);
CREATE INDEX idx_mixnodes_identity_key ON mixnodes (identity_key);
CREATE TABLE mixnode_description (
id SERIAL PRIMARY KEY,
mix_id BIGINT UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc BIGINT NOT NULL,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
);
-- Indexes for description table
CREATE INDEX idx_mixnode_description_mix_id ON mixnode_description (mix_id);
CREATE TABLE summary
(
key VARCHAR PRIMARY KEY,
value_json VARCHAR,
last_updated_utc BIGINT NOT NULL
);
CREATE TABLE summary_history
(
id SERIAL PRIMARY KEY,
date VARCHAR UNIQUE NOT NULL,
timestamp_utc BIGINT NOT NULL,
value_json VARCHAR
);
CREATE INDEX idx_summary_history_timestamp_utc ON summary_history (timestamp_utc);
CREATE INDEX idx_summary_history_date ON summary_history (date);
CREATE TABLE gateway_description (
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc BIGINT NOT NULL,
FOREIGN KEY (gateway_identity_key) REFERENCES gateways (gateway_identity_key)
);
CREATE TABLE mixnode_daily_stats (
id SERIAL PRIMARY KEY,
mix_id BIGINT NOT NULL,
total_stake BIGINT NOT NULL,
date_utc VARCHAR NOT NULL,
packets_received INTEGER DEFAULT 0,
packets_sent INTEGER DEFAULT 0,
packets_dropped INTEGER DEFAULT 0,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id),
UNIQUE (mix_id, date_utc) -- This constraint automatically creates an index
);
CREATE TABLE testruns
(
id SERIAL PRIMARY KEY,
gateway_id INTEGER NOT NULL,
status INTEGER NOT NULL, -- 0=pending, 1=in-progress, 2=complete
timestamp_utc BIGINT NOT NULL,
ip_address VARCHAR NOT NULL,
log VARCHAR NOT NULL,
FOREIGN KEY (gateway_id) REFERENCES gateways (id)
);
@@ -0,0 +1,5 @@
ALTER TABLE testruns
RENAME COLUMN timestamp_utc TO created_utc;
ALTER TABLE testruns
ADD COLUMN last_assigned_utc BIGINT;
@@ -0,0 +1,16 @@
CREATE TABLE gateway_session_stats (
id SERIAL PRIMARY KEY,
gateway_identity_key VARCHAR NOT NULL,
node_id BIGINT NOT NULL,
day DATE NOT NULL,
unique_active_clients BIGINT NOT NULL,
session_started BIGINT NOT NULL,
users_hashes VARCHAR,
vpn_sessions VARCHAR,
mixnet_sessions VARCHAR,
unknown_sessions VARCHAR,
UNIQUE (node_id, day) -- This constraint automatically creates an index
);
CREATE INDEX idx_gateway_session_stats_identity_key ON gateway_session_stats (gateway_identity_key);
CREATE INDEX idx_gateway_session_stats_day ON gateway_session_stats (day);
@@ -0,0 +1,11 @@
CREATE TABLE mixnode_packet_stats_raw (
id SERIAL PRIMARY KEY,
mix_id BIGINT NOT NULL,
timestamp_utc BIGINT NOT NULL,
packets_received INTEGER,
packets_sent INTEGER,
packets_dropped INTEGER,
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
);
CREATE INDEX idx_mixnode_packet_stats_raw_mix_id_timestamp_utc ON mixnode_packet_stats_raw (mix_id, timestamp_utc);
@@ -0,0 +1,54 @@
ALTER TABLE mixnodes DROP COLUMN blacklisted;
ALTER TABLE gateways DROP COLUMN blacklisted;
CREATE TABLE nym_nodes (
node_id INTEGER PRIMARY KEY,
ed25519_identity_pubkey VARCHAR NOT NULL UNIQUE,
total_stake BIGINT NOT NULL,
ip_addresses TEXT NOT NULL,
mix_port INTEGER NOT NULL,
x25519_sphinx_pubkey VARCHAR NOT NULL UNIQUE,
node_role TEXT NOT NULL,
supported_roles TEXT NOT NULL,
performance VARCHAR NOT NULL,
entry TEXT,
last_updated_utc INTEGER NOT NULL
);
CREATE INDEX idx_nym_nodes_node_id ON nym_nodes (node_id);
CREATE INDEX idx_nym_nodes_ed25519_identity_pubkey ON nym_nodes (ed25519_identity_pubkey);
CREATE TABLE nym_node_descriptions (
id SERIAL PRIMARY KEY,
node_id INTEGER UNIQUE NOT NULL,
moniker VARCHAR,
website VARCHAR,
security_contact VARCHAR,
details VARCHAR,
last_updated_utc INTEGER NOT NULL,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
);
CREATE TABLE nym_nodes_packet_stats_raw (
id SERIAL PRIMARY KEY,
node_id INTEGER NOT NULL,
timestamp_utc INTEGER NOT NULL,
packets_received INTEGER,
packets_sent INTEGER,
packets_dropped INTEGER,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
);
CREATE INDEX idx_nym_nodes_packet_stats_raw_node_id_timestamp_utc ON nym_nodes_packet_stats_raw (node_id, timestamp_utc);
CREATE TABLE nym_node_daily_mixing_stats (
id SERIAL PRIMARY KEY,
node_id INTEGER NOT NULL,
total_stake BIGINT NOT NULL,
date_utc VARCHAR NOT NULL,
packets_received INTEGER DEFAULT 0,
packets_sent INTEGER DEFAULT 0,
packets_dropped INTEGER DEFAULT 0,
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id),
UNIQUE (node_id, date_utc) -- This constraint automatically creates an index
);
@@ -0,0 +1,23 @@
ALTER TABLE nym_nodes ADD COLUMN self_described TEXT;
ALTER TABLE nym_nodes ADD COLUMN bond_info TEXT;
-- PostgreSQL doesn't need table recreation for adding ON DELETE CASCADE
-- We can drop and recreate the foreign key constraints directly
-- Drop existing foreign key constraints
ALTER TABLE nym_node_descriptions DROP CONSTRAINT IF EXISTS nym_node_descriptions_node_id_fkey;
ALTER TABLE nym_nodes_packet_stats_raw DROP CONSTRAINT IF EXISTS nym_nodes_packet_stats_raw_node_id_fkey;
ALTER TABLE nym_node_daily_mixing_stats DROP CONSTRAINT IF EXISTS nym_node_daily_mixing_stats_node_id_fkey;
-- Add foreign key constraints with ON DELETE CASCADE
ALTER TABLE nym_node_descriptions
ADD CONSTRAINT nym_node_descriptions_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
ALTER TABLE nym_nodes_packet_stats_raw
ADD CONSTRAINT nym_nodes_packet_stats_raw_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
ALTER TABLE nym_node_daily_mixing_stats
ADD CONSTRAINT nym_node_daily_mixing_stats_node_id_fkey
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
@@ -0,0 +1,9 @@
-- Removing UNIQUE constraints on nym_nodes
-- In PostgreSQL, we can drop constraints directly without recreating the table
-- Drop the unique constraints
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_ed25519_identity_pubkey_key;
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_x25519_sphinx_pubkey_key;
-- The columns and indexes remain, only the unique constraints are removed
-- The existing indexes idx_nym_nodes_node_id and idx_nym_nodes_ed25519_identity_pubkey remain unchanged
@@ -0,0 +1,112 @@
-- for a couple of days after migrating chrono -> time, we stored dates as
-- 2025-June-DD instead of 2025-06-DD. This migration fixes those entries.
--
-- Because of a UNIQUE constraint on (node_id, date_utc), we can't just rename in-place.
-- - merge (add) node stats back to the original table where conflict (node_id, date_utc) would exist
-- - delete invalid records from original table (those stats were merged into correct rows above)
-- - insert rows that did not have a conflicting (node_Id, date_utc) combo
-- Conflicts affect only the date which has both kinds of entries,
-- e.g. 2025-06-05 and 2025-June-05 (date when this change was deployed)
--
-- This applies to both affected tables.
-- ----------------------------------------
-- mixnode_daily_stats
-- ----------------------------------------
-- First, copy over rows with invalid date to a temp table (in the correct date format)
CREATE TEMP TABLE tmp_mix AS
SELECT
mix_id,
REPLACE(date_utc,'June','06') AS new_date,
SUM(total_stake) AS total_stake_sum,
SUM(packets_received) AS packets_received_sum,
SUM(packets_sent) AS packets_sent_sum,
SUM(packets_dropped) AS packets_dropped_sum
FROM mixnode_daily_stats
WHERE date_utc LIKE '%June%'
GROUP BY mix_id, new_date;
UPDATE mixnode_daily_stats AS m
SET
total_stake = m.total_stake,
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc)
WHERE EXISTS (
SELECT 1 FROM tmp_mix
WHERE mix_id = m.mix_id
AND new_date = m.date_utc
);
DELETE FROM mixnode_daily_stats
WHERE date_utc LIKE '%June%';
INSERT INTO mixnode_daily_stats
(mix_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
SELECT
mix_id,
new_date,
total_stake_sum,
packets_received_sum,
packets_sent_sum,
packets_dropped_sum
FROM tmp_mix AS t
-- only those whose new_date did _not_ already exist
WHERE NOT EXISTS (
SELECT 1 FROM mixnode_daily_stats AS m
WHERE m.mix_id = t.mix_id
AND m.date_utc = t.new_date
);
DROP TABLE tmp_mix;
-- ----------------------------------------
-- nym_node_daily_mixing_stats
-- ----------------------------------------
CREATE TEMP TABLE tmp_nym_node_stats AS
SELECT
node_id,
REPLACE(date_utc,'June','06') AS new_date,
SUM(total_stake) AS total_stake_sum,
SUM(packets_received) AS packets_received_sum,
SUM(packets_sent) AS packets_sent_sum,
SUM(packets_dropped) AS packets_dropped_sum
FROM nym_node_daily_mixing_stats
WHERE date_utc LIKE '%June%'
GROUP BY node_id, new_date;
UPDATE nym_node_daily_mixing_stats AS m
SET
total_stake = m.total_stake,
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc)
WHERE EXISTS (
SELECT 1 FROM tmp_nym_node_stats
WHERE node_id = m.node_id
AND new_date = m.date_utc
);
DELETE FROM nym_node_daily_mixing_stats
WHERE date_utc LIKE '%June%';
INSERT INTO nym_node_daily_mixing_stats
(node_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
SELECT
node_id,
new_date,
total_stake_sum,
packets_received_sum,
packets_sent_sum,
packets_dropped_sum
FROM tmp_nym_node_stats AS t
WHERE NOT EXISTS (
SELECT 1 FROM nym_node_daily_mixing_stats AS m
WHERE m.node_id = t.node_id
AND m.date_utc = t.new_date
);
DROP TABLE tmp_nym_node_stats;
@@ -0,0 +1,3 @@
-- Convert ip_addresses column from TEXT to JSONB for better type safety
ALTER TABLE nym_nodes
ALTER COLUMN ip_addresses TYPE JSONB USING ip_addresses::JSONB;
@@ -0,0 +1,7 @@
-- Convert remaining TEXT columns that store JSON to JSONB
ALTER TABLE nym_nodes
ALTER COLUMN node_role TYPE JSONB USING node_role::JSONB,
ALTER COLUMN supported_roles TYPE JSONB USING supported_roles::JSONB,
ALTER COLUMN entry TYPE JSONB USING entry::JSONB,
ALTER COLUMN self_described TYPE JSONB USING self_described::JSONB,
ALTER COLUMN bond_info TYPE JSONB USING bond_info::JSONB;
@@ -0,0 +1,17 @@
-- Add partial indexes for NOT NULL filtering to improve performance of /explorer/v3/nodes endpoint
-- PostgreSQL version
-- Index for queries filtering on self_described IS NOT NULL
CREATE INDEX IF NOT EXISTS idx_nym_nodes_self_described_not_null
ON nym_nodes(node_id)
WHERE self_described IS NOT NULL;
-- Index for queries filtering on bond_info IS NOT NULL
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_info_not_null
ON nym_nodes(node_id)
WHERE bond_info IS NOT NULL;
-- Composite index for queries filtering on both bond_info AND self_described
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_self_described
ON nym_nodes(node_id)
WHERE bond_info IS NOT NULL AND self_described IS NOT NULL;
@@ -1,24 +1,52 @@
use anyhow::{anyhow, Result};
use sqlx::{
migrate::Migrator,
query,
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
ConnectOptions, SqlitePool,
};
use std::{str::FromStr, time::Duration};
pub(crate) mod models;
pub(crate) mod queries;
pub(crate) mod query_wrapper;
#[cfg(test)]
mod tests;
// Re-export the query wrapper functions for easier access
pub(crate) use query_wrapper::query;
#[allow(unused_imports)]
pub(crate) use query_wrapper::query_as;
#[cfg(feature = "sqlite")]
use sqlx::{
migrate::Migrator,
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
ConnectOptions, SqlitePool,
};
#[cfg(feature = "pg")]
use sqlx::{migrate::Migrator, postgres::PgConnectOptions, ConnectOptions, PgPool};
#[cfg(feature = "sqlite")]
static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
#[cfg(feature = "pg")]
static MIGRATOR: Migrator = sqlx::migrate!("./migrations_pg");
#[cfg(feature = "sqlite")]
pub(crate) type DbPool = SqlitePool;
#[cfg(feature = "pg")]
pub(crate) type DbPool = PgPool;
#[cfg(feature = "sqlite")]
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Sqlite>;
#[cfg(feature = "pg")]
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Postgres>;
pub(crate) struct Storage {
pool: DbPool,
}
impl Storage {
#[cfg(feature = "sqlite")]
pub async fn init(connection_url: String, busy_timeout: Duration) -> Result<Self> {
let connect_options = SqliteConnectOptions::from_str(&connection_url)?
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal)
@@ -41,16 +69,39 @@ impl Storage {
Ok(Storage { pool })
}
#[cfg(feature = "pg")]
pub async fn init(connection_url: String, _busy_timeout: Duration) -> Result<Self> {
use std::env;
let mut connect_options =
PgConnectOptions::from_str(&connection_url)?.disable_statement_logging();
let ssl_cert_path = env::var("PG_CERT").ok();
if let Some(ssl_cert) = ssl_cert_path {
connect_options = connect_options
.ssl_mode(sqlx::postgres::PgSslMode::Require)
.ssl_root_cert(ssl_cert);
}
let pool = sqlx::PgPool::connect_with(connect_options)
.await
.map_err(|err| anyhow!("Failed to connect to {}: {}", &connection_url, err))?;
MIGRATOR.run(&pool).await?;
Ok(Storage { pool })
}
/// Cloning pool is cheap, it's the same underlying set of connections
pub fn pool_owned(&self) -> DbPool {
self.pool.clone()
}
#[cfg(feature = "sqlite")]
async fn assert_busy_timeout(pool: DbPool, expected_busy_timeout_s: i64) -> Result<()> {
let mut conn = pool.acquire().await?;
// Sqlite stores this value as miliseconds
// https://www.sqlite.org/pragma.html#pragma_busy_timeout
let busy_timeout_db = query!("PRAGMA busy_timeout;")
let busy_timeout_db = sqlx::query!("PRAGMA busy_timeout;")
.fetch_one(conn.as_mut())
.await?;
@@ -38,11 +38,11 @@ pub(crate) struct GatewayInsertRecord {
pub(crate) performance: u8,
}
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub(crate) struct GatewayDto {
pub(crate) gateway_identity_key: String,
pub(crate) bonded: bool,
pub(crate) performance: i64,
pub(crate) performance: i32,
pub(crate) self_described: Option<String>,
pub(crate) explorer_pretty_bond: Option<String>,
pub(crate) last_probe_result: Option<String>,
@@ -121,7 +121,7 @@ pub(crate) struct MixnodeRecord {
pub(crate) is_dp_delegatee: bool,
}
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub(crate) struct MixnodeDto {
pub(crate) mix_id: i64,
pub(crate) bonded: bool,
@@ -183,14 +183,14 @@ pub(crate) struct BondedStatusDto {
}
#[allow(unused)]
#[derive(Debug, Clone, Default)]
#[derive(Debug, Clone, Default, FromRow)]
pub(crate) struct SummaryDto {
pub(crate) key: String,
pub(crate) value_json: String,
pub(crate) last_updated_utc: i64,
}
#[derive(Debug, Clone, Default)]
#[derive(Debug, Clone, Default, FromRow)]
pub(crate) struct SummaryHistoryDto {
#[allow(dead_code)]
pub id: i64,
@@ -287,11 +287,11 @@ pub(crate) mod gateway {
}
#[allow(dead_code)] // not dead code, this is SQL data model
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub struct TestRunDto {
pub id: i64,
pub gateway_id: i64,
pub status: i64,
pub id: i32,
pub gateway_id: i32,
pub status: i32,
pub created_utc: i64,
pub ip_address: String,
pub log: String,
@@ -313,9 +313,9 @@ pub struct GatewayIdentityDto {
}
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
#[derive(Debug, Clone)]
#[derive(Debug, Clone, FromRow)]
pub struct GatewayInfoDto {
pub id: i64,
pub id: i32,
pub gateway_identity_key: String,
pub self_described: Option<String>,
pub explorer_pretty_bond: Option<String>,
@@ -379,6 +379,7 @@ impl ScrapeNodeKind {
}
}
#[derive(Clone)]
pub(crate) struct ScraperNodeInfo {
pub node_kind: ScrapeNodeKind,
pub hosts: Vec<String>,
@@ -412,11 +413,11 @@ impl ScraperNodeInfo {
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
#[derive(FromRow, Debug)]
pub(crate) struct NymNodeDto {
pub node_id: i64,
pub node_id: i32,
pub ed25519_identity_pubkey: String,
pub total_stake: i64,
pub ip_addresses: serde_json::Value,
pub mix_port: i64,
pub mix_port: i32,
pub x25519_sphinx_pubkey: String,
pub node_role: serde_json::Value,
pub supported_roles: serde_json::Value,
@@ -429,11 +430,11 @@ pub(crate) struct NymNodeDto {
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
#[derive(Debug)]
pub(crate) struct NymNodeInsertRecord {
pub node_id: i64,
pub node_id: i32,
pub ed25519_identity_pubkey: String,
pub total_stake: i64,
pub ip_addresses: serde_json::Value,
pub mix_port: i64,
pub mix_port: i32,
pub x25519_sphinx_pubkey: String,
pub node_role: serde_json::Value,
pub supported_roles: serde_json::Value,
@@ -441,7 +442,7 @@ pub(crate) struct NymNodeInsertRecord {
pub entry: Option<serde_json::Value>,
pub self_described: Option<serde_json::Value>,
pub bond_info: Option<serde_json::Value>,
pub last_updated_utc: String,
pub last_updated_utc: i64,
}
impl NymNodeInsertRecord {
@@ -450,7 +451,7 @@ impl NymNodeInsertRecord {
bond_info: Option<&NymNodeDetails>,
self_described: Option<&NymNodeDescription>,
) -> anyhow::Result<Self> {
let now = OffsetDateTime::now_utc().to_string();
let now = OffsetDateTime::now_utc().unix_timestamp();
// if bond info is missing, set stake to 0
let total_stake = bond_info
@@ -461,11 +462,11 @@ impl NymNodeInsertRecord {
let self_described = serialize_opt_to_value!(self_described)?;
let record = Self {
node_id: skimmed_node.node_id.into(),
node_id: skimmed_node.node_id as i32,
ed25519_identity_pubkey: skimmed_node.ed25519_identity_pubkey.to_base58_string(),
total_stake,
ip_addresses: serde_json::to_value(&skimmed_node.ip_addresses)?,
mix_port: skimmed_node.mix_port as i64,
mix_port: skimmed_node.mix_port as i32,
x25519_sphinx_pubkey: skimmed_node.x25519_sphinx_pubkey.to_base58_string(),
node_role: serde_json::to_value(&skimmed_node.role)?,
supported_roles: serde_json::to_value(skimmed_node.supported_roles)?,
@@ -514,11 +515,11 @@ impl TryFrom<NymNodeDto> for SkimmedNode {
}
}
#[derive(Debug, Serialize, Deserialize, sqlx::Decode)]
#[derive(Debug, Serialize, Deserialize, sqlx::Decode, FromRow)]
pub struct NodeStats {
pub packets_received: i64,
pub packets_sent: i64,
pub packets_dropped: i64,
pub packets_received: i32,
pub packets_sent: i32,
pub packets_dropped: i32,
}
pub struct InsertStatsRecord {
@@ -3,32 +3,32 @@ use std::collections::HashSet;
use crate::{
db::{
models::{GatewayDto, GatewayInsertRecord},
DbPool,
DbConnection, DbPool,
},
http::models::Gateway,
node_scraper::helpers::NodeDescriptionResponse,
};
use futures_util::TryStreamExt;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use tracing::error;
pub(crate) async fn select_gateway_identity(
conn: &mut PoolConnection<Sqlite>,
gateway_pk: i64,
conn: &mut DbConnection,
gateway_pk: i32,
) -> anyhow::Result<String> {
let record = sqlx::query!(
let record = crate::db::query(
r#"SELECT
gateway_identity_key
FROM
gateways
WHERE
id = ?"#,
gateway_pk
)
.bind(gateway_pk)
.fetch_one(conn.as_mut())
.await?;
Ok(record.gateway_identity_key)
Ok(record.try_get("gateway_identity_key")?)
}
pub(crate) async fn update_bonded_gateways(
@@ -37,7 +37,7 @@ pub(crate) async fn update_bonded_gateways(
) -> anyhow::Result<()> {
let mut tx = pool.begin().await?;
sqlx::query!(
crate::db::query(
r#"UPDATE
gateways
SET
@@ -48,7 +48,7 @@ pub(crate) async fn update_bonded_gateways(
.await?;
for record in gateways {
sqlx::query!(
crate::db::query(
"INSERT INTO gateways
(gateway_identity_key, bonded,
self_described, explorer_pretty_bond,
@@ -60,13 +60,13 @@ pub(crate) async fn update_bonded_gateways(
explorer_pretty_bond=excluded.explorer_pretty_bond,
last_updated_utc=excluded.last_updated_utc,
performance = excluded.performance;",
record.identity_key,
record.bonded,
record.self_described,
record.explorer_pretty_bond,
record.last_updated_utc,
record.performance
)
.bind(record.identity_key)
.bind(record.bonded)
.bind(record.self_described)
.bind(record.explorer_pretty_bond)
.bind(record.last_updated_utc)
.bind(record.performance as i32)
.execute(&mut *tx)
.await?;
}
@@ -78,22 +78,21 @@ pub(crate) async fn update_bonded_gateways(
pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gateway>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as!(
GatewayDto,
let items = crate::db::query_as::<GatewayDto>(
r#"SELECT
gw.gateway_identity_key as "gateway_identity_key!",
gw.bonded as "bonded: bool",
gw.performance as "performance!",
gw.self_described as "self_described?",
gw.explorer_pretty_bond as "explorer_pretty_bond?",
gw.last_probe_result as "last_probe_result?",
gw.last_probe_log as "last_probe_log?",
gw.last_testrun_utc as "last_testrun_utc?",
gw.last_updated_utc as "last_updated_utc!",
COALESCE(gd.moniker, "NA") as "moniker!",
COALESCE(gd.website, "NA") as "website!",
COALESCE(gd.security_contact, "NA") as "security_contact!",
COALESCE(gd.details, "NA") as "details!"
gw.gateway_identity_key,
gw.bonded,
gw.performance,
gw.self_described,
gw.explorer_pretty_bond,
gw.last_probe_result,
gw.last_probe_log,
gw.last_testrun_utc,
gw.last_updated_utc,
COALESCE(gd.moniker, 'NA') as moniker,
COALESCE(gd.website, 'NA') as website,
COALESCE(gd.security_contact, 'NA') as security_contact,
COALESCE(gd.details, 'NA') as details
FROM gateways gw
LEFT JOIN gateway_description gd
ON gw.gateway_identity_key = gd.gateway_identity_key
@@ -114,29 +113,29 @@ pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gatewa
pub(crate) async fn get_bonded_gateway_id_keys(pool: &DbPool) -> anyhow::Result<HashSet<String>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query!(
let items = crate::db::query(
r#"
SELECT gateway_identity_key
FROM gateways
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?
.into_iter()
.map(|record| record.gateway_identity_key)
.map(|record| record.try_get::<String, _>("gateway_identity_key").unwrap())
.collect::<HashSet<_>>();
Ok(items)
}
pub(crate) async fn insert_gateway_description(
conn: &mut PoolConnection<Sqlite>,
identity_key: &str,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
identity_key: String,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO gateway_description (
gateway_identity_key,
@@ -153,15 +152,54 @@ pub(crate) async fn insert_gateway_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
identity_key,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(identity_key)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn get_or_create_gateway(
conn: &mut DbConnection,
gateway_identity_key: &str,
) -> anyhow::Result<i32> {
// Try to find existing gateway
let existing = crate::db::query("SELECT id FROM gateways WHERE gateway_identity_key = ?")
.bind(gateway_identity_key.to_string())
.fetch_optional(conn.as_mut())
.await?;
if let Some(row) = existing {
return Ok(row.try_get("id")?);
}
// Create new gateway
tracing::info!("Creating new gateway record for {}", gateway_identity_key);
let now = crate::utils::now_utc().unix_timestamp();
let result = crate::db::query(
r#"INSERT INTO gateways (
gateway_identity_key,
bonded,
performance,
self_described,
last_updated_utc
) VALUES (?, ?, ?, ?, ?)
RETURNING id"#,
)
.bind(gateway_identity_key.to_string())
.bind(true) // Assume bonded since being tested
.bind(0) // Initial performance
.bind("null")
.bind(now)
.fetch_one(conn.as_mut())
.await?;
Ok(result.try_get("id")?)
}
@@ -6,6 +6,7 @@ use futures_util::TryStreamExt;
use time::Date;
use tracing::error;
#[cfg(feature = "sqlite")]
pub(crate) async fn insert_session_records(
pool: &DbPool,
records: Vec<GatewaySessionsRecord>,
@@ -36,6 +37,38 @@ pub(crate) async fn insert_session_records(
Ok(())
}
#[cfg(feature = "pg")]
pub(crate) async fn insert_session_records(
pool: &DbPool,
records: Vec<GatewaySessionsRecord>,
) -> anyhow::Result<()> {
let mut tx = pool.begin().await?;
for record in records {
sqlx::query!(
"INSERT INTO gateway_session_stats
(gateway_identity_key, node_id, day,
unique_active_clients, session_started, users_hashes,
vpn_sessions, mixnet_sessions, unknown_sessions)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT DO NOTHING",
record.gateway_identity_key,
record.node_id,
record.day,
record.unique_active_clients,
record.session_started,
record.users_hashes,
record.vpn_sessions,
record.mixnet_sessions,
record.unknown_sessions,
)
.execute(&mut *tx)
.await?;
}
tx.commit().await?;
Ok(())
}
pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<SessionStats>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as(
@@ -68,7 +101,8 @@ pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<Sess
pub(crate) async fn delete_old_records(pool: &DbPool, cut_off: Date) -> anyhow::Result<()> {
let mut conn = pool.acquire().await?;
sqlx::query!("DELETE FROM gateway_session_stats WHERE day <= ?", cut_off)
crate::db::query("DELETE FROM gateway_session_stats WHERE day <= ?")
.bind(cut_off)
.execute(&mut *conn)
.await?;
Ok(())
@@ -6,8 +6,8 @@ use crate::db::{models::NetworkSummary, DbPool};
/// `daily_summary`
pub(crate) async fn insert_summaries(
pool: &DbPool,
summaries: &[(&str, usize)],
summary: &NetworkSummary,
summaries: Vec<(String, usize)>,
summary: NetworkSummary,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
insert_summary(pool, summaries, last_updated).await?;
@@ -19,7 +19,7 @@ pub(crate) async fn insert_summaries(
async fn insert_summary(
pool: &DbPool,
summaries: &[(&str, usize)],
summaries: Vec<(String, usize)>,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
let timestamp = last_updated.unix_timestamp();
@@ -27,17 +27,17 @@ async fn insert_summary(
for (kind, value) in summaries {
let value = value.to_string();
sqlx::query!(
crate::db::query(
"INSERT INTO summary
(key, value_json, last_updated_utc)
VALUES (?, ?, ?)
ON CONFLICT(key) DO UPDATE SET
value_json=excluded.value_json,
last_updated_utc=excluded.last_updated_utc;",
kind,
value,
timestamp
)
.bind(kind.clone())
.bind(value)
.bind(timestamp)
.execute(&mut *tx)
.await
.map_err(|err| {
@@ -60,7 +60,7 @@ async fn insert_summary(
/// This is not aggregate data, it's a set of latest data points
async fn insert_summary_history(
pool: &DbPool,
summary: &NetworkSummary,
summary: NetworkSummary,
last_updated: UtcDateTime,
) -> anyhow::Result<()> {
let mut conn = pool.acquire().await?;
@@ -70,17 +70,17 @@ async fn insert_summary_history(
let date = datetime_to_only_date_str(last_updated);
sqlx::query!(
crate::db::query(
"INSERT INTO summary_history
(date, timestamp_utc, value_json)
VALUES (?, ?, ?)
ON CONFLICT(date) DO UPDATE SET
timestamp_utc=excluded.timestamp_utc,
value_json=excluded.value_json;",
date,
timestamp,
value_json
)
.bind(date)
.bind(timestamp)
.bind(value_json)
.execute(&mut *conn)
.await?;
@@ -1,13 +1,13 @@
use std::collections::HashSet;
use futures_util::TryStreamExt;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use tracing::error;
use crate::{
db::{
models::{MixnodeDto, MixnodeRecord},
DbPool,
DbConnection, DbPool,
},
http::models::{DailyStats, Mixnode},
node_scraper::helpers::NodeDescriptionResponse,
@@ -20,7 +20,7 @@ pub(crate) async fn update_mixnodes(
let mut tx = pool.begin().await?;
// mark all as unbonded
sqlx::query!(
crate::db::query(
r#"UPDATE
mixnodes
SET
@@ -31,9 +31,9 @@ pub(crate) async fn update_mixnodes(
.await?;
// existing nodes get updated on insert
for record in mixnodes.iter() {
for record in mixnodes.into_iter() {
// https://www.sqlite.org/lang_upsert.html
sqlx::query!(
crate::db::query(
"INSERT INTO mixnodes
(mix_id, identity_key, bonded, total_stake,
host, http_api_port, full_details,
@@ -46,17 +46,17 @@ pub(crate) async fn update_mixnodes(
full_details=excluded.full_details,self_described=excluded.self_described,
last_updated_utc=excluded.last_updated_utc,
is_dp_delegatee = excluded.is_dp_delegatee;",
record.mix_id,
record.identity_key,
record.bonded,
record.total_stake,
record.host,
record.http_port,
record.full_details,
record.self_described,
record.last_updated_utc,
record.is_dp_delegatee
)
.bind(record.mix_id as i64)
.bind(record.identity_key)
.bind(record.bonded)
.bind(record.total_stake)
.bind(record.host)
.bind(record.http_port as i32)
.bind(record.full_details)
.bind(record.self_described)
.bind(record.last_updated_utc)
.bind(record.is_dp_delegatee)
.execute(&mut *tx)
.await?;
}
@@ -78,10 +78,10 @@ pub(crate) async fn get_all_mixnodes(pool: &DbPool) -> anyhow::Result<Vec<Mixnod
mn.full_details as "full_details!",
mn.self_described as "self_described",
mn.last_updated_utc as "last_updated_utc!",
COALESCE(md.moniker, "NA") as "moniker!",
COALESCE(md.website, "NA") as "website!",
COALESCE(md.security_contact, "NA") as "security_contact!",
COALESCE(md.details, "NA") as "details!"
md.moniker as "moniker!",
md.website as "website!",
md.security_contact as "security_contact!",
md.details as "details!"
FROM mixnodes mn
LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id
ORDER BY mn.mix_id"#
@@ -144,29 +144,29 @@ pub(crate) async fn get_daily_stats(pool: &DbPool) -> anyhow::Result<Vec<DailySt
pub(crate) async fn get_bonded_mix_ids(pool: &DbPool) -> anyhow::Result<HashSet<i64>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query!(
let items = crate::db::query(
r#"
SELECT mix_id
FROM mixnodes
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?
.into_iter()
.map(|record| record.mix_id)
.map(|record| record.try_get::<i64, _>("mix_id").unwrap())
.collect::<HashSet<_>>();
Ok(items)
}
pub(crate) async fn insert_mixnode_description(
conn: &mut PoolConnection<Sqlite>,
mix_id: &i64,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
mix_id: i64,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO mixnode_description (
mix_id, moniker, website, security_contact, details, last_updated_utc
@@ -178,13 +178,13 @@ pub(crate) async fn insert_mixnode_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
mix_id,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(mix_id)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
@@ -9,7 +9,8 @@ mod summary;
pub(crate) mod testruns;
pub(crate) use gateways::{
get_all_gateways, get_bonded_gateway_id_keys, select_gateway_identity, update_bonded_gateways,
get_all_gateways, get_bonded_gateway_id_keys, get_or_create_gateway, select_gateway_identity,
update_bonded_gateways,
};
pub(crate) use gateways_stats::{delete_old_records, get_sessions_stats, insert_session_records};
pub(crate) use misc::insert_summaries;
@@ -4,14 +4,14 @@ use nym_validator_client::{
client::{NodeId, NymNodeDetails},
models::NymNodeDescription,
};
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use std::collections::HashMap;
use tracing::instrument;
use crate::{
db::{
models::{NymNodeDto, NymNodeInsertRecord},
DbPool,
DbConnection, DbPool,
},
node_scraper::helpers::NodeDescriptionResponse,
};
@@ -19,21 +19,20 @@ use crate::{
pub(crate) async fn get_all_nym_nodes(pool: &DbPool) -> anyhow::Result<Vec<NymNodeDto>> {
let mut conn = pool.acquire().await?;
sqlx::query_as!(
NymNodeDto,
crate::db::query_as::<NymNodeDto>(
r#"SELECT
node_id,
ed25519_identity_pubkey,
total_stake,
ip_addresses as "ip_addresses!: serde_json::Value",
ip_addresses,
mix_port,
x25519_sphinx_pubkey,
node_role as "node_role: serde_json::Value",
supported_roles as "supported_roles: serde_json::Value",
entry as "entry: serde_json::Value",
node_role,
supported_roles,
entry,
performance,
self_described as "self_described: serde_json::Value",
bond_info as "bond_info: serde_json::Value"
self_described,
bond_info
FROM
nym_nodes
ORDER BY
@@ -56,21 +55,20 @@ pub(crate) async fn get_described_bonded_nym_nodes(
) -> anyhow::Result<Vec<NymNodeDto>> {
let mut conn = pool.acquire().await?;
sqlx::query_as!(
NymNodeDto,
crate::db::query_as::<NymNodeDto>(
r#"SELECT
node_id,
ed25519_identity_pubkey,
total_stake,
ip_addresses as "ip_addresses!: serde_json::Value",
ip_addresses,
mix_port,
x25519_sphinx_pubkey,
node_role as "node_role: serde_json::Value",
supported_roles as "supported_roles: serde_json::Value",
entry as "entry: serde_json::Value",
node_role,
supported_roles,
entry,
performance,
self_described as "self_described: serde_json::Value",
bond_info as "bond_info: serde_json::Value"
self_described,
bond_info
FROM
nym_nodes
WHERE
@@ -92,7 +90,7 @@ pub(crate) async fn update_nym_nodes(
) -> anyhow::Result<usize> {
let mut tx = pool.begin().await?;
sqlx::query!(
crate::db::query(
"UPDATE nym_nodes
SET
self_described = NULL,
@@ -104,7 +102,7 @@ pub(crate) async fn update_nym_nodes(
let inserted = node_records.len();
for record in node_records {
// https://www.sqlite.org/lang_upsert.html
sqlx::query!(
crate::db::query(
"INSERT INTO nym_nodes
(node_id, ed25519_identity_pubkey,
total_stake,
@@ -129,20 +127,20 @@ pub(crate) async fn update_nym_nodes(
performance=excluded.performance,
last_updated_utc=excluded.last_updated_utc
;",
record.node_id,
record.ed25519_identity_pubkey,
record.total_stake,
record.ip_addresses,
record.mix_port,
record.x25519_sphinx_pubkey,
record.node_role,
record.supported_roles,
record.entry,
record.self_described,
record.bond_info,
record.performance,
record.last_updated_utc,
)
.bind(record.node_id)
.bind(record.ed25519_identity_pubkey)
.bind(record.total_stake)
.bind(record.ip_addresses)
.bind(record.mix_port)
.bind(record.x25519_sphinx_pubkey)
.bind(record.node_role)
.bind(record.supported_roles)
.bind(record.entry)
.bind(record.self_described)
.bind(record.bond_info)
.bind(record.performance)
.bind(record.last_updated_utc)
.execute(&mut *tx)
.await
.map_err(|e| anyhow::anyhow!("Failed to INSERT node_id={}: {}", record.node_id, e))?;
@@ -150,6 +148,10 @@ pub(crate) async fn update_nym_nodes(
tx.commit().await?;
tracing::debug!(
"Successfully inserted/updated {} nym_nodes records",
inserted
);
Ok(inserted)
}
@@ -158,10 +160,10 @@ pub(crate) async fn get_described_node_bond_info(
) -> anyhow::Result<HashMap<NodeId, NymNodeDetails>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
node_id,
bond_info as "bond_info: serde_json::Value"
bond_info
FROM
nym_nodes
WHERE
@@ -176,11 +178,11 @@ pub(crate) async fn get_described_node_bond_info(
records
.into_iter()
.filter_map(|record| {
record
.bond_info
// only return details for nodes which have details stored
.and_then(|bond_info| serde_json::from_value::<NymNodeDetails>(bond_info).ok())
.map(|res| (record.node_id as NodeId, res))
let node_id: i32 = record.try_get("node_id").ok()?;
let bond_info: serde_json::Value = record.try_get("bond_info").ok()?;
serde_json::from_value::<NymNodeDetails>(bond_info)
.ok()
.map(|res| (node_id as i64 as NodeId, res))
})
.collect::<HashMap<_, _>>()
})
@@ -192,10 +194,10 @@ pub(crate) async fn get_node_self_description(
) -> anyhow::Result<HashMap<NodeId, NymNodeDescription>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
node_id,
self_described as "self_described: serde_json::Value"
self_described
FROM
nym_nodes
WHERE
@@ -210,13 +212,11 @@ pub(crate) async fn get_node_self_description(
records
.into_iter()
.filter_map(|record| {
record
.self_described
// only return details for nodes which have details stored
.and_then(|description| {
serde_json::from_value::<NymNodeDescription>(description).ok()
})
.map(|res| (record.node_id as NodeId, res))
let node_id: i32 = record.try_get("node_id").ok()?;
let self_described: serde_json::Value = record.try_get("self_described").ok()?;
serde_json::from_value::<NymNodeDescription>(self_described)
.ok()
.map(|res| (node_id as i64 as NodeId, res))
})
.collect::<HashMap<_, _>>()
})
@@ -228,7 +228,7 @@ pub(crate) async fn get_bonded_node_description(
) -> anyhow::Result<HashMap<NodeId, NodeDescription>> {
let mut conn = pool.acquire().await?;
sqlx::query!(
crate::db::query(
r#"SELECT
nd.node_id,
moniker,
@@ -238,7 +238,7 @@ pub(crate) async fn get_bonded_node_description(
FROM
nym_node_descriptions nd
INNER JOIN
nym_nodes
nym_nodes nn on nd.node_id = nn.node_id
WHERE
bond_info IS NOT NULL
"#,
@@ -249,14 +249,15 @@ pub(crate) async fn get_bonded_node_description(
records
.into_iter()
.map(|elem| {
let node_id: NodeId = elem.node_id.try_into().unwrap_or_default();
let node_id: i64 = elem.try_get("node_id").unwrap_or(0);
let node_id: NodeId = node_id.try_into().unwrap_or_default();
(
node_id,
NodeDescription {
moniker: elem.moniker.unwrap_or_default(),
website: elem.website.unwrap_or_default(),
security_contact: elem.security_contact.unwrap_or_default(),
details: elem.details.unwrap_or_default(),
moniker: elem.try_get("moniker").unwrap_or_default(),
website: elem.try_get("website").unwrap_or_default(),
security_contact: elem.try_get("security_contact").unwrap_or_default(),
details: elem.try_get("details").unwrap_or_default(),
},
)
})
@@ -266,12 +267,12 @@ pub(crate) async fn get_bonded_node_description(
}
pub(crate) async fn insert_nym_node_description(
conn: &mut PoolConnection<Sqlite>,
node_id: &i64,
description: &NodeDescriptionResponse,
conn: &mut DbConnection,
node_id: i64,
description: NodeDescriptionResponse,
timestamp: i64,
) -> anyhow::Result<()> {
sqlx::query!(
crate::db::query(
r#"
INSERT INTO nym_node_descriptions (
node_id, moniker, website, security_contact, details, last_updated_utc
@@ -283,13 +284,13 @@ pub(crate) async fn insert_nym_node_description(
details = excluded.details,
last_updated_utc = excluded.last_updated_utc
"#,
node_id,
description.moniker,
description.website,
description.security_contact,
description.details,
timestamp,
)
.bind(node_id)
.bind(description.moniker)
.bind(description.website)
.bind(description.security_contact)
.bind(description.details)
.bind(timestamp)
.execute(conn.as_mut())
.await
.map(drop)
@@ -59,7 +59,8 @@ pub(crate) async fn batch_store_packet_stats(
.map_err(|err| anyhow::anyhow!("Failed to commit: {err}"))
}
async fn insert_node_packet_stats_uncommitted(
#[cfg(feature = "sqlite")]
pub(crate) async fn insert_node_packet_stats_uncommitted(
tx: &mut Transaction<'static, sqlx::Sqlite>,
node_kind: &ScrapeNodeKind,
stats: &NodeStats,
@@ -67,35 +68,35 @@ async fn insert_node_packet_stats_uncommitted(
) -> Result<()> {
match node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query!(
sqlx::query(
r#"
INSERT INTO mixnode_packet_stats_raw (
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES (?, ?, ?, ?, ?)
"#,
mix_id,
timestamp_utc,
stats.packets_received,
stats.packets_sent,
stats.packets_dropped,
)
.bind(mix_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(tx.as_mut())
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query!(
sqlx::query(
r#"
INSERT INTO nym_nodes_packet_stats_raw (
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES (?, ?, ?, ?, ?)
"#,
node_id,
timestamp_utc,
stats.packets_received,
stats.packets_sent,
stats.packets_dropped,
)
.bind(node_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(tx.as_mut())
.await?;
}
@@ -104,6 +105,53 @@ async fn insert_node_packet_stats_uncommitted(
Ok(())
}
#[cfg(feature = "pg")]
pub(crate) async fn insert_node_packet_stats_uncommitted(
tx: &mut Transaction<'static, sqlx::Postgres>,
node_kind: &ScrapeNodeKind,
stats: &NodeStats,
timestamp_utc: i64,
) -> Result<()> {
match node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query(
r#"
INSERT INTO mixnode_packet_stats_raw (
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES ($1, $2, $3, $4, $5)
"#,
)
.bind(mix_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(tx.as_mut())
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query(
r#"
INSERT INTO nym_nodes_packet_stats_raw (
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
) VALUES ($1, $2, $3, $4, $5)
"#,
)
.bind(node_id)
.bind(timestamp_utc)
.bind(stats.packets_received)
.bind(stats.packets_sent)
.bind(stats.packets_dropped)
.execute(tx.as_mut())
.await?;
}
}
Ok(())
}
#[cfg(feature = "sqlite")]
pub(crate) async fn get_raw_node_stats(
tx: &mut Transaction<'static, sqlx::Sqlite>,
node_kind: &ScrapeNodeKind,
@@ -112,39 +160,37 @@ pub(crate) async fn get_raw_node_stats(
// if no packets are found, it's fine to assume 0 because that's also
// SQL default value if none provided
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query_as!(
NodeStats,
sqlx::query_as::<_, NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as "packets_received!: _",
COALESCE(packets_sent, 0) as "packets_sent!: _",
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM mixnode_packet_stats_raw
WHERE mix_id = ?
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
mix_id
)
.bind(mix_id)
.fetch_optional(tx.as_mut())
.await?
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query_as!(
NodeStats,
sqlx::query_as::<_, NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as "packets_received!: _",
COALESCE(packets_sent, 0) as "packets_sent!: _",
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM nym_nodes_packet_stats_raw
WHERE node_id = ?
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
node_id
)
.bind(node_id)
.fetch_optional(tx.as_mut())
.await?
}
@@ -153,6 +199,55 @@ pub(crate) async fn get_raw_node_stats(
Ok(packets)
}
#[cfg(feature = "pg")]
pub(crate) async fn get_raw_node_stats(
tx: &mut Transaction<'static, sqlx::Postgres>,
node_kind: &ScrapeNodeKind,
) -> Result<Option<NodeStats>> {
let packets = match node_kind {
// if no packets are found, it's fine to assume 0 because that's also
// SQL default value if none provided
ScrapeNodeKind::LegacyMixnode { mix_id } => {
sqlx::query_as::<_, NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM mixnode_packet_stats_raw
WHERE mix_id = $1
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
)
.bind(mix_id)
.fetch_optional(tx.as_mut())
.await?
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
sqlx::query_as::<_, NodeStats>(
r#"
SELECT
COALESCE(packets_received, 0) as packets_received,
COALESCE(packets_sent, 0) as packets_sent,
COALESCE(packets_dropped, 0) as packets_dropped
FROM nym_nodes_packet_stats_raw
WHERE node_id = $1
ORDER BY timestamp_utc DESC
LIMIT 1 OFFSET 1
"#,
)
.bind(node_id)
.fetch_optional(tx.as_mut())
.await?
}
};
Ok(packets)
}
#[cfg(feature = "sqlite")]
pub(crate) async fn insert_daily_node_stats_uncommitted(
tx: &mut Transaction<'static, sqlx::Sqlite>,
node_kind: &ScrapeNodeKind,
@@ -161,19 +256,19 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
) -> Result<()> {
match node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
let total_stake = sqlx::query_scalar!(
let total_stake = sqlx::query_scalar::<_, i64>(
r#"
SELECT
total_stake
FROM mixnodes
WHERE mix_id = ?
"#,
mix_id
)
.bind(mix_id)
.fetch_one(tx.as_mut())
.await?;
sqlx::query!(
sqlx::query(
r#"
INSERT INTO mixnode_daily_stats (
mix_id, date_utc,
@@ -186,31 +281,31 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
"#,
mix_id,
date_utc,
total_stake,
packets.packets_received,
packets.packets_sent,
packets.packets_dropped,
)
.bind(mix_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(tx.as_mut())
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
let total_stake = sqlx::query_scalar!(
let total_stake = sqlx::query_scalar::<_, i64>(
r#"
SELECT
total_stake
FROM nym_nodes
WHERE node_id = ?
"#,
node_id
)
.bind(node_id)
.fetch_one(tx.as_mut())
.await?;
sqlx::query!(
sqlx::query(
r#"
INSERT INTO nym_node_daily_mixing_stats (
node_id, date_utc,
@@ -223,13 +318,99 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
"#,
node_id,
date_utc,
total_stake,
packets.packets_received,
packets.packets_sent,
packets.packets_dropped,
)
.bind(node_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(tx.as_mut())
.await?;
}
}
Ok(())
}
#[cfg(feature = "pg")]
pub(crate) async fn insert_daily_node_stats_uncommitted(
tx: &mut Transaction<'static, sqlx::Postgres>,
node_kind: &ScrapeNodeKind,
date_utc: &str,
packets: NodeStats,
) -> Result<()> {
match node_kind {
ScrapeNodeKind::LegacyMixnode { mix_id } => {
let total_stake = sqlx::query_scalar::<_, i64>(
r#"
SELECT
total_stake
FROM mixnodes
WHERE mix_id = $1
"#,
)
.bind(mix_id)
.fetch_one(tx.as_mut())
.await?;
sqlx::query(
r#"
INSERT INTO mixnode_daily_stats (
mix_id, date_utc,
total_stake, packets_received,
packets_sent, packets_dropped
) VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT(mix_id, date_utc) DO UPDATE SET
total_stake = excluded.total_stake,
packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
"#,
)
.bind(mix_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(tx.as_mut())
.await?;
}
ScrapeNodeKind::MixingNymNode { node_id }
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
let total_stake = sqlx::query_scalar::<_, i64>(
r#"
SELECT
total_stake
FROM nym_nodes
WHERE node_id = $1
"#,
)
.bind(node_id)
.fetch_one(tx.as_mut())
.await?;
sqlx::query(
r#"
INSERT INTO nym_node_daily_mixing_stats (
node_id, date_utc,
total_stake, packets_received,
packets_sent, packets_dropped
) VALUES ($1, $2, $3, $4, $5, $6)
ON CONFLICT(node_id, date_utc) DO UPDATE SET
total_stake = excluded.total_stake,
packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
"#,
)
.bind(node_id)
.bind(date_utc)
.bind(total_stake)
.bind(packets.packets_received)
.bind(packets.packets_sent)
.bind(packets.packets_dropped)
.execute(tx.as_mut())
.await?;
}
@@ -12,6 +12,7 @@ use crate::{
};
use anyhow::Result;
use nym_validator_client::nym_api::SkimmedNode;
use sqlx::Row;
pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperNodeInfo>> {
let mut nodes_to_scrape = Vec::new();
@@ -68,12 +69,12 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
tracing::debug!("Fetched {} 🚪 entry/exit nodes", entry_exit_nodes);
let mut conn = pool.acquire().await?;
let mixnodes = sqlx::query!(
let mixnodes = crate::db::query(
r#"
SELECT mix_id as node_id, host, http_api_port
FROM mixnodes
WHERE bonded = true
"#
"#,
)
.fetch_all(&mut *conn)
.await?;
@@ -85,18 +86,20 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
let mut legacy_not_in_nym_node_list = 0;
let total_legacy_mixnodes = mixnodes.len();
for mixnode in mixnodes {
let node_id: i64 = mixnode.try_get("node_id")?;
let host: String = mixnode.try_get("host")?;
let http_api_port: i64 = mixnode.try_get("http_api_port")?;
if nodes_to_scrape
.iter()
.all(|node| node.node_id() != &mixnode.node_id)
.all(|node| node.node_id() != &node_id)
{
// in case polyfilling on Nym API gets removed, this part ensures
// mixnodes are added to the final list of nodes to scrape
nodes_to_scrape.push(ScraperNodeInfo {
node_kind: ScrapeNodeKind::LegacyMixnode {
mix_id: mixnode.node_id,
},
hosts: vec![mixnode.host],
http_api_port: mixnode.http_api_port,
node_kind: ScrapeNodeKind::LegacyMixnode { mix_id: node_id },
hosts: vec![host],
http_api_port,
});
legacy_not_in_nym_node_list += 1;
@@ -121,8 +124,8 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
pub(crate) async fn insert_scraped_node_description(
pool: &DbPool,
node_kind: &ScrapeNodeKind,
description: &NodeDescriptionResponse,
node_kind: ScrapeNodeKind,
description: NodeDescriptionResponse,
) -> Result<()> {
let timestamp = now_utc().unix_timestamp();
let mut conn = pool.acquire().await?;
@@ -138,7 +141,7 @@ pub(crate) async fn insert_scraped_node_description(
node_id,
identity_key,
} => {
insert_nym_node_description(&mut conn, node_id, description, timestamp).await?;
insert_nym_node_description(&mut conn, node_id, description.clone(), timestamp).await?;
// for historic reasons (/gateways API), store this info into gateways table as well
insert_gateway_description(&mut conn, identity_key, description, timestamp).await?;
}
@@ -23,13 +23,12 @@ use crate::{
pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<SummaryHistory>> {
let mut conn = pool.acquire().await?;
let items = sqlx::query_as!(
SummaryHistoryDto,
let items = crate::db::query_as::<SummaryHistoryDto>(
r#"SELECT
id as "id!",
date as "date!",
timestamp_utc as "timestamp_utc!",
value_json as "value_json!"
id,
date,
timestamp_utc,
value_json
FROM summary_history
ORDER BY date DESC
LIMIT 30"#,
@@ -51,13 +50,12 @@ pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<Sum
async fn get_summary_dto(pool: &DbPool) -> anyhow::Result<Vec<SummaryDto>> {
let mut conn = pool.acquire().await?;
Ok(sqlx::query_as!(
SummaryDto,
Ok(crate::db::query_as::<SummaryDto>(
r#"SELECT
key as "key!",
value_json as "value_json!",
last_updated_utc as "last_updated_utc!"
FROM summary"#
key,
value_json,
last_updated_utc
FROM summary"#,
)
.fetch(&mut *conn)
.try_collect::<Vec<_>>()
@@ -1,40 +1,38 @@
use crate::db::models::{TestRunDto, TestRunStatus};
use crate::db::DbConnection;
use crate::db::DbPool;
use crate::http::models::TestrunAssignment;
use crate::utils::now_utc;
use sqlx::{pool::PoolConnection, Sqlite};
use sqlx::Row;
use time::Duration;
pub(crate) async fn count_testruns_in_progress(
conn: &mut PoolConnection<Sqlite>,
) -> anyhow::Result<i64> {
sqlx::query_scalar!(
r#"SELECT
COUNT(id) as "count: i64"
FROM testruns
WHERE
status = ?
"#,
TestRunStatus::InProgress as i64,
)
.fetch_one(conn.as_mut())
.await
.map_err(anyhow::Error::from)
pub(crate) async fn count_testruns_in_progress(conn: &mut DbConnection) -> anyhow::Result<i64> {
#[cfg(feature = "sqlite")]
let sql = "SELECT COUNT(id) FROM testruns WHERE status = ?";
#[cfg(feature = "pg")]
let sql = "SELECT COUNT(id) FROM testruns WHERE status = $1";
let count: i64 = sqlx::query_scalar(sql)
.bind(TestRunStatus::InProgress as i32)
.fetch_one(conn.as_mut())
.await?;
Ok(count)
}
pub(crate) async fn get_in_progress_testrun_by_id(
conn: &mut PoolConnection<Sqlite>,
testrun_id: i64,
conn: &mut DbConnection,
testrun_id: i32,
) -> anyhow::Result<TestRunDto> {
sqlx::query_as!(
TestRunDto,
crate::db::query_as::<TestRunDto>(
r#"SELECT
id as "id!",
gateway_id as "gateway_id!",
status as "status!",
created_utc as "created_utc!",
ip_address as "ip_address!",
log as "log!",
id,
gateway_id,
status,
created_utc,
ip_address,
log,
last_assigned_utc
FROM testruns
WHERE
@@ -43,12 +41,12 @@ pub(crate) async fn get_in_progress_testrun_by_id(
status = ?
ORDER BY created_utc
LIMIT 1"#,
testrun_id,
TestRunStatus::InProgress as i64,
)
.bind(testrun_id)
.bind(TestRunStatus::InProgress as i32)
.fetch_one(conn.as_mut())
.await
.map_err(|e| anyhow::anyhow!("Couldn't retrieve testrun {testrun_id}: {e}"))
.map_err(|e| anyhow::anyhow!("Failed to retrieve in-progress testrun {testrun_id}: {e}"))
}
pub(crate) async fn update_testruns_assigned_before(
@@ -59,7 +57,7 @@ pub(crate) async fn update_testruns_assigned_before(
let previous_run = now_utc() - max_age;
let cutoff_timestamp = previous_run.unix_timestamp();
let res = sqlx::query!(
let res = crate::db::query(
r#"UPDATE
testruns
SET
@@ -69,10 +67,10 @@ pub(crate) async fn update_testruns_assigned_before(
AND
last_assigned_utc < ?
"#,
TestRunStatus::Queued as i64,
TestRunStatus::InProgress as i64,
cutoff_timestamp
)
.bind(TestRunStatus::Queued as i32)
.bind(TestRunStatus::InProgress as i32)
.bind(cutoff_timestamp)
.execute(conn.as_mut())
.await?;
@@ -89,36 +87,36 @@ pub(crate) async fn update_testruns_assigned_before(
}
pub(crate) async fn assign_oldest_testrun(
conn: &mut PoolConnection<Sqlite>,
conn: &mut DbConnection,
) -> anyhow::Result<Option<TestrunAssignment>> {
let now = now_utc().unix_timestamp();
// find & mark as "In progress" in the same transaction to avoid race conditions
let returning = sqlx::query!(
let returning = crate::db::query(
r#"UPDATE testruns
SET
status = ?,
last_assigned_utc = ?
WHERE rowid =
WHERE id =
(
SELECT rowid
SELECT id
FROM testruns
WHERE status = ?
ORDER BY created_utc asc
LIMIT 1
)
RETURNING
id as "id!",
id,
gateway_id
"#,
TestRunStatus::InProgress as i64,
now,
TestRunStatus::Queued as i64,
)
.bind(TestRunStatus::InProgress as i32)
.bind(now)
.bind(TestRunStatus::Queued as i32)
.fetch_optional(conn.as_mut())
.await?;
if let Some(testrun) = returning {
let gw_identity = sqlx::query!(
let gw_identity = crate::db::query(
r#"
SELECT
id,
@@ -126,14 +124,14 @@ pub(crate) async fn assign_oldest_testrun(
FROM gateways
WHERE id = ?
LIMIT 1"#,
testrun.gateway_id
)
.bind(testrun.try_get::<i32, _>("gateway_id")?)
.fetch_one(conn.as_mut())
.await?;
Ok(Some(TestrunAssignment {
testrun_id: testrun.id,
gateway_identity_key: gw_identity.gateway_identity_key,
testrun_id: testrun.try_get("id")?,
gateway_identity_key: gw_identity.try_get("gateway_identity_key")?,
assigned_at_utc: now,
}))
} else {
@@ -142,67 +140,146 @@ pub(crate) async fn assign_oldest_testrun(
}
pub(crate) async fn update_testrun_status(
conn: &mut PoolConnection<Sqlite>,
testrun_id: i64,
conn: &mut DbConnection,
testrun_id: i32,
status: TestRunStatus,
) -> anyhow::Result<()> {
let status = status as u32;
sqlx::query!(
"UPDATE testruns SET status = ? WHERE id = ?",
status,
testrun_id
)
.execute(conn.as_mut())
.await?;
let status = status as i32;
crate::db::query("UPDATE testruns SET status = ? WHERE id = ?")
.bind(status)
.bind(testrun_id)
.execute(conn.as_mut())
.await?;
Ok(())
}
pub(crate) async fn update_gateway_last_probe_log(
conn: &mut PoolConnection<Sqlite>,
gateway_pk: i64,
log: &str,
conn: &mut DbConnection,
gateway_pk: i32,
log: String,
) -> anyhow::Result<()> {
sqlx::query!(
"UPDATE gateways SET last_probe_log = ? WHERE id = ?",
log,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_probe_log = ? WHERE id = ?")
.bind(log)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(|e| {
anyhow::anyhow!(
"Failed to update probe log for gateway {}: {}",
gateway_pk,
e
)
})
}
pub(crate) async fn update_gateway_last_probe_result(
conn: &mut PoolConnection<Sqlite>,
gateway_pk: i64,
result: &str,
conn: &mut DbConnection,
gateway_pk: i32,
result: String,
) -> anyhow::Result<()> {
sqlx::query!(
"UPDATE gateways SET last_probe_result = ? WHERE id = ?",
result,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_probe_result = ? WHERE id = ?")
.bind(result)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(|e| {
anyhow::anyhow!(
"Failed to update probe result for gateway {}: {}",
gateway_pk,
e
)
})
}
pub(crate) async fn update_gateway_score(
conn: &mut PoolConnection<Sqlite>,
gateway_pk: i64,
conn: &mut DbConnection,
gateway_pk: i32,
) -> anyhow::Result<()> {
let now = now_utc().unix_timestamp();
sqlx::query!(
"UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
now,
now,
gateway_pk
)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
crate::db::query("UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?")
.bind(now)
.bind(now)
.bind(gateway_pk)
.execute(conn.as_mut())
.await
.map(drop)
.map_err(From::from)
}
pub(crate) async fn get_testrun_by_id(
conn: &mut DbConnection,
testrun_id: i32,
) -> anyhow::Result<TestRunDto> {
crate::db::query_as::<TestRunDto>(
r#"SELECT
id,
gateway_id,
status,
created_utc,
ip_address,
log,
last_assigned_utc
FROM testruns
WHERE id = ?"#,
)
.bind(testrun_id)
.fetch_one(conn.as_mut())
.await
.map_err(|e| anyhow::anyhow!("Testrun {} not found: {}", testrun_id, e))
}
pub(crate) async fn insert_external_testrun(
conn: &mut DbConnection,
testrun_id: i32,
gateway_id: i32,
assigned_at_utc: i64,
) -> anyhow::Result<()> {
let now = crate::utils::now_utc().unix_timestamp();
crate::db::query(
r#"INSERT INTO testruns (
id,
gateway_id,
status,
created_utc,
last_assigned_utc,
ip_address,
log
) VALUES (?, ?, ?, ?, ?, ?, ?)"#,
)
.bind(testrun_id)
.bind(gateway_id)
.bind(TestRunStatus::InProgress as i32)
.bind(now)
.bind(assigned_at_utc)
.bind("external") // Marker for external origin
.bind("") // Empty initial log
.execute(conn.as_mut())
.await?;
tracing::debug!(
"Created external testrun {} for gateway {}",
testrun_id,
gateway_id
);
Ok(())
}
pub(crate) async fn update_testrun_status_by_gateway(
conn: &mut DbConnection,
gateway_id: i32,
status: TestRunStatus,
) -> anyhow::Result<()> {
let status = status as i32;
crate::db::query("UPDATE testruns SET status = ? WHERE gateway_id = ? AND status = ?")
.bind(status)
.bind(gateway_id)
.bind(TestRunStatus::InProgress as i32)
.execute(conn.as_mut())
.await?;
Ok(())
}
@@ -0,0 +1,251 @@
use sqlx::Database;
/// Converts SQLite-style ? placeholders to PostgreSQL $N format
#[cfg(feature = "pg")]
fn convert_placeholders(query: &str) -> String {
let mut result = String::with_capacity(query.len() + 10);
let mut placeholder_count = 0;
let mut chars = query.chars();
let mut in_string: Option<char> = None;
let mut escape_next = false;
#[allow(clippy::while_let_on_iterator)]
while let Some(ch) = chars.next() {
if escape_next {
result.push(ch);
escape_next = false;
continue;
}
if let Some(quote_char) = in_string {
result.push(ch);
if ch == quote_char {
in_string = None;
} else if ch == '\\' {
escape_next = true;
}
continue;
}
match ch {
'\\' => {
result.push(ch);
escape_next = true;
}
'\'' | '"' => {
result.push(ch);
in_string = Some(ch);
}
'?' => {
placeholder_count += 1;
result.push_str(&format!("${placeholder_count}"));
}
_ => {
result.push(ch);
}
}
}
result
}
/// Creates a query that automatically handles placeholder conversion
#[cfg(feature = "sqlite")]
pub fn query(
sql: &str,
) -> sqlx::query::Query<'_, sqlx::Sqlite, <sqlx::Sqlite as Database>::Arguments<'_>> {
sqlx::query(sql)
}
#[cfg(feature = "pg")]
pub fn query(
sql: &str,
) -> sqlx::query::Query<'static, sqlx::Postgres, <sqlx::Postgres as Database>::Arguments<'static>> {
let converted = convert_placeholders(sql);
sqlx::query(Box::leak(converted.into_boxed_str()))
}
/// Creates a query_as that automatically handles placeholder conversion
#[cfg(feature = "sqlite")]
pub fn query_as<O>(
sql: &str,
) -> sqlx::query::QueryAs<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
where
O: for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
{
sqlx::query_as(sql)
}
#[cfg(feature = "pg")]
pub fn query_as<O>(
sql: &str,
) -> sqlx::query::QueryAs<
'static,
sqlx::Postgres,
O,
<sqlx::Postgres as Database>::Arguments<'static>,
>
where
O: for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
{
let converted = convert_placeholders(sql);
sqlx::query_as(Box::leak(converted.into_boxed_str()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
#[cfg(feature = "pg")]
fn test_convert_placeholders() {
// Basic conversion
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE id = ?"),
r"SELECT * FROM table WHERE id = $1"
);
// Multiple placeholders
assert_eq!(
convert_placeholders(r"INSERT INTO table (a, b, c) VALUES (?, ?, ?)"),
r"INSERT INTO table (a, b, c) VALUES ($1, $2, $3)"
);
// Placeholder inside string literal should be ignored
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE name = 'test?' AND id = ?"),
r"SELECT * FROM table WHERE name = 'test?' AND id = $1"
);
// Update statement
assert_eq!(
convert_placeholders(r"UPDATE table SET a = ?, b = ? WHERE id = ?"),
r"UPDATE table SET a = $1, b = $2 WHERE id = $3"
);
// Test with 10 placeholders (like in update_mixnodes)
assert_eq!(
convert_placeholders(r"INSERT INTO t VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"),
r"INSERT INTO t VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)"
);
// No placeholders
assert_eq!(
convert_placeholders(r"SELECT * FROM table"),
r"SELECT * FROM table"
);
// Placeholder at the beginning
assert_eq!(convert_placeholders(r"? AND ?"), r"$1 AND $2");
// Placeholder at the end
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE id = ?"),
r"SELECT * FROM table WHERE id = $1"
);
// Adjacent placeholders
assert_eq!(
convert_placeholders(r"VALUES(?,? ,?)"),
r"VALUES($1,$2 ,$3)"
);
// Escaped single quote
assert_eq!(
convert_placeholders(r"SELECT * FROM foo WHERE bar = 'it\'s a test' AND baz = ?"),
r"SELECT * FROM foo WHERE bar = 'it\'s a test' AND baz = $1"
);
// Double quotes
assert_eq!(
convert_placeholders(r#"SELECT * FROM "table" WHERE "column" = ? AND name = "test?""#),
r#"SELECT * FROM "table" WHERE "column" = $1 AND name = "test?""#
);
// Mixed quotes
assert_eq!(
convert_placeholders(
r#"SELECT * FROM table WHERE a = 'single?' AND b = "double?" AND c = ?"#
),
r#"SELECT * FROM table WHERE a = 'single?' AND b = "double?" AND c = $1"#
);
// Escaped backslash before quote
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE path = 'C:\\?' AND id = ?"),
r"SELECT * FROM table WHERE path = 'C:\\?' AND id = $1"
);
// Multiple escaped quotes
assert_eq!(
convert_placeholders(
r#"INSERT INTO table (msg) VALUES ('it\'s "complex" test') WHERE id = ?"#
),
r#"INSERT INTO table (msg) VALUES ('it\'s "complex" test') WHERE id = $1"#
);
// Very long query with many placeholders
let long_query = r"INSERT INTO very_long_table_name (col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14, col15) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
let expected = r"INSERT INTO very_long_table_name (col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14, col15) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)";
assert_eq!(convert_placeholders(long_query), expected);
// Query with comments (question marks in comments are also converted)
assert_eq!(
convert_placeholders(
r"-- This is a comment with ?
SELECT * FROM table WHERE id = ? -- another comment ?"
),
r"-- This is a comment with $1
SELECT * FROM table WHERE id = $2 -- another comment $3"
);
// Multiline strings
assert_eq!(
convert_placeholders(
r"SELECT * FROM table
WHERE description = 'This is a
multiline string with ?'
AND id = ?"
),
r"SELECT * FROM table
WHERE description = 'This is a
multiline string with ?'
AND id = $1"
);
// Complex nested quotes
assert_eq!(
convert_placeholders(
r#"SELECT json_extract(data, '$.items[?(@.name=="test?")]') FROM table WHERE id = ?"#
),
r#"SELECT json_extract(data, '$.items[?(@.name=="test?")]') FROM table WHERE id = $1"#
);
// Empty string
assert_eq!(convert_placeholders(""), "");
// Only placeholders
assert_eq!(convert_placeholders("???"), "$1$2$3");
// Unicode in strings
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE name = '测试?' AND id = ?"),
r"SELECT * FROM table WHERE name = '测试?' AND id = $1"
);
// Test case with backslash at end of string
assert_eq!(
convert_placeholders(r"SELECT * FROM table WHERE path LIKE '%\\' AND id = ?"),
r"SELECT * FROM table WHERE path LIKE '%\\' AND id = $1"
);
// Mismatched quotes
assert_eq!(
convert_placeholders(r#"SELECT * FROM foo WHERE bar = "'" AND baz = ?"#),
r#"SELECT * FROM foo WHERE bar = "'" AND baz = $1"#
);
// Unmatched quote
assert_eq!(convert_placeholders(r"SELECT 'oops?"), r"SELECT 'oops?");
}
}
@@ -0,0 +1,461 @@
#[cfg(test)]
mod db_tests {
#[test]
fn test_gateway_dto_try_from() {
let gateway_dto = crate::db::models::GatewayDto {
gateway_identity_key: "test_identity".to_string(),
bonded: true,
performance: 100,
self_described: Some("{\"key\":\"value\"}".to_string()),
explorer_pretty_bond: Some("{\"key\":\"value\"}".to_string()),
last_probe_result: Some("{\"key\":\"value\"}".to_string()),
last_probe_log: Some("log".to_string()),
last_testrun_utc: Some(1672531200),
last_updated_utc: 1672531200,
moniker: "moniker".to_string(),
security_contact: "contact".to_string(),
details: "details".to_string(),
website: "website".to_string(),
};
let http_gateway: crate::http::models::Gateway = gateway_dto.try_into().unwrap();
assert_eq!(http_gateway.gateway_identity_key, "test_identity");
assert!(http_gateway.bonded);
assert_eq!(http_gateway.performance, 100);
assert!(http_gateway.self_described.is_some());
assert!(http_gateway.explorer_pretty_bond.is_some());
assert!(http_gateway.last_probe_result.is_some());
assert_eq!(http_gateway.last_probe_log, Some("log".to_string()));
assert!(http_gateway.last_testrun_utc.is_some());
assert!(!http_gateway.last_updated_utc.is_empty());
assert_eq!(http_gateway.description.moniker, "moniker");
assert_eq!(http_gateway.description.website, "website");
assert_eq!(http_gateway.description.security_contact, "contact");
assert_eq!(http_gateway.description.details, "details");
}
#[test]
fn test_mixnode_dto_try_from() {
let mixnode_dto = crate::db::models::MixnodeDto {
mix_id: 1,
bonded: true,
is_dp_delegatee: false,
total_stake: 1000000,
full_details: "{\"key\":\"value\"}".to_string(),
self_described: Some("{\"key\":\"value\"}".to_string()),
last_updated_utc: 1672531200,
moniker: "moniker".to_string(),
website: "website".to_string(),
security_contact: "contact".to_string(),
details: "details".to_string(),
};
let http_mixnode: crate::http::models::Mixnode = mixnode_dto.try_into().unwrap();
assert_eq!(http_mixnode.mix_id, 1);
assert!(http_mixnode.bonded);
assert!(!http_mixnode.is_dp_delegatee);
assert_eq!(http_mixnode.total_stake, 1000000);
assert!(http_mixnode.full_details.is_some());
assert!(http_mixnode.self_described.is_some());
assert!(!http_mixnode.last_updated_utc.is_empty());
assert_eq!(http_mixnode.description.moniker, "moniker");
assert_eq!(http_mixnode.description.website, "website");
assert_eq!(http_mixnode.description.security_contact, "contact");
assert_eq!(http_mixnode.description.details, "details");
}
#[test]
fn test_summary_history_dto_try_from() {
let summary_history_dto = crate::db::models::SummaryHistoryDto {
id: 1,
date: "2023-01-01".to_string(),
value_json: "{\"key\":\"value\"}".to_string(),
timestamp_utc: 1672531200,
};
let summary_history: crate::http::models::SummaryHistory =
summary_history_dto.try_into().unwrap();
assert_eq!(summary_history.date, "2023-01-01");
assert!(summary_history.value_json.is_object());
assert!(!summary_history.timestamp_utc.is_empty());
}
#[test]
fn test_gateway_sessions_record_try_from() {
let gateway_sessions_record = crate::db::models::GatewaySessionsRecord {
gateway_identity_key: "test_identity".to_string(),
node_id: 1,
day: time::macros::date!(2023 - 01 - 01),
unique_active_clients: 10,
session_started: 100,
users_hashes: Some("{\"key\":\"value\"}".to_string()),
vpn_sessions: Some("{\"key\":\"value\"}".to_string()),
mixnet_sessions: Some("{\"key\":\"value\"}".to_string()),
unknown_sessions: Some("{\"key\":\"value\"}".to_string()),
};
let session_stats: crate::http::models::SessionStats =
gateway_sessions_record.try_into().unwrap();
assert_eq!(session_stats.gateway_identity_key, "test_identity");
assert_eq!(session_stats.node_id, 1);
assert_eq!(session_stats.day, time::macros::date!(2023 - 01 - 01));
assert_eq!(session_stats.unique_active_clients, 10);
assert_eq!(session_stats.session_started, 100);
assert!(session_stats.users_hashes.is_some());
assert!(session_stats.vpn_sessions.is_some());
assert!(session_stats.mixnet_sessions.is_some());
assert!(session_stats.unknown_sessions.is_some());
}
#[test]
fn test_nym_node_dto_try_from() {
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
let nym_node_dto = crate::db::models::NymNodeDto {
node_id: 1,
ed25519_identity_pubkey: ed25519_pk.to_base58_string(),
total_stake: 1000000,
ip_addresses: serde_json::json!(["1.1.1.1"]),
mix_port: 1789,
x25519_sphinx_pubkey: x25519_pk.to_base58_string(),
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
layer: 1
}),
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
entry: false,
mixnode: true,
exit_nr: false,
exit_ipr: false,
}),
entry: None,
performance: "1.0".to_string(),
self_described: None,
bond_info: None,
};
let skimmed_node: nym_validator_client::nym_api::SkimmedNode =
nym_node_dto.try_into().unwrap();
assert_eq!(skimmed_node.node_id, 1);
assert_eq!(skimmed_node.ed25519_identity_pubkey, ed25519_pk);
assert_eq!(
skimmed_node.ip_addresses,
vec!["1.1.1.1".parse::<std::net::IpAddr>().unwrap()]
);
assert_eq!(skimmed_node.mix_port, 1789);
assert_eq!(skimmed_node.x25519_sphinx_pubkey, x25519_pk);
match skimmed_node.role {
nym_validator_client::nym_nodes::NodeRole::Mixnode { layer } => assert_eq!(layer, 1),
_ => panic!("Unexpected node role"),
}
assert!(!skimmed_node.supported_roles.entry);
assert!(skimmed_node.supported_roles.mixnode);
assert!(!skimmed_node.supported_roles.exit_nr);
assert!(!skimmed_node.supported_roles.exit_ipr);
assert!(skimmed_node.entry.is_none());
assert_eq!(
skimmed_node.performance,
nym_contracts_common::Percent::from_percentage_value(100).unwrap()
);
}
}
#[test]
fn test_nym_node_insert_record_new() {
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
let skimmed_node = nym_validator_client::nym_api::SkimmedNode {
node_id: 1,
ed25519_identity_pubkey: ed25519_pk,
ip_addresses: vec!["1.1.1.1".parse().unwrap()],
mix_port: 1789,
x25519_sphinx_pubkey: x25519_pk,
role: nym_validator_client::nym_nodes::NodeRole::Mixnode { layer: 1 },
supported_roles: nym_validator_client::models::DeclaredRoles {
entry: false,
mixnode: true,
exit_nr: false,
exit_ipr: false,
},
entry: None,
performance: nym_contracts_common::Percent::from_percentage_value(100).unwrap(),
};
let record = crate::db::models::NymNodeInsertRecord::new(skimmed_node, None, None).unwrap();
assert_eq!(record.node_id, 1);
assert_eq!(
record.ed25519_identity_pubkey,
ed25519_pk.to_base58_string()
);
assert_eq!(record.total_stake, 0);
assert_eq!(record.ip_addresses, serde_json::json!(["1.1.1.1"]));
assert_eq!(record.mix_port, 1789);
assert_eq!(record.x25519_sphinx_pubkey, x25519_pk.to_base58_string());
assert_eq!(
record.node_role,
serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode { layer: 1 })
);
assert_eq!(
record.supported_roles,
serde_json::json!(nym_validator_client::models::DeclaredRoles {
entry: false,
mixnode: true,
exit_nr: false,
exit_ipr: false,
})
);
assert_eq!(record.performance, "1");
assert!(record.entry.is_none());
assert!(record.self_described.is_none());
assert!(record.bond_info.is_none());
}
#[test]
fn test_nym_node_insert_record_with_entry() {
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
let skimmed_node = nym_validator_client::nym_api::SkimmedNode {
node_id: 1,
ed25519_identity_pubkey: ed25519_pk,
ip_addresses: vec!["1.1.1.1".parse().unwrap()],
mix_port: 1789,
x25519_sphinx_pubkey: x25519_pk,
role: nym_validator_client::nym_nodes::NodeRole::EntryGateway,
supported_roles: nym_validator_client::models::DeclaredRoles {
entry: true,
mixnode: false,
exit_nr: true,
exit_ipr: false,
},
entry: Some(nym_validator_client::nym_nodes::BasicEntryInformation {
hostname: Some("gateway.example.com".to_string()),
ws_port: 9001,
wss_port: Some(9002),
}),
performance: nym_contracts_common::Percent::from_percentage_value(99).unwrap(),
};
let record = crate::db::models::NymNodeInsertRecord::new(skimmed_node, None, None).unwrap();
assert_eq!(record.node_id, 1);
assert_eq!(record.total_stake, 0); // No bond info provided
assert!(record.entry.is_some());
assert!(record.self_described.is_none());
assert!(record.bond_info.is_none());
assert!(record.last_updated_utc > 0);
}
#[test]
fn test_gateway_dto_with_null_values() {
let gateway_dto = crate::db::models::GatewayDto {
gateway_identity_key: "test_identity".to_string(),
bonded: false,
performance: 0,
self_described: None,
explorer_pretty_bond: None,
last_probe_result: None,
last_probe_log: None,
last_testrun_utc: None,
last_updated_utc: 0,
moniker: "".to_string(),
security_contact: "".to_string(),
details: "".to_string(),
website: "".to_string(),
};
let http_gateway: crate::http::models::Gateway = gateway_dto.try_into().unwrap();
assert_eq!(http_gateway.gateway_identity_key, "test_identity");
assert!(!http_gateway.bonded);
assert_eq!(http_gateway.performance, 0);
assert!(http_gateway.self_described.is_none());
assert!(http_gateway.explorer_pretty_bond.is_none());
assert!(http_gateway.last_probe_result.is_none());
assert!(http_gateway.last_probe_log.is_none());
assert!(http_gateway.last_testrun_utc.is_none());
assert_eq!(http_gateway.last_updated_utc, "1970-01-01T00:00:00Z");
}
#[test]
fn test_mixnode_dto_with_invalid_json() {
let mixnode_dto = crate::db::models::MixnodeDto {
mix_id: 1,
bonded: true,
is_dp_delegatee: false,
total_stake: 1000000,
full_details: "invalid json".to_string(),
self_described: Some("also invalid".to_string()),
last_updated_utc: 1672531200,
moniker: "moniker".to_string(),
website: "website".to_string(),
security_contact: "contact".to_string(),
details: "details".to_string(),
};
let http_mixnode: crate::http::models::Mixnode = mixnode_dto.try_into().unwrap();
// Invalid JSON should result in None
assert!(http_mixnode.full_details.is_none());
assert_eq!(http_mixnode.self_described, Some(serde_json::Value::Null));
}
#[test]
fn test_summary_history_dto_with_invalid_json() {
let summary_history_dto = crate::db::models::SummaryHistoryDto {
id: 1,
date: "2023-01-01".to_string(),
value_json: "not valid json".to_string(),
timestamp_utc: 1672531200,
};
let summary_history: crate::http::models::SummaryHistory =
summary_history_dto.try_into().unwrap();
assert_eq!(summary_history.date, "2023-01-01");
// Invalid JSON should result in default (null)
assert!(summary_history.value_json.is_null());
}
#[test]
fn test_gateway_sessions_record_with_all_none() {
let gateway_sessions_record = crate::db::models::GatewaySessionsRecord {
gateway_identity_key: "test_identity".to_string(),
node_id: 1,
day: time::macros::date!(2023 - 01 - 01),
unique_active_clients: 0,
session_started: 0,
users_hashes: None,
vpn_sessions: None,
mixnet_sessions: None,
unknown_sessions: None,
};
let session_stats: crate::http::models::SessionStats =
gateway_sessions_record.try_into().unwrap();
assert_eq!(session_stats.gateway_identity_key, "test_identity");
assert_eq!(session_stats.node_id, 1);
assert_eq!(session_stats.unique_active_clients, 0);
assert_eq!(session_stats.session_started, 0);
assert!(session_stats.users_hashes.is_none());
assert!(session_stats.vpn_sessions.is_none());
assert!(session_stats.mixnet_sessions.is_none());
assert!(session_stats.unknown_sessions.is_none());
}
#[test]
fn test_scraper_node_info_contact_addresses() {
use crate::db::models::{ScrapeNodeKind, ScraperNodeInfo};
let node_info = ScraperNodeInfo {
node_kind: ScrapeNodeKind::MixingNymNode { node_id: 123 },
hosts: vec!["1.1.1.1".to_string(), "example.com".to_string()],
http_api_port: 8080,
};
let addresses = node_info.contact_addresses();
// Should generate multiple URLs for each host
// Custom port (8080) should be inserted at the beginning
assert!(addresses.contains(&"http://1.1.1.1:8080".to_string()));
assert!(addresses.contains(&"http://example.com:8080".to_string()));
assert!(addresses.contains(&"http://1.1.1.1:8000".to_string()));
assert!(addresses.contains(&"https://1.1.1.1".to_string()));
assert!(addresses.contains(&"http://example.com:8000".to_string()));
// Check that URLs follow the expected pattern
assert!(addresses.len() >= 8); // At least 4 URLs per host
}
#[test]
fn test_scrape_node_kind_node_id() {
use crate::db::models::ScrapeNodeKind;
let legacy = ScrapeNodeKind::LegacyMixnode { mix_id: 42 };
assert_eq!(*legacy.node_id(), 42);
let mixing = ScrapeNodeKind::MixingNymNode { node_id: 123 };
assert_eq!(*mixing.node_id(), 123);
let entry_exit = ScrapeNodeKind::EntryExitNymNode {
node_id: 456,
identity_key: "key123".to_string(),
};
assert_eq!(*entry_exit.node_id(), 456);
}
#[test]
fn test_nym_node_dto_with_invalid_keys() {
let nym_node_dto = crate::db::models::NymNodeDto {
node_id: 1,
ed25519_identity_pubkey: "invalid_base58".to_string(),
total_stake: 1000000,
ip_addresses: serde_json::json!(["1.1.1.1"]),
mix_port: 1789,
x25519_sphinx_pubkey: "also_invalid".to_string(),
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
layer: 1
}),
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
entry: false,
mixnode: true,
exit_nr: false,
exit_ipr: false,
}),
entry: None,
performance: "1.0".to_string(),
self_described: None,
bond_info: None,
};
let result: Result<nym_validator_client::nym_api::SkimmedNode, _> = nym_node_dto.try_into();
assert!(result.is_err());
assert!(result
.unwrap_err()
.to_string()
.contains("ed25519_identity_pubkey"));
}
#[test]
fn test_nym_node_dto_with_invalid_performance() {
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
let nym_node_dto = crate::db::models::NymNodeDto {
node_id: 1,
ed25519_identity_pubkey: ed25519_pk.to_base58_string(),
total_stake: 1000000,
ip_addresses: serde_json::json!(["1.1.1.1"]),
mix_port: 1789,
x25519_sphinx_pubkey: x25519_pk.to_base58_string(),
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
layer: 1
}),
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
entry: false,
mixnode: true,
exit_nr: false,
exit_ipr: false,
}),
entry: None,
performance: "invalid_percent".to_string(),
self_described: None,
bond_info: None,
};
let result: Result<nym_validator_client::nym_api::SkimmedNode, _> = nym_node_dto.try_into();
assert!(result.is_err());
assert!(result
.unwrap_err()
.to_string()
.contains("can't parse Percent"));
}
@@ -65,3 +65,36 @@ pub async fn dvpn_gateways(
.await,
))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_routes_construction() {
let router = routes();
// Verify the router builds without panic
let _routes = router;
}
#[test]
fn test_min_node_version_query_deserialization() {
// Test with version
let json = r#"{"min_node_version": "1.2.3"}"#;
let query: MinNodeVersionQuery = serde_json::from_str(json).unwrap();
assert_eq!(query.min_node_version, Some("1.2.3".to_string()));
// Test without version
let json_empty = r#"{}"#;
let query_empty: MinNodeVersionQuery = serde_json::from_str(json_empty).unwrap();
assert_eq!(query_empty.min_node_version, None);
}
#[test]
fn test_min_supported_version() {
// Test that the lazy static initializes correctly
assert_eq!(MIN_SUPPORTED_VERSION.major, 1);
assert_eq!(MIN_SUPPORTED_VERSION.minor, 6);
assert_eq!(MIN_SUPPORTED_VERSION.patch, 2);
}
}
@@ -57,21 +57,7 @@ async fn gateways_skinny(
) -> HttpResult<Json<PagedResult<GatewaySkinny>>> {
let db = state.db_pool();
let res = state.cache().get_gateway_list(db).await;
let res: Vec<GatewaySkinny> = res
.iter()
.filter(|g| g.bonded)
.map(|g| GatewaySkinny {
gateway_identity_key: g.gateway_identity_key.clone(),
self_described: g.self_described.clone(),
performance: g.performance,
explorer_pretty_bond: g.explorer_pretty_bond.clone(),
last_probe_result: g.last_probe_result.clone(),
last_testrun_utc: g.last_testrun_utc.clone(),
last_updated_utc: g.last_updated_utc.clone(),
routing_score: g.routing_score,
config_score: g.config_score,
})
.collect();
let res: Vec<GatewaySkinny> = filter_bonded_gateways_to_skinny(res);
Ok(Json(PagedResult::paginate(pagination, res)))
}
@@ -108,3 +94,126 @@ async fn get_gateway(
None => Err(HttpError::invalid_input(identity_key)),
}
}
// Extract filtering logic for testing
fn filter_bonded_gateways_to_skinny(gateways: Vec<Gateway>) -> Vec<GatewaySkinny> {
gateways
.iter()
.filter(|g| g.bonded)
.map(|g| GatewaySkinny {
gateway_identity_key: g.gateway_identity_key.clone(),
self_described: g.self_described.clone(),
performance: g.performance,
explorer_pretty_bond: g.explorer_pretty_bond.clone(),
last_probe_result: g.last_probe_result.clone(),
last_testrun_utc: g.last_testrun_utc.clone(),
last_updated_utc: g.last_updated_utc.clone(),
routing_score: g.routing_score,
config_score: g.config_score,
})
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::http::models::Gateway;
use nym_node_requests::api::v1::node::models::NodeDescription;
fn create_test_gateway(identity_key: &str, bonded: bool, performance: u8) -> Gateway {
Gateway {
gateway_identity_key: identity_key.to_string(),
bonded,
performance,
self_described: Some(serde_json::json!({"test": "data"})),
explorer_pretty_bond: Some(serde_json::json!({"bond": "info"})),
description: NodeDescription {
moniker: "Test Gateway".to_string(),
website: "".to_string(),
security_contact: "".to_string(),
details: "".to_string(),
},
last_probe_result: Some(serde_json::json!({"result": "ok"})),
last_probe_log: None,
last_testrun_utc: Some("2024-01-20T10:00:00Z".to_string()),
last_updated_utc: "2024-01-20T11:00:00Z".to_string(),
routing_score: 0.95,
config_score: 100,
}
}
#[test]
fn test_filter_bonded_gateways_to_skinny_empty_list() {
let gateways = vec![];
let result = filter_bonded_gateways_to_skinny(gateways);
assert_eq!(result.len(), 0);
}
#[test]
fn test_filter_bonded_gateways_to_skinny_all_bonded() {
let gateways = vec![
create_test_gateway("gw1", true, 90),
create_test_gateway("gw2", true, 95),
create_test_gateway("gw3", true, 85),
];
let result = filter_bonded_gateways_to_skinny(gateways);
assert_eq!(result.len(), 3);
assert_eq!(result[0].gateway_identity_key, "gw1");
assert_eq!(result[1].gateway_identity_key, "gw2");
assert_eq!(result[2].gateway_identity_key, "gw3");
}
#[test]
fn test_filter_bonded_gateways_to_skinny_mixed() {
let gateways = vec![
create_test_gateway("gw1", true, 90),
create_test_gateway("gw2", false, 95),
create_test_gateway("gw3", true, 85),
create_test_gateway("gw4", false, 100),
];
let result = filter_bonded_gateways_to_skinny(gateways);
assert_eq!(result.len(), 2);
assert_eq!(result[0].gateway_identity_key, "gw1");
assert_eq!(result[1].gateway_identity_key, "gw3");
}
#[test]
fn test_filter_bonded_gateways_to_skinny_none_bonded() {
let gateways = vec![
create_test_gateway("gw1", false, 90),
create_test_gateway("gw2", false, 95),
];
let result = filter_bonded_gateways_to_skinny(gateways);
assert_eq!(result.len(), 0);
}
#[test]
fn test_gateway_to_skinny_conversion() {
let gateway = create_test_gateway("test_gw", true, 98);
let gateways = vec![gateway.clone()];
let result = filter_bonded_gateways_to_skinny(gateways);
assert_eq!(result.len(), 1);
let skinny = &result[0];
assert_eq!(skinny.gateway_identity_key, gateway.gateway_identity_key);
assert_eq!(skinny.performance, gateway.performance);
assert_eq!(skinny.self_described, gateway.self_described);
assert_eq!(skinny.explorer_pretty_bond, gateway.explorer_pretty_bond);
assert_eq!(skinny.last_probe_result, gateway.last_probe_result);
assert_eq!(skinny.last_testrun_utc, gateway.last_testrun_utc);
assert_eq!(skinny.last_updated_utc, gateway.last_updated_utc);
assert_eq!(skinny.routing_score, gateway.routing_score);
assert_eq!(skinny.config_score, gateway.config_score);
}
#[test]
fn test_identity_key_param_deserialization() {
let json = r#"{"identity_key": "test_key_123"}"#;
let param: IdentityKeyParam = serde_json::from_str(json).unwrap();
assert_eq!(param.identity_key, "test_key_123");
}
}
@@ -8,3 +8,15 @@ pub(crate) fn routes() -> Router<AppState> {
Router::new().nest("/sessions", sessions::routes())
//eventually add other metrics type
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_routes_construction() {
let router = routes();
// Verify the router builds without panic
let _routes = router;
}
}

Some files were not shown because too many files have changed in this diff Show More