Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| ba27d1cdcd | |||
| 401f8d1fd2 | |||
| 038234e5de | |||
| be403c6ee8 | |||
| 0b51d98e2b | |||
| 76d749baaf |
@@ -0,0 +1,686 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Nym is a privacy platform that uses mixnet technology to protect against metadata surveillance. The platform consists of several key components:
|
||||
- Mixnet nodes (mixnodes) for packet mixing
|
||||
- Gateways (entry/exit points for the network)
|
||||
- Clients for interacting with the network
|
||||
- Network monitoring tools
|
||||
- Validators for network consensus
|
||||
- Various service providers and integrations
|
||||
|
||||
## Build Commands
|
||||
|
||||
### Rust Components
|
||||
|
||||
```bash
|
||||
# Default build (debug)
|
||||
cargo build
|
||||
|
||||
# Release build
|
||||
cargo build --release
|
||||
|
||||
# Build a specific package
|
||||
cargo build -p <package-name>
|
||||
|
||||
# Build main components
|
||||
make build
|
||||
|
||||
# Build release versions of main binaries and contracts
|
||||
make build-release
|
||||
|
||||
# Build specific binaries
|
||||
make build-nym-cli
|
||||
cargo build -p nym-node --release
|
||||
cargo build -p nym-api --release
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run clippy, unit tests, and formatting
|
||||
make test
|
||||
|
||||
# Run all tests including slow tests
|
||||
make test-all
|
||||
|
||||
# Run clippy on all workspaces
|
||||
make clippy
|
||||
|
||||
# Run unit tests for a specific package
|
||||
cargo test -p <package-name>
|
||||
|
||||
# Run only expensive/ignored tests
|
||||
cargo test --workspace -- --ignored
|
||||
|
||||
# Run API tests
|
||||
dotenv -f envs/sandbox.env -- cargo test --test public-api-tests
|
||||
|
||||
# Run tests with specific log level
|
||||
RUST_LOG=debug cargo test -p <package-name>
|
||||
|
||||
# Run specific test scripts
|
||||
./nym-node/tests/test_apis.sh
|
||||
./scripts/wireguard-exit-policy/exit-policy-tests.sh
|
||||
```
|
||||
|
||||
### Linting and Formatting
|
||||
|
||||
```bash
|
||||
# Run rustfmt on all code
|
||||
make fmt
|
||||
|
||||
# Check formatting without modifying
|
||||
cargo fmt --all -- --check
|
||||
|
||||
# Run clippy with all targets
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
|
||||
# TypeScript linting
|
||||
yarn lint
|
||||
yarn lint:fix
|
||||
yarn types:lint:fix
|
||||
|
||||
# Check dependencies for security/licensing issues
|
||||
cargo deny check
|
||||
```
|
||||
|
||||
### WASM Components
|
||||
|
||||
```bash
|
||||
# Build all WASM components
|
||||
make sdk-wasm-build
|
||||
|
||||
# Build TypeScript SDK
|
||||
yarn build:sdk
|
||||
npx lerna run --scope @nymproject/sdk build --stream
|
||||
|
||||
# Build and test WASM components
|
||||
make sdk-wasm
|
||||
|
||||
# Build specific WASM packages
|
||||
cd wasm/client && make
|
||||
cd wasm/mix-fetch && make
|
||||
cd wasm/node-tester && make
|
||||
```
|
||||
|
||||
### Contract Development
|
||||
|
||||
```bash
|
||||
# Build all contracts
|
||||
make contracts
|
||||
|
||||
# Build contracts in release mode
|
||||
make build-release-contracts
|
||||
|
||||
# Generate contract schemas
|
||||
make contract-schema
|
||||
|
||||
# Run wasm-opt on contracts
|
||||
make wasm-opt-contracts
|
||||
|
||||
# Check contracts with cosmwasm-check
|
||||
make cosmwasm-check-contracts
|
||||
```
|
||||
|
||||
### Running Components
|
||||
|
||||
```bash
|
||||
# Run nym-node as a mixnode
|
||||
cargo run -p nym-node -- run --mode mixnode
|
||||
|
||||
# Run nym-node as a gateway
|
||||
cargo run -p nym-node -- run --mode gateway
|
||||
|
||||
# Run the network monitor
|
||||
cargo run -p nym-network-monitor
|
||||
|
||||
# Run the API server
|
||||
cargo run -p nym-api
|
||||
|
||||
# Run with specific environment
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Start a local network
|
||||
./scripts/localnet_start.sh
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
The Nym platform consists of various components organized as a monorepo:
|
||||
|
||||
1. **Core Mixnet Infrastructure**:
|
||||
- `nym-node`: Core binary supporting mixnode and gateway modes
|
||||
- `common/nymsphinx`: Implementation of the Sphinx packet format
|
||||
- `common/topology`: Network topology management
|
||||
- `common/types`: Shared data types across components
|
||||
|
||||
2. **Network Monitoring**:
|
||||
- `nym-network-monitor`: Monitors the network's reliability and performance
|
||||
- `nym-api`: API server for network stats and monitoring data
|
||||
- Metrics tracking for nodes, routes, and overall network health
|
||||
|
||||
3. **Client Implementations**:
|
||||
- `clients/native`: Native Rust client implementation
|
||||
- `clients/socks5`: SOCKS5 proxy client for standard applications
|
||||
- `wasm`: WebAssembly client implementations (for browsers)
|
||||
- `nym-connect`: Desktop and mobile clients
|
||||
|
||||
4. **Blockchain & Smart Contracts**:
|
||||
- `common/cosmwasm-smart-contracts`: Smart contract implementations
|
||||
- `contracts`: CosmWasm contracts for the Nym network
|
||||
- `common/ledger`: Blockchain integration
|
||||
|
||||
5. **Utilities & Tools**:
|
||||
- `tools`: Various CLI tools and utilities
|
||||
- `sdk`: SDKs for different languages and platforms
|
||||
- `documentation`: Documentation generation and management
|
||||
|
||||
## Packet System
|
||||
|
||||
Nym uses a modified Sphinx packet format for its mixnet:
|
||||
|
||||
1. **Message Chunking**:
|
||||
- Messages are divided into "sets" and "fragments"
|
||||
- Each fragment fits in a single Sphinx packet
|
||||
- The `common/nymsphinx/chunking` module handles message fragmentation
|
||||
|
||||
2. **Routing**:
|
||||
- Packets traverse through 3 layers of mixnodes
|
||||
- Routing information is encrypted in layers (onion routing)
|
||||
- The final gateway receives and processes the messages
|
||||
|
||||
3. **Monitoring**:
|
||||
- Monitoring system tracks packet delivery through the network
|
||||
- Routes are analyzed for reliability statistics
|
||||
- Node performance metrics are collected
|
||||
|
||||
## Network Protocol
|
||||
|
||||
Nym implements the Loopix mixnet design with several key privacy features:
|
||||
|
||||
1. **Continuous-time Mixing**:
|
||||
- Each mixnode delays messages independently with an exponential distribution
|
||||
- This creates random reordering of packets, destroying timing correlations
|
||||
- Offers better anonymity properties than batch mixing approaches
|
||||
|
||||
2. **Cover Traffic**:
|
||||
- Clients and nodes generate dummy "loop" packets that circulate through the network
|
||||
- These packets are indistinguishable from real traffic
|
||||
- Creates a baseline level of traffic that hides actual communication patterns
|
||||
- Provides unobservability (hiding when and how much real traffic is being sent)
|
||||
|
||||
3. **Stratified Network Architecture**:
|
||||
- Traffic flows through Entry Gateway → 3 Mixnode Layers → Exit Gateway
|
||||
- Path selection is independent per-message (unlike Tor)
|
||||
- Each node connects only to adjacent layers
|
||||
|
||||
4. **Anonymous Replies**:
|
||||
- Single-Use Reply Blocks (SURBs) allow receiving messages without revealing identity
|
||||
- Enables bidirectional communication while maintaining privacy
|
||||
|
||||
## Network Monitoring Architecture
|
||||
|
||||
The network monitoring system is a core component that measures mixnet reliability:
|
||||
|
||||
1. The `nym-network-monitor` sends test packets through the network
|
||||
2. These packets follow predefined routes through multiple mixnodes
|
||||
3. Metrics are collected about:
|
||||
- Successful and failed packet deliveries
|
||||
- Node reliability (percentage of successful packet handling)
|
||||
- Route reliability (which specific route combinations work best)
|
||||
4. Results are stored in the database and used by `nym-api` to:
|
||||
- Present node performance statistics
|
||||
- Determine network rewards
|
||||
- Provide route selection guidance to clients
|
||||
|
||||
In the current branch, metrics collection is being enhanced with a fanout approach to submit to multiple API endpoints.
|
||||
|
||||
## Development Environment
|
||||
|
||||
### Required Dependencies
|
||||
|
||||
- Rust toolchain (stable, 1.80+)
|
||||
- Node.js (v20+) and yarn for TypeScript components
|
||||
- SQLite for local database development
|
||||
- PostgreSQL for API database (optional, for full API functionality)
|
||||
- CosmWasm tools for contract development
|
||||
- For building contracts: `wasm-opt` tool from `binaryen`
|
||||
- Python 3.8+ for some scripts
|
||||
- Docker (optional, for containerized development)
|
||||
- protoc (Protocol Buffers compiler) for some components
|
||||
|
||||
### Environment Configurations
|
||||
|
||||
The `envs/` directory contains pre-configured environments:
|
||||
|
||||
#### Available Environments
|
||||
|
||||
- **`local.env`**: Local development environment
|
||||
- Points to local services (localhost)
|
||||
- Uses test mnemonics and keys
|
||||
- Ideal for testing without external dependencies
|
||||
|
||||
- **`sandbox.env`**: Sandbox test network
|
||||
- Public test network with real nodes
|
||||
- Test tokens available from faucet
|
||||
- Contract addresses for sandbox deployment
|
||||
- API: https://sandbox-nym-api1.nymtech.net
|
||||
|
||||
- **`mainnet.env`**: Production mainnet
|
||||
- Real network with real tokens
|
||||
- Production contract addresses
|
||||
- API: https://validator.nymtech.net
|
||||
- Use with caution!
|
||||
|
||||
- **`canary.env`**: Canary deployment
|
||||
- Pre-release testing environment
|
||||
- Tests new features before mainnet
|
||||
|
||||
- **`mainnet-local-api.env`**: Hybrid environment
|
||||
- Uses mainnet contracts but local API
|
||||
- Useful for API development against mainnet data
|
||||
|
||||
#### Key Environment Variables
|
||||
|
||||
```bash
|
||||
# Network configuration
|
||||
NETWORK_NAME=sandbox # Network identifier
|
||||
BECH32_PREFIX=n # Address prefix (n for sandbox, n for mainnet)
|
||||
NYM_API=https://sandbox-nym-api1.nymtech.net/api
|
||||
NYXD=https://rpc.sandbox.nymtech.net
|
||||
NYM_API_NETWORK=sandbox
|
||||
|
||||
# Contract addresses (network-specific)
|
||||
MIXNET_CONTRACT_ADDRESS=n1xr3rq8yvd7qplsw5yx90ftsr2zdhg4e9z60h5duusgxpv72hud3sjkxkav
|
||||
VESTING_CONTRACT_ADDRESS=n1unyuj8qnmygvzuex3dwmg9yzt9alhvyeat0uu0jedg2wj33efl5qackslz
|
||||
# ... other contract addresses
|
||||
|
||||
# Mnemonic for testing (NEVER use in production)
|
||||
MNEMONIC="clutch captain shoe salt awake harvest setup primary inmate ugly among become"
|
||||
|
||||
# API Keys and tokens
|
||||
IPINFO_API_TOKEN=your_token_here
|
||||
AUTHENTICATOR_PASSWORD=password_here
|
||||
|
||||
# Logging
|
||||
RUST_LOG=info # Options: error, warn, info, debug, trace
|
||||
RUST_BACKTRACE=1 # Enable backtraces
|
||||
|
||||
# Database
|
||||
DATABASE_URL=postgresql://user:pass@localhost/nym_api
|
||||
```
|
||||
|
||||
#### Using Environment Files
|
||||
|
||||
```bash
|
||||
# Load environment and run command
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Export to shell
|
||||
source envs/sandbox.env
|
||||
|
||||
# Use with make targets
|
||||
dotenv -f envs/sandbox.env -- make run-api-tests
|
||||
```
|
||||
|
||||
## Initial Setup
|
||||
|
||||
### First Time Setup
|
||||
|
||||
1. **Install Prerequisites**
|
||||
```bash
|
||||
# Install Rust
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
|
||||
# Install Node.js and yarn
|
||||
# Via nvm (recommended):
|
||||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
|
||||
nvm install 20
|
||||
npm install -g yarn
|
||||
|
||||
# Install build tools
|
||||
# Ubuntu/Debian:
|
||||
sudo apt-get install build-essential pkg-config libssl-dev protobuf-compiler libpq-dev
|
||||
|
||||
# macOS:
|
||||
brew install protobuf postgresql
|
||||
|
||||
# Install wasm-opt for contract builds
|
||||
npm install -g wasm-opt
|
||||
|
||||
# Add wasm target for Rust
|
||||
rustup target add wasm32-unknown-unknown
|
||||
```
|
||||
|
||||
2. **Clone and Setup Repository**
|
||||
```bash
|
||||
git clone https://github.com/nymtech/nym.git
|
||||
cd nym/nym
|
||||
|
||||
# Install JavaScript dependencies
|
||||
yarn install
|
||||
|
||||
# Build the project
|
||||
make build
|
||||
```
|
||||
|
||||
3. **Database Setup (Optional, for API development)**
|
||||
```bash
|
||||
# Install PostgreSQL
|
||||
# Create database
|
||||
createdb nym_api
|
||||
|
||||
# Run migrations (from nym-api directory)
|
||||
cd nym-api
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Run a mixnode locally
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode mixnode --id my-mixnode
|
||||
|
||||
# Run a gateway locally
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode gateway --id my-gateway
|
||||
|
||||
# Run the API server
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Run a client
|
||||
cargo run -p nym-client -- init --id my-client
|
||||
cargo run -p nym-client -- run --id my-client
|
||||
```
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
The project uses GitHub Actions for CI/CD with several key workflows:
|
||||
|
||||
1. **Build and Test**:
|
||||
- `ci-build.yml`: Main build workflow for Rust components
|
||||
- Tests are run on multiple platforms (Linux, Windows, macOS)
|
||||
- Includes formatting check (rustfmt) and linting (clippy)
|
||||
|
||||
2. **Release Process**:
|
||||
- Binary artifacts are published on release tags
|
||||
- Multiple platform builds are created
|
||||
|
||||
3. **Documentation**:
|
||||
- Documentation is automatically built and deployed
|
||||
|
||||
## Database Structure
|
||||
|
||||
The system uses SQLite databases with tables like:
|
||||
- `mixnode_status`: Status information about mixnodes
|
||||
- `gateway_status`: Status information about gateways
|
||||
- `routes`: Route performance information (success/failure of specific paths)
|
||||
- `monitor_run`: Information about monitoring test runs
|
||||
|
||||
## Development Workflows
|
||||
|
||||
### Running a Node
|
||||
|
||||
To run the mixnode or gateway:
|
||||
|
||||
```bash
|
||||
# Run nym-node as a mixnode with specified identity
|
||||
cargo run -p nym-node -- run --mode mixnode --id my-mixnode
|
||||
|
||||
# Run nym-node as a gateway
|
||||
cargo run -p nym-node -- run --mode gateway --id my-gateway
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Nodes can be configured with files in various locations:
|
||||
- Command-line arguments
|
||||
- Environment variables
|
||||
- `.env` files specified with `--config-env-file`
|
||||
|
||||
### Monitoring
|
||||
|
||||
To monitor the health of your node:
|
||||
- View logs for real-time information
|
||||
- Use the node's HTTP API for status information
|
||||
- Check the explorer for public node statistics
|
||||
|
||||
## Common Libraries
|
||||
|
||||
- `common/types`: Shared data types across all components
|
||||
- `common/crypto`: Cryptographic primitives and wrappers
|
||||
- `common/client-core`: Core client functionality
|
||||
- `common/gateway-client`: Client-gateway communication
|
||||
- `common/task`: Task management and concurrency utilities
|
||||
- `common/nymsphinx`: Sphinx packet implementation for mixnet
|
||||
- `common/topology`: Network topology management
|
||||
- `common/credentials`: Credential system for privacy-preserving authentication
|
||||
- `common/bandwidth-controller`: Bandwidth management and accounting
|
||||
|
||||
## Code Conventions
|
||||
|
||||
- Error handling: Use anyhow/thiserror for structured error handling
|
||||
- Logging: Use the tracing framework for logging and diagnostics
|
||||
- State management: Generally use Tokio/futures for async code
|
||||
- Configuration: Use the config crate and env vars with defaults
|
||||
- Database: Use sqlx for type-safe database queries
|
||||
- Follow clippy recommendations and rustfmt formatting
|
||||
- Use semantic commit messages: feat, fix, docs, refactor, test, chore
|
||||
|
||||
## When Making Changes
|
||||
|
||||
- Run `make test` before submitting PRs
|
||||
- Follow Rust naming conventions
|
||||
- Use `clippy` to check for common issues
|
||||
- Update SQLx query caches when modifying DB queries: `cargo sqlx prepare`
|
||||
- Consider backward compatibility for protocol changes
|
||||
- Use lefthook pre-commit hooks for TypeScript formatting
|
||||
- Run `cargo deny check` to verify dependency compliance
|
||||
- Test against both sandbox and local environments when possible
|
||||
- Update relevant documentation and CHANGELOG.md
|
||||
|
||||
## Development Tools
|
||||
|
||||
### Useful Cargo Commands
|
||||
|
||||
```bash
|
||||
# Check for outdated dependencies
|
||||
cargo outdated
|
||||
|
||||
# Analyze binary size
|
||||
cargo bloat --release -p nym-node
|
||||
|
||||
# Generate dependency graph
|
||||
cargo tree -p nym-api
|
||||
|
||||
# Run with instrumentation
|
||||
cargo run --features profiling -p nym-node
|
||||
|
||||
# Check for security advisories
|
||||
cargo audit
|
||||
```
|
||||
|
||||
### Database Tools
|
||||
|
||||
```bash
|
||||
# SQLx CLI for migrations
|
||||
cargo install sqlx-cli
|
||||
|
||||
# Create new migration
|
||||
cd nym-api && sqlx migrate add <migration_name>
|
||||
|
||||
# Prepare query metadata for offline compilation
|
||||
cargo sqlx prepare --workspace
|
||||
|
||||
# View database schema
|
||||
./nym-api/enter_db.sh
|
||||
```
|
||||
|
||||
### Development Scripts
|
||||
|
||||
- `scripts/build_topology.py`: Generate network topology files
|
||||
- `scripts/node_api_check.py`: Verify node API endpoints
|
||||
- `scripts/network_tunnel_manager.sh`: Manage network tunnels
|
||||
- `scripts/localnet_start.sh`: Start a local test network
|
||||
- Various deployment scripts in `deployment/` for different environments
|
||||
|
||||
## Debugging
|
||||
|
||||
- Enable more verbose logging with the RUST_LOG environment variable:
|
||||
```
|
||||
RUST_LOG=debug,nym_node=trace cargo run -p nym-node -- run --mode mixnode
|
||||
```
|
||||
- Use the HTTP API endpoints for status information
|
||||
- Check monitoring data in the database for network performance metrics
|
||||
- For complex issues, use tracing tools to follow packet flow
|
||||
- Enable backtraces: `RUST_BACKTRACE=full`
|
||||
- For WASM debugging: Use browser developer tools with source maps
|
||||
|
||||
## Deployment and Advanced Configurations
|
||||
|
||||
### Deployment Structure
|
||||
|
||||
The `deployment/` directory contains Ansible playbooks and configurations for various deployment scenarios:
|
||||
|
||||
- **`aws/`**: AWS-specific deployment configurations
|
||||
- **`mixnode/`**: Mixnode deployment playbooks
|
||||
- **`gateway/`**: Gateway deployment playbooks
|
||||
- **`validator/`**: Validator node deployment
|
||||
- **`sandbox-v2/`**: Complete sandbox environment setup
|
||||
- **`big-dipper-2/`**: Block explorer deployment
|
||||
|
||||
### Sandbox V2 Deployment
|
||||
|
||||
The sandbox-v2 deployment (`deployment/sandbox-v2/`) provides a complete test environment:
|
||||
|
||||
```bash
|
||||
# Key playbooks:
|
||||
- deploy.yaml # Main deployment orchestrator
|
||||
- deploy-mixnodes.yaml # Deploy mixnodes
|
||||
- deploy-gateways.yaml # Deploy gateways
|
||||
- deploy-validators.yaml # Deploy validator nodes
|
||||
- deploy-nym-api.yaml # Deploy API services
|
||||
```
|
||||
|
||||
### Custom Environment Setup
|
||||
|
||||
To create a custom environment:
|
||||
|
||||
1. Copy an existing env file: `cp envs/sandbox.env envs/custom.env`
|
||||
2. Modify the network endpoints and contract addresses
|
||||
3. Update the `NETWORK_NAME` to your identifier
|
||||
4. Set appropriate mnemonics and keys (use fresh ones for production!)
|
||||
|
||||
### Contract Addresses
|
||||
|
||||
Contract addresses are network-specific and defined in environment files:
|
||||
- Mixnet contract: Manages mixnode/gateway registry
|
||||
- Vesting contract: Handles token vesting schedules
|
||||
- Coconut contracts: Privacy-preserving credentials
|
||||
- Name service: Human-readable address mapping
|
||||
- Ecash contract: Electronic cash functionality
|
||||
|
||||
### Local Network Setup
|
||||
|
||||
For a completely local network:
|
||||
```bash
|
||||
# Start local chain
|
||||
./scripts/localnet_start.sh
|
||||
|
||||
# Deploy contracts
|
||||
cd contracts
|
||||
make deploy-local
|
||||
|
||||
# Start nodes with local config
|
||||
dotenv -f envs/local.env -- cargo run -p nym-node -- run --mode mixnode
|
||||
```
|
||||
|
||||
## Common Issues and Troubleshooting
|
||||
|
||||
### Database Issues
|
||||
|
||||
- When modifying database queries, you must update SQLx query caches:
|
||||
```bash
|
||||
cargo sqlx prepare
|
||||
```
|
||||
- If you see SQLx errors about missing query files, this is likely the cause
|
||||
- For "database is locked" errors with SQLite, ensure only one process accesses the DB
|
||||
- For PostgreSQL connection issues, verify DATABASE_URL and that the server is running
|
||||
|
||||
### API Connection Issues
|
||||
|
||||
- Check the environment variables pointing to the APIs (NYM_API, NYXD)
|
||||
- Verify network connectivity and API health endpoints
|
||||
- For authentication issues, check node keys and credentials
|
||||
- Common endpoints to verify:
|
||||
- API health: `$NYM_API/health`
|
||||
- Chain status: `$NYXD/status`
|
||||
- Contract info: `$NYXD/cosmwasm/wasm/v1/contract/$CONTRACT_ADDRESS`
|
||||
|
||||
### Build Problems
|
||||
|
||||
- Clean dependencies with `cargo clean` for a fresh build
|
||||
- Check for compatible Rust version (1.80+ recommended)
|
||||
- For smart contract builds, ensure wasm-opt is installed: `npm install -g wasm-opt`
|
||||
- For cross-compilation issues, check target-specific dependencies
|
||||
- WASM build issues: Ensure wasm32-unknown-unknown target is installed:
|
||||
```bash
|
||||
rustup target add wasm32-unknown-unknown
|
||||
```
|
||||
- For "cannot find -lpq" errors, install PostgreSQL development files:
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install libpq-dev
|
||||
# macOS
|
||||
brew install postgresql
|
||||
```
|
||||
|
||||
### Environment Issues
|
||||
|
||||
- Contract address mismatches: Ensure you're using the correct environment file
|
||||
- "Account sequence mismatch": The account nonce is out of sync, wait and retry
|
||||
- Token decimal issues: Sandbox uses different decimal places than mainnet
|
||||
- API version mismatches: Ensure your local API version matches the network
|
||||
- "Insufficient funds": Get test tokens from faucet (sandbox) or check balance
|
||||
- Gateway/mixnode bonding issues: Verify minimum stake requirements
|
||||
|
||||
## Working with Routes and Monitoring
|
||||
|
||||
1. Route monitoring metrics are stored in a `routes` table with:
|
||||
- Layer node IDs (layer1, layer2, layer3, gw)
|
||||
- Success flag (boolean)
|
||||
- Timestamp
|
||||
|
||||
2. To analyze routes:
|
||||
- Check `NetworkAccount` and `AccountingRoute` in `nym-network-monitor/src/accounting.rs`
|
||||
- View monitoring logic in `common/nymsphinx/chunking/monitoring.rs`
|
||||
- Observe how routes are submitted to the database in the `submit_accounting_routes_to_db` function
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Profiling and Benchmarking
|
||||
|
||||
```bash
|
||||
# Run benchmarks
|
||||
cargo bench -p nym-node
|
||||
|
||||
# Profile with perf (Linux)
|
||||
cargo build --release --features profiling
|
||||
perf record --call-graph=dwarf ./target/release/nym-node run --mode mixnode
|
||||
perf report
|
||||
|
||||
# Generate flamegraph
|
||||
cargo install flamegraph
|
||||
cargo flamegraph --bin nym-node -- run --mode mixnode
|
||||
```
|
||||
|
||||
### Common Performance Considerations
|
||||
|
||||
- Use bounded channels for backpressure
|
||||
- Batch database operations where possible
|
||||
- Monitor memory usage with `RUST_LOG=nym_node::metrics=debug`
|
||||
- Use connection pooling for database connections
|
||||
- Consider using `jemalloc` for better memory allocation performance
|
||||
Generated
+841
-932
File diff suppressed because it is too large
Load Diff
@@ -4,3 +4,61 @@ The Node Status API serves information about individual `nym-nodes` in the Mixne
|
||||
We recommend that developers building applications such as explorers or analytics interfaces about the Mixnet run their own instance of the API, in order to promote a robust network of downstream services, and spread the load of API calls amongst as many endpoints as possible.
|
||||
|
||||
You can find build and operation instructions in the [docs](https://nym.com/docs/apis/ns-api).
|
||||
|
||||
## Database Support
|
||||
|
||||
The Node Status API supports both SQLite and PostgreSQL databases through Cargo feature flags:
|
||||
|
||||
- **SQLite** (default): Lightweight, file-based database suitable for development and small deployments
|
||||
- **PostgreSQL**: Full-featured database recommended for production deployments
|
||||
|
||||
### Building with Different Database Backends
|
||||
|
||||
```bash
|
||||
# Build with SQLite (default)
|
||||
cargo build --features sqlite --no-default-features
|
||||
|
||||
# Build with PostgreSQL
|
||||
cargo build --features pg --no-default-features
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Test with SQLite
|
||||
cargo test --features sqlite --no-default-features
|
||||
|
||||
# Test with PostgreSQL
|
||||
make test-db # This sets up a test PostgreSQL instance
|
||||
```
|
||||
|
||||
### Development Commands
|
||||
|
||||
The project includes a Makefile with helpful commands for both database backends:
|
||||
|
||||
```bash
|
||||
# Check code compilation
|
||||
make check-sqlite # Check with SQLite
|
||||
make check-pg # Check with PostgreSQL
|
||||
|
||||
# Run clippy linter
|
||||
make clippy-sqlite # Lint with SQLite
|
||||
make clippy-pg # Lint with PostgreSQL
|
||||
make clippy # Run both
|
||||
|
||||
# PostgreSQL development
|
||||
make dev-db # Start a PostgreSQL instance for development
|
||||
make prepare-pg # Prepare SQLx offline cache for PostgreSQL
|
||||
```
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The database abstraction is implemented using a query wrapper that automatically converts SQLite-style `?` placeholders to PostgreSQL-style `$1, $2, ...` placeholders at runtime. This allows writing queries once using SQLite syntax while maintaining compatibility with both databases.
|
||||
|
||||
Key differences handled:
|
||||
- Placeholder syntax (`?` vs `$1, $2, ...`)
|
||||
- Type conversions (SQLite uses i64, PostgreSQL uses i32 for many fields)
|
||||
- SQL dialect differences (e.g., `INSERT OR IGNORE` vs `ON CONFLICT DO NOTHING`)
|
||||
- RETURNING clause behavior
|
||||
|
||||
For more details on PostgreSQL setup, see [README_PG.md](nym-node-status-api/README_PG.md).
|
||||
|
||||
@@ -14,13 +14,22 @@ rust-version.workspace = true
|
||||
readme.workspace = true
|
||||
|
||||
[dependencies]
|
||||
anyhow = { workspace = true}
|
||||
anyhow = { workspace = true }
|
||||
clap = { workspace = true, features = ["derive", "env"] }
|
||||
nym-bin-common = { path = "../../common/bin-common", features = ["models"]}
|
||||
futures = { workspace = true }
|
||||
nym-bin-common = { path = "../../common/bin-common", features = ["models"] }
|
||||
nym-node-status-client = { path = "../nym-node-status-client" }
|
||||
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "rand"] }
|
||||
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"asymmetric",
|
||||
"rand",
|
||||
] }
|
||||
rand = { workspace = true }
|
||||
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "process", "fs"] }
|
||||
tokio = { workspace = true, features = [
|
||||
"macros",
|
||||
"rt-multi-thread",
|
||||
"process",
|
||||
"fs",
|
||||
] }
|
||||
tracing = { workspace = true }
|
||||
tracing-subscriber = { workspace = true, features = ["env-filter"] }
|
||||
|
||||
|
||||
@@ -1,17 +1,45 @@
|
||||
use crate::probe::GwProbe;
|
||||
use clap::{Parser, Subcommand};
|
||||
use nym_bin_common::bin_info;
|
||||
use nym_crypto::asymmetric::ed25519::PrivateKey;
|
||||
use std::sync::OnceLock;
|
||||
|
||||
pub(crate) mod generate_keypair;
|
||||
pub(crate) mod run_probe;
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct ServerConfig {
|
||||
pub(crate) address: String,
|
||||
pub(crate) port: u16,
|
||||
pub(crate) auth_key: PrivateKey,
|
||||
}
|
||||
|
||||
// Helper for passing LONG_VERSION to clap
|
||||
fn pretty_build_info_static() -> &'static str {
|
||||
static PRETTY_BUILD_INFORMATION: OnceLock<String> = OnceLock::new();
|
||||
PRETTY_BUILD_INFORMATION.get_or_init(|| bin_info!().pretty_print())
|
||||
}
|
||||
|
||||
fn parse_server_config(s: &str) -> Result<ServerConfig, String> {
|
||||
let parts: Vec<&str> = s.split(':').collect();
|
||||
if parts.len() != 3 {
|
||||
return Err("Server config must be in format 'address:port:auth_key'".to_string());
|
||||
}
|
||||
|
||||
let address = parts[0].to_string();
|
||||
let port = parts[1]
|
||||
.parse::<u16>()
|
||||
.map_err(|_| "Invalid port number".to_string())?;
|
||||
let auth_key =
|
||||
PrivateKey::from_base58_string(parts[2]).map_err(|_| "Invalid auth key".to_string())?;
|
||||
|
||||
Ok(ServerConfig {
|
||||
address,
|
||||
port,
|
||||
auth_key,
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[clap(author = "Nymtech", version, long_version = pretty_build_info_static(), about)]
|
||||
pub(crate) struct Args {
|
||||
@@ -22,15 +50,10 @@ pub(crate) struct Args {
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub(crate) enum Command {
|
||||
RunProbe {
|
||||
#[arg(short, long, env = "NODE_STATUS_AGENT_SERVER_ADDRESS")]
|
||||
server_address: String,
|
||||
|
||||
#[arg(short = 'p', long, env = "NODE_STATUS_AGENT_SERVER_PORT")]
|
||||
server_port: u16,
|
||||
|
||||
/// base58-encoded private key
|
||||
#[arg(long, env = "NODE_STATUS_AGENT_AUTH_KEY")]
|
||||
ns_api_auth_key: String,
|
||||
/// Server configurations in format "address:port:auth_key"
|
||||
/// Can be specified multiple times for multiple servers
|
||||
#[arg(short, long, required = true)]
|
||||
server: Vec<String>,
|
||||
|
||||
/// path of binary to run
|
||||
#[arg(long, env = "NODE_STATUS_AGENT_PROBE_PATH")]
|
||||
@@ -54,22 +77,28 @@ impl Args {
|
||||
pub(crate) async fn execute(&self) -> anyhow::Result<()> {
|
||||
match &self.command {
|
||||
Command::RunProbe {
|
||||
server_address,
|
||||
server_port,
|
||||
ns_api_auth_key,
|
||||
server,
|
||||
probe_path,
|
||||
probe_extra_args,
|
||||
} => run_probe::run_probe(
|
||||
server_address,
|
||||
server_port.to_owned(),
|
||||
ns_api_auth_key,
|
||||
probe_path,
|
||||
probe_extra_args,
|
||||
)
|
||||
.await
|
||||
.inspect_err(|err| {
|
||||
tracing::error!("{err}");
|
||||
})?,
|
||||
} => {
|
||||
// Parse server configs
|
||||
let mut servers = Vec::new();
|
||||
for s in server {
|
||||
match parse_server_config(s) {
|
||||
Ok(config) => servers.push(config),
|
||||
Err(e) => {
|
||||
tracing::error!("Invalid server config '{}': {}", s, e);
|
||||
anyhow::bail!("Invalid server config '{}': {}", s, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
run_probe::run_probe(&servers, probe_path, probe_extra_args)
|
||||
.await
|
||||
.inspect_err(|err| {
|
||||
tracing::error!("{err}");
|
||||
})?
|
||||
}
|
||||
Command::GenerateKeypair { path } => {
|
||||
let path = path
|
||||
.to_owned()
|
||||
|
||||
@@ -1,33 +1,117 @@
|
||||
use crate::cli::GwProbe;
|
||||
use anyhow::Context;
|
||||
use nym_crypto::asymmetric::ed25519::PrivateKey;
|
||||
use crate::cli::{GwProbe, ServerConfig};
|
||||
|
||||
pub(crate) async fn run_probe(
|
||||
server_ip: &str,
|
||||
server_port: u16,
|
||||
ns_api_auth_key: &str,
|
||||
servers: &[ServerConfig],
|
||||
probe_path: &str,
|
||||
probe_extra_args: &Vec<String>,
|
||||
) -> anyhow::Result<()> {
|
||||
let auth_key = PrivateKey::from_base58_string(ns_api_auth_key)
|
||||
.context("Couldn't parse auth key, exiting")?;
|
||||
|
||||
let ns_api_client = nym_node_status_client::NsApiClient::new(server_ip, server_port, auth_key);
|
||||
if servers.is_empty() {
|
||||
anyhow::bail!("No servers configured");
|
||||
}
|
||||
|
||||
let probe = GwProbe::new(probe_path.to_string());
|
||||
|
||||
let version = probe.version().await;
|
||||
tracing::info!("Probe version:\n{}", version);
|
||||
|
||||
if let Some(testrun) = ns_api_client.request_testrun().await? {
|
||||
let log = probe.run_and_get_log(&Some(testrun.gateway_identity_key), probe_extra_args);
|
||||
// Always use first server as primary
|
||||
let primary_server = &servers[0];
|
||||
tracing::info!(
|
||||
"Requesting testrun from primary server: {}:{}",
|
||||
primary_server.address,
|
||||
primary_server.port
|
||||
);
|
||||
|
||||
ns_api_client
|
||||
.submit_results(testrun.testrun_id, log, testrun.assigned_at_utc)
|
||||
.await?;
|
||||
} else {
|
||||
tracing::info!("No testruns available, exiting")
|
||||
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
|
||||
&primary_server.auth_key.to_bytes(),
|
||||
)
|
||||
.expect("Failed to clone auth key");
|
||||
let ns_api_client = nym_node_status_client::NsApiClient::new(
|
||||
&primary_server.address,
|
||||
primary_server.port,
|
||||
auth_key,
|
||||
);
|
||||
|
||||
match ns_api_client.request_testrun().await {
|
||||
Ok(Some(testrun)) => {
|
||||
tracing::info!(
|
||||
"Received testrun {} for gateway {} from primary",
|
||||
testrun.testrun_id,
|
||||
testrun.gateway_identity_key
|
||||
);
|
||||
|
||||
// Run the probe
|
||||
let log = probe.run_and_get_log(
|
||||
&Some(testrun.gateway_identity_key.clone()),
|
||||
probe_extra_args,
|
||||
);
|
||||
|
||||
// Submit to ALL servers in parallel
|
||||
let handles = servers
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(idx, server)| {
|
||||
let testrun = testrun.clone();
|
||||
let log = log.clone();
|
||||
|
||||
async move {
|
||||
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
|
||||
&server.auth_key.to_bytes(),
|
||||
)
|
||||
.expect("Failed to clone auth key");
|
||||
let client = nym_node_status_client::NsApiClient::new(
|
||||
&server.address,
|
||||
server.port,
|
||||
auth_key,
|
||||
);
|
||||
|
||||
let result = client
|
||||
.submit_results_with_context(
|
||||
testrun.testrun_id,
|
||||
log,
|
||||
testrun.assigned_at_utc,
|
||||
testrun.gateway_identity_key,
|
||||
)
|
||||
.await;
|
||||
|
||||
(idx, server.address.clone(), server.port, result)
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let results = futures::future::join_all(handles).await;
|
||||
|
||||
for result in results {
|
||||
match result.3 {
|
||||
Ok(()) => {
|
||||
tracing::info!(
|
||||
"✅ Successfully submitted to server[{}] {}:{}",
|
||||
result.0,
|
||||
result.1,
|
||||
result.2
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
"❌ Failed to submit to server[{}] {}:{} - {}",
|
||||
result.0,
|
||||
result.1,
|
||||
result.2,
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
Ok(None) => {
|
||||
tracing::info!("No testruns available from primary server");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to contact primary server: {}", e);
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Example environment variables for nym-node-status-api
|
||||
|
||||
# Database configuration
|
||||
# For SQLite:
|
||||
# DATABASE_URL=sqlite://nym-node-status-api.sqlite
|
||||
|
||||
# For PostgreSQL:
|
||||
# DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# Network configuration
|
||||
NETWORK_NAME=sandbox
|
||||
NYM_API=https://sandbox-nym-api1.nymtech.net/api
|
||||
NYXD=https://rpc.sandbox.nymtech.net
|
||||
|
||||
# API configuration
|
||||
NYM_NODE_STATUS_API_HTTP_PORT=8000
|
||||
NYM_API_CLIENT_TIMEOUT=15
|
||||
SQLX_BUSY_TIMEOUT_S=5
|
||||
|
||||
# Monitoring intervals
|
||||
NODE_STATUS_API_MONITOR_REFRESH_INTERVAL=300
|
||||
NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL=300
|
||||
NODE_STATUS_API_GEODATA_TTL=86400
|
||||
|
||||
# Agent keys (comma-separated list)
|
||||
NODE_STATUS_API_AGENT_KEY_LIST=
|
||||
|
||||
# External service tokens
|
||||
IPINFO_API_TOKEN=your_token_here
|
||||
|
||||
# MixNodes (Optional)
|
||||
DATA_PROVIDER_DELEGATION_LIST=
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_daily_stats (\n mix_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,\n packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,\n packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "01ee4a30bc3104712e5bc371a45d614a89d88adf02358800433e06100c13c548"
|
||||
}
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT mix_id as node_id, host, http_api_port\n FROM mixnodes\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "host",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "http_api_port",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "021c6c65d1ed806d8430bef7883906b42a7e4b280c8efb32db15d7c6a51d7a27"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_description (\n mix_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (mix_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "06065394c157927e4002ddd5c7c1af626ae15728d615f539470cd7c189312385"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n gateway_identity_key\n FROM\n gateways\n WHERE\n id = ?",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "06b17d1e5f61201a1b7542896ba55c69cd5c1a7e7d87073c94600c783a0a3984"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n testruns\n SET\n status = ?\n WHERE\n status = ?\n AND\n last_assigned_utc < ?\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "0cf0e4d4f30e90caecffd6255ef85dab12730e538be194438f19ed7f198bd50e"
|
||||
}
|
||||
+17
-17
@@ -1,43 +1,43 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT\n date_utc as \"date_utc!\",\n SUM(total_stake) as \"total_stake!: i64\",\n SUM(packets_received) as \"total_packets_received!: i64\",\n SUM(packets_sent) as \"total_packets_sent!: i64\",\n SUM(packets_dropped) as \"total_packets_dropped!: i64\"\n FROM (\n SELECT\n date_utc,\n n.total_stake,\n n.packets_received,\n n.packets_sent,\n n.packets_dropped\n FROM nym_node_daily_mixing_stats n\n UNION ALL\n SELECT\n m.date_utc,\n m.total_stake,\n m.packets_received,\n m.packets_sent,\n m.packets_dropped\n FROM mixnode_daily_stats m\n LEFT JOIN nym_node_daily_mixing_stats ON m.mix_id = nym_node_daily_mixing_stats.node_id\n WHERE nym_node_daily_mixing_stats.node_id IS NULL\n )\n GROUP BY date_utc\n ORDER BY date_utc ASC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "date_utc!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
"name": "date_utc!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "total_stake!: i64",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "total_stake!: i64",
|
||||
"type_info": "Numeric"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_received!: i64",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_received!: i64",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_sent!: i64",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_sent!: i64",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_dropped!: i64",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_dropped!: i64",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "124d45b9604439584650f401607c46bdbd162c7c689f74fe9ddfdfd48f5ddc07"
|
||||
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n key as \"key!\",\n value_json as \"value_json!\",\n last_updated_utc as \"last_updated_utc!\"\n FROM summary",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "value_json!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "1327b5118f9144dddbcf8edb11f7dc549cf503409fd6dfedcdc02dbcd61d5454"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_packet_stats_raw (\n mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "21e44766729777756f6eb04bf3b81df3e591008a1e3fd664ed83ca86ac51bd8c"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n id,\n gateway_identity_key\n FROM gateways\n WHERE id = ?\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "2236299f9f691376db54cbd58ec5ceb89b9925cba46efcf4ed79ef0759a01129"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n AND\n self_described IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "227539374e7473f6f9642289c5b5d1bcd636315ab23537cb5f6d2f82a2bcb7bf"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT gateway_identity_key\n FROM gateways\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "25300e435780101fa207c8e26ef2f49ba5db84d63e89440bb494e8327fe73686"
|
||||
}
|
||||
-86
@@ -1,86 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n AND\n bond_info IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ed25519_identity_pubkey",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_addresses!: serde_json::Value",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "mix_port",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "x25519_sphinx_pubkey",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "node_role: serde_json::Value",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "supported_roles: serde_json::Value",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "entry: serde_json::Value",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "performance",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "283f49a65c7d70bf271702ff6a5c7ad6e68c81932d295ff18ed198c54706a57c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT OR IGNORE INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 9
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "3243cf5646255a9430d1e6710970505d0dbcc62703f40e090e80ff48c77723c4"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO mixnodes\n (mix_id, identity_key, bonded, total_stake,\n host, http_api_port, full_details,\n self_described, last_updated_utc, is_dp_delegatee)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id) DO UPDATE SET\n bonded=excluded.bonded,\n total_stake=excluded.total_stake, host=excluded.host,\n http_api_port=excluded.http_api_port,\n full_details=excluded.full_details,self_described=excluded.self_described,\n last_updated_utc=excluded.last_updated_utc,\n is_dp_delegatee = excluded.is_dp_delegatee;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 10
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "3cd5cb4bfca4243925da4ddbccd811e842090e98982e1032670df77961870b32"
|
||||
}
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_identity_key as \"gateway_identity_key!\",\n self_described as \"self_described?\",\n explorer_pretty_bond as \"explorer_pretty_bond?\"\n FROM gateways\n WHERE gateway_identity_key = ?\n AND bonded = true\n ORDER BY gateway_identity_key\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described?",
|
||||
"ordinal": 2,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "explorer_pretty_bond?",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "3e7e987780937873cdb393b157d7708c9f01047b0689eb0d4f7a973b328c609d"
|
||||
}
|
||||
-92
@@ -1,92 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n gw.gateway_identity_key as \"gateway_identity_key!\",\n gw.bonded as \"bonded: bool\",\n gw.performance as \"performance!\",\n gw.self_described as \"self_described?\",\n gw.explorer_pretty_bond as \"explorer_pretty_bond?\",\n gw.last_probe_result as \"last_probe_result?\",\n gw.last_probe_log as \"last_probe_log?\",\n gw.last_testrun_utc as \"last_testrun_utc?\",\n gw.last_updated_utc as \"last_updated_utc!\",\n COALESCE(gd.moniker, \"NA\") as \"moniker!\",\n COALESCE(gd.website, \"NA\") as \"website!\",\n COALESCE(gd.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(gd.details, \"NA\") as \"details!\"\n FROM gateways gw\n LEFT JOIN gateway_description gd\n ON gw.gateway_identity_key = gd.gateway_identity_key\n ORDER BY gw.gateway_identity_key",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "performance!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "self_described?",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "explorer_pretty_bond?",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_probe_result?",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_probe_log?",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_testrun_utc?",
|
||||
"ordinal": 7,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 8,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "moniker!",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "website!",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "security_contact!",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "details!",
|
||||
"ordinal": 12,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "3eb1d8491bda3c1d6e071b6eb364b9a979f4bdb11ea81b2d0f022555bab51ecb"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n total_stake\n FROM mixnodes\n WHERE mix_id = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "3fc2baabf194b147b20be2a49401cc0c100a1d7a7c347393adde2410fa6f4dfe"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE testruns SET status = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "418944f2eccb838cb3882f34469203c8569f03fdd39ce09d7b74177896e52a8c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_probe_log = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4afcc6673890f795c2793f1e2f8570ee787fc7daf00fcb916f18d1cb7d6c8f08"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4d865e873c9cb133883da94db72dcdebd4969e1f240def9fb1bf946f4a1f342f"
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)\n ON CONFLICT DO NOTHING",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Int8",
|
||||
"Date",
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4fca38abbb416d9457c34a8ba4faf481a837eda4f3e1bee1d430a4eb102a5b3d"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_node_daily_mixing_stats (\n node_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,\n packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,\n packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "5912ea335a957d217f5e2b3a63a25b31715c2098310fe7a9db688bc2fd36aad4"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE testruns\n SET\n status = ?,\n last_assigned_utc = ?\n WHERE rowid =\n (\n SELECT rowid\n FROM testruns\n WHERE status = ?\n ORDER BY created_utc asc\n LIMIT 1\n )\n RETURNING\n id as \"id!\",\n gateway_id\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "5e9cbb39f5fb53774803270f422989e199aac4d4a71913c7074359b4bd676b02"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO nym_nodes\n (node_id, ed25519_identity_pubkey,\n total_stake,\n ip_addresses, mix_port,\n x25519_sphinx_pubkey, node_role,\n supported_roles, entry,\n self_described,\n bond_info,\n performance, last_updated_utc\n )\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id) DO UPDATE SET\n ed25519_identity_pubkey=excluded.ed25519_identity_pubkey,\n ip_addresses=excluded.ip_addresses,\n mix_port=excluded.mix_port,\n x25519_sphinx_pubkey=excluded.x25519_sphinx_pubkey,\n node_role=excluded.node_role,\n supported_roles=excluded.supported_roles,\n entry=excluded.entry,\n self_described=excluded.self_described,\n bond_info=excluded.bond_info,\n performance=excluded.performance,\n last_updated_utc=excluded.last_updated_utc\n ;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 13
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "664e059ac2c58e1115fe214376a6b326b31c93298f20019772cce2e277a194f8"
|
||||
}
|
||||
+5
-5
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT count(id) FROM mixnodes",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count(id)",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "670b7ed7d57a6986181b24be24ca667e8cacdf677ccb906415b3fe92be0c436b"
|
||||
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_node_descriptions (\n node_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (node_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "6a9780aad1f2f0c8ef780e51fe856679d5e28f95143f4e2a2b409009dc0f55ba"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int4",
|
||||
"Int4",
|
||||
"Varchar",
|
||||
"Int8",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "6de15de62c0caa545910a17877a3ac5ebe44a398b199ab0120207a5569f54d0b"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_probe_result = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "6ef3efde571d46961244cd90420f3de5949a5ff2083453cb879af8a1689efe2f"
|
||||
}
|
||||
+30
-30
@@ -1,66 +1,66 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n COALESCE(md.moniker, \"NA\") as \"moniker!\",\n COALESCE(md.website, \"NA\") as \"website!\",\n COALESCE(md.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(md.details, \"NA\") as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n md.moniker as \"moniker!\",\n md.website as \"website!\",\n md.security_contact as \"security_contact!\",\n md.details as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "mix_id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "mix_id!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "bonded: bool",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"name": "is_dp_delegatee: bool",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
"name": "is_dp_delegatee: bool",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"name": "total_stake!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
"name": "total_stake!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "full_details!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
"name": "full_details!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "self_described",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
"name": "self_described",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
"name": "last_updated_utc!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "moniker!",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
"name": "moniker!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "website!",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
"name": "website!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "security_contact!",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
"name": "security_contact!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "details!",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
"name": "details!",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
@@ -70,11 +70,11 @@
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "f75af377da33db1455c6e0f612e0fa9583888f343b8b59faf37fc6799b244379"
|
||||
"hash": "74b76cd0d94c1afc51c21c14c12236a2b964b981c8cbcc282117fe9bc38338dd"
|
||||
}
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n date as \"date!\",\n timestamp_utc as \"timestamp_utc!\",\n value_json as \"value_json!\"\n FROM summary_history\n ORDER BY date DESC\n LIMIT 30",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "date!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "timestamp_utc!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "value_json!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "7600823da7ce80b8ffda933608603a2752e28df775d1af8fd943a5fc8d7dc00d"
|
||||
}
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM mixnode_packet_stats_raw\n WHERE mix_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "packets_received!: _",
|
||||
"ordinal": 0,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_sent!: _",
|
||||
"ordinal": 1,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_dropped!: _",
|
||||
"ordinal": 2,
|
||||
"type_info": "Null"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "786b6a5d954e38b204cebf322711c74c8cf1c08e5e2896a1d6d5b85c91991753"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO summary_history\n (date, timestamp_utc, value_json)\n VALUES (?, ?, ?)\n ON CONFLICT(date) DO UPDATE SET\n timestamp_utc=excluded.timestamp_utc,\n value_json=excluded.value_json;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "788515c34588aec352773df4b6e6c5e41f3c0bb56a27648b5e25466b8634a578"
|
||||
}
|
||||
+5
-5
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT count(id) FROM gateways",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count(id)",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "86ff64db477a1d6235179b0b88d86b86d1b9be62336c9eac0eef44987a5451b5"
|
||||
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM nym_nodes_packet_stats_raw\n WHERE node_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "packets_received!: _",
|
||||
"ordinal": 0,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_sent!: _",
|
||||
"ordinal": 1,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_dropped!: _",
|
||||
"ordinal": 2,
|
||||
"type_info": "Null"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "8bdf85a61e443fa5f4835bffd0bffc8ed1011f56714fde6007e50951e569854b"
|
||||
}
|
||||
+6
-6
@@ -1,21 +1,21 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n gateway_identity_key as \"gateway_identity_key!\",\n bonded as \"bonded: bool\"\n FROM gateways\n ORDER BY last_testrun_utc",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
"name": "gateway_identity_key!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "bonded: bool",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
|
||||
-86
@@ -1,86 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ed25519_identity_pubkey",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_addresses!: serde_json::Value",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "mix_port",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "x25519_sphinx_pubkey",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "node_role: serde_json::Value",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "supported_roles: serde_json::Value",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "entry: serde_json::Value",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "performance",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "9334f0c91252fcd7ec72558a271222615bb282e5334665700709ae475a5daea2"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO gateways\n (gateway_identity_key, bonded,\n self_described, explorer_pretty_bond,\n last_updated_utc, performance)\n VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(gateway_identity_key) DO UPDATE SET\n bonded=excluded.bonded,\n self_described=excluded.self_described,\n explorer_pretty_bond=excluded.explorer_pretty_bond,\n last_updated_utc=excluded.last_updated_utc,\n performance = excluded.performance;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "9796d354ae075eab4cbd3438839c39da94025494395ec7b093aefef696f2d0c5"
|
||||
}
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE gateway_id = ? AND status != 2\n ORDER BY id DESC\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "status!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "created_utc!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_address!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "log!",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_assigned_utc",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "a1d9eb816acd1a91ed0975c801c9295c01a78861a2a0597ad28b1579a14bf008"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n COUNT(id) as \"count: i64\"\n FROM testruns\n WHERE\n status = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count: i64",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "a79a61b87325f3f1d9c5a4fb386ccd585be0641e5878acb6283b879f22ed2b4c"
|
||||
}
|
||||
-44
@@ -1,44 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n nd.node_id,\n moniker,\n website,\n security_contact,\n details\n FROM\n nym_node_descriptions nd\n INNER JOIN\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "moniker",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "website",
|
||||
"ordinal": 2,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "security_contact",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "details",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "aa602604c0e7b6eef41ea3cd83c16610e15be2d7ee3f6c4d3debf23f95fb9c2e"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE nym_nodes\n SET\n self_described = NULL,\n bond_info = NULL",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "b68796d1d8d2384b30f1aace06269682c4ae96f774261f5c298264d3c12e5b67"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c214c001acbbf79fa499816f36ec586c4c29c03efb4cf0c40b73a5c76159cf5c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n mixnodes\n SET\n bonded = false\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c42917c9542c1d720d92035863064741aefc9f7a7d1630f6b863ebd8174b6684"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n self_described as \"self_described: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n ORDER BY\n node_id\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "c7656b2b1b4328415772ce69d0568bd5438d6c8496ca9cbdcfb70bb5375b345e"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT mix_id\n FROM mixnodes\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "mix_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "c910788edefe64bbb34379702bcbde9ec6159c9fa03b13652e1f620dcd92125e"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "DELETE FROM gateway_session_stats WHERE day <= ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c9e61180ec35dfab8623333fafa3b216b93440d0fddc0a37dd1b6c1813741f6a"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n total_stake\n FROM nym_nodes\n WHERE node_id = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "d2e07d44594ca5b44a6100482ff432c39d761f2a0ac1d6515cf73416f2eb6c61"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO summary\n (key, value_json, last_updated_utc)\n VALUES (?, ?, ?)\n ON CONFLICT(key) DO UPDATE SET\n value_json=excluded.value_json,\n last_updated_utc=excluded.last_updated_utc;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "e0c76a959276e3b0f44c720af9c74a5bf4912ee73468e62e7d0d96b1d9074cbe"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO gateway_description (\n gateway_identity_key,\n moniker,\n website,\n security_contact,\n details,\n last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (gateway_identity_key) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "e9790b63ebe4bff5172bb8cb7bfc288364855003cf0e4d63e95047e7b502c650"
|
||||
}
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE\n id = ?\n AND\n status = ?\n ORDER BY created_utc\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "status!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "created_utc!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_address!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "log!",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_assigned_utc",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "f0c64794cbaed87a1d3166251d8e6720c9a9fc929144188460849be85d915004"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n gateways\n SET\n bonded = false\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "f7e3fa31d68c028bf39cc95389f29f8758ec922dd2e7ea064a1e537e580c9ee5"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_nodes_packet_stats_raw (\n node_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "fcb1698d9e0e3a14524c92e7c99a811588c2bbc50d4975487a0464321a1b18c9"
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
[package]
|
||||
name = "nym-node-status-api"
|
||||
version = "3.1.1"
|
||||
version = "3.1.3"
|
||||
authors.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
@@ -28,11 +28,15 @@ moka = { workspace = true, features = ["future"] }
|
||||
# Nym API: revert after Cheddar is out
|
||||
nym-contracts-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-mixnet-contract-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["openapi"]}
|
||||
nym-node-status-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"openapi",
|
||||
] }
|
||||
nym-node-status-client = { path = "../nym-node-status-client" }
|
||||
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["middleware"]}
|
||||
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"middleware",
|
||||
] }
|
||||
nym-network-defaults = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-serde-helpers = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-statistics-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
@@ -62,7 +66,7 @@ serde_json = { workspace = true }
|
||||
serde_json_path = { workspace = true }
|
||||
strum = { workspace = true }
|
||||
strum_macros = { workspace = true }
|
||||
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "sqlite", "time"] }
|
||||
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "time"] }
|
||||
thiserror = { workspace = true }
|
||||
time = { workspace = true, features = ["formatting"] }
|
||||
tokio = { workspace = true, features = ["rt-multi-thread"] }
|
||||
@@ -77,13 +81,16 @@ utoipauto = { workspace = true }
|
||||
|
||||
nym-node-metrics = { path = "../../nym-node/nym-node-metrics" }
|
||||
|
||||
[features]
|
||||
default = ["pg"]
|
||||
sqlite = ["sqlx/sqlite"]
|
||||
pg = ["sqlx/postgres"]
|
||||
|
||||
[build-dependencies]
|
||||
anyhow = { workspace = true }
|
||||
tokio = { workspace = true, features = ["macros"] }
|
||||
sqlx = { workspace = true, features = [
|
||||
"runtime-tokio-rustls",
|
||||
"sqlite",
|
||||
"macros",
|
||||
"migrate",
|
||||
] }
|
||||
|
||||
@@ -0,0 +1,117 @@
|
||||
# Makefile for nym-node-status-api database management
|
||||
|
||||
# --- Configuration ---
|
||||
TEST_DATABASE_URL := postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# Docker compose service names
|
||||
DB_SERVICE_NAME := postgres-test
|
||||
DB_CONTAINER_NAME := nym_node_status_api_postgres_test
|
||||
|
||||
# Default target
|
||||
.PHONY: default
|
||||
default: help
|
||||
|
||||
# --- Main Targets ---
|
||||
.PHONY: prepare-pg
|
||||
prepare-pg: test-db-up test-db-wait test-db-migrate test-db-prepare test-db-down ## Setup PostgreSQL and prepare SQLx offline cache
|
||||
|
||||
.PHONY: test-db
|
||||
test-db: test-db-up test-db-wait test-db-migrate test-db-run test-db-down ## Run tests with PostgreSQL database
|
||||
|
||||
.PHONY: dev-db
|
||||
dev-db: test-db-up test-db-wait test-db-migrate ## Start PostgreSQL for development (keeps running)
|
||||
@echo "PostgreSQL is running on port 5433"
|
||||
@echo "Connection string: $(TEST_DATABASE_URL)"
|
||||
|
||||
# --- Docker Compose Targets ---
|
||||
.PHONY: test-db-up
|
||||
test-db-up: ## Start the PostgreSQL test database in the background
|
||||
@echo "Starting PostgreSQL test database..."
|
||||
docker compose up -d $(DB_SERVICE_NAME)
|
||||
|
||||
.PHONY: test-db-wait
|
||||
test-db-wait: ## Wait for the PostgreSQL database to be healthy
|
||||
@echo "Waiting for PostgreSQL database..."
|
||||
@while ! docker inspect --format='{{.State.Health.Status}}' $(DB_CONTAINER_NAME) 2>/dev/null | grep -q 'healthy'; do \
|
||||
echo -n "."; \
|
||||
sleep 1; \
|
||||
done; \
|
||||
echo " Database is healthy!"
|
||||
|
||||
.PHONY: test-db-down
|
||||
test-db-down: ## Stop and remove the test database
|
||||
@echo "Stopping PostgreSQL test database..."
|
||||
docker compose down
|
||||
|
||||
# --- SQLx Targets ---
|
||||
.PHONY: test-db-migrate
|
||||
test-db-migrate: ## Run database migrations against PostgreSQL
|
||||
@echo "Running PostgreSQL migrations..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" sqlx migrate run --source migrations_pg
|
||||
|
||||
.PHONY: test-db-prepare
|
||||
test-db-prepare: ## Run sqlx prepare for compile-time query verification
|
||||
@echo "Running sqlx prepare for PostgreSQL..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" cargo sqlx prepare -- --features pg
|
||||
|
||||
# --- Build and Test Targets ---
|
||||
.PHONY: test-db-run
|
||||
test-db-run: ## Run tests with PostgreSQL feature
|
||||
@echo "Running tests with PostgreSQL..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" cargo test --features pg --no-default-features
|
||||
|
||||
.PHONY: build-pg
|
||||
build-pg: ## Build with PostgreSQL feature
|
||||
@echo "Building with PostgreSQL feature..."
|
||||
cargo build --features pg --no-default-features
|
||||
|
||||
.PHONY: build-sqlite
|
||||
build-sqlite: ## Build with SQLite feature (default)
|
||||
@echo "Building with SQLite feature..."
|
||||
cargo build --features sqlite --no-default-features
|
||||
|
||||
.PHONY: check-pg
|
||||
check-pg: ## Check code with PostgreSQL feature
|
||||
@echo "Checking code with PostgreSQL feature..."
|
||||
cargo check --features pg --no-default-features
|
||||
|
||||
.PHONY: check-sqlite
|
||||
check-sqlite: ## Check code with SQLite feature
|
||||
@echo "Checking code with SQLite feature..."
|
||||
cargo check --features sqlite --no-default-features
|
||||
|
||||
.PHONY: clippy
|
||||
clippy: clippy-pg clippy-sqlite
|
||||
|
||||
.PHONY: clippy-pg
|
||||
clippy-pg: ## Run clippy with PostgreSQL feature
|
||||
@echo "Running clippy with PostgreSQL feature..."
|
||||
cargo clippy --features pg --no-default-features -- -D warnings
|
||||
|
||||
.PHONY: clippy-sqlite
|
||||
clippy-sqlite: ## Run clippy with SQLite feature (default)
|
||||
@echo "Running clippy with SQLite feature..."
|
||||
cargo clippy --features sqlite --no-default-features -- -D warnings
|
||||
|
||||
# --- Cleanup Targets ---
|
||||
.PHONY: clean
|
||||
clean: ## Clean build artifacts and SQLx cache
|
||||
cargo clean
|
||||
rm -rf .sqlx
|
||||
|
||||
.PHONY: clean-db
|
||||
clean-db: test-db-down ## Stop database and clean volumes
|
||||
docker volume rm -f nym-node-status-api_postgres_test_data 2>/dev/null || true
|
||||
|
||||
# --- Utility Targets ---
|
||||
.PHONY: sqlx-cli
|
||||
sqlx-cli: ## Install sqlx-cli if not already installed
|
||||
@command -v sqlx >/dev/null 2>&1 || cargo install sqlx-cli --features postgres,sqlite
|
||||
|
||||
.PHONY: psql
|
||||
psql: ## Connect to the running PostgreSQL database with psql
|
||||
@docker exec -it $(DB_CONTAINER_NAME) psql -U testuser -d nym_node_status_api_test
|
||||
|
||||
.PHONY: help
|
||||
help: ## Show help for Makefile targets
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
|
||||
@@ -0,0 +1,104 @@
|
||||
# PostgreSQL Support for nym-node-status-api
|
||||
|
||||
This project now supports both SQLite (default) and PostgreSQL databases.
|
||||
|
||||
## Quick Start with PostgreSQL
|
||||
|
||||
### 1. Install Prerequisites
|
||||
|
||||
```bash
|
||||
# Install sqlx-cli if not already installed
|
||||
make sqlx-cli
|
||||
```
|
||||
|
||||
### 2. Prepare PostgreSQL for Development
|
||||
|
||||
```bash
|
||||
# This will:
|
||||
# - Start PostgreSQL in Docker
|
||||
# - Run migrations
|
||||
# - Generate SQLx offline query cache
|
||||
# - Stop the database
|
||||
make prepare-pg
|
||||
```
|
||||
|
||||
### 3. Build with PostgreSQL
|
||||
|
||||
```bash
|
||||
# Build with PostgreSQL feature
|
||||
make build-pg
|
||||
|
||||
# Or manually:
|
||||
cargo build --features pg --no-default-features
|
||||
```
|
||||
|
||||
### 4. Run with PostgreSQL
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL for development (keeps running)
|
||||
make dev-db
|
||||
|
||||
# In another terminal, run the application
|
||||
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test \
|
||||
cargo run --features pg --no-default-features
|
||||
```
|
||||
|
||||
## Database Features
|
||||
|
||||
- `sqlite` (default): Uses SQLite database
|
||||
- `pg`: Uses PostgreSQL database
|
||||
|
||||
Only one database feature can be active at a time.
|
||||
|
||||
## Migration Differences
|
||||
|
||||
SQLite migrations are in `migrations/`, PostgreSQL migrations are in `migrations_pg/`.
|
||||
|
||||
Key differences:
|
||||
- **AUTOINCREMENT** → **SERIAL**
|
||||
- **INTEGER CHECK (0,1)** → **BOOLEAN**
|
||||
- **REAL** → **DOUBLE PRECISION**
|
||||
- No table recreation needed for constraint changes in PostgreSQL
|
||||
|
||||
## Makefile Targets
|
||||
|
||||
```bash
|
||||
make help # Show all available targets
|
||||
make prepare-pg # Setup PostgreSQL and prepare SQLx cache
|
||||
make dev-db # Start PostgreSQL for development
|
||||
make test-db # Run tests with PostgreSQL
|
||||
make build-pg # Build with PostgreSQL
|
||||
make build-sqlite # Build with SQLite
|
||||
make psql # Connect to running PostgreSQL
|
||||
make clean # Clean build artifacts
|
||||
make clean-db # Stop database and clean volumes
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See `.env.example` for all configuration options. Key variable:
|
||||
|
||||
```bash
|
||||
# For PostgreSQL:
|
||||
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# For SQLite:
|
||||
DATABASE_URL=sqlite://nym-node-status-api.sqlite
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SQLx Offline Mode
|
||||
|
||||
If you see "no cached data for this query" errors:
|
||||
|
||||
1. Ensure PostgreSQL is running: `make dev-db`
|
||||
2. Run: `make test-db-prepare`
|
||||
|
||||
### Connection Refused
|
||||
|
||||
If you see "Connection refused" errors:
|
||||
|
||||
1. Check Docker is running: `docker ps`
|
||||
2. Check PostgreSQL container: `docker ps | grep nym_node_status_api_postgres_test`
|
||||
3. Restart database: `make test-db-down && make dev-db`
|
||||
@@ -1,17 +1,20 @@
|
||||
use anyhow::{anyhow, Result};
|
||||
use anyhow::Result;
|
||||
#[cfg(feature = "sqlite")]
|
||||
use sqlx::{Connection, SqliteConnection};
|
||||
#[cfg(feature = "sqlite")]
|
||||
#[cfg(target_family = "unix")]
|
||||
use std::fs::Permissions;
|
||||
#[cfg(feature = "sqlite")]
|
||||
#[cfg(target_family = "unix")]
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
#[cfg(feature = "sqlite")]
|
||||
use tokio::{fs::File, io::AsyncWriteExt};
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
const SQLITE_DB_FILENAME: &str = "nym-node-status-api.sqlite";
|
||||
|
||||
/// If you need to re-run migrations or reset the db, just run
|
||||
/// cargo clean -p nym-node-status-api
|
||||
#[tokio::main(flavor = "current_thread")]
|
||||
async fn main() -> Result<()> {
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn init_db() -> Result<()> {
|
||||
let out_dir = read_env_var("OUT_DIR")?;
|
||||
let database_path = format!("{out_dir}/{SQLITE_DB_FILENAME}?mode=rwc");
|
||||
|
||||
@@ -30,11 +33,30 @@ async fn main() -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "pg", not(feature = "sqlite")))]
|
||||
async fn init_db() -> Result<()> {
|
||||
// PostgreSQL doesn't need build-time initialization
|
||||
// Just ensure DATABASE_URL is available for runtime
|
||||
if let Ok(database_url) = std::env::var("DATABASE_URL") {
|
||||
println!("cargo::rustc-env=DATABASE_URL={database_url}");
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// If you need to re-run migrations or reset the db, just run
|
||||
/// cargo clean -p nym-node-status-api
|
||||
#[tokio::main(flavor = "current_thread")]
|
||||
async fn main() -> Result<()> {
|
||||
init_db().await
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
fn read_env_var(var: &str) -> Result<String> {
|
||||
std::env::var(var).map_err(|_| anyhow!("You need to set {} env var", var))
|
||||
std::env::var(var).map_err(|_| anyhow::anyhow!("You need to set {} env var", var))
|
||||
}
|
||||
|
||||
/// use `./enter_db.sh` to inspect DB
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn write_db_path_to_file(out_dir: &str, db_filename: &str) -> anyhow::Result<()> {
|
||||
let mut file = File::create("settings.sql").await?;
|
||||
let settings = ".mode columns
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
services:
|
||||
postgres-test:
|
||||
image: postgres:16-alpine
|
||||
container_name: nym_node_status_api_postgres_test
|
||||
environment:
|
||||
POSTGRES_DB: nym_node_status_api_test
|
||||
POSTGRES_USER: testuser
|
||||
POSTGRES_PASSWORD: testpass
|
||||
ports:
|
||||
- '5433:5432' # Map to 5433 to avoid conflicts with default PostgreSQL
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U testuser -d nym_node_status_api_test']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
# Optional: Add volume for persistent data during development
|
||||
# volumes:
|
||||
# - postgres_test_data:/var/lib/postgresql/data
|
||||
|
||||
# volumes:
|
||||
# postgres_test_data:
|
||||
@@ -0,0 +1,113 @@
|
||||
CREATE TABLE gateways
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR NOT NULL UNIQUE,
|
||||
self_described VARCHAR NOT NULL,
|
||||
explorer_pretty_bond VARCHAR,
|
||||
last_probe_result VARCHAR,
|
||||
last_probe_log VARCHAR,
|
||||
config_score INTEGER NOT NULL DEFAULT 0,
|
||||
config_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
config_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
routing_score INTEGER NOT NULL DEFAULT 0,
|
||||
routing_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
routing_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
test_run_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
last_testrun_utc BIGINT,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
bonded BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
performance INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
|
||||
CREATE INDEX idx_gateway_description_gateway_identity_key ON gateways (gateway_identity_key);
|
||||
|
||||
|
||||
CREATE TABLE mixnodes (
|
||||
id SERIAL PRIMARY KEY,
|
||||
identity_key VARCHAR NOT NULL UNIQUE,
|
||||
mix_id BIGINT NOT NULL UNIQUE,
|
||||
bonded BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
total_stake BIGINT NOT NULL,
|
||||
host VARCHAR NOT NULL,
|
||||
http_api_port BIGINT NOT NULL,
|
||||
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
full_details VARCHAR,
|
||||
self_described VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
is_dp_delegatee BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_mixnodes_mix_id ON mixnodes (mix_id);
|
||||
CREATE INDEX idx_mixnodes_identity_key ON mixnodes (identity_key);
|
||||
|
||||
CREATE TABLE mixnode_description (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
|
||||
);
|
||||
|
||||
-- Indexes for description table
|
||||
CREATE INDEX idx_mixnode_description_mix_id ON mixnode_description (mix_id);
|
||||
|
||||
|
||||
CREATE TABLE summary
|
||||
(
|
||||
key VARCHAR PRIMARY KEY,
|
||||
value_json VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE summary_history
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
date VARCHAR UNIQUE NOT NULL,
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
value_json VARCHAR
|
||||
);
|
||||
|
||||
CREATE INDEX idx_summary_history_timestamp_utc ON summary_history (timestamp_utc);
|
||||
CREATE INDEX idx_summary_history_date ON summary_history (date);
|
||||
|
||||
|
||||
CREATE TABLE gateway_description (
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
FOREIGN KEY (gateway_identity_key) REFERENCES gateways (gateway_identity_key)
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE mixnode_daily_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT NOT NULL,
|
||||
total_stake BIGINT NOT NULL,
|
||||
date_utc VARCHAR NOT NULL,
|
||||
packets_received INTEGER DEFAULT 0,
|
||||
packets_sent INTEGER DEFAULT 0,
|
||||
packets_dropped INTEGER DEFAULT 0,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id),
|
||||
UNIQUE (mix_id, date_utc) -- This constraint automatically creates an index
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE testruns
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_id INTEGER NOT NULL,
|
||||
status INTEGER NOT NULL, -- 0=pending, 1=in-progress, 2=complete
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
ip_address VARCHAR NOT NULL,
|
||||
log VARCHAR NOT NULL,
|
||||
FOREIGN KEY (gateway_id) REFERENCES gateways (id)
|
||||
);
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
ALTER TABLE testruns
|
||||
RENAME COLUMN timestamp_utc TO created_utc;
|
||||
|
||||
ALTER TABLE testruns
|
||||
ADD COLUMN last_assigned_utc BIGINT;
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
CREATE TABLE gateway_session_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR NOT NULL,
|
||||
node_id BIGINT NOT NULL,
|
||||
day DATE NOT NULL,
|
||||
unique_active_clients BIGINT NOT NULL,
|
||||
session_started BIGINT NOT NULL,
|
||||
users_hashes VARCHAR,
|
||||
vpn_sessions VARCHAR,
|
||||
mixnet_sessions VARCHAR,
|
||||
unknown_sessions VARCHAR,
|
||||
UNIQUE (node_id, day) -- This constraint automatically creates an index
|
||||
);
|
||||
|
||||
CREATE INDEX idx_gateway_session_stats_identity_key ON gateway_session_stats (gateway_identity_key);
|
||||
CREATE INDEX idx_gateway_session_stats_day ON gateway_session_stats (day);
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
CREATE TABLE mixnode_packet_stats_raw (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT NOT NULL,
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
packets_received INTEGER,
|
||||
packets_sent INTEGER,
|
||||
packets_dropped INTEGER,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_mixnode_packet_stats_raw_mix_id_timestamp_utc ON mixnode_packet_stats_raw (mix_id, timestamp_utc);
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
ALTER TABLE mixnodes DROP COLUMN blacklisted;
|
||||
ALTER TABLE gateways DROP COLUMN blacklisted;
|
||||
|
||||
CREATE TABLE nym_nodes (
|
||||
node_id INTEGER PRIMARY KEY,
|
||||
ed25519_identity_pubkey VARCHAR NOT NULL UNIQUE,
|
||||
total_stake BIGINT NOT NULL,
|
||||
ip_addresses TEXT NOT NULL,
|
||||
mix_port INTEGER NOT NULL,
|
||||
x25519_sphinx_pubkey VARCHAR NOT NULL UNIQUE,
|
||||
node_role TEXT NOT NULL,
|
||||
supported_roles TEXT NOT NULL,
|
||||
performance VARCHAR NOT NULL,
|
||||
entry TEXT,
|
||||
last_updated_utc INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_nym_nodes_node_id ON nym_nodes (node_id);
|
||||
CREATE INDEX idx_nym_nodes_ed25519_identity_pubkey ON nym_nodes (ed25519_identity_pubkey);
|
||||
|
||||
CREATE TABLE nym_node_descriptions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc INTEGER NOT NULL,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
|
||||
);
|
||||
|
||||
CREATE TABLE nym_nodes_packet_stats_raw (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
timestamp_utc INTEGER NOT NULL,
|
||||
packets_received INTEGER,
|
||||
packets_sent INTEGER,
|
||||
packets_dropped INTEGER,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_nym_nodes_packet_stats_raw_node_id_timestamp_utc ON nym_nodes_packet_stats_raw (node_id, timestamp_utc);
|
||||
|
||||
CREATE TABLE nym_node_daily_mixing_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
total_stake BIGINT NOT NULL,
|
||||
date_utc VARCHAR NOT NULL,
|
||||
packets_received INTEGER DEFAULT 0,
|
||||
packets_sent INTEGER DEFAULT 0,
|
||||
packets_dropped INTEGER DEFAULT 0,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id),
|
||||
UNIQUE (node_id, date_utc) -- This constraint automatically creates an index
|
||||
);
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
ALTER TABLE nym_nodes ADD COLUMN self_described TEXT;
|
||||
ALTER TABLE nym_nodes ADD COLUMN bond_info TEXT;
|
||||
|
||||
-- PostgreSQL doesn't need table recreation for adding ON DELETE CASCADE
|
||||
-- We can drop and recreate the foreign key constraints directly
|
||||
|
||||
-- Drop existing foreign key constraints
|
||||
ALTER TABLE nym_node_descriptions DROP CONSTRAINT IF EXISTS nym_node_descriptions_node_id_fkey;
|
||||
ALTER TABLE nym_nodes_packet_stats_raw DROP CONSTRAINT IF EXISTS nym_nodes_packet_stats_raw_node_id_fkey;
|
||||
ALTER TABLE nym_node_daily_mixing_stats DROP CONSTRAINT IF EXISTS nym_node_daily_mixing_stats_node_id_fkey;
|
||||
|
||||
-- Add foreign key constraints with ON DELETE CASCADE
|
||||
ALTER TABLE nym_node_descriptions
|
||||
ADD CONSTRAINT nym_node_descriptions_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
|
||||
ALTER TABLE nym_nodes_packet_stats_raw
|
||||
ADD CONSTRAINT nym_nodes_packet_stats_raw_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
|
||||
ALTER TABLE nym_node_daily_mixing_stats
|
||||
ADD CONSTRAINT nym_node_daily_mixing_stats_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
-- Removing UNIQUE constraints on nym_nodes
|
||||
-- In PostgreSQL, we can drop constraints directly without recreating the table
|
||||
|
||||
-- Drop the unique constraints
|
||||
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_ed25519_identity_pubkey_key;
|
||||
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_x25519_sphinx_pubkey_key;
|
||||
|
||||
-- The columns and indexes remain, only the unique constraints are removed
|
||||
-- The existing indexes idx_nym_nodes_node_id and idx_nym_nodes_ed25519_identity_pubkey remain unchanged
|
||||
@@ -0,0 +1,112 @@
|
||||
-- for a couple of days after migrating chrono -> time, we stored dates as
|
||||
-- 2025-June-DD instead of 2025-06-DD. This migration fixes those entries.
|
||||
--
|
||||
-- Because of a UNIQUE constraint on (node_id, date_utc), we can't just rename in-place.
|
||||
-- - merge (add) node stats back to the original table where conflict (node_id, date_utc) would exist
|
||||
-- - delete invalid records from original table (those stats were merged into correct rows above)
|
||||
-- - insert rows that did not have a conflicting (node_Id, date_utc) combo
|
||||
-- Conflicts affect only the date which has both kinds of entries,
|
||||
-- e.g. 2025-06-05 and 2025-June-05 (date when this change was deployed)
|
||||
--
|
||||
-- This applies to both affected tables.
|
||||
|
||||
-- ----------------------------------------
|
||||
-- mixnode_daily_stats
|
||||
-- ----------------------------------------
|
||||
|
||||
-- First, copy over rows with invalid date to a temp table (in the correct date format)
|
||||
CREATE TEMP TABLE tmp_mix AS
|
||||
SELECT
|
||||
mix_id,
|
||||
REPLACE(date_utc,'June','06') AS new_date,
|
||||
SUM(total_stake) AS total_stake_sum,
|
||||
SUM(packets_received) AS packets_received_sum,
|
||||
SUM(packets_sent) AS packets_sent_sum,
|
||||
SUM(packets_dropped) AS packets_dropped_sum
|
||||
FROM mixnode_daily_stats
|
||||
WHERE date_utc LIKE '%June%'
|
||||
GROUP BY mix_id, new_date;
|
||||
|
||||
UPDATE mixnode_daily_stats AS m
|
||||
SET
|
||||
total_stake = m.total_stake,
|
||||
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
|
||||
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
|
||||
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc)
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM tmp_mix
|
||||
WHERE mix_id = m.mix_id
|
||||
AND new_date = m.date_utc
|
||||
);
|
||||
|
||||
DELETE FROM mixnode_daily_stats
|
||||
WHERE date_utc LIKE '%June%';
|
||||
|
||||
INSERT INTO mixnode_daily_stats
|
||||
(mix_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
|
||||
SELECT
|
||||
mix_id,
|
||||
new_date,
|
||||
total_stake_sum,
|
||||
packets_received_sum,
|
||||
packets_sent_sum,
|
||||
packets_dropped_sum
|
||||
FROM tmp_mix AS t
|
||||
-- only those whose new_date did _not_ already exist
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM mixnode_daily_stats AS m
|
||||
WHERE m.mix_id = t.mix_id
|
||||
AND m.date_utc = t.new_date
|
||||
);
|
||||
|
||||
DROP TABLE tmp_mix;
|
||||
|
||||
|
||||
-- ----------------------------------------
|
||||
-- nym_node_daily_mixing_stats
|
||||
-- ----------------------------------------
|
||||
|
||||
CREATE TEMP TABLE tmp_nym_node_stats AS
|
||||
SELECT
|
||||
node_id,
|
||||
REPLACE(date_utc,'June','06') AS new_date,
|
||||
SUM(total_stake) AS total_stake_sum,
|
||||
SUM(packets_received) AS packets_received_sum,
|
||||
SUM(packets_sent) AS packets_sent_sum,
|
||||
SUM(packets_dropped) AS packets_dropped_sum
|
||||
FROM nym_node_daily_mixing_stats
|
||||
WHERE date_utc LIKE '%June%'
|
||||
GROUP BY node_id, new_date;
|
||||
|
||||
UPDATE nym_node_daily_mixing_stats AS m
|
||||
SET
|
||||
total_stake = m.total_stake,
|
||||
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
|
||||
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
|
||||
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc)
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM tmp_nym_node_stats
|
||||
WHERE node_id = m.node_id
|
||||
AND new_date = m.date_utc
|
||||
);
|
||||
|
||||
DELETE FROM nym_node_daily_mixing_stats
|
||||
WHERE date_utc LIKE '%June%';
|
||||
|
||||
INSERT INTO nym_node_daily_mixing_stats
|
||||
(node_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
|
||||
SELECT
|
||||
node_id,
|
||||
new_date,
|
||||
total_stake_sum,
|
||||
packets_received_sum,
|
||||
packets_sent_sum,
|
||||
packets_dropped_sum
|
||||
FROM tmp_nym_node_stats AS t
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM nym_node_daily_mixing_stats AS m
|
||||
WHERE m.node_id = t.node_id
|
||||
AND m.date_utc = t.new_date
|
||||
);
|
||||
|
||||
DROP TABLE tmp_nym_node_stats;
|
||||
@@ -1,24 +1,49 @@
|
||||
use anyhow::{anyhow, Result};
|
||||
use sqlx::{
|
||||
migrate::Migrator,
|
||||
query,
|
||||
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
|
||||
ConnectOptions, SqlitePool,
|
||||
};
|
||||
use std::{str::FromStr, time::Duration};
|
||||
|
||||
pub(crate) mod models;
|
||||
pub(crate) mod queries;
|
||||
pub(crate) mod query_wrapper;
|
||||
|
||||
// Re-export the query wrapper functions for easier access
|
||||
pub(crate) use query_wrapper::query;
|
||||
#[allow(unused_imports)]
|
||||
pub(crate) use query_wrapper::{query_as, query_scalar};
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
use sqlx::{
|
||||
migrate::Migrator,
|
||||
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
|
||||
ConnectOptions, SqlitePool,
|
||||
};
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
use sqlx::{migrate::Migrator, postgres::PgConnectOptions, ConnectOptions, PgPool};
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
static MIGRATOR: Migrator = sqlx::migrate!("./migrations_pg");
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) type DbPool = SqlitePool;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) type DbPool = PgPool;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Sqlite>;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Postgres>;
|
||||
|
||||
pub(crate) struct Storage {
|
||||
pool: DbPool,
|
||||
}
|
||||
|
||||
impl Storage {
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub async fn init(connection_url: String, busy_timeout: Duration) -> Result<Self> {
|
||||
let connect_options = SqliteConnectOptions::from_str(&connection_url)?
|
||||
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal)
|
||||
@@ -41,16 +66,31 @@ impl Storage {
|
||||
Ok(Storage { pool })
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub async fn init(connection_url: String, _busy_timeout: Duration) -> Result<Self> {
|
||||
let connect_options =
|
||||
PgConnectOptions::from_str(&connection_url)?.disable_statement_logging();
|
||||
|
||||
let pool = sqlx::PgPool::connect_with(connect_options)
|
||||
.await
|
||||
.map_err(|err| anyhow!("Failed to connect to {}: {}", &connection_url, err))?;
|
||||
|
||||
MIGRATOR.run(&pool).await?;
|
||||
|
||||
Ok(Storage { pool })
|
||||
}
|
||||
|
||||
/// Cloning pool is cheap, it's the same underlying set of connections
|
||||
pub fn pool_owned(&self) -> DbPool {
|
||||
self.pool.clone()
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn assert_busy_timeout(pool: DbPool, expected_busy_timeout_s: i64) -> Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
// Sqlite stores this value as miliseconds
|
||||
// https://www.sqlite.org/pragma.html#pragma_busy_timeout
|
||||
let busy_timeout_db = query!("PRAGMA busy_timeout;")
|
||||
let busy_timeout_db = sqlx::query!("PRAGMA busy_timeout;")
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -38,7 +38,7 @@ pub(crate) struct GatewayInsertRecord {
|
||||
pub(crate) performance: u8,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub(crate) struct GatewayDto {
|
||||
pub(crate) gateway_identity_key: String,
|
||||
pub(crate) bonded: bool,
|
||||
@@ -121,7 +121,7 @@ pub(crate) struct MixnodeRecord {
|
||||
pub(crate) is_dp_delegatee: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub(crate) struct MixnodeDto {
|
||||
pub(crate) mix_id: i64,
|
||||
pub(crate) bonded: bool,
|
||||
@@ -183,14 +183,14 @@ pub(crate) struct BondedStatusDto {
|
||||
}
|
||||
|
||||
#[allow(unused)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
#[derive(Debug, Clone, Default, FromRow)]
|
||||
pub(crate) struct SummaryDto {
|
||||
pub(crate) key: String,
|
||||
pub(crate) value_json: String,
|
||||
pub(crate) last_updated_utc: i64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
#[derive(Debug, Clone, Default, FromRow)]
|
||||
pub(crate) struct SummaryHistoryDto {
|
||||
#[allow(dead_code)]
|
||||
pub id: i64,
|
||||
@@ -287,7 +287,7 @@ pub(crate) mod gateway {
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // not dead code, this is SQL data model
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub struct TestRunDto {
|
||||
pub id: i64,
|
||||
pub gateway_id: i64,
|
||||
@@ -313,7 +313,7 @@ pub struct GatewayIdentityDto {
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub struct GatewayInfoDto {
|
||||
pub id: i64,
|
||||
pub gateway_identity_key: String,
|
||||
@@ -362,7 +362,7 @@ impl TryFrom<GatewaySessionsRecord> for http::models::SessionStats {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(strum_macros::Display)]
|
||||
#[derive(strum_macros::Display, Clone)]
|
||||
pub(crate) enum ScrapeNodeKind {
|
||||
LegacyMixnode { mix_id: i64 },
|
||||
MixingNymNode { node_id: i64 },
|
||||
@@ -379,6 +379,7 @@ impl ScrapeNodeKind {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct ScraperNodeInfo {
|
||||
pub node_kind: ScrapeNodeKind,
|
||||
pub hosts: Vec<String>,
|
||||
@@ -514,7 +515,7 @@ impl TryFrom<NymNodeDto> for SkimmedNode {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize, sqlx::Decode)]
|
||||
#[derive(Debug, Serialize, Deserialize, sqlx::Decode, FromRow)]
|
||||
pub struct NodeStats {
|
||||
pub packets_received: i64,
|
||||
pub packets_sent: i64,
|
||||
|
||||
@@ -3,32 +3,32 @@ use std::collections::HashSet;
|
||||
use crate::{
|
||||
db::{
|
||||
models::{GatewayDto, GatewayInsertRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
http::models::Gateway,
|
||||
mixnet_scraper::helpers::NodeDescriptionResponse,
|
||||
};
|
||||
use futures_util::TryStreamExt;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use tracing::error;
|
||||
|
||||
pub(crate) async fn select_gateway_identity(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i64,
|
||||
) -> anyhow::Result<String> {
|
||||
let record = sqlx::query!(
|
||||
let record = crate::db::query(
|
||||
r#"SELECT
|
||||
gateway_identity_key
|
||||
FROM
|
||||
gateways
|
||||
WHERE
|
||||
id = ?"#,
|
||||
gateway_pk
|
||||
)
|
||||
.bind(gateway_pk)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(record.gateway_identity_key)
|
||||
Ok(record.try_get("gateway_identity_key")?)
|
||||
}
|
||||
|
||||
pub(crate) async fn update_bonded_gateways(
|
||||
@@ -37,7 +37,7 @@ pub(crate) async fn update_bonded_gateways(
|
||||
) -> anyhow::Result<()> {
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"UPDATE
|
||||
gateways
|
||||
SET
|
||||
@@ -48,7 +48,7 @@ pub(crate) async fn update_bonded_gateways(
|
||||
.await?;
|
||||
|
||||
for record in gateways {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO gateways
|
||||
(gateway_identity_key, bonded,
|
||||
self_described, explorer_pretty_bond,
|
||||
@@ -60,13 +60,13 @@ pub(crate) async fn update_bonded_gateways(
|
||||
explorer_pretty_bond=excluded.explorer_pretty_bond,
|
||||
last_updated_utc=excluded.last_updated_utc,
|
||||
performance = excluded.performance;",
|
||||
record.identity_key,
|
||||
record.bonded,
|
||||
record.self_described,
|
||||
record.explorer_pretty_bond,
|
||||
record.last_updated_utc,
|
||||
record.performance
|
||||
)
|
||||
.bind(record.identity_key)
|
||||
.bind(record.bonded)
|
||||
.bind(record.self_described)
|
||||
.bind(record.explorer_pretty_bond)
|
||||
.bind(record.last_updated_utc)
|
||||
.bind(record.performance as i8)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
@@ -78,22 +78,21 @@ pub(crate) async fn update_bonded_gateways(
|
||||
|
||||
pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gateway>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as!(
|
||||
GatewayDto,
|
||||
let items = crate::db::query_as::<GatewayDto>(
|
||||
r#"SELECT
|
||||
gw.gateway_identity_key as "gateway_identity_key!",
|
||||
gw.bonded as "bonded: bool",
|
||||
gw.performance as "performance!",
|
||||
gw.self_described as "self_described?",
|
||||
gw.explorer_pretty_bond as "explorer_pretty_bond?",
|
||||
gw.last_probe_result as "last_probe_result?",
|
||||
gw.last_probe_log as "last_probe_log?",
|
||||
gw.last_testrun_utc as "last_testrun_utc?",
|
||||
gw.last_updated_utc as "last_updated_utc!",
|
||||
COALESCE(gd.moniker, "NA") as "moniker!",
|
||||
COALESCE(gd.website, "NA") as "website!",
|
||||
COALESCE(gd.security_contact, "NA") as "security_contact!",
|
||||
COALESCE(gd.details, "NA") as "details!"
|
||||
gw.gateway_identity_key,
|
||||
gw.bonded,
|
||||
gw.performance,
|
||||
gw.self_described,
|
||||
gw.explorer_pretty_bond,
|
||||
gw.last_probe_result,
|
||||
gw.last_probe_log,
|
||||
gw.last_testrun_utc,
|
||||
gw.last_updated_utc,
|
||||
COALESCE(gd.moniker, 'NA') as moniker,
|
||||
COALESCE(gd.website, 'NA') as website,
|
||||
COALESCE(gd.security_contact, 'NA') as security_contact,
|
||||
COALESCE(gd.details, 'NA') as details
|
||||
FROM gateways gw
|
||||
LEFT JOIN gateway_description gd
|
||||
ON gw.gateway_identity_key = gd.gateway_identity_key
|
||||
@@ -114,29 +113,29 @@ pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gatewa
|
||||
|
||||
pub(crate) async fn get_bonded_gateway_id_keys(pool: &DbPool) -> anyhow::Result<HashSet<String>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query!(
|
||||
let items = crate::db::query(
|
||||
r#"
|
||||
SELECT gateway_identity_key
|
||||
FROM gateways
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|record| record.gateway_identity_key)
|
||||
.map(|record| record.try_get::<String, _>("gateway_identity_key").unwrap())
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_gateway_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
identity_key: &str,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
identity_key: String,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO gateway_description (
|
||||
gateway_identity_key,
|
||||
@@ -153,15 +152,54 @@ pub(crate) async fn insert_gateway_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
identity_key,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(identity_key)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_or_create_gateway(
|
||||
conn: &mut DbConnection,
|
||||
gateway_identity_key: &str,
|
||||
) -> anyhow::Result<i64> {
|
||||
// Try to find existing gateway
|
||||
let existing = crate::db::query("SELECT id FROM gateways WHERE gateway_identity_key = ?")
|
||||
.bind(gateway_identity_key.to_string())
|
||||
.fetch_optional(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
if let Some(row) = existing {
|
||||
return Ok(row.try_get("id")?);
|
||||
}
|
||||
|
||||
// Create new gateway
|
||||
tracing::info!("Creating new gateway record for {}", gateway_identity_key);
|
||||
let now = crate::utils::now_utc().unix_timestamp();
|
||||
|
||||
let result = crate::db::query(
|
||||
r#"INSERT INTO gateways (
|
||||
gateway_identity_key,
|
||||
bonded,
|
||||
performance,
|
||||
self_described,
|
||||
last_updated_utc
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
RETURNING id"#,
|
||||
)
|
||||
.bind(gateway_identity_key.to_string())
|
||||
.bind(true) // Assume bonded since being tested
|
||||
.bind(0) // Initial performance
|
||||
.bind("null")
|
||||
.bind(now)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(result.try_get("id")?)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ use futures_util::TryStreamExt;
|
||||
use time::Date;
|
||||
use tracing::error;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) async fn insert_session_records(
|
||||
pool: &DbPool,
|
||||
records: Vec<GatewaySessionsRecord>,
|
||||
@@ -36,6 +37,38 @@ pub(crate) async fn insert_session_records(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) async fn insert_session_records(
|
||||
pool: &DbPool,
|
||||
records: Vec<GatewaySessionsRecord>,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut tx = pool.begin().await?;
|
||||
for record in records {
|
||||
sqlx::query!(
|
||||
"INSERT INTO gateway_session_stats
|
||||
(gateway_identity_key, node_id, day,
|
||||
unique_active_clients, session_started, users_hashes,
|
||||
vpn_sessions, mixnet_sessions, unknown_sessions)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT DO NOTHING",
|
||||
record.gateway_identity_key,
|
||||
record.node_id,
|
||||
record.day,
|
||||
record.unique_active_clients,
|
||||
record.session_started,
|
||||
record.users_hashes,
|
||||
record.vpn_sessions,
|
||||
record.mixnet_sessions,
|
||||
record.unknown_sessions,
|
||||
)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<SessionStats>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as(
|
||||
@@ -68,7 +101,8 @@ pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<Sess
|
||||
|
||||
pub(crate) async fn delete_old_records(pool: &DbPool, cut_off: Date) -> anyhow::Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
sqlx::query!("DELETE FROM gateway_session_stats WHERE day <= ?", cut_off)
|
||||
crate::db::query("DELETE FROM gateway_session_stats WHERE day <= ?")
|
||||
.bind(cut_off)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
Ok(())
|
||||
|
||||
@@ -6,8 +6,8 @@ use crate::db::{models::NetworkSummary, DbPool};
|
||||
/// `daily_summary`
|
||||
pub(crate) async fn insert_summaries(
|
||||
pool: &DbPool,
|
||||
summaries: &[(&str, usize)],
|
||||
summary: &NetworkSummary,
|
||||
summaries: Vec<(String, usize)>,
|
||||
summary: NetworkSummary,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
insert_summary(pool, summaries, last_updated).await?;
|
||||
@@ -19,7 +19,7 @@ pub(crate) async fn insert_summaries(
|
||||
|
||||
async fn insert_summary(
|
||||
pool: &DbPool,
|
||||
summaries: &[(&str, usize)],
|
||||
summaries: Vec<(String, usize)>,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
let timestamp = last_updated.unix_timestamp();
|
||||
@@ -27,17 +27,17 @@ async fn insert_summary(
|
||||
|
||||
for (kind, value) in summaries {
|
||||
let value = value.to_string();
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO summary
|
||||
(key, value_json, last_updated_utc)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET
|
||||
value_json=excluded.value_json,
|
||||
last_updated_utc=excluded.last_updated_utc;",
|
||||
kind,
|
||||
value,
|
||||
timestamp
|
||||
)
|
||||
.bind(kind.clone())
|
||||
.bind(value)
|
||||
.bind(timestamp)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.map_err(|err| {
|
||||
@@ -60,7 +60,7 @@ async fn insert_summary(
|
||||
/// This is not aggregate data, it's a set of latest data points
|
||||
async fn insert_summary_history(
|
||||
pool: &DbPool,
|
||||
summary: &NetworkSummary,
|
||||
summary: NetworkSummary,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
@@ -70,17 +70,17 @@ async fn insert_summary_history(
|
||||
|
||||
let date = datetime_to_only_date_str(last_updated);
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO summary_history
|
||||
(date, timestamp_utc, value_json)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(date) DO UPDATE SET
|
||||
timestamp_utc=excluded.timestamp_utc,
|
||||
value_json=excluded.value_json;",
|
||||
date,
|
||||
timestamp,
|
||||
value_json
|
||||
)
|
||||
.bind(date)
|
||||
.bind(timestamp)
|
||||
.bind(value_json)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
use std::collections::HashSet;
|
||||
|
||||
use futures_util::TryStreamExt;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use tracing::error;
|
||||
|
||||
use crate::{
|
||||
db::{
|
||||
models::{MixnodeDto, MixnodeRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
http::models::{DailyStats, Mixnode},
|
||||
mixnet_scraper::helpers::NodeDescriptionResponse,
|
||||
@@ -20,7 +20,7 @@ pub(crate) async fn update_mixnodes(
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
// mark all as unbonded
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"UPDATE
|
||||
mixnodes
|
||||
SET
|
||||
@@ -31,9 +31,9 @@ pub(crate) async fn update_mixnodes(
|
||||
.await?;
|
||||
|
||||
// existing nodes get updated on insert
|
||||
for record in mixnodes.iter() {
|
||||
for record in mixnodes.into_iter() {
|
||||
// https://www.sqlite.org/lang_upsert.html
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO mixnodes
|
||||
(mix_id, identity_key, bonded, total_stake,
|
||||
host, http_api_port, full_details,
|
||||
@@ -46,17 +46,17 @@ pub(crate) async fn update_mixnodes(
|
||||
full_details=excluded.full_details,self_described=excluded.self_described,
|
||||
last_updated_utc=excluded.last_updated_utc,
|
||||
is_dp_delegatee = excluded.is_dp_delegatee;",
|
||||
record.mix_id,
|
||||
record.identity_key,
|
||||
record.bonded,
|
||||
record.total_stake,
|
||||
record.host,
|
||||
record.http_port,
|
||||
record.full_details,
|
||||
record.self_described,
|
||||
record.last_updated_utc,
|
||||
record.is_dp_delegatee
|
||||
)
|
||||
.bind(record.mix_id as i64)
|
||||
.bind(record.identity_key)
|
||||
.bind(record.bonded)
|
||||
.bind(record.total_stake)
|
||||
.bind(record.host)
|
||||
.bind(record.http_port as i32)
|
||||
.bind(record.full_details)
|
||||
.bind(record.self_described)
|
||||
.bind(record.last_updated_utc)
|
||||
.bind(record.is_dp_delegatee)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
@@ -78,10 +78,10 @@ pub(crate) async fn get_all_mixnodes(pool: &DbPool) -> anyhow::Result<Vec<Mixnod
|
||||
mn.full_details as "full_details!",
|
||||
mn.self_described as "self_described",
|
||||
mn.last_updated_utc as "last_updated_utc!",
|
||||
COALESCE(md.moniker, "NA") as "moniker!",
|
||||
COALESCE(md.website, "NA") as "website!",
|
||||
COALESCE(md.security_contact, "NA") as "security_contact!",
|
||||
COALESCE(md.details, "NA") as "details!"
|
||||
md.moniker as "moniker!",
|
||||
md.website as "website!",
|
||||
md.security_contact as "security_contact!",
|
||||
md.details as "details!"
|
||||
FROM mixnodes mn
|
||||
LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id
|
||||
ORDER BY mn.mix_id"#
|
||||
@@ -144,29 +144,29 @@ pub(crate) async fn get_daily_stats(pool: &DbPool) -> anyhow::Result<Vec<DailySt
|
||||
|
||||
pub(crate) async fn get_bonded_mix_ids(pool: &DbPool) -> anyhow::Result<HashSet<i64>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query!(
|
||||
let items = crate::db::query(
|
||||
r#"
|
||||
SELECT mix_id
|
||||
FROM mixnodes
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|record| record.mix_id)
|
||||
.map(|record| record.try_get::<i64, _>("mix_id").unwrap())
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_mixnode_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
mix_id: &i64,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
mix_id: i64,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_description (
|
||||
mix_id, moniker, website, security_contact, details, last_updated_utc
|
||||
@@ -178,13 +178,13 @@ pub(crate) async fn insert_mixnode_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
mix_id,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
|
||||
@@ -9,7 +9,8 @@ mod summary;
|
||||
pub(crate) mod testruns;
|
||||
|
||||
pub(crate) use gateways::{
|
||||
get_all_gateways, get_bonded_gateway_id_keys, select_gateway_identity, update_bonded_gateways,
|
||||
get_all_gateways, get_bonded_gateway_id_keys, get_or_create_gateway, select_gateway_identity,
|
||||
update_bonded_gateways,
|
||||
};
|
||||
pub(crate) use gateways_stats::{delete_old_records, get_sessions_stats, insert_session_records};
|
||||
pub(crate) use misc::insert_summaries;
|
||||
|
||||
@@ -4,14 +4,14 @@ use nym_validator_client::{
|
||||
client::{NodeId, NymNodeDetails},
|
||||
models::NymNodeDescription,
|
||||
};
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use std::collections::HashMap;
|
||||
use tracing::instrument;
|
||||
|
||||
use crate::{
|
||||
db::{
|
||||
models::{NymNodeDto, NymNodeInsertRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
mixnet_scraper::helpers::NodeDescriptionResponse,
|
||||
};
|
||||
@@ -19,21 +19,20 @@ use crate::{
|
||||
pub(crate) async fn get_all_nym_nodes(pool: &DbPool) -> anyhow::Result<Vec<NymNodeDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query_as!(
|
||||
NymNodeDto,
|
||||
crate::db::query_as::<NymNodeDto>(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
ip_addresses as "ip_addresses!: serde_json::Value",
|
||||
ip_addresses,
|
||||
mix_port,
|
||||
x25519_sphinx_pubkey,
|
||||
node_role as "node_role: serde_json::Value",
|
||||
supported_roles as "supported_roles: serde_json::Value",
|
||||
entry as "entry: serde_json::Value",
|
||||
node_role,
|
||||
supported_roles,
|
||||
entry,
|
||||
performance,
|
||||
self_described as "self_described: serde_json::Value",
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
self_described,
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
ORDER BY
|
||||
@@ -56,21 +55,20 @@ pub(crate) async fn get_described_bonded_nym_nodes(
|
||||
) -> anyhow::Result<Vec<NymNodeDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query_as!(
|
||||
NymNodeDto,
|
||||
crate::db::query_as::<NymNodeDto>(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
ip_addresses as "ip_addresses!: serde_json::Value",
|
||||
ip_addresses,
|
||||
mix_port,
|
||||
x25519_sphinx_pubkey,
|
||||
node_role as "node_role: serde_json::Value",
|
||||
supported_roles as "supported_roles: serde_json::Value",
|
||||
entry as "entry: serde_json::Value",
|
||||
node_role,
|
||||
supported_roles,
|
||||
entry,
|
||||
performance,
|
||||
self_described as "self_described: serde_json::Value",
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
self_described,
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -92,7 +90,7 @@ pub(crate) async fn update_nym_nodes(
|
||||
) -> anyhow::Result<usize> {
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"UPDATE nym_nodes
|
||||
SET
|
||||
self_described = NULL,
|
||||
@@ -104,7 +102,7 @@ pub(crate) async fn update_nym_nodes(
|
||||
let inserted = node_records.len();
|
||||
for record in node_records {
|
||||
// https://www.sqlite.org/lang_upsert.html
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO nym_nodes
|
||||
(node_id, ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
@@ -129,20 +127,20 @@ pub(crate) async fn update_nym_nodes(
|
||||
performance=excluded.performance,
|
||||
last_updated_utc=excluded.last_updated_utc
|
||||
;",
|
||||
record.node_id,
|
||||
record.ed25519_identity_pubkey,
|
||||
record.total_stake,
|
||||
record.ip_addresses,
|
||||
record.mix_port,
|
||||
record.x25519_sphinx_pubkey,
|
||||
record.node_role,
|
||||
record.supported_roles,
|
||||
record.entry,
|
||||
record.self_described,
|
||||
record.bond_info,
|
||||
record.performance,
|
||||
record.last_updated_utc,
|
||||
)
|
||||
.bind(record.node_id)
|
||||
.bind(record.ed25519_identity_pubkey)
|
||||
.bind(record.total_stake)
|
||||
.bind(record.ip_addresses)
|
||||
.bind(record.mix_port as i32)
|
||||
.bind(record.x25519_sphinx_pubkey)
|
||||
.bind(record.node_role)
|
||||
.bind(record.supported_roles)
|
||||
.bind(record.entry)
|
||||
.bind(record.self_described)
|
||||
.bind(record.bond_info)
|
||||
.bind(record.performance)
|
||||
.bind(record.last_updated_utc)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Failed to INSERT node_id={}: {}", record.node_id, e))?;
|
||||
@@ -158,10 +156,10 @@ pub(crate) async fn get_described_node_bond_info(
|
||||
) -> anyhow::Result<HashMap<NodeId, NymNodeDetails>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -176,11 +174,11 @@ pub(crate) async fn get_described_node_bond_info(
|
||||
records
|
||||
.into_iter()
|
||||
.filter_map(|record| {
|
||||
record
|
||||
.bond_info
|
||||
// only return details for nodes which have details stored
|
||||
.and_then(|bond_info| serde_json::from_value::<NymNodeDetails>(bond_info).ok())
|
||||
.map(|res| (record.node_id as NodeId, res))
|
||||
let node_id: i64 = record.try_get("node_id").ok()?;
|
||||
let bond_info: serde_json::Value = record.try_get("bond_info").ok()?;
|
||||
serde_json::from_value::<NymNodeDetails>(bond_info)
|
||||
.ok()
|
||||
.map(|res| (node_id as NodeId, res))
|
||||
})
|
||||
.collect::<HashMap<_, _>>()
|
||||
})
|
||||
@@ -192,10 +190,10 @@ pub(crate) async fn get_node_self_description(
|
||||
) -> anyhow::Result<HashMap<NodeId, NymNodeDescription>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
self_described as "self_described: serde_json::Value"
|
||||
self_described
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -210,13 +208,11 @@ pub(crate) async fn get_node_self_description(
|
||||
records
|
||||
.into_iter()
|
||||
.filter_map(|record| {
|
||||
record
|
||||
.self_described
|
||||
// only return details for nodes which have details stored
|
||||
.and_then(|description| {
|
||||
serde_json::from_value::<NymNodeDescription>(description).ok()
|
||||
})
|
||||
.map(|res| (record.node_id as NodeId, res))
|
||||
let node_id: i64 = record.try_get("node_id").ok()?;
|
||||
let self_described: serde_json::Value = record.try_get("self_described").ok()?;
|
||||
serde_json::from_value::<NymNodeDescription>(self_described)
|
||||
.ok()
|
||||
.map(|res| (node_id as NodeId, res))
|
||||
})
|
||||
.collect::<HashMap<_, _>>()
|
||||
})
|
||||
@@ -228,7 +224,7 @@ pub(crate) async fn get_bonded_node_description(
|
||||
) -> anyhow::Result<HashMap<NodeId, NodeDescription>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
nd.node_id,
|
||||
moniker,
|
||||
@@ -249,14 +245,15 @@ pub(crate) async fn get_bonded_node_description(
|
||||
records
|
||||
.into_iter()
|
||||
.map(|elem| {
|
||||
let node_id: NodeId = elem.node_id.try_into().unwrap_or_default();
|
||||
let node_id: i64 = elem.try_get("node_id").unwrap_or(0);
|
||||
let node_id: NodeId = node_id.try_into().unwrap_or_default();
|
||||
(
|
||||
node_id,
|
||||
NodeDescription {
|
||||
moniker: elem.moniker.unwrap_or_default(),
|
||||
website: elem.website.unwrap_or_default(),
|
||||
security_contact: elem.security_contact.unwrap_or_default(),
|
||||
details: elem.details.unwrap_or_default(),
|
||||
moniker: elem.try_get("moniker").unwrap_or_default(),
|
||||
website: elem.try_get("website").unwrap_or_default(),
|
||||
security_contact: elem.try_get("security_contact").unwrap_or_default(),
|
||||
details: elem.try_get("details").unwrap_or_default(),
|
||||
},
|
||||
)
|
||||
})
|
||||
@@ -266,12 +263,12 @@ pub(crate) async fn get_bonded_node_description(
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_nym_node_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
node_id: &i64,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
node_id: i64,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO nym_node_descriptions (
|
||||
node_id, moniker, website, security_contact, details, last_updated_utc
|
||||
@@ -283,13 +280,13 @@ pub(crate) async fn insert_nym_node_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
node_id,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
|
||||
@@ -6,7 +6,7 @@ use anyhow::Result;
|
||||
|
||||
pub(crate) async fn insert_node_packet_stats(
|
||||
pool: &DbPool,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
node_kind: ScrapeNodeKind,
|
||||
stats: &NodeStats,
|
||||
timestamp_utc: i64,
|
||||
) -> Result<()> {
|
||||
@@ -14,35 +14,35 @@ pub(crate) async fn insert_node_packet_stats(
|
||||
|
||||
match node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_packet_stats_raw (
|
||||
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
"#,
|
||||
mix_id,
|
||||
timestamp_utc,
|
||||
stats.packets_received,
|
||||
stats.packets_sent,
|
||||
stats.packets_dropped,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO nym_nodes_packet_stats_raw (
|
||||
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
"#,
|
||||
node_id,
|
||||
timestamp_utc,
|
||||
stats.packets_received,
|
||||
stats.packets_sent,
|
||||
stats.packets_dropped,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
}
|
||||
@@ -53,7 +53,7 @@ pub(crate) async fn insert_node_packet_stats(
|
||||
|
||||
pub(crate) async fn get_raw_node_stats(
|
||||
pool: &DbPool,
|
||||
node: &ScraperNodeInfo,
|
||||
node: ScraperNodeInfo,
|
||||
) -> Result<Option<NodeStats>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
@@ -61,39 +61,37 @@ pub(crate) async fn get_raw_node_stats(
|
||||
// if no packets are found, it's fine to assume 0 because that's also
|
||||
// SQL default value if none provided
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query_as!(
|
||||
NodeStats,
|
||||
crate::db::query_as::<NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as "packets_received!: _",
|
||||
COALESCE(packets_sent, 0) as "packets_sent!: _",
|
||||
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM mixnode_packet_stats_raw
|
||||
WHERE mix_id = ?
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
mix_id
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_optional(&mut *conn)
|
||||
.await?
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query_as!(
|
||||
NodeStats,
|
||||
crate::db::query_as::<NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as "packets_received!: _",
|
||||
COALESCE(packets_sent, 0) as "packets_sent!: _",
|
||||
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM nym_nodes_packet_stats_raw
|
||||
WHERE node_id = ?
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
node_id
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_optional(&mut *conn)
|
||||
.await?
|
||||
}
|
||||
@@ -104,27 +102,27 @@ pub(crate) async fn get_raw_node_stats(
|
||||
|
||||
pub(crate) async fn insert_daily_node_stats(
|
||||
pool: &DbPool,
|
||||
node: &ScraperNodeInfo,
|
||||
date_utc: &str,
|
||||
node: ScraperNodeInfo,
|
||||
date_utc: String,
|
||||
packets: NodeStats,
|
||||
) -> Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
match node.node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
let total_stake = sqlx::query_scalar!(
|
||||
let total_stake = crate::db::query_scalar::<i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM mixnodes
|
||||
WHERE mix_id = ?
|
||||
"#,
|
||||
mix_id
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_daily_stats (
|
||||
mix_id, date_utc,
|
||||
@@ -137,31 +135,31 @@ pub(crate) async fn insert_daily_node_stats(
|
||||
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
mix_id,
|
||||
date_utc,
|
||||
total_stake,
|
||||
packets.packets_received,
|
||||
packets.packets_sent,
|
||||
packets.packets_dropped,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
let total_stake = sqlx::query_scalar!(
|
||||
let total_stake = crate::db::query_scalar::<i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM nym_nodes
|
||||
WHERE node_id = ?
|
||||
"#,
|
||||
node_id
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO nym_node_daily_mixing_stats (
|
||||
node_id, date_utc,
|
||||
@@ -174,13 +172,13 @@ pub(crate) async fn insert_daily_node_stats(
|
||||
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
node_id,
|
||||
date_utc,
|
||||
total_stake,
|
||||
packets.packets_received,
|
||||
packets.packets_sent,
|
||||
packets.packets_dropped,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ use crate::{
|
||||
};
|
||||
use anyhow::Result;
|
||||
use nym_validator_client::nym_api::SkimmedNode;
|
||||
use sqlx::Row;
|
||||
|
||||
pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperNodeInfo>> {
|
||||
let mut nodes_to_scrape = Vec::new();
|
||||
@@ -68,12 +69,12 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
tracing::debug!("Fetched {} 🚪 entry/exit nodes", entry_exit_nodes);
|
||||
|
||||
let mut conn = pool.acquire().await?;
|
||||
let mixnodes = sqlx::query!(
|
||||
let mixnodes = crate::db::query(
|
||||
r#"
|
||||
SELECT mix_id as node_id, host, http_api_port
|
||||
FROM mixnodes
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?;
|
||||
@@ -85,18 +86,20 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
let mut legacy_not_in_nym_node_list = 0;
|
||||
let total_legacy_mixnodes = mixnodes.len();
|
||||
for mixnode in mixnodes {
|
||||
let node_id: i64 = mixnode.try_get("node_id")?;
|
||||
let host: String = mixnode.try_get("host")?;
|
||||
let http_api_port: i64 = mixnode.try_get("http_api_port")?;
|
||||
|
||||
if nodes_to_scrape
|
||||
.iter()
|
||||
.all(|node| node.node_id() != &mixnode.node_id)
|
||||
.all(|node| node.node_id() != &node_id)
|
||||
{
|
||||
// in case polyfilling on Nym API gets removed, this part ensures
|
||||
// mixnodes are added to the final list of nodes to scrape
|
||||
nodes_to_scrape.push(ScraperNodeInfo {
|
||||
node_kind: ScrapeNodeKind::LegacyMixnode {
|
||||
mix_id: mixnode.node_id,
|
||||
},
|
||||
hosts: vec![mixnode.host],
|
||||
http_api_port: mixnode.http_api_port,
|
||||
node_kind: ScrapeNodeKind::LegacyMixnode { mix_id: node_id },
|
||||
hosts: vec![host],
|
||||
http_api_port,
|
||||
});
|
||||
|
||||
legacy_not_in_nym_node_list += 1;
|
||||
@@ -121,8 +124,8 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
|
||||
pub(crate) async fn insert_scraped_node_description(
|
||||
pool: &DbPool,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
description: &NodeDescriptionResponse,
|
||||
node_kind: ScrapeNodeKind,
|
||||
description: NodeDescriptionResponse,
|
||||
) -> Result<()> {
|
||||
let timestamp = now_utc().unix_timestamp();
|
||||
let mut conn = pool.acquire().await?;
|
||||
@@ -138,7 +141,7 @@ pub(crate) async fn insert_scraped_node_description(
|
||||
node_id,
|
||||
identity_key,
|
||||
} => {
|
||||
insert_nym_node_description(&mut conn, node_id, description, timestamp).await?;
|
||||
insert_nym_node_description(&mut conn, node_id, description.clone(), timestamp).await?;
|
||||
// for historic reasons (/gateways API), store this info into gateways table as well
|
||||
insert_gateway_description(&mut conn, identity_key, description, timestamp).await?;
|
||||
}
|
||||
|
||||
@@ -23,13 +23,12 @@ use crate::{
|
||||
|
||||
pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<SummaryHistory>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as!(
|
||||
SummaryHistoryDto,
|
||||
let items = crate::db::query_as::<SummaryHistoryDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
date as "date!",
|
||||
timestamp_utc as "timestamp_utc!",
|
||||
value_json as "value_json!"
|
||||
id,
|
||||
date,
|
||||
timestamp_utc,
|
||||
value_json
|
||||
FROM summary_history
|
||||
ORDER BY date DESC
|
||||
LIMIT 30"#,
|
||||
@@ -51,13 +50,12 @@ pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<Sum
|
||||
|
||||
async fn get_summary_dto(pool: &DbPool) -> anyhow::Result<Vec<SummaryDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
Ok(sqlx::query_as!(
|
||||
SummaryDto,
|
||||
Ok(crate::db::query_as::<SummaryDto>(
|
||||
r#"SELECT
|
||||
key as "key!",
|
||||
value_json as "value_json!",
|
||||
last_updated_utc as "last_updated_utc!"
|
||||
FROM summary"#
|
||||
key,
|
||||
value_json,
|
||||
last_updated_utc
|
||||
FROM summary"#,
|
||||
)
|
||||
.fetch(&mut *conn)
|
||||
.try_collect::<Vec<_>>()
|
||||
|
||||
@@ -1,40 +1,38 @@
|
||||
use crate::db::models::{TestRunDto, TestRunStatus};
|
||||
use crate::db::DbConnection;
|
||||
use crate::db::DbPool;
|
||||
use crate::http::models::TestrunAssignment;
|
||||
use crate::utils::now_utc;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use time::Duration;
|
||||
|
||||
pub(crate) async fn count_testruns_in_progress(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
) -> anyhow::Result<i64> {
|
||||
sqlx::query_scalar!(
|
||||
r#"SELECT
|
||||
COUNT(id) as "count: i64"
|
||||
FROM testruns
|
||||
WHERE
|
||||
status = ?
|
||||
"#,
|
||||
TestRunStatus::InProgress as i64,
|
||||
)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map_err(anyhow::Error::from)
|
||||
pub(crate) async fn count_testruns_in_progress(conn: &mut DbConnection) -> anyhow::Result<i64> {
|
||||
#[cfg(feature = "sqlite")]
|
||||
let sql = "SELECT COUNT(id) FROM testruns WHERE status = ?";
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
let sql = "SELECT COUNT(id) FROM testruns WHERE status = $1";
|
||||
|
||||
let count: i64 = sqlx::query_scalar(sql)
|
||||
.bind(TestRunStatus::InProgress as i64)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_in_progress_testrun_by_id(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i64,
|
||||
) -> anyhow::Result<TestRunDto> {
|
||||
sqlx::query_as!(
|
||||
TestRunDto,
|
||||
crate::db::query_as::<TestRunDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
gateway_id as "gateway_id!",
|
||||
status as "status!",
|
||||
created_utc as "created_utc!",
|
||||
ip_address as "ip_address!",
|
||||
log as "log!",
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
ip_address,
|
||||
log,
|
||||
last_assigned_utc
|
||||
FROM testruns
|
||||
WHERE
|
||||
@@ -43,9 +41,9 @@ pub(crate) async fn get_in_progress_testrun_by_id(
|
||||
status = ?
|
||||
ORDER BY created_utc
|
||||
LIMIT 1"#,
|
||||
testrun_id,
|
||||
TestRunStatus::InProgress as i64,
|
||||
)
|
||||
.bind(testrun_id)
|
||||
.bind(TestRunStatus::InProgress as i64)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Couldn't retrieve testrun {testrun_id}: {e}"))
|
||||
@@ -59,7 +57,7 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
let previous_run = now_utc() - max_age;
|
||||
let cutoff_timestamp = previous_run.unix_timestamp();
|
||||
|
||||
let res = sqlx::query!(
|
||||
let res = crate::db::query(
|
||||
r#"UPDATE
|
||||
testruns
|
||||
SET
|
||||
@@ -69,10 +67,10 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
AND
|
||||
last_assigned_utc < ?
|
||||
"#,
|
||||
TestRunStatus::Queued as i64,
|
||||
TestRunStatus::InProgress as i64,
|
||||
cutoff_timestamp
|
||||
)
|
||||
.bind(TestRunStatus::Queued as i64)
|
||||
.bind(TestRunStatus::InProgress as i64)
|
||||
.bind(cutoff_timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
@@ -89,11 +87,11 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
}
|
||||
|
||||
pub(crate) async fn assign_oldest_testrun(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
) -> anyhow::Result<Option<TestrunAssignment>> {
|
||||
let now = now_utc().unix_timestamp();
|
||||
// find & mark as "In progress" in the same transaction to avoid race conditions
|
||||
let returning = sqlx::query!(
|
||||
let returning = crate::db::query(
|
||||
r#"UPDATE testruns
|
||||
SET
|
||||
status = ?,
|
||||
@@ -107,18 +105,18 @@ pub(crate) async fn assign_oldest_testrun(
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING
|
||||
id as "id!",
|
||||
id,
|
||||
gateway_id
|
||||
"#,
|
||||
TestRunStatus::InProgress as i64,
|
||||
now,
|
||||
TestRunStatus::Queued as i64,
|
||||
)
|
||||
.bind(TestRunStatus::InProgress as i64)
|
||||
.bind(now)
|
||||
.bind(TestRunStatus::Queued as i64)
|
||||
.fetch_optional(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
if let Some(testrun) = returning {
|
||||
let gw_identity = sqlx::query!(
|
||||
let gw_identity = crate::db::query(
|
||||
r#"
|
||||
SELECT
|
||||
id,
|
||||
@@ -126,14 +124,14 @@ pub(crate) async fn assign_oldest_testrun(
|
||||
FROM gateways
|
||||
WHERE id = ?
|
||||
LIMIT 1"#,
|
||||
testrun.gateway_id
|
||||
)
|
||||
.bind(testrun.try_get::<i64, _>("gateway_id")?)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(Some(TestrunAssignment {
|
||||
testrun_id: testrun.id,
|
||||
gateway_identity_key: gw_identity.gateway_identity_key,
|
||||
testrun_id: testrun.try_get("id")?,
|
||||
gateway_identity_key: gw_identity.try_get("gateway_identity_key")?,
|
||||
assigned_at_utc: now,
|
||||
}))
|
||||
} else {
|
||||
@@ -142,67 +140,134 @@ pub(crate) async fn assign_oldest_testrun(
|
||||
}
|
||||
|
||||
pub(crate) async fn update_testrun_status(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i64,
|
||||
status: TestRunStatus,
|
||||
) -> anyhow::Result<()> {
|
||||
let status = status as u32;
|
||||
sqlx::query!(
|
||||
"UPDATE testruns SET status = ? WHERE id = ?",
|
||||
status,
|
||||
testrun_id
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
let status = status as i32;
|
||||
crate::db::query("UPDATE testruns SET status = ? WHERE id = ?")
|
||||
.bind(status)
|
||||
.bind(testrun_id)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_last_probe_log(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i64,
|
||||
log: &str,
|
||||
log: String,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_probe_log = ? WHERE id = ?",
|
||||
log,
|
||||
gateway_pk
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
crate::db::query("UPDATE gateways SET last_probe_log = ? WHERE id = ?")
|
||||
.bind(log)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_last_probe_result(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i64,
|
||||
result: &str,
|
||||
result: String,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_probe_result = ? WHERE id = ?",
|
||||
result,
|
||||
gateway_pk
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
crate::db::query("UPDATE gateways SET last_probe_result = ? WHERE id = ?")
|
||||
.bind(result)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_score(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
let now = now_utc().unix_timestamp();
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
|
||||
now,
|
||||
now,
|
||||
gateway_pk
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
crate::db::query("UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?")
|
||||
.bind(now)
|
||||
.bind(now)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_testrun_by_id(
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i64,
|
||||
) -> anyhow::Result<TestRunDto> {
|
||||
crate::db::query_as::<TestRunDto>(
|
||||
r#"SELECT
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
ip_address,
|
||||
log,
|
||||
last_assigned_utc
|
||||
FROM testruns
|
||||
WHERE id = ?"#,
|
||||
)
|
||||
.bind(testrun_id)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Testrun {} not found: {}", testrun_id, e))
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_external_testrun(
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i64,
|
||||
gateway_id: i64,
|
||||
assigned_at_utc: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
let now = crate::utils::now_utc().unix_timestamp();
|
||||
|
||||
crate::db::query(
|
||||
r#"INSERT INTO testruns (
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
last_assigned_utc,
|
||||
ip_address,
|
||||
log
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)"#,
|
||||
)
|
||||
.bind(testrun_id)
|
||||
.bind(gateway_id)
|
||||
.bind(TestRunStatus::InProgress as i64)
|
||||
.bind(now)
|
||||
.bind(assigned_at_utc)
|
||||
.bind("external") // Marker for external origin
|
||||
.bind("") // Empty initial log
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
tracing::debug!(
|
||||
"Created external testrun {} for gateway {}",
|
||||
testrun_id,
|
||||
gateway_id
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn update_testrun_status_by_gateway(
|
||||
conn: &mut DbConnection,
|
||||
gateway_id: i64,
|
||||
status: TestRunStatus,
|
||||
) -> anyhow::Result<()> {
|
||||
let status = status as i32;
|
||||
crate::db::query("UPDATE testruns SET status = ? WHERE gateway_id = ? AND status = ?")
|
||||
.bind(status)
|
||||
.bind(gateway_id)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,144 @@
|
||||
use sqlx::Database;
|
||||
|
||||
/// Converts SQLite-style ? placeholders to PostgreSQL $N format
|
||||
#[cfg(feature = "pg")]
|
||||
fn convert_placeholders(query: &str) -> String {
|
||||
let mut result = String::with_capacity(query.len() + 10);
|
||||
let mut placeholder_count = 0;
|
||||
let chars = query.chars().peekable();
|
||||
let mut in_string = false;
|
||||
let mut escape_next = false;
|
||||
|
||||
for ch in chars {
|
||||
if escape_next {
|
||||
result.push(ch);
|
||||
escape_next = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
match ch {
|
||||
'\\' => {
|
||||
result.push(ch);
|
||||
escape_next = true;
|
||||
}
|
||||
'\'' => {
|
||||
result.push(ch);
|
||||
in_string = !in_string;
|
||||
}
|
||||
'?' if !in_string => {
|
||||
placeholder_count += 1;
|
||||
result.push_str(&format!("${placeholder_count}"));
|
||||
}
|
||||
_ => {
|
||||
result.push(ch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Creates a query that automatically handles placeholder conversion
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub fn query(
|
||||
sql: &str,
|
||||
) -> sqlx::query::Query<'_, sqlx::Sqlite, <sqlx::Sqlite as Database>::Arguments<'_>> {
|
||||
sqlx::query(sql)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub fn query(
|
||||
sql: &str,
|
||||
) -> sqlx::query::Query<'static, sqlx::Postgres, <sqlx::Postgres as Database>::Arguments<'static>> {
|
||||
let converted = convert_placeholders(sql);
|
||||
sqlx::query(Box::leak(converted.into_boxed_str()))
|
||||
}
|
||||
|
||||
/// Creates a query_as that automatically handles placeholder conversion
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub fn query_as<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryAs<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
|
||||
where
|
||||
O: for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
|
||||
{
|
||||
sqlx::query_as(sql)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub fn query_as<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryAs<
|
||||
'static,
|
||||
sqlx::Postgres,
|
||||
O,
|
||||
<sqlx::Postgres as Database>::Arguments<'static>,
|
||||
>
|
||||
where
|
||||
O: for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
|
||||
{
|
||||
let converted = convert_placeholders(sql);
|
||||
sqlx::query_as(Box::leak(converted.into_boxed_str()))
|
||||
}
|
||||
|
||||
/// Creates a query_scalar that automatically handles placeholder conversion
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub fn query_scalar<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryScalar<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
|
||||
where
|
||||
(O,): for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
|
||||
{
|
||||
sqlx::query_scalar(sql)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub fn query_scalar<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryScalar<
|
||||
'static,
|
||||
sqlx::Postgres,
|
||||
O,
|
||||
<sqlx::Postgres as Database>::Arguments<'static>,
|
||||
>
|
||||
where
|
||||
(O,): for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
|
||||
{
|
||||
let converted = convert_placeholders(sql);
|
||||
sqlx::query_scalar(Box::leak(converted.into_boxed_str()))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
#[cfg(feature = "pg")]
|
||||
fn test_convert_placeholders() {
|
||||
assert_eq!(
|
||||
convert_placeholders("SELECT * FROM table WHERE id = ?"),
|
||||
"SELECT * FROM table WHERE id = $1"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
convert_placeholders("INSERT INTO table (a, b, c) VALUES (?, ?, ?)"),
|
||||
"INSERT INTO table (a, b, c) VALUES ($1, $2, $3)"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
convert_placeholders("SELECT * FROM table WHERE name = 'test?' AND id = ?"),
|
||||
"SELECT * FROM table WHERE name = 'test?' AND id = $1"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
convert_placeholders("UPDATE table SET a = ?, b = ? WHERE id = ?"),
|
||||
"UPDATE table SET a = $1, b = $2 WHERE id = $3"
|
||||
);
|
||||
|
||||
// Test with 10 placeholders (like in update_mixnodes)
|
||||
assert_eq!(
|
||||
convert_placeholders("INSERT INTO t VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"),
|
||||
"INSERT INTO t VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)"
|
||||
);
|
||||
}
|
||||
}
|
||||
@@ -1,5 +1,6 @@
|
||||
use crate::db::models::TestRunStatus;
|
||||
use crate::db::models::{TestRunDto, TestRunStatus};
|
||||
use crate::db::queries;
|
||||
use crate::db::DbConnection;
|
||||
use crate::utils::{now_utc, unix_timestamp_to_utc_rfc3339};
|
||||
use crate::{
|
||||
db,
|
||||
@@ -17,7 +18,7 @@ use axum::{
|
||||
};
|
||||
use nym_node_status_client::{
|
||||
auth::VerifiableRequest,
|
||||
models::{get_testrun, submit_results},
|
||||
models::{get_testrun, submit_results, submit_results_v2},
|
||||
};
|
||||
use reqwest::StatusCode;
|
||||
use tracing::warn;
|
||||
@@ -29,6 +30,7 @@ pub(crate) fn routes() -> Router<AppState> {
|
||||
Router::new()
|
||||
.route("/", axum::routing::get(request_testrun))
|
||||
.route("/:testrun_id", axum::routing::post(submit_testrun))
|
||||
.route("/:testrun_id/v2", axum::routing::post(submit_testrun_v2))
|
||||
.layer(DefaultBodyLimit::max(1024 * 1024 * 5))
|
||||
}
|
||||
|
||||
@@ -138,7 +140,7 @@ async fn submit_testrun(
|
||||
queries::testruns::update_gateway_last_probe_log(
|
||||
&mut conn,
|
||||
assigned_testrun.gateway_id,
|
||||
&submitted_result.payload.probe_result,
|
||||
submitted_result.payload.probe_result.clone(),
|
||||
)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
@@ -146,7 +148,7 @@ async fn submit_testrun(
|
||||
queries::testruns::update_gateway_last_probe_result(
|
||||
&mut conn,
|
||||
assigned_testrun.gateway_id,
|
||||
&result,
|
||||
result,
|
||||
)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
@@ -170,6 +172,72 @@ async fn submit_testrun(
|
||||
Ok(StatusCode::CREATED)
|
||||
}
|
||||
|
||||
#[tracing::instrument(level = "debug", skip_all)]
|
||||
async fn submit_testrun_v2(
|
||||
Path(submitted_testrun_id): Path<i64>,
|
||||
State(state): State<AppState>,
|
||||
Json(submission): Json<submit_results_v2::SubmitResultsV2>,
|
||||
) -> HttpResult<StatusCode> {
|
||||
authenticate(&submission, &state)?;
|
||||
is_fresh(&submission.payload.assigned_at_utc)?;
|
||||
|
||||
let db = state.db_pool();
|
||||
let mut conn = db
|
||||
.acquire()
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
// Try to find existing testrun
|
||||
match queries::testruns::get_testrun_by_id(&mut conn, submitted_testrun_id).await {
|
||||
Ok(testrun) => {
|
||||
// Validate it matches the submission
|
||||
let gw_identity = queries::select_gateway_identity(&mut conn, testrun.gateway_id)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
if gw_identity != submission.payload.gateway_identity_key {
|
||||
tracing::warn!(
|
||||
"Gateway mismatch for testrun {}: expected {}, got {}",
|
||||
submitted_testrun_id,
|
||||
gw_identity,
|
||||
submission.payload.gateway_identity_key
|
||||
);
|
||||
return Err(HttpError::invalid_input("Gateway identity mismatch"));
|
||||
}
|
||||
|
||||
// Process normally using existing testrun
|
||||
process_testrun_submission(testrun, submission.payload, &mut conn).await
|
||||
}
|
||||
Err(_) => {
|
||||
// External testrun - create records
|
||||
tracing::info!(
|
||||
"Creating external testrun {} for gateway {}",
|
||||
submitted_testrun_id,
|
||||
submission.payload.gateway_identity_key
|
||||
);
|
||||
|
||||
// Get or create gateway
|
||||
let gateway_id =
|
||||
queries::get_or_create_gateway(&mut conn, &submission.payload.gateway_identity_key)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
// Create testrun
|
||||
queries::testruns::insert_external_testrun(
|
||||
&mut conn,
|
||||
submitted_testrun_id,
|
||||
gateway_id,
|
||||
submission.payload.assigned_at_utc,
|
||||
)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
// Process submission
|
||||
process_testrun_submission_by_gateway(gateway_id, submission.payload, &mut conn).await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TODO dz this should be middleware
|
||||
#[tracing::instrument(level = "debug", skip_all)]
|
||||
fn authenticate(request: &impl VerifiableRequest, state: &AppState) -> HttpResult<()> {
|
||||
@@ -212,3 +280,70 @@ fn get_result_from_log(log: &str) -> String {
|
||||
}
|
||||
"".to_string()
|
||||
}
|
||||
|
||||
async fn process_testrun_submission(
|
||||
testrun: TestRunDto,
|
||||
payload: submit_results_v2::Payload,
|
||||
conn: &mut DbConnection,
|
||||
) -> HttpResult<StatusCode> {
|
||||
// Validate timestamp matches
|
||||
if Some(payload.assigned_at_utc) != testrun.last_assigned_utc {
|
||||
tracing::warn!(
|
||||
"Submitted testrun timestamp mismatch: {} != {:?}, rejecting",
|
||||
payload.assigned_at_utc,
|
||||
testrun.last_assigned_utc
|
||||
);
|
||||
return Err(HttpError::invalid_input("Invalid testrun submitted"));
|
||||
}
|
||||
|
||||
// Process the submission
|
||||
process_testrun_submission_by_gateway(testrun.gateway_id, payload, conn).await
|
||||
}
|
||||
|
||||
async fn process_testrun_submission_by_gateway(
|
||||
gateway_id: i64,
|
||||
payload: submit_results_v2::Payload,
|
||||
conn: &mut DbConnection,
|
||||
) -> HttpResult<StatusCode> {
|
||||
let gw_identity = &payload.gateway_identity_key;
|
||||
|
||||
tracing::debug!(
|
||||
"Processing testrun submission for gateway {} ({} bytes)",
|
||||
gw_identity,
|
||||
payload.probe_result.len(),
|
||||
);
|
||||
|
||||
// Update testrun status to complete
|
||||
queries::testruns::update_testrun_status_by_gateway(conn, gateway_id, TestRunStatus::Complete)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
// Update gateway with results
|
||||
queries::testruns::update_gateway_last_probe_log(
|
||||
conn,
|
||||
gateway_id,
|
||||
payload.probe_result.clone(),
|
||||
)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
let result = get_result_from_log(&payload.probe_result);
|
||||
queries::testruns::update_gateway_last_probe_result(conn, gateway_id, result)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
queries::testruns::update_gateway_score(conn, gateway_id)
|
||||
.await
|
||||
.map_err(HttpError::internal_with_logging)?;
|
||||
|
||||
let assigned_at = unix_timestamp_to_utc_rfc3339(payload.assigned_at_utc);
|
||||
let now = now_utc();
|
||||
tracing::info!(
|
||||
"✅ Testrun for gateway {} complete (assigned at {}, current time {})",
|
||||
gw_identity,
|
||||
assigned_at,
|
||||
now
|
||||
);
|
||||
|
||||
Ok(StatusCode::CREATED)
|
||||
}
|
||||
|
||||
@@ -5,6 +5,12 @@ use nym_task::signal::wait_for_signal;
|
||||
use nym_validator_client::nyxd::NyxdClient;
|
||||
use std::sync::Arc;
|
||||
|
||||
#[cfg(all(feature = "sqlite", feature = "pg"))]
|
||||
compile_error!("Features 'sqlite' and 'pg' are mutually exclusive");
|
||||
|
||||
#[cfg(not(any(feature = "sqlite", feature = "pg")))]
|
||||
compile_error!("Either 'sqlite' or 'pg' feature must be enabled");
|
||||
|
||||
mod cli;
|
||||
mod db;
|
||||
mod http;
|
||||
|
||||
@@ -5,6 +5,7 @@ use crate::{
|
||||
get_raw_node_stats, insert_daily_node_stats, insert_node_packet_stats,
|
||||
insert_scraped_node_description,
|
||||
},
|
||||
DbPool,
|
||||
},
|
||||
utils::{generate_node_name, now_utc},
|
||||
};
|
||||
@@ -12,11 +13,10 @@ use ammonia::Builder;
|
||||
use anyhow::{anyhow, Result};
|
||||
use reqwest;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::SqlitePool;
|
||||
use std::time::Duration;
|
||||
use time::UtcDateTime;
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize)]
|
||||
#[derive(Debug, Serialize, Deserialize, Clone)]
|
||||
pub struct NodeDescriptionResponse {
|
||||
pub moniker: Option<String>,
|
||||
pub website: Option<String>,
|
||||
@@ -116,7 +116,7 @@ pub fn sanitize_description(
|
||||
}
|
||||
}
|
||||
|
||||
pub async fn scrape_and_store_description(pool: &SqlitePool, node: &ScraperNodeInfo) -> Result<()> {
|
||||
pub async fn scrape_and_store_description(pool: &DbPool, node: ScraperNodeInfo) -> Result<()> {
|
||||
let client = build_client()?;
|
||||
let urls = node.contact_addresses();
|
||||
|
||||
@@ -151,15 +151,12 @@ pub async fn scrape_and_store_description(pool: &SqlitePool, node: &ScraperNodeI
|
||||
})?;
|
||||
|
||||
let sanitized_description = sanitize_description(description, *node.node_id());
|
||||
insert_scraped_node_description(pool, &node.node_kind, &sanitized_description).await?;
|
||||
insert_scraped_node_description(pool, node.node_kind.clone(), sanitized_description).await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub async fn scrape_and_store_packet_stats(
|
||||
pool: &SqlitePool,
|
||||
node: &ScraperNodeInfo,
|
||||
) -> Result<()> {
|
||||
pub async fn scrape_and_store_packet_stats(pool: &DbPool, node: ScraperNodeInfo) -> Result<()> {
|
||||
let client = build_client()?;
|
||||
let urls = node.contact_addresses();
|
||||
|
||||
@@ -189,7 +186,7 @@ pub async fn scrape_and_store_packet_stats(
|
||||
|
||||
let timestamp = now_utc();
|
||||
let timestamp_utc = timestamp.unix_timestamp();
|
||||
insert_node_packet_stats(pool, &node.node_kind, &stats, timestamp_utc).await?;
|
||||
insert_node_packet_stats(pool, node.node_kind.clone(), &stats, timestamp_utc).await?;
|
||||
|
||||
// Update daily stats
|
||||
update_daily_stats(pool, node, timestamp, &stats).await?;
|
||||
@@ -198,8 +195,8 @@ pub async fn scrape_and_store_packet_stats(
|
||||
}
|
||||
|
||||
pub async fn update_daily_stats(
|
||||
pool: &SqlitePool,
|
||||
node: &ScraperNodeInfo,
|
||||
pool: &DbPool,
|
||||
node: ScraperNodeInfo,
|
||||
timestamp: UtcDateTime,
|
||||
current_stats: &NodeStats,
|
||||
) -> Result<()> {
|
||||
@@ -211,7 +208,7 @@ pub async fn update_daily_stats(
|
||||
);
|
||||
|
||||
// Get previous stats
|
||||
let previous_stats = get_raw_node_stats(pool, node).await?;
|
||||
let previous_stats = get_raw_node_stats(pool, node.clone()).await?;
|
||||
|
||||
let (diff_received, diff_sent, diff_dropped) = if let Some(prev) = previous_stats {
|
||||
(
|
||||
@@ -226,7 +223,7 @@ pub async fn update_daily_stats(
|
||||
insert_daily_node_stats(
|
||||
pool,
|
||||
node,
|
||||
&date_utc,
|
||||
date_utc,
|
||||
NodeStats {
|
||||
packets_received: diff_received,
|
||||
packets_sent: diff_sent,
|
||||
|
||||
@@ -2,9 +2,9 @@ use std::sync::atomic::{AtomicUsize, Ordering};
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
pub mod helpers;
|
||||
use crate::db::DbPool;
|
||||
use anyhow::Result;
|
||||
use helpers::{scrape_and_store_description, scrape_and_store_packet_stats};
|
||||
use sqlx::SqlitePool;
|
||||
use tracing::{debug, error, instrument, warn};
|
||||
|
||||
use crate::db::models::ScraperNodeInfo;
|
||||
@@ -19,13 +19,13 @@ static TASK_COUNTER: AtomicUsize = AtomicUsize::new(0);
|
||||
static TASK_ID_COUNTER: AtomicUsize = AtomicUsize::new(0);
|
||||
|
||||
pub struct Scraper {
|
||||
pool: SqlitePool,
|
||||
pool: DbPool,
|
||||
description_queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
|
||||
packet_queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
|
||||
}
|
||||
|
||||
impl Scraper {
|
||||
pub fn new(pool: SqlitePool) -> Self {
|
||||
pub fn new(pool: DbPool) -> Self {
|
||||
Self {
|
||||
pool,
|
||||
description_queue: Arc::new(Mutex::new(Vec::new())),
|
||||
@@ -71,7 +71,7 @@ impl Scraper {
|
||||
|
||||
#[instrument(level = "info", name = "description_scraper", skip_all)]
|
||||
async fn run_description_scraper(
|
||||
pool: &SqlitePool,
|
||||
pool: &DbPool,
|
||||
queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
|
||||
) -> Result<()> {
|
||||
let nodes = get_nodes_for_scraping(pool).await?;
|
||||
@@ -88,7 +88,7 @@ impl Scraper {
|
||||
|
||||
#[instrument(level = "info", name = "packet_scraper", skip_all)]
|
||||
async fn run_packet_scraper(
|
||||
pool: &SqlitePool,
|
||||
pool: &DbPool,
|
||||
queue: Arc<Mutex<Vec<ScraperNodeInfo>>>,
|
||||
) -> Result<()> {
|
||||
let nodes = get_nodes_for_scraping(pool).await?;
|
||||
@@ -104,7 +104,7 @@ impl Scraper {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn process_description_queue(pool: &SqlitePool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
|
||||
async fn process_description_queue(pool: &DbPool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
|
||||
loop {
|
||||
let running_tasks = TASK_COUNTER.load(Ordering::Relaxed);
|
||||
|
||||
@@ -127,7 +127,7 @@ impl Scraper {
|
||||
let pool = pool.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
match scrape_and_store_description(&pool, &node).await {
|
||||
match scrape_and_store_description(&pool, node.clone()).await {
|
||||
Ok(_) => debug!(
|
||||
"📝 ✅ Description task #{} for node {} complete",
|
||||
task_id,
|
||||
@@ -149,7 +149,7 @@ impl Scraper {
|
||||
}
|
||||
}
|
||||
|
||||
async fn process_packet_queue(pool: &SqlitePool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
|
||||
async fn process_packet_queue(pool: &DbPool, queue: Arc<Mutex<Vec<ScraperNodeInfo>>>) {
|
||||
loop {
|
||||
let running_tasks = TASK_COUNTER.load(Ordering::Relaxed);
|
||||
|
||||
@@ -172,7 +172,7 @@ impl Scraper {
|
||||
let pool = pool.clone();
|
||||
|
||||
tokio::spawn(async move {
|
||||
match scrape_and_store_packet_stats(&pool, &node).await {
|
||||
match scrape_and_store_packet_stats(&pool, node.clone()).await {
|
||||
Ok(_) => debug!(
|
||||
"📊 ✅ Packet stats task #{} for node {} complete",
|
||||
task_id,
|
||||
|
||||
@@ -252,17 +252,23 @@ impl Monitor {
|
||||
//
|
||||
|
||||
let nodes_summary = vec![
|
||||
(NYMNODE_COUNT, nym_node_count),
|
||||
(ASSIGNED_MIXING_COUNT, assigned_mixing_count),
|
||||
(MIXNODES_LEGACY_COUNT, count_legacy_mixnodes),
|
||||
(NYMNODES_DESCRIBED_COUNT, described_nodes.len()),
|
||||
(GATEWAYS_BONDED_COUNT, count_bonded_gateways),
|
||||
(ASSIGNED_ENTRY_COUNT, assigned_entry_count),
|
||||
(ASSIGNED_EXIT_COUNT, assigned_exit_count),
|
||||
(NYMNODE_COUNT.to_string(), nym_node_count),
|
||||
(ASSIGNED_MIXING_COUNT.to_string(), assigned_mixing_count),
|
||||
(MIXNODES_LEGACY_COUNT.to_string(), count_legacy_mixnodes),
|
||||
(NYMNODES_DESCRIBED_COUNT.to_string(), described_nodes.len()),
|
||||
(GATEWAYS_BONDED_COUNT.to_string(), count_bonded_gateways),
|
||||
(ASSIGNED_ENTRY_COUNT.to_string(), assigned_entry_count),
|
||||
(ASSIGNED_EXIT_COUNT.to_string(), assigned_exit_count),
|
||||
// TODO dz doesn't make sense, could make sense with historical Nym
|
||||
// Nodes if we really need this data
|
||||
(MIXNODES_HISTORICAL_COUNT, all_historical_mixnodes),
|
||||
(GATEWAYS_HISTORICAL_COUNT, all_historical_gateways),
|
||||
(
|
||||
MIXNODES_HISTORICAL_COUNT.to_string(),
|
||||
all_historical_mixnodes,
|
||||
),
|
||||
(
|
||||
GATEWAYS_HISTORICAL_COUNT.to_string(),
|
||||
all_historical_gateways,
|
||||
),
|
||||
];
|
||||
|
||||
let last_updated = now_utc();
|
||||
@@ -295,7 +301,8 @@ impl Monitor {
|
||||
},
|
||||
};
|
||||
|
||||
queries::insert_summaries(&pool, &nodes_summary, &network_summary, last_updated).await?;
|
||||
queries::insert_summaries(&pool, nodes_summary.clone(), network_summary, last_updated)
|
||||
.await?;
|
||||
|
||||
let mut log_lines: Vec<String> = vec![];
|
||||
for (key, value) in nodes_summary.iter() {
|
||||
@@ -495,15 +502,31 @@ impl Monitor {
|
||||
async fn historical_count(pool: &DbPool) -> anyhow::Result<(usize, usize)> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
let all_historical_gateways = sqlx::query_scalar!(r#"SELECT count(id) FROM gateways"#)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?
|
||||
.cast_checked()?;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
let all_historical_gateways = sqlx::query_scalar!(r#"SELECT count(id) FROM gateways"#)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?
|
||||
.unwrap_or(0)
|
||||
.cast_checked()?;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
let all_historical_mixnodes = sqlx::query_scalar!(r#"SELECT count(id) FROM mixnodes"#)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?
|
||||
.cast_checked()?;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
let all_historical_mixnodes = sqlx::query_scalar!(r#"SELECT count(id) FROM mixnodes"#)
|
||||
.fetch_one(&mut *conn)
|
||||
.await?
|
||||
.unwrap_or(0)
|
||||
.cast_checked()?;
|
||||
|
||||
Ok((all_historical_gateways, all_historical_mixnodes))
|
||||
}
|
||||
|
||||
@@ -9,7 +9,7 @@ pub struct GatewayIdentityDto {
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize, utoipa::ToSchema)]
|
||||
pub struct TestRun {
|
||||
pub id: u32,
|
||||
pub id: i32,
|
||||
pub identity_key: String,
|
||||
pub status: String,
|
||||
pub log: String,
|
||||
|
||||
@@ -1,13 +1,12 @@
|
||||
use crate::db::models::{GatewayInfoDto, TestRunDto, TestRunStatus};
|
||||
use crate::db::DbConnection;
|
||||
use crate::testruns::models::TestRun;
|
||||
use crate::utils::now_utc;
|
||||
use anyhow::anyhow;
|
||||
use futures_util::TryStreamExt;
|
||||
use sqlx::pool::PoolConnection;
|
||||
use sqlx::Sqlite;
|
||||
|
||||
pub(crate) async fn try_queue_testrun(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
identity_key: String,
|
||||
ip_address: String,
|
||||
) -> anyhow::Result<TestRun> {
|
||||
@@ -15,20 +14,19 @@ pub(crate) async fn try_queue_testrun(
|
||||
let timestamp = now.unix_timestamp();
|
||||
let timestamp_pretty = now.to_string();
|
||||
|
||||
let items = sqlx::query_as!(
|
||||
GatewayInfoDto,
|
||||
let items = crate::db::query_as::<GatewayInfoDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
gateway_identity_key as "gateway_identity_key!",
|
||||
self_described as "self_described?",
|
||||
explorer_pretty_bond as "explorer_pretty_bond?"
|
||||
id,
|
||||
gateway_identity_key,
|
||||
self_described,
|
||||
explorer_pretty_bond
|
||||
FROM gateways
|
||||
WHERE gateway_identity_key = ?
|
||||
AND bonded = true
|
||||
ORDER BY gateway_identity_key
|
||||
LIMIT 1"#,
|
||||
identity_key,
|
||||
)
|
||||
.bind(identity_key.clone())
|
||||
// TODO dz should call .fetch_one
|
||||
// TODO dz replace this in other queries as well
|
||||
.fetch(conn.as_mut())
|
||||
@@ -48,22 +46,21 @@ pub(crate) async fn try_queue_testrun(
|
||||
//
|
||||
// check if there is already a test run for this gateway
|
||||
//
|
||||
let items = sqlx::query_as!(
|
||||
TestRunDto,
|
||||
let items = crate::db::query_as::<TestRunDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
gateway_id as "gateway_id!",
|
||||
status as "status!",
|
||||
created_utc as "created_utc!",
|
||||
ip_address as "ip_address!",
|
||||
log as "log!",
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
ip_address,
|
||||
log,
|
||||
last_assigned_utc
|
||||
FROM testruns
|
||||
WHERE gateway_id = ? AND status != 2
|
||||
ORDER BY id DESC
|
||||
LIMIT 1"#,
|
||||
gateway_id,
|
||||
)
|
||||
.bind(gateway_id)
|
||||
.fetch(conn.as_mut())
|
||||
.try_collect::<Vec<_>>()
|
||||
.await?;
|
||||
@@ -71,8 +68,8 @@ pub(crate) async fn try_queue_testrun(
|
||||
if !items.is_empty() {
|
||||
let testrun = items.first().unwrap();
|
||||
return Ok(TestRun {
|
||||
id: testrun.id as u32,
|
||||
identity_key,
|
||||
id: testrun.id as i32,
|
||||
identity_key: identity_key.clone(),
|
||||
status: format!(
|
||||
"{}",
|
||||
TestRunStatus::from_repr(testrun.status as u8).unwrap()
|
||||
@@ -84,24 +81,43 @@ pub(crate) async fn try_queue_testrun(
|
||||
//
|
||||
// save test run
|
||||
//
|
||||
let status = TestRunStatus::Queued as u32;
|
||||
let status = TestRunStatus::Queued as i32;
|
||||
let log = format!("Test for {identity_key} requested at {timestamp_pretty} UTC\n\n");
|
||||
|
||||
let id = sqlx::query!(
|
||||
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
|
||||
gateway_id,
|
||||
status,
|
||||
ip_address,
|
||||
timestamp,
|
||||
log,
|
||||
)
|
||||
#[cfg(feature = "sqlite")]
|
||||
let id = {
|
||||
sqlx::query!(
|
||||
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
|
||||
gateway_id,
|
||||
status,
|
||||
ip_address,
|
||||
timestamp,
|
||||
log,
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.await?
|
||||
.last_insert_rowid();
|
||||
.last_insert_rowid()
|
||||
};
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
let id = {
|
||||
let record = sqlx::query!(
|
||||
"INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
|
||||
gateway_id as i32,
|
||||
status,
|
||||
ip_address,
|
||||
timestamp,
|
||||
log,
|
||||
)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
record.id
|
||||
};
|
||||
|
||||
Ok(TestRun {
|
||||
id: id as u32,
|
||||
identity_key,
|
||||
#[allow(clippy::useless_conversion)]
|
||||
id: id.try_into().unwrap(),
|
||||
identity_key: identity_key.clone(),
|
||||
status: format!("{}", TestRunStatus::Queued),
|
||||
log,
|
||||
})
|
||||
|
||||
@@ -16,8 +16,11 @@ readme.workspace = true
|
||||
[dependencies]
|
||||
anyhow = { workspace = true }
|
||||
bincode = { workspace = true }
|
||||
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "serde"] }
|
||||
nym-http-api-client = { path = "../../common/http-api-client" }
|
||||
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"asymmetric",
|
||||
"serde",
|
||||
] }
|
||||
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
reqwest = { workspace = true }
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
serde_json = { workspace = true }
|
||||
|
||||
@@ -15,4 +15,11 @@ impl ApiPaths {
|
||||
pub(super) fn submit_results(&self, testrun_id: impl Display) -> String {
|
||||
format!("{}/internal/testruns/{}", self.server_address, testrun_id)
|
||||
}
|
||||
|
||||
pub(super) fn submit_results_v2(&self, testrun_id: impl Display) -> String {
|
||||
format!(
|
||||
"{}/internal/testruns/{}/v2",
|
||||
self.server_address, testrun_id
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
use crate::models::{get_testrun, submit_results, TestrunAssignment};
|
||||
use crate::models::{get_testrun, submit_results, submit_results_v2, TestrunAssignment};
|
||||
use anyhow::bail;
|
||||
use api::ApiPaths;
|
||||
use nym_crypto::asymmetric::ed25519::{PrivateKey, Signature};
|
||||
@@ -94,6 +94,37 @@ impl NsApiClient {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[instrument(level = "debug", skip(self, probe_result))]
|
||||
pub async fn submit_results_with_context(
|
||||
&self,
|
||||
testrun_id: i64,
|
||||
probe_result: String,
|
||||
assigned_at_utc: i64,
|
||||
gateway_identity_key: String,
|
||||
) -> anyhow::Result<()> {
|
||||
let target_url = self.api.submit_results_v2(testrun_id);
|
||||
|
||||
let payload = submit_results_v2::Payload {
|
||||
probe_result,
|
||||
agent_public_key: self.auth_key.public_key(),
|
||||
assigned_at_utc,
|
||||
gateway_identity_key,
|
||||
};
|
||||
let signature = self.sign_message(&payload)?;
|
||||
let submit_results = submit_results_v2::SubmitResultsV2 { payload, signature };
|
||||
|
||||
let res = self
|
||||
.client
|
||||
.post(target_url)
|
||||
.json(&submit_results)
|
||||
.send()
|
||||
.await
|
||||
.and_then(|response| response.error_for_status())?;
|
||||
|
||||
tracing::debug!("Submitted results with context: {})", res.status());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn sign_message<T>(&self, message: &T) -> anyhow::Result<Signature>
|
||||
where
|
||||
T: serde::Serialize,
|
||||
|
||||
@@ -74,3 +74,38 @@ pub mod submit_results {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub mod submit_results_v2 {
|
||||
use crate::auth::SignedRequest;
|
||||
|
||||
use super::*;
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
pub struct Payload {
|
||||
pub probe_result: String,
|
||||
pub agent_public_key: PublicKey,
|
||||
pub assigned_at_utc: i64,
|
||||
pub gateway_identity_key: String,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Deserialize, Serialize)]
|
||||
pub struct SubmitResultsV2 {
|
||||
pub payload: Payload,
|
||||
pub signature: Signature,
|
||||
}
|
||||
|
||||
impl SignedRequest for SubmitResultsV2 {
|
||||
type Payload = Payload;
|
||||
|
||||
fn public_key(&self) -> &PublicKey {
|
||||
&self.payload.agent_public_key
|
||||
}
|
||||
|
||||
fn signature(&self) -> &Signature {
|
||||
&self.signature
|
||||
}
|
||||
|
||||
fn payload(&self) -> &Self::Payload {
|
||||
&self.payload
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user