Allow PG database backend (#5880)
* feat(db): add SQL query wrapper for PostgreSQL placeholder conversion - Created query_wrapper module with functions to automatically convert SQLite ? placeholders to PostgreSQL $1, $2, ... format - Updated build.rs to handle mutually exclusive feature flags - Modified one query in mixnodes.rs as proof of concept - Added type conversions for PostgreSQL compatibility (u32->i64, u16->i32) This is a checkpoint commit before converting all queries to use the wrapper. * feat(nym-node-status-api): add PostgreSQL database support via feature flags Implement dual database support for SQLite and PostgreSQL through Cargo feature flags. The implementation uses a query wrapper that automatically converts SQLite-style ? placeholders to PostgreSQL-style $1, $2, ... placeholders at runtime. Key changes: - Add query wrapper functions that handle placeholder conversion - Convert all sqlx::query\! macros to use wrapper functions - Handle type conversions between databases (i64 vs i32) - Add feature-gated implementations for database-specific SQL syntax - Update Makefile with clippy targets for both database features - Document database support in README * feat(nym-node-status-agent): add multi-API support with random selection Agents can now connect to multiple APIs and randomly select one for each testrun: - Accept multiple --server arguments in format "address:port:auth_key" - Randomly shuffle server list before attempting connections - Try each server until a testrun is obtained - Submit results back only to the API that provided the testrun - Continue to next server if one is down or has no testruns available * feat(nym-node-status): implement primary/secondary server architecture - Agent now requests testruns only from primary server (first in list) - Results are submitted to all configured servers in parallel - Secondary servers accept external testruns via new v2 endpoint - Added auto-creation of gateway and testrun records on secondary servers - New database queries: get_or_create_gateway, insert_external_testrun - Client library enhanced with submit_results_with_context method * Bump Node status API version * Fix build workdir * Bump to 3.1.4 * Fix types and queries * 3.1.6 * Fix gateway perf, bump 3.1.7 * NodeId -> i32, 3.1.8 * Bump agent version * i64 -> i32 * Use image yq * Migration and more types * Update remaining JSONB columns * Simplify server config * Update build path * Change delimiter * bump agent * Split up pg and sqlite builds * More typing fixes, build-and-push script * Fix Dockerfile-pg * Bump node-status-api * TYping * Agent build script * More logging around testruns * Fail loudly on read errors * Cleanup * Debug get gateways query * Fix get_gateways query * Use pg cert, 3.1.16 * Submit regular results to primary server * Bump freshenss cutoff * Update Cargo.lock * fix: resolve rebase conflicts and compilation errors After rebasing onto develop, fixed several issues: - Fixed borrowed data escapes error by using sqlx::query directly in transaction functions - Removed unused imports and cleaned up code - Maintained database-specific implementations for transaction functions * fmt * Make PG default to make lives easier * Performance improvements for Explorer v2 * Fix sqlite build * Fix PG migration * Tests round 1 * DB tests * More tests * And some more tests * And some more, more tests * cargo fmt * Fix some failing lints * Fix lioness version problems * Clippy in tests --------- Co-authored-by: dynco-nym <173912580+dynco-nym@users.noreply.github.com>
This commit is contained in:
Drazen Urch
@@ -38,10 +38,9 @@ jobs:
|
||||
git config --global user.name "Lawrence Stalder"
|
||||
|
||||
- name: Get version from cargo.toml
|
||||
uses: mikefarah/yq@v4.45.4
|
||||
id: get_version
|
||||
with:
|
||||
cmd: yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
|
||||
run: |
|
||||
yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
|
||||
|
||||
- name: cleanup-gateway-probe-ref
|
||||
id: cleanup_gateway_probe_ref
|
||||
|
||||
@@ -32,10 +32,9 @@ jobs:
|
||||
git config --global user.name "Lawrence Stalder"
|
||||
|
||||
- name: Get version from cargo.toml
|
||||
uses: mikefarah/yq@v4.45.4
|
||||
id: get_version
|
||||
with:
|
||||
cmd: yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
|
||||
run: |
|
||||
yq -oy '.package.version' ${{ env.WORKING_DIRECTORY }}/Cargo.toml
|
||||
|
||||
- name: Set GIT_TAG variable
|
||||
run: echo "GIT_TAG=${{ env.WORKING_DIRECTORY }}-${{ steps.get_version.outputs.result }}" >> $GITHUB_ENV
|
||||
|
||||
@@ -0,0 +1,686 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Nym is a privacy platform that uses mixnet technology to protect against metadata surveillance. The platform consists of several key components:
|
||||
- Mixnet nodes (mixnodes) for packet mixing
|
||||
- Gateways (entry/exit points for the network)
|
||||
- Clients for interacting with the network
|
||||
- Network monitoring tools
|
||||
- Validators for network consensus
|
||||
- Various service providers and integrations
|
||||
|
||||
## Build Commands
|
||||
|
||||
### Rust Components
|
||||
|
||||
```bash
|
||||
# Default build (debug)
|
||||
cargo build
|
||||
|
||||
# Release build
|
||||
cargo build --release
|
||||
|
||||
# Build a specific package
|
||||
cargo build -p <package-name>
|
||||
|
||||
# Build main components
|
||||
make build
|
||||
|
||||
# Build release versions of main binaries and contracts
|
||||
make build-release
|
||||
|
||||
# Build specific binaries
|
||||
make build-nym-cli
|
||||
cargo build -p nym-node --release
|
||||
cargo build -p nym-api --release
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
```bash
|
||||
# Run clippy, unit tests, and formatting
|
||||
make test
|
||||
|
||||
# Run all tests including slow tests
|
||||
make test-all
|
||||
|
||||
# Run clippy on all workspaces
|
||||
make clippy
|
||||
|
||||
# Run unit tests for a specific package
|
||||
cargo test -p <package-name>
|
||||
|
||||
# Run only expensive/ignored tests
|
||||
cargo test --workspace -- --ignored
|
||||
|
||||
# Run API tests
|
||||
dotenv -f envs/sandbox.env -- cargo test --test public-api-tests
|
||||
|
||||
# Run tests with specific log level
|
||||
RUST_LOG=debug cargo test -p <package-name>
|
||||
|
||||
# Run specific test scripts
|
||||
./nym-node/tests/test_apis.sh
|
||||
./scripts/wireguard-exit-policy/exit-policy-tests.sh
|
||||
```
|
||||
|
||||
### Linting and Formatting
|
||||
|
||||
```bash
|
||||
# Run rustfmt on all code
|
||||
make fmt
|
||||
|
||||
# Check formatting without modifying
|
||||
cargo fmt --all -- --check
|
||||
|
||||
# Run clippy with all targets
|
||||
cargo clippy --workspace --all-targets -- -D warnings
|
||||
|
||||
# TypeScript linting
|
||||
yarn lint
|
||||
yarn lint:fix
|
||||
yarn types:lint:fix
|
||||
|
||||
# Check dependencies for security/licensing issues
|
||||
cargo deny check
|
||||
```
|
||||
|
||||
### WASM Components
|
||||
|
||||
```bash
|
||||
# Build all WASM components
|
||||
make sdk-wasm-build
|
||||
|
||||
# Build TypeScript SDK
|
||||
yarn build:sdk
|
||||
npx lerna run --scope @nymproject/sdk build --stream
|
||||
|
||||
# Build and test WASM components
|
||||
make sdk-wasm
|
||||
|
||||
# Build specific WASM packages
|
||||
cd wasm/client && make
|
||||
cd wasm/mix-fetch && make
|
||||
cd wasm/node-tester && make
|
||||
```
|
||||
|
||||
### Contract Development
|
||||
|
||||
```bash
|
||||
# Build all contracts
|
||||
make contracts
|
||||
|
||||
# Build contracts in release mode
|
||||
make build-release-contracts
|
||||
|
||||
# Generate contract schemas
|
||||
make contract-schema
|
||||
|
||||
# Run wasm-opt on contracts
|
||||
make wasm-opt-contracts
|
||||
|
||||
# Check contracts with cosmwasm-check
|
||||
make cosmwasm-check-contracts
|
||||
```
|
||||
|
||||
### Running Components
|
||||
|
||||
```bash
|
||||
# Run nym-node as a mixnode
|
||||
cargo run -p nym-node -- run --mode mixnode
|
||||
|
||||
# Run nym-node as a gateway
|
||||
cargo run -p nym-node -- run --mode gateway
|
||||
|
||||
# Run the network monitor
|
||||
cargo run -p nym-network-monitor
|
||||
|
||||
# Run the API server
|
||||
cargo run -p nym-api
|
||||
|
||||
# Run with specific environment
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Start a local network
|
||||
./scripts/localnet_start.sh
|
||||
```
|
||||
|
||||
## Architecture
|
||||
|
||||
The Nym platform consists of various components organized as a monorepo:
|
||||
|
||||
1. **Core Mixnet Infrastructure**:
|
||||
- `nym-node`: Core binary supporting mixnode and gateway modes
|
||||
- `common/nymsphinx`: Implementation of the Sphinx packet format
|
||||
- `common/topology`: Network topology management
|
||||
- `common/types`: Shared data types across components
|
||||
|
||||
2. **Network Monitoring**:
|
||||
- `nym-network-monitor`: Monitors the network's reliability and performance
|
||||
- `nym-api`: API server for network stats and monitoring data
|
||||
- Metrics tracking for nodes, routes, and overall network health
|
||||
|
||||
3. **Client Implementations**:
|
||||
- `clients/native`: Native Rust client implementation
|
||||
- `clients/socks5`: SOCKS5 proxy client for standard applications
|
||||
- `wasm`: WebAssembly client implementations (for browsers)
|
||||
- `nym-connect`: Desktop and mobile clients
|
||||
|
||||
4. **Blockchain & Smart Contracts**:
|
||||
- `common/cosmwasm-smart-contracts`: Smart contract implementations
|
||||
- `contracts`: CosmWasm contracts for the Nym network
|
||||
- `common/ledger`: Blockchain integration
|
||||
|
||||
5. **Utilities & Tools**:
|
||||
- `tools`: Various CLI tools and utilities
|
||||
- `sdk`: SDKs for different languages and platforms
|
||||
- `documentation`: Documentation generation and management
|
||||
|
||||
## Packet System
|
||||
|
||||
Nym uses a modified Sphinx packet format for its mixnet:
|
||||
|
||||
1. **Message Chunking**:
|
||||
- Messages are divided into "sets" and "fragments"
|
||||
- Each fragment fits in a single Sphinx packet
|
||||
- The `common/nymsphinx/chunking` module handles message fragmentation
|
||||
|
||||
2. **Routing**:
|
||||
- Packets traverse through 3 layers of mixnodes
|
||||
- Routing information is encrypted in layers (onion routing)
|
||||
- The final gateway receives and processes the messages
|
||||
|
||||
3. **Monitoring**:
|
||||
- Monitoring system tracks packet delivery through the network
|
||||
- Routes are analyzed for reliability statistics
|
||||
- Node performance metrics are collected
|
||||
|
||||
## Network Protocol
|
||||
|
||||
Nym implements the Loopix mixnet design with several key privacy features:
|
||||
|
||||
1. **Continuous-time Mixing**:
|
||||
- Each mixnode delays messages independently with an exponential distribution
|
||||
- This creates random reordering of packets, destroying timing correlations
|
||||
- Offers better anonymity properties than batch mixing approaches
|
||||
|
||||
2. **Cover Traffic**:
|
||||
- Clients and nodes generate dummy "loop" packets that circulate through the network
|
||||
- These packets are indistinguishable from real traffic
|
||||
- Creates a baseline level of traffic that hides actual communication patterns
|
||||
- Provides unobservability (hiding when and how much real traffic is being sent)
|
||||
|
||||
3. **Stratified Network Architecture**:
|
||||
- Traffic flows through Entry Gateway → 3 Mixnode Layers → Exit Gateway
|
||||
- Path selection is independent per-message (unlike Tor)
|
||||
- Each node connects only to adjacent layers
|
||||
|
||||
4. **Anonymous Replies**:
|
||||
- Single-Use Reply Blocks (SURBs) allow receiving messages without revealing identity
|
||||
- Enables bidirectional communication while maintaining privacy
|
||||
|
||||
## Network Monitoring Architecture
|
||||
|
||||
The network monitoring system is a core component that measures mixnet reliability:
|
||||
|
||||
1. The `nym-network-monitor` sends test packets through the network
|
||||
2. These packets follow predefined routes through multiple mixnodes
|
||||
3. Metrics are collected about:
|
||||
- Successful and failed packet deliveries
|
||||
- Node reliability (percentage of successful packet handling)
|
||||
- Route reliability (which specific route combinations work best)
|
||||
4. Results are stored in the database and used by `nym-api` to:
|
||||
- Present node performance statistics
|
||||
- Determine network rewards
|
||||
- Provide route selection guidance to clients
|
||||
|
||||
In the current branch, metrics collection is being enhanced with a fanout approach to submit to multiple API endpoints.
|
||||
|
||||
## Development Environment
|
||||
|
||||
### Required Dependencies
|
||||
|
||||
- Rust toolchain (stable, 1.80+)
|
||||
- Node.js (v20+) and yarn for TypeScript components
|
||||
- SQLite for local database development
|
||||
- PostgreSQL for API database (optional, for full API functionality)
|
||||
- CosmWasm tools for contract development
|
||||
- For building contracts: `wasm-opt` tool from `binaryen`
|
||||
- Python 3.8+ for some scripts
|
||||
- Docker (optional, for containerized development)
|
||||
- protoc (Protocol Buffers compiler) for some components
|
||||
|
||||
### Environment Configurations
|
||||
|
||||
The `envs/` directory contains pre-configured environments:
|
||||
|
||||
#### Available Environments
|
||||
|
||||
- **`local.env`**: Local development environment
|
||||
- Points to local services (localhost)
|
||||
- Uses test mnemonics and keys
|
||||
- Ideal for testing without external dependencies
|
||||
|
||||
- **`sandbox.env`**: Sandbox test network
|
||||
- Public test network with real nodes
|
||||
- Test tokens available from faucet
|
||||
- Contract addresses for sandbox deployment
|
||||
- API: https://sandbox-nym-api1.nymtech.net
|
||||
|
||||
- **`mainnet.env`**: Production mainnet
|
||||
- Real network with real tokens
|
||||
- Production contract addresses
|
||||
- API: https://validator.nymtech.net
|
||||
- Use with caution!
|
||||
|
||||
- **`canary.env`**: Canary deployment
|
||||
- Pre-release testing environment
|
||||
- Tests new features before mainnet
|
||||
|
||||
- **`mainnet-local-api.env`**: Hybrid environment
|
||||
- Uses mainnet contracts but local API
|
||||
- Useful for API development against mainnet data
|
||||
|
||||
#### Key Environment Variables
|
||||
|
||||
```bash
|
||||
# Network configuration
|
||||
NETWORK_NAME=sandbox # Network identifier
|
||||
BECH32_PREFIX=n # Address prefix (n for sandbox, n for mainnet)
|
||||
NYM_API=https://sandbox-nym-api1.nymtech.net/api
|
||||
NYXD=https://rpc.sandbox.nymtech.net
|
||||
NYM_API_NETWORK=sandbox
|
||||
|
||||
# Contract addresses (network-specific)
|
||||
MIXNET_CONTRACT_ADDRESS=n1xr3rq8yvd7qplsw5yx90ftsr2zdhg4e9z60h5duusgxpv72hud3sjkxkav
|
||||
VESTING_CONTRACT_ADDRESS=n1unyuj8qnmygvzuex3dwmg9yzt9alhvyeat0uu0jedg2wj33efl5qackslz
|
||||
# ... other contract addresses
|
||||
|
||||
# Mnemonic for testing (NEVER use in production)
|
||||
MNEMONIC="clutch captain shoe salt awake harvest setup primary inmate ugly among become"
|
||||
|
||||
# API Keys and tokens
|
||||
IPINFO_API_TOKEN=your_token_here
|
||||
AUTHENTICATOR_PASSWORD=password_here
|
||||
|
||||
# Logging
|
||||
RUST_LOG=info # Options: error, warn, info, debug, trace
|
||||
RUST_BACKTRACE=1 # Enable backtraces
|
||||
|
||||
# Database
|
||||
DATABASE_URL=postgresql://user:pass@localhost/nym_api
|
||||
```
|
||||
|
||||
#### Using Environment Files
|
||||
|
||||
```bash
|
||||
# Load environment and run command
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Export to shell
|
||||
source envs/sandbox.env
|
||||
|
||||
# Use with make targets
|
||||
dotenv -f envs/sandbox.env -- make run-api-tests
|
||||
```
|
||||
|
||||
## Initial Setup
|
||||
|
||||
### First Time Setup
|
||||
|
||||
1. **Install Prerequisites**
|
||||
```bash
|
||||
# Install Rust
|
||||
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
|
||||
|
||||
# Install Node.js and yarn
|
||||
# Via nvm (recommended):
|
||||
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.0/install.sh | bash
|
||||
nvm install 20
|
||||
npm install -g yarn
|
||||
|
||||
# Install build tools
|
||||
# Ubuntu/Debian:
|
||||
sudo apt-get install build-essential pkg-config libssl-dev protobuf-compiler libpq-dev
|
||||
|
||||
# macOS:
|
||||
brew install protobuf postgresql
|
||||
|
||||
# Install wasm-opt for contract builds
|
||||
npm install -g wasm-opt
|
||||
|
||||
# Add wasm target for Rust
|
||||
rustup target add wasm32-unknown-unknown
|
||||
```
|
||||
|
||||
2. **Clone and Setup Repository**
|
||||
```bash
|
||||
git clone https://github.com/nymtech/nym.git
|
||||
cd nym/nym
|
||||
|
||||
# Install JavaScript dependencies
|
||||
yarn install
|
||||
|
||||
# Build the project
|
||||
make build
|
||||
```
|
||||
|
||||
3. **Database Setup (Optional, for API development)**
|
||||
```bash
|
||||
# Install PostgreSQL
|
||||
# Create database
|
||||
createdb nym_api
|
||||
|
||||
# Run migrations (from nym-api directory)
|
||||
cd nym-api
|
||||
sqlx migrate run
|
||||
```
|
||||
|
||||
### Quick Start
|
||||
|
||||
```bash
|
||||
# Run a mixnode locally
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode mixnode --id my-mixnode
|
||||
|
||||
# Run a gateway locally
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-node -- run --mode gateway --id my-gateway
|
||||
|
||||
# Run the API server
|
||||
dotenv -f envs/sandbox.env -- cargo run -p nym-api
|
||||
|
||||
# Run a client
|
||||
cargo run -p nym-client -- init --id my-client
|
||||
cargo run -p nym-client -- run --id my-client
|
||||
```
|
||||
|
||||
## CI/CD Pipeline
|
||||
|
||||
The project uses GitHub Actions for CI/CD with several key workflows:
|
||||
|
||||
1. **Build and Test**:
|
||||
- `ci-build.yml`: Main build workflow for Rust components
|
||||
- Tests are run on multiple platforms (Linux, Windows, macOS)
|
||||
- Includes formatting check (rustfmt) and linting (clippy)
|
||||
|
||||
2. **Release Process**:
|
||||
- Binary artifacts are published on release tags
|
||||
- Multiple platform builds are created
|
||||
|
||||
3. **Documentation**:
|
||||
- Documentation is automatically built and deployed
|
||||
|
||||
## Database Structure
|
||||
|
||||
The system uses SQLite databases with tables like:
|
||||
- `mixnode_status`: Status information about mixnodes
|
||||
- `gateway_status`: Status information about gateways
|
||||
- `routes`: Route performance information (success/failure of specific paths)
|
||||
- `monitor_run`: Information about monitoring test runs
|
||||
|
||||
## Development Workflows
|
||||
|
||||
### Running a Node
|
||||
|
||||
To run the mixnode or gateway:
|
||||
|
||||
```bash
|
||||
# Run nym-node as a mixnode with specified identity
|
||||
cargo run -p nym-node -- run --mode mixnode --id my-mixnode
|
||||
|
||||
# Run nym-node as a gateway
|
||||
cargo run -p nym-node -- run --mode gateway --id my-gateway
|
||||
```
|
||||
|
||||
### Configuration
|
||||
|
||||
Nodes can be configured with files in various locations:
|
||||
- Command-line arguments
|
||||
- Environment variables
|
||||
- `.env` files specified with `--config-env-file`
|
||||
|
||||
### Monitoring
|
||||
|
||||
To monitor the health of your node:
|
||||
- View logs for real-time information
|
||||
- Use the node's HTTP API for status information
|
||||
- Check the explorer for public node statistics
|
||||
|
||||
## Common Libraries
|
||||
|
||||
- `common/types`: Shared data types across all components
|
||||
- `common/crypto`: Cryptographic primitives and wrappers
|
||||
- `common/client-core`: Core client functionality
|
||||
- `common/gateway-client`: Client-gateway communication
|
||||
- `common/task`: Task management and concurrency utilities
|
||||
- `common/nymsphinx`: Sphinx packet implementation for mixnet
|
||||
- `common/topology`: Network topology management
|
||||
- `common/credentials`: Credential system for privacy-preserving authentication
|
||||
- `common/bandwidth-controller`: Bandwidth management and accounting
|
||||
|
||||
## Code Conventions
|
||||
|
||||
- Error handling: Use anyhow/thiserror for structured error handling
|
||||
- Logging: Use the tracing framework for logging and diagnostics
|
||||
- State management: Generally use Tokio/futures for async code
|
||||
- Configuration: Use the config crate and env vars with defaults
|
||||
- Database: Use sqlx for type-safe database queries
|
||||
- Follow clippy recommendations and rustfmt formatting
|
||||
- Use semantic commit messages: feat, fix, docs, refactor, test, chore
|
||||
|
||||
## When Making Changes
|
||||
|
||||
- Run `make test` before submitting PRs
|
||||
- Follow Rust naming conventions
|
||||
- Use `clippy` to check for common issues
|
||||
- Update SQLx query caches when modifying DB queries: `cargo sqlx prepare`
|
||||
- Consider backward compatibility for protocol changes
|
||||
- Use lefthook pre-commit hooks for TypeScript formatting
|
||||
- Run `cargo deny check` to verify dependency compliance
|
||||
- Test against both sandbox and local environments when possible
|
||||
- Update relevant documentation and CHANGELOG.md
|
||||
|
||||
## Development Tools
|
||||
|
||||
### Useful Cargo Commands
|
||||
|
||||
```bash
|
||||
# Check for outdated dependencies
|
||||
cargo outdated
|
||||
|
||||
# Analyze binary size
|
||||
cargo bloat --release -p nym-node
|
||||
|
||||
# Generate dependency graph
|
||||
cargo tree -p nym-api
|
||||
|
||||
# Run with instrumentation
|
||||
cargo run --features profiling -p nym-node
|
||||
|
||||
# Check for security advisories
|
||||
cargo audit
|
||||
```
|
||||
|
||||
### Database Tools
|
||||
|
||||
```bash
|
||||
# SQLx CLI for migrations
|
||||
cargo install sqlx-cli
|
||||
|
||||
# Create new migration
|
||||
cd nym-api && sqlx migrate add <migration_name>
|
||||
|
||||
# Prepare query metadata for offline compilation
|
||||
cargo sqlx prepare --workspace
|
||||
|
||||
# View database schema
|
||||
./nym-api/enter_db.sh
|
||||
```
|
||||
|
||||
### Development Scripts
|
||||
|
||||
- `scripts/build_topology.py`: Generate network topology files
|
||||
- `scripts/node_api_check.py`: Verify node API endpoints
|
||||
- `scripts/network_tunnel_manager.sh`: Manage network tunnels
|
||||
- `scripts/localnet_start.sh`: Start a local test network
|
||||
- Various deployment scripts in `deployment/` for different environments
|
||||
|
||||
## Debugging
|
||||
|
||||
- Enable more verbose logging with the RUST_LOG environment variable:
|
||||
```
|
||||
RUST_LOG=debug,nym_node=trace cargo run -p nym-node -- run --mode mixnode
|
||||
```
|
||||
- Use the HTTP API endpoints for status information
|
||||
- Check monitoring data in the database for network performance metrics
|
||||
- For complex issues, use tracing tools to follow packet flow
|
||||
- Enable backtraces: `RUST_BACKTRACE=full`
|
||||
- For WASM debugging: Use browser developer tools with source maps
|
||||
|
||||
## Deployment and Advanced Configurations
|
||||
|
||||
### Deployment Structure
|
||||
|
||||
The `deployment/` directory contains Ansible playbooks and configurations for various deployment scenarios:
|
||||
|
||||
- **`aws/`**: AWS-specific deployment configurations
|
||||
- **`mixnode/`**: Mixnode deployment playbooks
|
||||
- **`gateway/`**: Gateway deployment playbooks
|
||||
- **`validator/`**: Validator node deployment
|
||||
- **`sandbox-v2/`**: Complete sandbox environment setup
|
||||
- **`big-dipper-2/`**: Block explorer deployment
|
||||
|
||||
### Sandbox V2 Deployment
|
||||
|
||||
The sandbox-v2 deployment (`deployment/sandbox-v2/`) provides a complete test environment:
|
||||
|
||||
```bash
|
||||
# Key playbooks:
|
||||
- deploy.yaml # Main deployment orchestrator
|
||||
- deploy-mixnodes.yaml # Deploy mixnodes
|
||||
- deploy-gateways.yaml # Deploy gateways
|
||||
- deploy-validators.yaml # Deploy validator nodes
|
||||
- deploy-nym-api.yaml # Deploy API services
|
||||
```
|
||||
|
||||
### Custom Environment Setup
|
||||
|
||||
To create a custom environment:
|
||||
|
||||
1. Copy an existing env file: `cp envs/sandbox.env envs/custom.env`
|
||||
2. Modify the network endpoints and contract addresses
|
||||
3. Update the `NETWORK_NAME` to your identifier
|
||||
4. Set appropriate mnemonics and keys (use fresh ones for production!)
|
||||
|
||||
### Contract Addresses
|
||||
|
||||
Contract addresses are network-specific and defined in environment files:
|
||||
- Mixnet contract: Manages mixnode/gateway registry
|
||||
- Vesting contract: Handles token vesting schedules
|
||||
- Coconut contracts: Privacy-preserving credentials
|
||||
- Name service: Human-readable address mapping
|
||||
- Ecash contract: Electronic cash functionality
|
||||
|
||||
### Local Network Setup
|
||||
|
||||
For a completely local network:
|
||||
```bash
|
||||
# Start local chain
|
||||
./scripts/localnet_start.sh
|
||||
|
||||
# Deploy contracts
|
||||
cd contracts
|
||||
make deploy-local
|
||||
|
||||
# Start nodes with local config
|
||||
dotenv -f envs/local.env -- cargo run -p nym-node -- run --mode mixnode
|
||||
```
|
||||
|
||||
## Common Issues and Troubleshooting
|
||||
|
||||
### Database Issues
|
||||
|
||||
- When modifying database queries, you must update SQLx query caches:
|
||||
```bash
|
||||
cargo sqlx prepare
|
||||
```
|
||||
- If you see SQLx errors about missing query files, this is likely the cause
|
||||
- For "database is locked" errors with SQLite, ensure only one process accesses the DB
|
||||
- For PostgreSQL connection issues, verify DATABASE_URL and that the server is running
|
||||
|
||||
### API Connection Issues
|
||||
|
||||
- Check the environment variables pointing to the APIs (NYM_API, NYXD)
|
||||
- Verify network connectivity and API health endpoints
|
||||
- For authentication issues, check node keys and credentials
|
||||
- Common endpoints to verify:
|
||||
- API health: `$NYM_API/health`
|
||||
- Chain status: `$NYXD/status`
|
||||
- Contract info: `$NYXD/cosmwasm/wasm/v1/contract/$CONTRACT_ADDRESS`
|
||||
|
||||
### Build Problems
|
||||
|
||||
- Clean dependencies with `cargo clean` for a fresh build
|
||||
- Check for compatible Rust version (1.80+ recommended)
|
||||
- For smart contract builds, ensure wasm-opt is installed: `npm install -g wasm-opt`
|
||||
- For cross-compilation issues, check target-specific dependencies
|
||||
- WASM build issues: Ensure wasm32-unknown-unknown target is installed:
|
||||
```bash
|
||||
rustup target add wasm32-unknown-unknown
|
||||
```
|
||||
- For "cannot find -lpq" errors, install PostgreSQL development files:
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install libpq-dev
|
||||
# macOS
|
||||
brew install postgresql
|
||||
```
|
||||
|
||||
### Environment Issues
|
||||
|
||||
- Contract address mismatches: Ensure you're using the correct environment file
|
||||
- "Account sequence mismatch": The account nonce is out of sync, wait and retry
|
||||
- Token decimal issues: Sandbox uses different decimal places than mainnet
|
||||
- API version mismatches: Ensure your local API version matches the network
|
||||
- "Insufficient funds": Get test tokens from faucet (sandbox) or check balance
|
||||
- Gateway/mixnode bonding issues: Verify minimum stake requirements
|
||||
|
||||
## Working with Routes and Monitoring
|
||||
|
||||
1. Route monitoring metrics are stored in a `routes` table with:
|
||||
- Layer node IDs (layer1, layer2, layer3, gw)
|
||||
- Success flag (boolean)
|
||||
- Timestamp
|
||||
|
||||
2. To analyze routes:
|
||||
- Check `NetworkAccount` and `AccountingRoute` in `nym-network-monitor/src/accounting.rs`
|
||||
- View monitoring logic in `common/nymsphinx/chunking/monitoring.rs`
|
||||
- Observe how routes are submitted to the database in the `submit_accounting_routes_to_db` function
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Profiling and Benchmarking
|
||||
|
||||
```bash
|
||||
# Run benchmarks
|
||||
cargo bench -p nym-node
|
||||
|
||||
# Profile with perf (Linux)
|
||||
cargo build --release --features profiling
|
||||
perf record --call-graph=dwarf ./target/release/nym-node run --mode mixnode
|
||||
perf report
|
||||
|
||||
# Generate flamegraph
|
||||
cargo install flamegraph
|
||||
cargo flamegraph --bin nym-node -- run --mode mixnode
|
||||
```
|
||||
|
||||
### Common Performance Considerations
|
||||
|
||||
- Use bounded channels for backpressure
|
||||
- Batch database operations where possible
|
||||
- Monitor memory usage with `RUST_LOG=nym_node::metrics=debug`
|
||||
- Use connection pooling for database connections
|
||||
- Consider using `jemalloc` for better memory allocation performance
|
||||
Generated
+1005
-949
File diff suppressed because it is too large
Load Diff
@@ -4,3 +4,61 @@ The Node Status API serves information about individual `nym-nodes` in the Mixne
|
||||
We recommend that developers building applications such as explorers or analytics interfaces about the Mixnet run their own instance of the API, in order to promote a robust network of downstream services, and spread the load of API calls amongst as many endpoints as possible.
|
||||
|
||||
You can find build and operation instructions in the [docs](https://nym.com/docs/apis/ns-api).
|
||||
|
||||
## Database Support
|
||||
|
||||
The Node Status API supports both SQLite and PostgreSQL databases through Cargo feature flags:
|
||||
|
||||
- **SQLite** (default): Lightweight, file-based database suitable for development and small deployments
|
||||
- **PostgreSQL**: Full-featured database recommended for production deployments
|
||||
|
||||
### Building with Different Database Backends
|
||||
|
||||
```bash
|
||||
# Build with SQLite (default)
|
||||
cargo build --features sqlite --no-default-features
|
||||
|
||||
# Build with PostgreSQL
|
||||
cargo build --features pg --no-default-features
|
||||
```
|
||||
|
||||
### Running Tests
|
||||
|
||||
```bash
|
||||
# Test with SQLite
|
||||
cargo test --features sqlite --no-default-features
|
||||
|
||||
# Test with PostgreSQL
|
||||
make test-db # This sets up a test PostgreSQL instance
|
||||
```
|
||||
|
||||
### Development Commands
|
||||
|
||||
The project includes a Makefile with helpful commands for both database backends:
|
||||
|
||||
```bash
|
||||
# Check code compilation
|
||||
make check-sqlite # Check with SQLite
|
||||
make check-pg # Check with PostgreSQL
|
||||
|
||||
# Run clippy linter
|
||||
make clippy-sqlite # Lint with SQLite
|
||||
make clippy-pg # Lint with PostgreSQL
|
||||
make clippy # Run both
|
||||
|
||||
# PostgreSQL development
|
||||
make dev-db # Start a PostgreSQL instance for development
|
||||
make prepare-pg # Prepare SQLx offline cache for PostgreSQL
|
||||
```
|
||||
|
||||
### Implementation Details
|
||||
|
||||
The database abstraction is implemented using a query wrapper that automatically converts SQLite-style `?` placeholders to PostgreSQL-style `$1, $2, ...` placeholders at runtime. This allows writing queries once using SQLite syntax while maintaining compatibility with both databases.
|
||||
|
||||
Key differences handled:
|
||||
- Placeholder syntax (`?` vs `$1, $2, ...`)
|
||||
- Type conversions (SQLite uses i64, PostgreSQL uses i32 for many fields)
|
||||
- SQL dialect differences (e.g., `INSERT OR IGNORE` vs `ON CONFLICT DO NOTHING`)
|
||||
- RETURNING clause behavior
|
||||
|
||||
For more details on PostgreSQL setup, see [README_PG.md](nym-node-status-api/README_PG.md).
|
||||
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
[package]
|
||||
name = "nym-node-status-agent"
|
||||
version = "1.0.0"
|
||||
version = "1.0.4"
|
||||
authors.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
@@ -16,11 +16,23 @@ readme.workspace = true
|
||||
[dependencies]
|
||||
anyhow = { workspace = true }
|
||||
clap = { workspace = true, features = ["derive", "env"] }
|
||||
nym-bin-common = { path = "../../common/bin-common", features = ["models"]}
|
||||
futures = { workspace = true }
|
||||
# nym-bin-common = { path = "../../common/bin-common", features = ["models"] }
|
||||
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"models",
|
||||
] }
|
||||
nym-node-status-client = { path = "../nym-node-status-client" }
|
||||
nym-crypto = { path = "../../common/crypto", features = ["asymmetric", "rand"] }
|
||||
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"asymmetric",
|
||||
"rand",
|
||||
] }
|
||||
rand = { workspace = true }
|
||||
tokio = { workspace = true, features = ["macros", "rt-multi-thread", "process", "fs"] }
|
||||
tokio = { workspace = true, features = [
|
||||
"macros",
|
||||
"rt-multi-thread",
|
||||
"process",
|
||||
"fs",
|
||||
] }
|
||||
tracing = { workspace = true }
|
||||
tracing-subscriber = { workspace = true, features = ["env-filter"] }
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ WORKDIR /usr/src/nym-vpn-client/nym-vpn-core
|
||||
RUN cargo build --release --package nym-gateway-probe
|
||||
|
||||
COPY ./ /usr/src/nym
|
||||
WORKDIR /usr/src/nym/nym-node-status-agent
|
||||
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-agent
|
||||
RUN cargo build --release
|
||||
|
||||
#-------------------------------------------------------------------
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Build and push Node Status Agent container to harbor.nymte.ch
|
||||
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
WORKING_DIRECTORY="${SCRIPT_DIR}"
|
||||
CONTAINER_NAME="node-status-agent"
|
||||
REGISTRY="harbor.nymte.ch"
|
||||
NAMESPACE="nym"
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Function to display usage
|
||||
usage() {
|
||||
echo "Usage: $0 <gateway-probe-git-ref>"
|
||||
echo " gateway-probe-git-ref - Git reference (branch/tag/commit) for gateway probe"
|
||||
echo ""
|
||||
echo "Example: $0 main"
|
||||
echo "Example: $0 release/2025.11-cheddar"
|
||||
echo "Example: $0 v1.2.3"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Parse arguments
|
||||
if [ $# -ne 1 ]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
GATEWAY_PROBE_GIT_REF="$1"
|
||||
|
||||
# Get version from Cargo.toml
|
||||
VERSION=$(grep "^version = " "${WORKING_DIRECTORY}/Cargo.toml" | sed -E 's/version = "(.*)"/\1/')
|
||||
if [ -z "$VERSION" ]; then
|
||||
echo -e "${RED}Error: Could not extract version from Cargo.toml${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Clean up git ref for use in tag (replace / with -)
|
||||
GIT_REF_SLUG="${GATEWAY_PROBE_GIT_REF//\//-}"
|
||||
|
||||
echo -e "${YELLOW}Building Node Status Agent${NC}"
|
||||
echo -e "${YELLOW}Version: ${VERSION}${NC}"
|
||||
echo -e "${YELLOW}Gateway Probe Git Ref: ${GATEWAY_PROBE_GIT_REF} (slug: ${GIT_REF_SLUG})${NC}"
|
||||
|
||||
# Login to Harbor
|
||||
echo -e "${GREEN}Logging into Harbor...${NC}"
|
||||
docker login "${REGISTRY}"
|
||||
|
||||
# Build the container
|
||||
echo -e "${GREEN}Building container with gateway probe from ${GATEWAY_PROBE_GIT_REF}...${NC}"
|
||||
# Build from repository root (two levels up from script location)
|
||||
docker build \
|
||||
--build-arg GIT_REF="${GATEWAY_PROBE_GIT_REF}" \
|
||||
-f "${WORKING_DIRECTORY}/Dockerfile" \
|
||||
"${SCRIPT_DIR}/../.." \
|
||||
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}" \
|
||||
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${GIT_REF_SLUG}"
|
||||
|
||||
# Push to Harbor
|
||||
echo -e "${GREEN}Pushing container to Harbor...${NC}"
|
||||
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}"
|
||||
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${GIT_REF_SLUG}"
|
||||
|
||||
echo -e "${GREEN}Successfully built and pushed ${CONTAINER_NAME}:${VERSION}-${GIT_REF_SLUG}${NC}"
|
||||
@@ -1,17 +1,45 @@
|
||||
use crate::probe::GwProbe;
|
||||
use clap::{Parser, Subcommand};
|
||||
use nym_bin_common::bin_info;
|
||||
use std::sync::OnceLock;
|
||||
use nym_crypto::asymmetric::ed25519::PrivateKey;
|
||||
use std::{env, sync::OnceLock};
|
||||
|
||||
pub(crate) mod generate_keypair;
|
||||
pub(crate) mod run_probe;
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct ServerConfig {
|
||||
pub(crate) address: String,
|
||||
pub(crate) port: u16,
|
||||
pub(crate) auth_key: PrivateKey,
|
||||
}
|
||||
|
||||
// Helper for passing LONG_VERSION to clap
|
||||
fn pretty_build_info_static() -> &'static str {
|
||||
static PRETTY_BUILD_INFORMATION: OnceLock<String> = OnceLock::new();
|
||||
PRETTY_BUILD_INFORMATION.get_or_init(|| bin_info!().pretty_print())
|
||||
}
|
||||
|
||||
fn parse_server_config(s: &str) -> Result<ServerConfig, String> {
|
||||
let parts: Vec<&str> = s.split('|').collect();
|
||||
if parts.len() != 2 {
|
||||
return Err("Server config must be in format 'address|port'".to_string());
|
||||
}
|
||||
|
||||
let address = parts[0].to_string();
|
||||
let port = parts[1]
|
||||
.parse::<u16>()
|
||||
.map_err(|_| "Invalid port number".to_string())?;
|
||||
let auth_key =
|
||||
PrivateKey::from_base58_string(env::var("NODE_STATUS_AGENT_AUTH_KEY").unwrap()).unwrap();
|
||||
|
||||
Ok(ServerConfig {
|
||||
address,
|
||||
port,
|
||||
auth_key,
|
||||
})
|
||||
}
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[clap(author = "Nymtech", version, long_version = pretty_build_info_static(), about)]
|
||||
pub(crate) struct Args {
|
||||
@@ -22,15 +50,10 @@ pub(crate) struct Args {
|
||||
#[derive(Subcommand, Debug)]
|
||||
pub(crate) enum Command {
|
||||
RunProbe {
|
||||
#[arg(short, long, env = "NODE_STATUS_AGENT_SERVER_ADDRESS")]
|
||||
server_address: String,
|
||||
|
||||
#[arg(short = 'p', long, env = "NODE_STATUS_AGENT_SERVER_PORT")]
|
||||
server_port: u16,
|
||||
|
||||
/// base58-encoded private key
|
||||
#[arg(long, env = "NODE_STATUS_AGENT_AUTH_KEY")]
|
||||
ns_api_auth_key: String,
|
||||
/// Server configurations in format "address:port:auth_key"
|
||||
/// Can be specified multiple times for multiple servers
|
||||
#[arg(short, long, required = true)]
|
||||
server: Vec<String>,
|
||||
|
||||
/// path of binary to run
|
||||
#[arg(long, env = "NODE_STATUS_AGENT_PROBE_PATH")]
|
||||
@@ -58,24 +81,29 @@ impl Args {
|
||||
pub(crate) async fn execute(&self) -> anyhow::Result<()> {
|
||||
match &self.command {
|
||||
Command::RunProbe {
|
||||
server_address,
|
||||
server_port,
|
||||
ns_api_auth_key,
|
||||
server,
|
||||
probe_path,
|
||||
mnemonic,
|
||||
probe_extra_args,
|
||||
} => run_probe::run_probe(
|
||||
server_address,
|
||||
server_port.to_owned(),
|
||||
ns_api_auth_key,
|
||||
probe_path,
|
||||
mnemonic,
|
||||
probe_extra_args,
|
||||
)
|
||||
} => {
|
||||
// Parse server configs
|
||||
let mut servers = Vec::new();
|
||||
for s in server {
|
||||
match parse_server_config(s) {
|
||||
Ok(config) => servers.push(config),
|
||||
Err(e) => {
|
||||
tracing::error!("Invalid server config '{}': {}", s, e);
|
||||
anyhow::bail!("Invalid server config '{}': {}", s, e);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
run_probe::run_probe(&servers, probe_path, mnemonic, probe_extra_args)
|
||||
.await
|
||||
.inspect_err(|err| {
|
||||
tracing::error!("{err}");
|
||||
})?,
|
||||
})?
|
||||
}
|
||||
Command::GenerateKeypair { path } => {
|
||||
let path = path
|
||||
.to_owned()
|
||||
|
||||
@@ -1,38 +1,143 @@
|
||||
use crate::cli::GwProbe;
|
||||
use anyhow::Context;
|
||||
use nym_crypto::asymmetric::ed25519::PrivateKey;
|
||||
use crate::cli::{GwProbe, ServerConfig};
|
||||
|
||||
pub(crate) async fn run_probe(
|
||||
server_ip: &str,
|
||||
server_port: u16,
|
||||
ns_api_auth_key: &str,
|
||||
servers: &[ServerConfig],
|
||||
probe_path: &str,
|
||||
mnemonic: &str,
|
||||
probe_extra_args: &Vec<String>,
|
||||
) -> anyhow::Result<()> {
|
||||
let auth_key = PrivateKey::from_base58_string(ns_api_auth_key)
|
||||
.context("Couldn't parse auth key, exiting")?;
|
||||
|
||||
let ns_api_client = nym_node_status_client::NsApiClient::new(server_ip, server_port, auth_key);
|
||||
if servers.is_empty() {
|
||||
anyhow::bail!("No servers configured");
|
||||
}
|
||||
|
||||
let probe = GwProbe::new(probe_path.to_string());
|
||||
|
||||
let version = probe.version().await;
|
||||
tracing::info!("Probe version:\n{}", version);
|
||||
|
||||
if let Some(testrun) = ns_api_client.request_testrun().await? {
|
||||
// Always use first server as primary
|
||||
let primary_server = &servers[0];
|
||||
tracing::info!(
|
||||
"Requesting testrun from primary server: {}:{}",
|
||||
primary_server.address,
|
||||
primary_server.port
|
||||
);
|
||||
|
||||
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
|
||||
&primary_server.auth_key.to_bytes(),
|
||||
)
|
||||
.expect("Failed to clone auth key");
|
||||
let ns_api_client = nym_node_status_client::NsApiClient::new(
|
||||
&primary_server.address,
|
||||
primary_server.port,
|
||||
auth_key,
|
||||
);
|
||||
|
||||
match ns_api_client.request_testrun().await {
|
||||
Ok(Some(testrun)) => {
|
||||
tracing::info!(
|
||||
"Received testrun {} for gateway {} from primary",
|
||||
testrun.testrun_id,
|
||||
testrun.gateway_identity_key
|
||||
);
|
||||
|
||||
// Run the probe
|
||||
let log = probe.run_and_get_log(
|
||||
&Some(testrun.gateway_identity_key),
|
||||
&Some(testrun.gateway_identity_key.clone()),
|
||||
mnemonic,
|
||||
probe_extra_args,
|
||||
);
|
||||
|
||||
ns_api_client
|
||||
.submit_results(testrun.testrun_id, log, testrun.assigned_at_utc)
|
||||
.await?;
|
||||
// Submit to ALL servers in parallel
|
||||
let handles = servers
|
||||
.iter()
|
||||
.enumerate()
|
||||
.map(|(idx, server)| {
|
||||
let testrun = testrun.clone();
|
||||
let log = log.clone();
|
||||
|
||||
async move {
|
||||
let auth_key = nym_crypto::asymmetric::ed25519::PrivateKey::from_bytes(
|
||||
&server.auth_key.to_bytes(),
|
||||
)
|
||||
.expect("Failed to clone auth key");
|
||||
let client = nym_node_status_client::NsApiClient::new(
|
||||
&server.address,
|
||||
server.port,
|
||||
auth_key,
|
||||
);
|
||||
|
||||
let result = if idx == 0 {
|
||||
// Primary server: submit regular results without context
|
||||
client
|
||||
.submit_results(
|
||||
testrun.testrun_id as i64,
|
||||
log,
|
||||
testrun.assigned_at_utc,
|
||||
)
|
||||
.await
|
||||
} else {
|
||||
tracing::info!("No testruns available, exiting")
|
||||
// Other servers: submit results with context
|
||||
client
|
||||
.submit_results_with_context(
|
||||
testrun.testrun_id,
|
||||
log,
|
||||
testrun.assigned_at_utc,
|
||||
testrun.gateway_identity_key,
|
||||
)
|
||||
.await
|
||||
};
|
||||
|
||||
(idx, server.address.clone(), server.port, result)
|
||||
}
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
|
||||
let results = futures::future::join_all(handles).await;
|
||||
|
||||
for result in results {
|
||||
match result.3 {
|
||||
Ok(()) => {
|
||||
let method = if result.0 == 0 {
|
||||
"regular"
|
||||
} else {
|
||||
"with context"
|
||||
};
|
||||
tracing::info!(
|
||||
"✅ Successfully submitted {} to server[{}] {}:{}",
|
||||
method,
|
||||
result.0,
|
||||
result.1,
|
||||
result.2
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
let method = if result.0 == 0 {
|
||||
"regular"
|
||||
} else {
|
||||
"with context"
|
||||
};
|
||||
tracing::warn!(
|
||||
"❌ Failed to submit {} to server[{}] {}:{} - {}",
|
||||
method,
|
||||
result.0,
|
||||
result.1,
|
||||
result.2,
|
||||
e
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
Ok(None) => {
|
||||
tracing::info!("No testruns available from primary server");
|
||||
Ok(())
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Failed to contact primary server: {}", e);
|
||||
Err(e)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Example environment variables for nym-node-status-api
|
||||
|
||||
# Database configuration
|
||||
# For SQLite:
|
||||
# DATABASE_URL=sqlite://nym-node-status-api.sqlite
|
||||
|
||||
# For PostgreSQL:
|
||||
# DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# Network configuration
|
||||
NETWORK_NAME=sandbox
|
||||
NYM_API=https://sandbox-nym-api1.nymtech.net/api
|
||||
NYXD=https://rpc.sandbox.nymtech.net
|
||||
|
||||
# API configuration
|
||||
NYM_NODE_STATUS_API_HTTP_PORT=8000
|
||||
NYM_API_CLIENT_TIMEOUT=15
|
||||
SQLX_BUSY_TIMEOUT_S=5
|
||||
|
||||
# Monitoring intervals
|
||||
NODE_STATUS_API_MONITOR_REFRESH_INTERVAL=300
|
||||
NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL=300
|
||||
NODE_STATUS_API_GEODATA_TTL=86400
|
||||
|
||||
# Agent keys (comma-separated list)
|
||||
NODE_STATUS_API_AGENT_KEY_LIST=
|
||||
|
||||
# External service tokens
|
||||
IPINFO_API_TOKEN=your_token_here
|
||||
|
||||
# MixNodes (Optional)
|
||||
DATA_PROVIDER_DELEGATION_LIST=
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_daily_stats (\n mix_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,\n packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,\n packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "01ee4a30bc3104712e5bc371a45d614a89d88adf02358800433e06100c13c548"
|
||||
}
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT mix_id as node_id, host, http_api_port\n FROM mixnodes\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "host",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "http_api_port",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "021c6c65d1ed806d8430bef7883906b42a7e4b280c8efb32db15d7c6a51d7a27"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_description (\n mix_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (mix_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "06065394c157927e4002ddd5c7c1af626ae15728d615f539470cd7c189312385"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n gateway_identity_key\n FROM\n gateways\n WHERE\n id = ?",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "06b17d1e5f61201a1b7542896ba55c69cd5c1a7e7d87073c94600c783a0a3984"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n testruns\n SET\n status = ?\n WHERE\n status = ?\n AND\n last_assigned_utc < ?\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "0cf0e4d4f30e90caecffd6255ef85dab12730e538be194438f19ed7f198bd50e"
|
||||
}
|
||||
+17
-17
@@ -1,43 +1,43 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "\n SELECT\n date_utc as \"date_utc!\",\n SUM(total_stake) as \"total_stake!: i64\",\n SUM(packets_received) as \"total_packets_received!: i64\",\n SUM(packets_sent) as \"total_packets_sent!: i64\",\n SUM(packets_dropped) as \"total_packets_dropped!: i64\"\n FROM (\n SELECT\n date_utc,\n n.total_stake,\n n.packets_received,\n n.packets_sent,\n n.packets_dropped\n FROM nym_node_daily_mixing_stats n\n UNION ALL\n SELECT\n m.date_utc,\n m.total_stake,\n m.packets_received,\n m.packets_sent,\n m.packets_dropped\n FROM mixnode_daily_stats m\n LEFT JOIN nym_node_daily_mixing_stats ON m.mix_id = nym_node_daily_mixing_stats.node_id\n WHERE nym_node_daily_mixing_stats.node_id IS NULL\n )\n GROUP BY date_utc\n ORDER BY date_utc ASC\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "date_utc!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
"name": "date_utc!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "total_stake!: i64",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "total_stake!: i64",
|
||||
"type_info": "Numeric"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_received!: i64",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_received!: i64",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_sent!: i64",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_sent!: i64",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "total_packets_dropped!: i64",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
"name": "total_packets_dropped!: i64",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "124d45b9604439584650f401607c46bdbd162c7c689f74fe9ddfdfd48f5ddc07"
|
||||
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n key as \"key!\",\n value_json as \"value_json!\",\n last_updated_utc as \"last_updated_utc!\"\n FROM summary",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "value_json!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "1327b5118f9144dddbcf8edb11f7dc549cf503409fd6dfedcdc02dbcd61d5454"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO mixnode_packet_stats_raw (\n mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "21e44766729777756f6eb04bf3b81df3e591008a1e3fd664ed83ca86ac51bd8c"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n id,\n gateway_identity_key\n FROM gateways\n WHERE id = ?\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "2236299f9f691376db54cbd58ec5ceb89b9925cba46efcf4ed79ef0759a01129"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n AND\n self_described IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "227539374e7473f6f9642289c5b5d1bcd636315ab23537cb5f6d2f82a2bcb7bf"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT gateway_identity_key\n FROM gateways\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "25300e435780101fa207c8e26ef2f49ba5db84d63e89440bb494e8327fe73686"
|
||||
}
|
||||
-86
@@ -1,86 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n AND\n bond_info IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ed25519_identity_pubkey",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_addresses!: serde_json::Value",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "mix_port",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "x25519_sphinx_pubkey",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "node_role: serde_json::Value",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "supported_roles: serde_json::Value",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "entry: serde_json::Value",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "performance",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "283f49a65c7d70bf271702ff6a5c7ad6e68c81932d295ff18ed198c54706a57c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT OR IGNORE INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 9
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "3243cf5646255a9430d1e6710970505d0dbcc62703f40e090e80ff48c77723c4"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO mixnodes\n (mix_id, identity_key, bonded, total_stake,\n host, http_api_port, full_details,\n self_described, last_updated_utc, is_dp_delegatee)\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(mix_id) DO UPDATE SET\n bonded=excluded.bonded,\n total_stake=excluded.total_stake, host=excluded.host,\n http_api_port=excluded.http_api_port,\n full_details=excluded.full_details,self_described=excluded.self_described,\n last_updated_utc=excluded.last_updated_utc,\n is_dp_delegatee = excluded.is_dp_delegatee;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 10
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "3cd5cb4bfca4243925da4ddbccd811e842090e98982e1032670df77961870b32"
|
||||
}
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_identity_key as \"gateway_identity_key!\",\n self_described as \"self_described?\",\n explorer_pretty_bond as \"explorer_pretty_bond?\"\n FROM gateways\n WHERE gateway_identity_key = ?\n AND bonded = true\n ORDER BY gateway_identity_key\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described?",
|
||||
"ordinal": 2,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "explorer_pretty_bond?",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "3e7e987780937873cdb393b157d7708c9f01047b0689eb0d4f7a973b328c609d"
|
||||
}
|
||||
-92
@@ -1,92 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n gw.gateway_identity_key as \"gateway_identity_key!\",\n gw.bonded as \"bonded: bool\",\n gw.performance as \"performance!\",\n gw.self_described as \"self_described?\",\n gw.explorer_pretty_bond as \"explorer_pretty_bond?\",\n gw.last_probe_result as \"last_probe_result?\",\n gw.last_probe_log as \"last_probe_log?\",\n gw.last_testrun_utc as \"last_testrun_utc?\",\n gw.last_updated_utc as \"last_updated_utc!\",\n COALESCE(gd.moniker, \"NA\") as \"moniker!\",\n COALESCE(gd.website, \"NA\") as \"website!\",\n COALESCE(gd.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(gd.details, \"NA\") as \"details!\"\n FROM gateways gw\n LEFT JOIN gateway_description gd\n ON gw.gateway_identity_key = gd.gateway_identity_key\n ORDER BY gw.gateway_identity_key",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "performance!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "self_described?",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "explorer_pretty_bond?",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_probe_result?",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_probe_log?",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_testrun_utc?",
|
||||
"ordinal": 7,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 8,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "moniker!",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "website!",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "security_contact!",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "details!",
|
||||
"ordinal": 12,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "3eb1d8491bda3c1d6e071b6eb364b9a979f4bdb11ea81b2d0f022555bab51ecb"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n total_stake\n FROM mixnodes\n WHERE mix_id = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "3fc2baabf194b147b20be2a49401cc0c100a1d7a7c347393adde2410fa6f4dfe"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE testruns SET status = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "418944f2eccb838cb3882f34469203c8569f03fdd39ce09d7b74177896e52a8c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_probe_log = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4afcc6673890f795c2793f1e2f8570ee787fc7daf00fcb916f18d1cb7d6c8f08"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES (?, ?, ?, ?, ?)",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4d865e873c9cb133883da94db72dcdebd4969e1f240def9fb1bf946f4a1f342f"
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO gateway_session_stats\n (gateway_identity_key, node_id, day,\n unique_active_clients, session_started, users_hashes,\n vpn_sessions, mixnet_sessions, unknown_sessions)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)\n ON CONFLICT DO NOTHING",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Varchar",
|
||||
"Int8",
|
||||
"Date",
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Varchar",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "4fca38abbb416d9457c34a8ba4faf481a837eda4f3e1bee1d430a4eb102a5b3d"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_node_daily_mixing_stats (\n node_id, date_utc,\n total_stake, packets_received,\n packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id, date_utc) DO UPDATE SET\n total_stake = excluded.total_stake,\n packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,\n packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,\n packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "5912ea335a957d217f5e2b3a63a25b31715c2098310fe7a9db688bc2fd36aad4"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE testruns\n SET\n status = ?,\n last_assigned_utc = ?\n WHERE rowid =\n (\n SELECT rowid\n FROM testruns\n WHERE status = ?\n ORDER BY created_utc asc\n LIMIT 1\n )\n RETURNING\n id as \"id!\",\n gateway_id\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "5e9cbb39f5fb53774803270f422989e199aac4d4a71913c7074359b4bd676b02"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO nym_nodes\n (node_id, ed25519_identity_pubkey,\n total_stake,\n ip_addresses, mix_port,\n x25519_sphinx_pubkey, node_role,\n supported_roles, entry,\n self_described,\n bond_info,\n performance, last_updated_utc\n )\n VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)\n ON CONFLICT(node_id) DO UPDATE SET\n ed25519_identity_pubkey=excluded.ed25519_identity_pubkey,\n ip_addresses=excluded.ip_addresses,\n mix_port=excluded.mix_port,\n x25519_sphinx_pubkey=excluded.x25519_sphinx_pubkey,\n node_role=excluded.node_role,\n supported_roles=excluded.supported_roles,\n entry=excluded.entry,\n self_described=excluded.self_described,\n bond_info=excluded.bond_info,\n performance=excluded.performance,\n last_updated_utc=excluded.last_updated_utc\n ;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 13
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "664e059ac2c58e1115fe214376a6b326b31c93298f20019772cce2e277a194f8"
|
||||
}
|
||||
+5
-5
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT count(id) FROM mixnodes",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count(id)",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "670b7ed7d57a6986181b24be24ca667e8cacdf677ccb906415b3fe92be0c436b"
|
||||
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_node_descriptions (\n node_id, moniker, website, security_contact, details, last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (node_id) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "6a9780aad1f2f0c8ef780e51fe856679d5e28f95143f4e2a2b409009dc0f55ba"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "INSERT INTO testruns (gateway_id, status, ip_address, created_utc, log) VALUES ($1, $2, $3, $4, $5) RETURNING id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int4",
|
||||
"Int4",
|
||||
"Varchar",
|
||||
"Int8",
|
||||
"Varchar"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "6de15de62c0caa545910a17877a3ac5ebe44a398b199ab0120207a5569f54d0b"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_probe_result = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "6ef3efde571d46961244cd90420f3de5949a5ff2083453cb879af8a1689efe2f"
|
||||
}
|
||||
+30
-30
@@ -1,66 +1,66 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n COALESCE(md.moniker, \"NA\") as \"moniker!\",\n COALESCE(md.website, \"NA\") as \"website!\",\n COALESCE(md.security_contact, \"NA\") as \"security_contact!\",\n COALESCE(md.details, \"NA\") as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n mn.mix_id as \"mix_id!\",\n mn.bonded as \"bonded: bool\",\n mn.is_dp_delegatee as \"is_dp_delegatee: bool\",\n mn.total_stake as \"total_stake!\",\n mn.full_details as \"full_details!\",\n mn.self_described as \"self_described\",\n mn.last_updated_utc as \"last_updated_utc!\",\n md.moniker as \"moniker!\",\n md.website as \"website!\",\n md.security_contact as \"security_contact!\",\n md.details as \"details!\"\n FROM mixnodes mn\n LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id\n ORDER BY mn.mix_id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "mix_id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "mix_id!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "bonded: bool",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"name": "is_dp_delegatee: bool",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
"name": "is_dp_delegatee: bool",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"name": "total_stake!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
"name": "total_stake!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "full_details!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
"name": "full_details!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "self_described",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
"name": "self_described",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "last_updated_utc!",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
"name": "last_updated_utc!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"name": "moniker!",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
"name": "moniker!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "website!",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
"name": "website!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "security_contact!",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
"name": "security_contact!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "details!",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
"name": "details!",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
@@ -70,11 +70,11 @@
|
||||
true,
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "f75af377da33db1455c6e0f612e0fa9583888f343b8b59faf37fc6799b244379"
|
||||
"hash": "74b76cd0d94c1afc51c21c14c12236a2b964b981c8cbcc282117fe9bc38338dd"
|
||||
}
|
||||
-38
@@ -1,38 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n date as \"date!\",\n timestamp_utc as \"timestamp_utc!\",\n value_json as \"value_json!\"\n FROM summary_history\n ORDER BY date DESC\n LIMIT 30",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "date!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "timestamp_utc!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "value_json!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "7600823da7ce80b8ffda933608603a2752e28df775d1af8fd943a5fc8d7dc00d"
|
||||
}
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM mixnode_packet_stats_raw\n WHERE mix_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "packets_received!: _",
|
||||
"ordinal": 0,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_sent!: _",
|
||||
"ordinal": 1,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_dropped!: _",
|
||||
"ordinal": 2,
|
||||
"type_info": "Null"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "786b6a5d954e38b204cebf322711c74c8cf1c08e5e2896a1d6d5b85c91991753"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO summary_history\n (date, timestamp_utc, value_json)\n VALUES (?, ?, ?)\n ON CONFLICT(date) DO UPDATE SET\n timestamp_utc=excluded.timestamp_utc,\n value_json=excluded.value_json;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "788515c34588aec352773df4b6e6c5e41f3c0bb56a27648b5e25466b8634a578"
|
||||
}
|
||||
+5
-5
@@ -1,19 +1,19 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT count(id) FROM gateways",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count(id)",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "86ff64db477a1d6235179b0b88d86b86d1b9be62336c9eac0eef44987a5451b5"
|
||||
|
||||
-32
@@ -1,32 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n COALESCE(packets_received, 0) as \"packets_received!: _\",\n COALESCE(packets_sent, 0) as \"packets_sent!: _\",\n COALESCE(packets_dropped, 0) as \"packets_dropped!: _\"\n FROM nym_nodes_packet_stats_raw\n WHERE node_id = ?\n ORDER BY timestamp_utc DESC\n LIMIT 1 OFFSET 1\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "packets_received!: _",
|
||||
"ordinal": 0,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_sent!: _",
|
||||
"ordinal": 1,
|
||||
"type_info": "Null"
|
||||
},
|
||||
{
|
||||
"name": "packets_dropped!: _",
|
||||
"ordinal": 2,
|
||||
"type_info": "Null"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "8bdf85a61e443fa5f4835bffd0bffc8ed1011f56714fde6007e50951e569854b"
|
||||
}
|
||||
+6
-6
@@ -1,21 +1,21 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n gateway_identity_key as \"gateway_identity_key!\",\n bonded as \"bonded: bool\"\n FROM gateways\n ORDER BY last_testrun_utc",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "gateway_identity_key!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Text"
|
||||
"name": "gateway_identity_key!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"name": "bonded: bool",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
"name": "bonded: bool",
|
||||
"type_info": "Bool"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
|
||||
-86
@@ -1,86 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n ed25519_identity_pubkey,\n total_stake,\n ip_addresses as \"ip_addresses!: serde_json::Value\",\n mix_port,\n x25519_sphinx_pubkey,\n node_role as \"node_role: serde_json::Value\",\n supported_roles as \"supported_roles: serde_json::Value\",\n entry as \"entry: serde_json::Value\",\n performance,\n self_described as \"self_described: serde_json::Value\",\n bond_info as \"bond_info: serde_json::Value\"\n FROM\n nym_nodes\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ed25519_identity_pubkey",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_addresses!: serde_json::Value",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "mix_port",
|
||||
"ordinal": 4,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "x25519_sphinx_pubkey",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "node_role: serde_json::Value",
|
||||
"ordinal": 6,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "supported_roles: serde_json::Value",
|
||||
"ordinal": 7,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "entry: serde_json::Value",
|
||||
"ordinal": 8,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "performance",
|
||||
"ordinal": 9,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 10,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "bond_info: serde_json::Value",
|
||||
"ordinal": 11,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true,
|
||||
false,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "9334f0c91252fcd7ec72558a271222615bb282e5334665700709ae475a5daea2"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO gateways\n (gateway_identity_key, bonded,\n self_described, explorer_pretty_bond,\n last_updated_utc, performance)\n VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT(gateway_identity_key) DO UPDATE SET\n bonded=excluded.bonded,\n self_described=excluded.self_described,\n explorer_pretty_bond=excluded.explorer_pretty_bond,\n last_updated_utc=excluded.last_updated_utc,\n performance = excluded.performance;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "9796d354ae075eab4cbd3438839c39da94025494395ec7b093aefef696f2d0c5"
|
||||
}
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE gateway_id = ? AND status != 2\n ORDER BY id DESC\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "status!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "created_utc!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_address!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "log!",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_assigned_utc",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "a1d9eb816acd1a91ed0975c801c9295c01a78861a2a0597ad28b1579a14bf008"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n COUNT(id) as \"count: i64\"\n FROM testruns\n WHERE\n status = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "count: i64",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "a79a61b87325f3f1d9c5a4fb386ccd585be0641e5878acb6283b879f22ed2b4c"
|
||||
}
|
||||
-44
@@ -1,44 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n nd.node_id,\n moniker,\n website,\n security_contact,\n details\n FROM\n nym_node_descriptions nd\n INNER JOIN\n nym_nodes\n WHERE\n bond_info IS NOT NULL\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "moniker",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "website",
|
||||
"ordinal": 2,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "security_contact",
|
||||
"ordinal": 3,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "details",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true,
|
||||
true,
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "aa602604c0e7b6eef41ea3cd83c16610e15be2d7ee3f6c4d3debf23f95fb9c2e"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE nym_nodes\n SET\n self_described = NULL,\n bond_info = NULL",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "b68796d1d8d2384b30f1aace06269682c4ae96f774261f5c298264d3c12e5b67"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c214c001acbbf79fa499816f36ec586c4c29c03efb4cf0c40b73a5c76159cf5c"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n mixnodes\n SET\n bonded = false\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c42917c9542c1d720d92035863064741aefc9f7a7d1630f6b863ebd8174b6684"
|
||||
}
|
||||
-26
@@ -1,26 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n node_id,\n self_described as \"self_described: serde_json::Value\"\n FROM\n nym_nodes\n WHERE\n self_described IS NOT NULL\n ORDER BY\n node_id\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "node_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "self_described: serde_json::Value",
|
||||
"ordinal": 1,
|
||||
"type_info": "Text"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "c7656b2b1b4328415772ce69d0568bd5438d6c8496ca9cbdcfb70bb5375b345e"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT mix_id\n FROM mixnodes\n WHERE bonded = true\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "mix_id",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "c910788edefe64bbb34379702bcbde9ec6159c9fa03b13652e1f620dcd92125e"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "DELETE FROM gateway_session_stats WHERE day <= ?",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "c9e61180ec35dfab8623333fafa3b216b93440d0fddc0a37dd1b6c1813741f6a"
|
||||
}
|
||||
-20
@@ -1,20 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n SELECT\n total_stake\n FROM nym_nodes\n WHERE node_id = ?\n ",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "total_stake",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 1
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "d2e07d44594ca5b44a6100482ff432c39d761f2a0ac1d6515cf73416f2eb6c61"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "INSERT INTO summary\n (key, value_json, last_updated_utc)\n VALUES (?, ?, ?)\n ON CONFLICT(key) DO UPDATE SET\n value_json=excluded.value_json,\n last_updated_utc=excluded.last_updated_utc;",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 3
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "e0c76a959276e3b0f44c720af9c74a5bf4912ee73468e62e7d0d96b1d9074cbe"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO gateway_description (\n gateway_identity_key,\n moniker,\n website,\n security_contact,\n details,\n last_updated_utc\n ) VALUES (?, ?, ?, ?, ?, ?)\n ON CONFLICT (gateway_identity_key) DO UPDATE SET\n moniker = excluded.moniker,\n website = excluded.website,\n security_contact = excluded.security_contact,\n details = excluded.details,\n last_updated_utc = excluded.last_updated_utc\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 6
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "e9790b63ebe4bff5172bb8cb7bfc288364855003cf0e4d63e95047e7b502c650"
|
||||
}
|
||||
-56
@@ -1,56 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "SELECT\n id as \"id!\",\n gateway_id as \"gateway_id!\",\n status as \"status!\",\n created_utc as \"created_utc!\",\n ip_address as \"ip_address!\",\n log as \"log!\",\n last_assigned_utc\n FROM testruns\n WHERE\n id = ?\n AND\n status = ?\n ORDER BY created_utc\n LIMIT 1",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"name": "id!",
|
||||
"ordinal": 0,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "gateway_id!",
|
||||
"ordinal": 1,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "status!",
|
||||
"ordinal": 2,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "created_utc!",
|
||||
"ordinal": 3,
|
||||
"type_info": "Integer"
|
||||
},
|
||||
{
|
||||
"name": "ip_address!",
|
||||
"ordinal": 4,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "log!",
|
||||
"ordinal": 5,
|
||||
"type_info": "Text"
|
||||
},
|
||||
{
|
||||
"name": "last_assigned_utc",
|
||||
"ordinal": 6,
|
||||
"type_info": "Integer"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Right": 2
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
false,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "f0c64794cbaed87a1d3166251d8e6720c9a9fc929144188460849be85d915004"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "UPDATE\n gateways\n SET\n bonded = false\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 0
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "f7e3fa31d68c028bf39cc95389f29f8758ec922dd2e7ea064a1e537e580c9ee5"
|
||||
}
|
||||
-12
@@ -1,12 +0,0 @@
|
||||
{
|
||||
"db_name": "SQLite",
|
||||
"query": "\n INSERT INTO nym_nodes_packet_stats_raw (\n node_id, timestamp_utc, packets_received, packets_sent, packets_dropped\n ) VALUES (?, ?, ?, ?, ?)\n ",
|
||||
"describe": {
|
||||
"columns": [],
|
||||
"parameters": {
|
||||
"Right": 5
|
||||
},
|
||||
"nullable": []
|
||||
},
|
||||
"hash": "fcb1698d9e0e3a14524c92e7c99a811588c2bbc50d4975487a0464321a1b18c9"
|
||||
}
|
||||
@@ -3,7 +3,7 @@
|
||||
|
||||
[package]
|
||||
name = "nym-node-status-api"
|
||||
version = "3.1.2"
|
||||
version = "3.2.2"
|
||||
authors.workspace = true
|
||||
repository.workspace = true
|
||||
homepage.workspace = true
|
||||
@@ -28,11 +28,15 @@ moka = { workspace = true, features = ["future"] }
|
||||
# Nym API: revert after Cheddar is out
|
||||
nym-contracts-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-mixnet-contract-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["openapi"]}
|
||||
nym-node-status-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-bin-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"openapi",
|
||||
] }
|
||||
nym-node-status-client = { path = "../nym-node-status-client" }
|
||||
nym-crypto = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-http-api-client = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = ["middleware"]}
|
||||
nym-http-api-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar", features = [
|
||||
"middleware",
|
||||
] }
|
||||
nym-network-defaults = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-serde-helpers = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
nym-statistics-common = { git = "https://github.com/nymtech/nym.git", branch = "release/2025.11-cheddar" }
|
||||
@@ -62,7 +66,7 @@ serde_json = { workspace = true }
|
||||
serde_json_path = { workspace = true }
|
||||
strum = { workspace = true }
|
||||
strum_macros = { workspace = true }
|
||||
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "sqlite", "time"] }
|
||||
sqlx = { workspace = true, features = ["runtime-tokio-rustls", "time"] }
|
||||
thiserror = { workspace = true }
|
||||
time = { workspace = true, features = ["formatting"] }
|
||||
tokio = { workspace = true, features = ["rt-multi-thread"] }
|
||||
@@ -77,16 +81,20 @@ utoipauto = { workspace = true }
|
||||
|
||||
nym-node-metrics = { path = "../../nym-node/nym-node-metrics" }
|
||||
|
||||
[features]
|
||||
default = ["pg"]
|
||||
sqlite = ["sqlx/sqlite"]
|
||||
pg = ["sqlx/postgres"]
|
||||
|
||||
[build-dependencies]
|
||||
anyhow = { workspace = true }
|
||||
tokio = { workspace = true, features = ["macros"] }
|
||||
sqlx = { workspace = true, features = [
|
||||
"runtime-tokio-rustls",
|
||||
"sqlite",
|
||||
"macros",
|
||||
"migrate",
|
||||
] }
|
||||
|
||||
[dev-dependencies]
|
||||
axum-test = "17.3.0"
|
||||
time = { workspace = true, features = ["macros"] }
|
||||
|
||||
+2
-2
@@ -2,9 +2,9 @@
|
||||
FROM harbor.nymte.ch/dockerhub/rust:latest AS builder
|
||||
|
||||
COPY ./ /usr/src/nym
|
||||
WORKDIR /usr/src/nym/nym-node-status-api
|
||||
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-api/
|
||||
|
||||
RUN cargo build --release
|
||||
RUN cargo build --release --features pg
|
||||
|
||||
|
||||
#-------------------------------------------------------------------
|
||||
@@ -0,0 +1,37 @@
|
||||
# this will only work with VPN, otherwise remove the harbor part
|
||||
FROM harbor.nymte.ch/dockerhub/rust:latest AS builder
|
||||
|
||||
COPY ./ /usr/src/nym
|
||||
WORKDIR /usr/src/nym/nym-node-status-api/nym-node-status-api/
|
||||
|
||||
RUN cargo build --release --features sqlite --no-default-features
|
||||
|
||||
|
||||
#-------------------------------------------------------------------
|
||||
# The following environment variables are required at runtime:
|
||||
#
|
||||
# EXPLORER_API
|
||||
# NYXD
|
||||
# NYM_API
|
||||
# DATABASE_URL
|
||||
#
|
||||
# And optionally:
|
||||
#
|
||||
# NYM_NODE_STATUS_API_NYM_HTTP_CACHE_TTL
|
||||
# NYM_NODE_STATUS_API_HTTP_PORT
|
||||
# NYM_API_CLIENT_TIMEOUT
|
||||
# EXPLORER_CLIENT_TIMEOUT
|
||||
# NODE_STATUS_API_MONITOR_REFRESH_INTERVAL
|
||||
# NODE_STATUS_API_TESTRUN_REFRESH_INTERVAL
|
||||
#
|
||||
# see https://github.com/nymtech/nym/blob/develop/nym-node-status-api/src/cli.rs for details
|
||||
#-------------------------------------------------------------------
|
||||
|
||||
FROM harbor.nymte.ch/dockerhub/ubuntu:24.04
|
||||
|
||||
RUN apt-get update && apt-get install -y ca-certificates
|
||||
|
||||
WORKDIR /nym
|
||||
|
||||
COPY --from=builder /usr/src/nym/target/release/nym-node-status-api ./
|
||||
ENTRYPOINT [ "/nym/nym-node-status-api" ]
|
||||
@@ -0,0 +1,117 @@
|
||||
# Makefile for nym-node-status-api database management
|
||||
|
||||
# --- Configuration ---
|
||||
TEST_DATABASE_URL := postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# Docker compose service names
|
||||
DB_SERVICE_NAME := postgres-test
|
||||
DB_CONTAINER_NAME := nym_node_status_api_postgres_test
|
||||
|
||||
# Default target
|
||||
.PHONY: default
|
||||
default: help
|
||||
|
||||
# --- Main Targets ---
|
||||
.PHONY: prepare-pg
|
||||
prepare-pg: test-db-up test-db-wait test-db-migrate test-db-prepare test-db-down ## Setup PostgreSQL and prepare SQLx offline cache
|
||||
|
||||
.PHONY: test-db
|
||||
test-db: test-db-up test-db-wait test-db-migrate test-db-run test-db-down ## Run tests with PostgreSQL database
|
||||
|
||||
.PHONY: dev-db
|
||||
dev-db: test-db-up test-db-wait test-db-migrate ## Start PostgreSQL for development (keeps running)
|
||||
@echo "PostgreSQL is running on port 5433"
|
||||
@echo "Connection string: $(TEST_DATABASE_URL)"
|
||||
|
||||
# --- Docker Compose Targets ---
|
||||
.PHONY: test-db-up
|
||||
test-db-up: ## Start the PostgreSQL test database in the background
|
||||
@echo "Starting PostgreSQL test database..."
|
||||
docker compose up -d $(DB_SERVICE_NAME)
|
||||
|
||||
.PHONY: test-db-wait
|
||||
test-db-wait: ## Wait for the PostgreSQL database to be healthy
|
||||
@echo "Waiting for PostgreSQL database..."
|
||||
@while ! docker inspect --format='{{.State.Health.Status}}' $(DB_CONTAINER_NAME) 2>/dev/null | grep -q 'healthy'; do \
|
||||
echo -n "."; \
|
||||
sleep 1; \
|
||||
done; \
|
||||
echo " Database is healthy!"
|
||||
|
||||
.PHONY: test-db-down
|
||||
test-db-down: ## Stop and remove the test database
|
||||
@echo "Stopping PostgreSQL test database..."
|
||||
docker compose down
|
||||
|
||||
# --- SQLx Targets ---
|
||||
.PHONY: test-db-migrate
|
||||
test-db-migrate: ## Run database migrations against PostgreSQL
|
||||
@echo "Running PostgreSQL migrations..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" sqlx migrate run --source migrations_pg
|
||||
|
||||
.PHONY: test-db-prepare
|
||||
test-db-prepare: ## Run sqlx prepare for compile-time query verification
|
||||
@echo "Running sqlx prepare for PostgreSQL..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" cargo sqlx prepare -- --features pg
|
||||
|
||||
# --- Build and Test Targets ---
|
||||
.PHONY: test-db-run
|
||||
test-db-run: ## Run tests with PostgreSQL feature
|
||||
@echo "Running tests with PostgreSQL..."
|
||||
DATABASE_URL="$(TEST_DATABASE_URL)" cargo test --features pg --no-default-features
|
||||
|
||||
.PHONY: build-pg
|
||||
build-pg: ## Build with PostgreSQL feature
|
||||
@echo "Building with PostgreSQL feature..."
|
||||
cargo build --features pg --no-default-features
|
||||
|
||||
.PHONY: build-sqlite
|
||||
build-sqlite: ## Build with SQLite feature (default)
|
||||
@echo "Building with SQLite feature..."
|
||||
cargo build --features sqlite --no-default-features
|
||||
|
||||
.PHONY: check-pg
|
||||
check-pg: ## Check code with PostgreSQL feature
|
||||
@echo "Checking code with PostgreSQL feature..."
|
||||
cargo check --features pg --no-default-features
|
||||
|
||||
.PHONY: check-sqlite
|
||||
check-sqlite: ## Check code with SQLite feature
|
||||
@echo "Checking code with SQLite feature..."
|
||||
cargo check --features sqlite --no-default-features
|
||||
|
||||
.PHONY: clippy
|
||||
clippy: clippy-pg clippy-sqlite
|
||||
|
||||
.PHONY: clippy-pg
|
||||
clippy-pg: ## Run clippy with PostgreSQL feature
|
||||
@echo "Running clippy with PostgreSQL feature..."
|
||||
cargo clippy --features pg --no-default-features -- -D warnings
|
||||
|
||||
.PHONY: clippy-sqlite
|
||||
clippy-sqlite: ## Run clippy with SQLite feature (default)
|
||||
@echo "Running clippy with SQLite feature..."
|
||||
cargo clippy --features sqlite --no-default-features -- -D warnings
|
||||
|
||||
# --- Cleanup Targets ---
|
||||
.PHONY: clean
|
||||
clean: ## Clean build artifacts and SQLx cache
|
||||
cargo clean
|
||||
rm -rf .sqlx
|
||||
|
||||
.PHONY: clean-db
|
||||
clean-db: test-db-down ## Stop database and clean volumes
|
||||
docker volume rm -f nym-node-status-api_postgres_test_data 2>/dev/null || true
|
||||
|
||||
# --- Utility Targets ---
|
||||
.PHONY: sqlx-cli
|
||||
sqlx-cli: ## Install sqlx-cli if not already installed
|
||||
@command -v sqlx >/dev/null 2>&1 || cargo install sqlx-cli --features postgres,sqlite
|
||||
|
||||
.PHONY: psql
|
||||
psql: ## Connect to the running PostgreSQL database with psql
|
||||
@docker exec -it $(DB_CONTAINER_NAME) psql -U testuser -d nym_node_status_api_test
|
||||
|
||||
.PHONY: help
|
||||
help: ## Show help for Makefile targets
|
||||
@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
|
||||
@@ -0,0 +1,104 @@
|
||||
# PostgreSQL Support for nym-node-status-api
|
||||
|
||||
This project now supports both SQLite (default) and PostgreSQL databases.
|
||||
|
||||
## Quick Start with PostgreSQL
|
||||
|
||||
### 1. Install Prerequisites
|
||||
|
||||
```bash
|
||||
# Install sqlx-cli if not already installed
|
||||
make sqlx-cli
|
||||
```
|
||||
|
||||
### 2. Prepare PostgreSQL for Development
|
||||
|
||||
```bash
|
||||
# This will:
|
||||
# - Start PostgreSQL in Docker
|
||||
# - Run migrations
|
||||
# - Generate SQLx offline query cache
|
||||
# - Stop the database
|
||||
make prepare-pg
|
||||
```
|
||||
|
||||
### 3. Build with PostgreSQL
|
||||
|
||||
```bash
|
||||
# Build with PostgreSQL feature
|
||||
make build-pg
|
||||
|
||||
# Or manually:
|
||||
cargo build --features pg --no-default-features
|
||||
```
|
||||
|
||||
### 4. Run with PostgreSQL
|
||||
|
||||
```bash
|
||||
# Start PostgreSQL for development (keeps running)
|
||||
make dev-db
|
||||
|
||||
# In another terminal, run the application
|
||||
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test \
|
||||
cargo run --features pg --no-default-features
|
||||
```
|
||||
|
||||
## Database Features
|
||||
|
||||
- `sqlite` (default): Uses SQLite database
|
||||
- `pg`: Uses PostgreSQL database
|
||||
|
||||
Only one database feature can be active at a time.
|
||||
|
||||
## Migration Differences
|
||||
|
||||
SQLite migrations are in `migrations/`, PostgreSQL migrations are in `migrations_pg/`.
|
||||
|
||||
Key differences:
|
||||
- **AUTOINCREMENT** → **SERIAL**
|
||||
- **INTEGER CHECK (0,1)** → **BOOLEAN**
|
||||
- **REAL** → **DOUBLE PRECISION**
|
||||
- No table recreation needed for constraint changes in PostgreSQL
|
||||
|
||||
## Makefile Targets
|
||||
|
||||
```bash
|
||||
make help # Show all available targets
|
||||
make prepare-pg # Setup PostgreSQL and prepare SQLx cache
|
||||
make dev-db # Start PostgreSQL for development
|
||||
make test-db # Run tests with PostgreSQL
|
||||
make build-pg # Build with PostgreSQL
|
||||
make build-sqlite # Build with SQLite
|
||||
make psql # Connect to running PostgreSQL
|
||||
make clean # Clean build artifacts
|
||||
make clean-db # Stop database and clean volumes
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See `.env.example` for all configuration options. Key variable:
|
||||
|
||||
```bash
|
||||
# For PostgreSQL:
|
||||
DATABASE_URL=postgres://testuser:testpass@localhost:5433/nym_node_status_api_test
|
||||
|
||||
# For SQLite:
|
||||
DATABASE_URL=sqlite://nym-node-status-api.sqlite
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### SQLx Offline Mode
|
||||
|
||||
If you see "no cached data for this query" errors:
|
||||
|
||||
1. Ensure PostgreSQL is running: `make dev-db`
|
||||
2. Run: `make test-db-prepare`
|
||||
|
||||
### Connection Refused
|
||||
|
||||
If you see "Connection refused" errors:
|
||||
|
||||
1. Check Docker is running: `docker ps`
|
||||
2. Check PostgreSQL container: `docker ps | grep nym_node_status_api_postgres_test`
|
||||
3. Restart database: `make test-db-down && make dev-db`
|
||||
@@ -0,0 +1,81 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Build and push Node Status API container to harbor.nymte.ch
|
||||
|
||||
set -e
|
||||
|
||||
# Configuration
|
||||
SCRIPT_DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
|
||||
WORKING_DIRECTORY="${SCRIPT_DIR}"
|
||||
CONTAINER_NAME="node-status-api"
|
||||
REGISTRY="harbor.nymte.ch"
|
||||
NAMESPACE="nym"
|
||||
|
||||
# Colors for output
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# Function to display usage
|
||||
usage() {
|
||||
echo "Usage: $0 [pg|sqlite|both]"
|
||||
echo " pg - Build and push PostgreSQL version"
|
||||
echo " sqlite - Build and push SQLite version"
|
||||
echo " both - Build and push both versions (default)"
|
||||
exit 1
|
||||
}
|
||||
|
||||
# Parse arguments
|
||||
DB_TYPE="${1:-both}"
|
||||
|
||||
if [[ ! "$DB_TYPE" =~ ^(pg|sqlite|both)$ ]]; then
|
||||
usage
|
||||
fi
|
||||
|
||||
# Get version from Cargo.toml
|
||||
VERSION=$(grep "^version = " "${WORKING_DIRECTORY}/Cargo.toml" | sed -E 's/version = "(.*)"/\1/')
|
||||
if [ -z "$VERSION" ]; then
|
||||
echo -e "${RED}Error: Could not extract version from Cargo.toml${NC}"
|
||||
exit 1
|
||||
fi
|
||||
echo -e "${YELLOW}Version: ${VERSION}${NC}"
|
||||
|
||||
# Login to Harbor
|
||||
echo -e "${GREEN}Logging into Harbor...${NC}"
|
||||
docker login "${REGISTRY}"
|
||||
|
||||
# Function to build and push
|
||||
build_and_push() {
|
||||
local db_type=$1
|
||||
local dockerfile="Dockerfile-${db_type}"
|
||||
|
||||
echo -e "${GREEN}Building ${db_type} container...${NC}"
|
||||
# Build from repository root (two levels up from script location)
|
||||
docker build -f "${WORKING_DIRECTORY}/${dockerfile}" "${SCRIPT_DIR}/../.." \
|
||||
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${db_type}" \
|
||||
-t "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${db_type}"
|
||||
|
||||
echo -e "${GREEN}Pushing ${db_type} container to Harbor...${NC}"
|
||||
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:${VERSION}-${db_type}"
|
||||
docker push "${REGISTRY}/${NAMESPACE}/${CONTAINER_NAME}:latest-${db_type}"
|
||||
|
||||
echo -e "${GREEN}Successfully built and pushed ${CONTAINER_NAME}:${VERSION}-${db_type}${NC}"
|
||||
}
|
||||
|
||||
# Build based on selection
|
||||
case "$DB_TYPE" in
|
||||
pg)
|
||||
build_and_push "pg"
|
||||
;;
|
||||
sqlite)
|
||||
build_and_push "sqlite"
|
||||
;;
|
||||
both)
|
||||
build_and_push "pg"
|
||||
echo ""
|
||||
build_and_push "sqlite"
|
||||
;;
|
||||
esac
|
||||
|
||||
echo -e "${GREEN}All builds completed successfully!${NC}"
|
||||
@@ -1,17 +1,20 @@
|
||||
use anyhow::{anyhow, Result};
|
||||
use anyhow::Result;
|
||||
#[cfg(feature = "sqlite")]
|
||||
use sqlx::{Connection, SqliteConnection};
|
||||
#[cfg(feature = "sqlite")]
|
||||
#[cfg(target_family = "unix")]
|
||||
use std::fs::Permissions;
|
||||
#[cfg(feature = "sqlite")]
|
||||
#[cfg(target_family = "unix")]
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
#[cfg(feature = "sqlite")]
|
||||
use tokio::{fs::File, io::AsyncWriteExt};
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
const SQLITE_DB_FILENAME: &str = "nym-node-status-api.sqlite";
|
||||
|
||||
/// If you need to re-run migrations or reset the db, just run
|
||||
/// cargo clean -p nym-node-status-api
|
||||
#[tokio::main(flavor = "current_thread")]
|
||||
async fn main() -> Result<()> {
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn init_db() -> Result<()> {
|
||||
let out_dir = read_env_var("OUT_DIR")?;
|
||||
let database_path = format!("{out_dir}/{SQLITE_DB_FILENAME}?mode=rwc");
|
||||
|
||||
@@ -30,11 +33,22 @@ async fn main() -> Result<()> {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// If you need to re-run migrations or reset the db, just run
|
||||
/// cargo clean -p nym-node-status-api
|
||||
#[tokio::main(flavor = "current_thread")]
|
||||
async fn main() -> Result<()> {
|
||||
#[cfg(feature = "sqlite")]
|
||||
init_db().await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
fn read_env_var(var: &str) -> Result<String> {
|
||||
std::env::var(var).map_err(|_| anyhow!("You need to set {} env var", var))
|
||||
std::env::var(var).map_err(|_| anyhow::anyhow!("You need to set {} env var", var))
|
||||
}
|
||||
|
||||
/// use `./enter_db.sh` to inspect DB
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn write_db_path_to_file(out_dir: &str, db_filename: &str) -> anyhow::Result<()> {
|
||||
let mut file = File::create("settings.sql").await?;
|
||||
let settings = ".mode columns
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
services:
|
||||
postgres-test:
|
||||
image: postgres:16-alpine
|
||||
container_name: nym_node_status_api_postgres_test
|
||||
environment:
|
||||
POSTGRES_DB: nym_node_status_api_test
|
||||
POSTGRES_USER: testuser
|
||||
POSTGRES_PASSWORD: testpass
|
||||
ports:
|
||||
- '5433:5432' # Map to 5433 to avoid conflicts with default PostgreSQL
|
||||
healthcheck:
|
||||
test: ['CMD-SHELL', 'pg_isready -U testuser -d nym_node_status_api_test']
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
# Optional: Add volume for persistent data during development
|
||||
# volumes:
|
||||
# - postgres_test_data:/var/lib/postgresql/data
|
||||
|
||||
# volumes:
|
||||
# postgres_test_data:
|
||||
@@ -0,0 +1,16 @@
|
||||
-- Add partial indexes for NOT NULL filtering to improve performance of /explorer/v3/nodes endpoint
|
||||
|
||||
-- Index for queries filtering on self_described IS NOT NULL
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_self_described_not_null
|
||||
ON nym_nodes(node_id)
|
||||
WHERE self_described IS NOT NULL;
|
||||
|
||||
-- Index for queries filtering on bond_info IS NOT NULL
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_info_not_null
|
||||
ON nym_nodes(node_id)
|
||||
WHERE bond_info IS NOT NULL;
|
||||
|
||||
-- Composite index for queries filtering on both bond_info AND self_described
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_self_described
|
||||
ON nym_nodes(node_id)
|
||||
WHERE bond_info IS NOT NULL AND self_described IS NOT NULL;
|
||||
@@ -0,0 +1,113 @@
|
||||
CREATE TABLE gateways
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR NOT NULL UNIQUE,
|
||||
self_described VARCHAR NOT NULL,
|
||||
explorer_pretty_bond VARCHAR,
|
||||
last_probe_result VARCHAR,
|
||||
last_probe_log VARCHAR,
|
||||
config_score INTEGER NOT NULL DEFAULT 0,
|
||||
config_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
config_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
routing_score INTEGER NOT NULL DEFAULT 0,
|
||||
routing_score_successes DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
routing_score_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
test_run_samples DOUBLE PRECISION NOT NULL DEFAULT 0,
|
||||
last_testrun_utc BIGINT,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
bonded BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
performance INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
|
||||
CREATE INDEX idx_gateway_description_gateway_identity_key ON gateways (gateway_identity_key);
|
||||
|
||||
|
||||
CREATE TABLE mixnodes (
|
||||
id SERIAL PRIMARY KEY,
|
||||
identity_key VARCHAR NOT NULL UNIQUE,
|
||||
mix_id BIGINT NOT NULL UNIQUE,
|
||||
bonded BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
total_stake BIGINT NOT NULL,
|
||||
host VARCHAR NOT NULL,
|
||||
http_api_port BIGINT NOT NULL,
|
||||
blacklisted BOOLEAN NOT NULL DEFAULT FALSE,
|
||||
full_details VARCHAR,
|
||||
self_described VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
is_dp_delegatee BOOLEAN NOT NULL DEFAULT FALSE
|
||||
);
|
||||
|
||||
CREATE INDEX idx_mixnodes_mix_id ON mixnodes (mix_id);
|
||||
CREATE INDEX idx_mixnodes_identity_key ON mixnodes (identity_key);
|
||||
|
||||
CREATE TABLE mixnode_description (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
|
||||
);
|
||||
|
||||
-- Indexes for description table
|
||||
CREATE INDEX idx_mixnode_description_mix_id ON mixnode_description (mix_id);
|
||||
|
||||
|
||||
CREATE TABLE summary
|
||||
(
|
||||
key VARCHAR PRIMARY KEY,
|
||||
value_json VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE summary_history
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
date VARCHAR UNIQUE NOT NULL,
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
value_json VARCHAR
|
||||
);
|
||||
|
||||
CREATE INDEX idx_summary_history_timestamp_utc ON summary_history (timestamp_utc);
|
||||
CREATE INDEX idx_summary_history_date ON summary_history (date);
|
||||
|
||||
|
||||
CREATE TABLE gateway_description (
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc BIGINT NOT NULL,
|
||||
FOREIGN KEY (gateway_identity_key) REFERENCES gateways (gateway_identity_key)
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE mixnode_daily_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT NOT NULL,
|
||||
total_stake BIGINT NOT NULL,
|
||||
date_utc VARCHAR NOT NULL,
|
||||
packets_received INTEGER DEFAULT 0,
|
||||
packets_sent INTEGER DEFAULT 0,
|
||||
packets_dropped INTEGER DEFAULT 0,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id),
|
||||
UNIQUE (mix_id, date_utc) -- This constraint automatically creates an index
|
||||
);
|
||||
|
||||
|
||||
CREATE TABLE testruns
|
||||
(
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_id INTEGER NOT NULL,
|
||||
status INTEGER NOT NULL, -- 0=pending, 1=in-progress, 2=complete
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
ip_address VARCHAR NOT NULL,
|
||||
log VARCHAR NOT NULL,
|
||||
FOREIGN KEY (gateway_id) REFERENCES gateways (id)
|
||||
);
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
ALTER TABLE testruns
|
||||
RENAME COLUMN timestamp_utc TO created_utc;
|
||||
|
||||
ALTER TABLE testruns
|
||||
ADD COLUMN last_assigned_utc BIGINT;
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
CREATE TABLE gateway_session_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
gateway_identity_key VARCHAR NOT NULL,
|
||||
node_id BIGINT NOT NULL,
|
||||
day DATE NOT NULL,
|
||||
unique_active_clients BIGINT NOT NULL,
|
||||
session_started BIGINT NOT NULL,
|
||||
users_hashes VARCHAR,
|
||||
vpn_sessions VARCHAR,
|
||||
mixnet_sessions VARCHAR,
|
||||
unknown_sessions VARCHAR,
|
||||
UNIQUE (node_id, day) -- This constraint automatically creates an index
|
||||
);
|
||||
|
||||
CREATE INDEX idx_gateway_session_stats_identity_key ON gateway_session_stats (gateway_identity_key);
|
||||
CREATE INDEX idx_gateway_session_stats_day ON gateway_session_stats (day);
|
||||
+11
@@ -0,0 +1,11 @@
|
||||
CREATE TABLE mixnode_packet_stats_raw (
|
||||
id SERIAL PRIMARY KEY,
|
||||
mix_id BIGINT NOT NULL,
|
||||
timestamp_utc BIGINT NOT NULL,
|
||||
packets_received INTEGER,
|
||||
packets_sent INTEGER,
|
||||
packets_dropped INTEGER,
|
||||
FOREIGN KEY (mix_id) REFERENCES mixnodes (mix_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_mixnode_packet_stats_raw_mix_id_timestamp_utc ON mixnode_packet_stats_raw (mix_id, timestamp_utc);
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
ALTER TABLE mixnodes DROP COLUMN blacklisted;
|
||||
ALTER TABLE gateways DROP COLUMN blacklisted;
|
||||
|
||||
CREATE TABLE nym_nodes (
|
||||
node_id INTEGER PRIMARY KEY,
|
||||
ed25519_identity_pubkey VARCHAR NOT NULL UNIQUE,
|
||||
total_stake BIGINT NOT NULL,
|
||||
ip_addresses TEXT NOT NULL,
|
||||
mix_port INTEGER NOT NULL,
|
||||
x25519_sphinx_pubkey VARCHAR NOT NULL UNIQUE,
|
||||
node_role TEXT NOT NULL,
|
||||
supported_roles TEXT NOT NULL,
|
||||
performance VARCHAR NOT NULL,
|
||||
entry TEXT,
|
||||
last_updated_utc INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX idx_nym_nodes_node_id ON nym_nodes (node_id);
|
||||
CREATE INDEX idx_nym_nodes_ed25519_identity_pubkey ON nym_nodes (ed25519_identity_pubkey);
|
||||
|
||||
CREATE TABLE nym_node_descriptions (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER UNIQUE NOT NULL,
|
||||
moniker VARCHAR,
|
||||
website VARCHAR,
|
||||
security_contact VARCHAR,
|
||||
details VARCHAR,
|
||||
last_updated_utc INTEGER NOT NULL,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
|
||||
);
|
||||
|
||||
CREATE TABLE nym_nodes_packet_stats_raw (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
timestamp_utc INTEGER NOT NULL,
|
||||
packets_received INTEGER,
|
||||
packets_sent INTEGER,
|
||||
packets_dropped INTEGER,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_nym_nodes_packet_stats_raw_node_id_timestamp_utc ON nym_nodes_packet_stats_raw (node_id, timestamp_utc);
|
||||
|
||||
CREATE TABLE nym_node_daily_mixing_stats (
|
||||
id SERIAL PRIMARY KEY,
|
||||
node_id INTEGER NOT NULL,
|
||||
total_stake BIGINT NOT NULL,
|
||||
date_utc VARCHAR NOT NULL,
|
||||
packets_received INTEGER DEFAULT 0,
|
||||
packets_sent INTEGER DEFAULT 0,
|
||||
packets_dropped INTEGER DEFAULT 0,
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id),
|
||||
UNIQUE (node_id, date_utc) -- This constraint automatically creates an index
|
||||
);
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
ALTER TABLE nym_nodes ADD COLUMN self_described TEXT;
|
||||
ALTER TABLE nym_nodes ADD COLUMN bond_info TEXT;
|
||||
|
||||
-- PostgreSQL doesn't need table recreation for adding ON DELETE CASCADE
|
||||
-- We can drop and recreate the foreign key constraints directly
|
||||
|
||||
-- Drop existing foreign key constraints
|
||||
ALTER TABLE nym_node_descriptions DROP CONSTRAINT IF EXISTS nym_node_descriptions_node_id_fkey;
|
||||
ALTER TABLE nym_nodes_packet_stats_raw DROP CONSTRAINT IF EXISTS nym_nodes_packet_stats_raw_node_id_fkey;
|
||||
ALTER TABLE nym_node_daily_mixing_stats DROP CONSTRAINT IF EXISTS nym_node_daily_mixing_stats_node_id_fkey;
|
||||
|
||||
-- Add foreign key constraints with ON DELETE CASCADE
|
||||
ALTER TABLE nym_node_descriptions
|
||||
ADD CONSTRAINT nym_node_descriptions_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
|
||||
ALTER TABLE nym_nodes_packet_stats_raw
|
||||
ADD CONSTRAINT nym_nodes_packet_stats_raw_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
|
||||
ALTER TABLE nym_node_daily_mixing_stats
|
||||
ADD CONSTRAINT nym_node_daily_mixing_stats_node_id_fkey
|
||||
FOREIGN KEY (node_id) REFERENCES nym_nodes (node_id) ON DELETE CASCADE;
|
||||
+9
@@ -0,0 +1,9 @@
|
||||
-- Removing UNIQUE constraints on nym_nodes
|
||||
-- In PostgreSQL, we can drop constraints directly without recreating the table
|
||||
|
||||
-- Drop the unique constraints
|
||||
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_ed25519_identity_pubkey_key;
|
||||
ALTER TABLE nym_nodes DROP CONSTRAINT IF EXISTS nym_nodes_x25519_sphinx_pubkey_key;
|
||||
|
||||
-- The columns and indexes remain, only the unique constraints are removed
|
||||
-- The existing indexes idx_nym_nodes_node_id and idx_nym_nodes_ed25519_identity_pubkey remain unchanged
|
||||
@@ -0,0 +1,112 @@
|
||||
-- for a couple of days after migrating chrono -> time, we stored dates as
|
||||
-- 2025-June-DD instead of 2025-06-DD. This migration fixes those entries.
|
||||
--
|
||||
-- Because of a UNIQUE constraint on (node_id, date_utc), we can't just rename in-place.
|
||||
-- - merge (add) node stats back to the original table where conflict (node_id, date_utc) would exist
|
||||
-- - delete invalid records from original table (those stats were merged into correct rows above)
|
||||
-- - insert rows that did not have a conflicting (node_Id, date_utc) combo
|
||||
-- Conflicts affect only the date which has both kinds of entries,
|
||||
-- e.g. 2025-06-05 and 2025-June-05 (date when this change was deployed)
|
||||
--
|
||||
-- This applies to both affected tables.
|
||||
|
||||
-- ----------------------------------------
|
||||
-- mixnode_daily_stats
|
||||
-- ----------------------------------------
|
||||
|
||||
-- First, copy over rows with invalid date to a temp table (in the correct date format)
|
||||
CREATE TEMP TABLE tmp_mix AS
|
||||
SELECT
|
||||
mix_id,
|
||||
REPLACE(date_utc,'June','06') AS new_date,
|
||||
SUM(total_stake) AS total_stake_sum,
|
||||
SUM(packets_received) AS packets_received_sum,
|
||||
SUM(packets_sent) AS packets_sent_sum,
|
||||
SUM(packets_dropped) AS packets_dropped_sum
|
||||
FROM mixnode_daily_stats
|
||||
WHERE date_utc LIKE '%June%'
|
||||
GROUP BY mix_id, new_date;
|
||||
|
||||
UPDATE mixnode_daily_stats AS m
|
||||
SET
|
||||
total_stake = m.total_stake,
|
||||
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
|
||||
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc),
|
||||
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_mix WHERE mix_id = m.mix_id AND new_date = m.date_utc)
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM tmp_mix
|
||||
WHERE mix_id = m.mix_id
|
||||
AND new_date = m.date_utc
|
||||
);
|
||||
|
||||
DELETE FROM mixnode_daily_stats
|
||||
WHERE date_utc LIKE '%June%';
|
||||
|
||||
INSERT INTO mixnode_daily_stats
|
||||
(mix_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
|
||||
SELECT
|
||||
mix_id,
|
||||
new_date,
|
||||
total_stake_sum,
|
||||
packets_received_sum,
|
||||
packets_sent_sum,
|
||||
packets_dropped_sum
|
||||
FROM tmp_mix AS t
|
||||
-- only those whose new_date did _not_ already exist
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM mixnode_daily_stats AS m
|
||||
WHERE m.mix_id = t.mix_id
|
||||
AND m.date_utc = t.new_date
|
||||
);
|
||||
|
||||
DROP TABLE tmp_mix;
|
||||
|
||||
|
||||
-- ----------------------------------------
|
||||
-- nym_node_daily_mixing_stats
|
||||
-- ----------------------------------------
|
||||
|
||||
CREATE TEMP TABLE tmp_nym_node_stats AS
|
||||
SELECT
|
||||
node_id,
|
||||
REPLACE(date_utc,'June','06') AS new_date,
|
||||
SUM(total_stake) AS total_stake_sum,
|
||||
SUM(packets_received) AS packets_received_sum,
|
||||
SUM(packets_sent) AS packets_sent_sum,
|
||||
SUM(packets_dropped) AS packets_dropped_sum
|
||||
FROM nym_node_daily_mixing_stats
|
||||
WHERE date_utc LIKE '%June%'
|
||||
GROUP BY node_id, new_date;
|
||||
|
||||
UPDATE nym_node_daily_mixing_stats AS m
|
||||
SET
|
||||
total_stake = m.total_stake,
|
||||
packets_received = m.packets_received + (SELECT packets_received_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
|
||||
packets_sent = m.packets_sent + (SELECT packets_sent_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc),
|
||||
packets_dropped = m.packets_dropped + (SELECT packets_dropped_sum FROM tmp_nym_node_stats WHERE node_id = m.node_id AND new_date = m.date_utc)
|
||||
WHERE EXISTS (
|
||||
SELECT 1 FROM tmp_nym_node_stats
|
||||
WHERE node_id = m.node_id
|
||||
AND new_date = m.date_utc
|
||||
);
|
||||
|
||||
DELETE FROM nym_node_daily_mixing_stats
|
||||
WHERE date_utc LIKE '%June%';
|
||||
|
||||
INSERT INTO nym_node_daily_mixing_stats
|
||||
(node_id, date_utc, total_stake, packets_received, packets_sent, packets_dropped)
|
||||
SELECT
|
||||
node_id,
|
||||
new_date,
|
||||
total_stake_sum,
|
||||
packets_received_sum,
|
||||
packets_sent_sum,
|
||||
packets_dropped_sum
|
||||
FROM tmp_nym_node_stats AS t
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1 FROM nym_node_daily_mixing_stats AS m
|
||||
WHERE m.node_id = t.node_id
|
||||
AND m.date_utc = t.new_date
|
||||
);
|
||||
|
||||
DROP TABLE tmp_nym_node_stats;
|
||||
@@ -0,0 +1,3 @@
|
||||
-- Convert ip_addresses column from TEXT to JSONB for better type safety
|
||||
ALTER TABLE nym_nodes
|
||||
ALTER COLUMN ip_addresses TYPE JSONB USING ip_addresses::JSONB;
|
||||
+7
@@ -0,0 +1,7 @@
|
||||
-- Convert remaining TEXT columns that store JSON to JSONB
|
||||
ALTER TABLE nym_nodes
|
||||
ALTER COLUMN node_role TYPE JSONB USING node_role::JSONB,
|
||||
ALTER COLUMN supported_roles TYPE JSONB USING supported_roles::JSONB,
|
||||
ALTER COLUMN entry TYPE JSONB USING entry::JSONB,
|
||||
ALTER COLUMN self_described TYPE JSONB USING self_described::JSONB,
|
||||
ALTER COLUMN bond_info TYPE JSONB USING bond_info::JSONB;
|
||||
+17
@@ -0,0 +1,17 @@
|
||||
-- Add partial indexes for NOT NULL filtering to improve performance of /explorer/v3/nodes endpoint
|
||||
-- PostgreSQL version
|
||||
|
||||
-- Index for queries filtering on self_described IS NOT NULL
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_self_described_not_null
|
||||
ON nym_nodes(node_id)
|
||||
WHERE self_described IS NOT NULL;
|
||||
|
||||
-- Index for queries filtering on bond_info IS NOT NULL
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_info_not_null
|
||||
ON nym_nodes(node_id)
|
||||
WHERE bond_info IS NOT NULL;
|
||||
|
||||
-- Composite index for queries filtering on both bond_info AND self_described
|
||||
CREATE INDEX IF NOT EXISTS idx_nym_nodes_bond_self_described
|
||||
ON nym_nodes(node_id)
|
||||
WHERE bond_info IS NOT NULL AND self_described IS NOT NULL;
|
||||
@@ -1,24 +1,52 @@
|
||||
use anyhow::{anyhow, Result};
|
||||
use sqlx::{
|
||||
migrate::Migrator,
|
||||
query,
|
||||
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
|
||||
ConnectOptions, SqlitePool,
|
||||
};
|
||||
use std::{str::FromStr, time::Duration};
|
||||
|
||||
pub(crate) mod models;
|
||||
pub(crate) mod queries;
|
||||
pub(crate) mod query_wrapper;
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests;
|
||||
|
||||
// Re-export the query wrapper functions for easier access
|
||||
pub(crate) use query_wrapper::query;
|
||||
#[allow(unused_imports)]
|
||||
pub(crate) use query_wrapper::query_as;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
use sqlx::{
|
||||
migrate::Migrator,
|
||||
sqlite::{SqliteAutoVacuum, SqliteConnectOptions, SqliteSynchronous},
|
||||
ConnectOptions, SqlitePool,
|
||||
};
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
use sqlx::{migrate::Migrator, postgres::PgConnectOptions, ConnectOptions, PgPool};
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
static MIGRATOR: Migrator = sqlx::migrate!("./migrations");
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
static MIGRATOR: Migrator = sqlx::migrate!("./migrations_pg");
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) type DbPool = SqlitePool;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) type DbPool = PgPool;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Sqlite>;
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) type DbConnection = sqlx::pool::PoolConnection<sqlx::Postgres>;
|
||||
|
||||
pub(crate) struct Storage {
|
||||
pool: DbPool,
|
||||
}
|
||||
|
||||
impl Storage {
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub async fn init(connection_url: String, busy_timeout: Duration) -> Result<Self> {
|
||||
let connect_options = SqliteConnectOptions::from_str(&connection_url)?
|
||||
.journal_mode(sqlx::sqlite::SqliteJournalMode::Wal)
|
||||
@@ -41,16 +69,39 @@ impl Storage {
|
||||
Ok(Storage { pool })
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub async fn init(connection_url: String, _busy_timeout: Duration) -> Result<Self> {
|
||||
use std::env;
|
||||
let mut connect_options =
|
||||
PgConnectOptions::from_str(&connection_url)?.disable_statement_logging();
|
||||
|
||||
let ssl_cert_path = env::var("PG_CERT").ok();
|
||||
|
||||
if let Some(ssl_cert) = ssl_cert_path {
|
||||
connect_options = connect_options
|
||||
.ssl_mode(sqlx::postgres::PgSslMode::Require)
|
||||
.ssl_root_cert(ssl_cert);
|
||||
}
|
||||
let pool = sqlx::PgPool::connect_with(connect_options)
|
||||
.await
|
||||
.map_err(|err| anyhow!("Failed to connect to {}: {}", &connection_url, err))?;
|
||||
|
||||
MIGRATOR.run(&pool).await?;
|
||||
|
||||
Ok(Storage { pool })
|
||||
}
|
||||
|
||||
/// Cloning pool is cheap, it's the same underlying set of connections
|
||||
pub fn pool_owned(&self) -> DbPool {
|
||||
self.pool.clone()
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
async fn assert_busy_timeout(pool: DbPool, expected_busy_timeout_s: i64) -> Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
// Sqlite stores this value as miliseconds
|
||||
// https://www.sqlite.org/pragma.html#pragma_busy_timeout
|
||||
let busy_timeout_db = query!("PRAGMA busy_timeout;")
|
||||
let busy_timeout_db = sqlx::query!("PRAGMA busy_timeout;")
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -38,11 +38,11 @@ pub(crate) struct GatewayInsertRecord {
|
||||
pub(crate) performance: u8,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub(crate) struct GatewayDto {
|
||||
pub(crate) gateway_identity_key: String,
|
||||
pub(crate) bonded: bool,
|
||||
pub(crate) performance: i64,
|
||||
pub(crate) performance: i32,
|
||||
pub(crate) self_described: Option<String>,
|
||||
pub(crate) explorer_pretty_bond: Option<String>,
|
||||
pub(crate) last_probe_result: Option<String>,
|
||||
@@ -121,7 +121,7 @@ pub(crate) struct MixnodeRecord {
|
||||
pub(crate) is_dp_delegatee: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub(crate) struct MixnodeDto {
|
||||
pub(crate) mix_id: i64,
|
||||
pub(crate) bonded: bool,
|
||||
@@ -183,14 +183,14 @@ pub(crate) struct BondedStatusDto {
|
||||
}
|
||||
|
||||
#[allow(unused)]
|
||||
#[derive(Debug, Clone, Default)]
|
||||
#[derive(Debug, Clone, Default, FromRow)]
|
||||
pub(crate) struct SummaryDto {
|
||||
pub(crate) key: String,
|
||||
pub(crate) value_json: String,
|
||||
pub(crate) last_updated_utc: i64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Default)]
|
||||
#[derive(Debug, Clone, Default, FromRow)]
|
||||
pub(crate) struct SummaryHistoryDto {
|
||||
#[allow(dead_code)]
|
||||
pub id: i64,
|
||||
@@ -287,11 +287,11 @@ pub(crate) mod gateway {
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // not dead code, this is SQL data model
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub struct TestRunDto {
|
||||
pub id: i64,
|
||||
pub gateway_id: i64,
|
||||
pub status: i64,
|
||||
pub id: i32,
|
||||
pub gateway_id: i32,
|
||||
pub status: i32,
|
||||
pub created_utc: i64,
|
||||
pub ip_address: String,
|
||||
pub log: String,
|
||||
@@ -313,9 +313,9 @@ pub struct GatewayIdentityDto {
|
||||
}
|
||||
|
||||
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
|
||||
#[derive(Debug, Clone)]
|
||||
#[derive(Debug, Clone, FromRow)]
|
||||
pub struct GatewayInfoDto {
|
||||
pub id: i64,
|
||||
pub id: i32,
|
||||
pub gateway_identity_key: String,
|
||||
pub self_described: Option<String>,
|
||||
pub explorer_pretty_bond: Option<String>,
|
||||
@@ -379,6 +379,7 @@ impl ScrapeNodeKind {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone)]
|
||||
pub(crate) struct ScraperNodeInfo {
|
||||
pub node_kind: ScrapeNodeKind,
|
||||
pub hosts: Vec<String>,
|
||||
@@ -412,11 +413,11 @@ impl ScraperNodeInfo {
|
||||
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
|
||||
#[derive(FromRow, Debug)]
|
||||
pub(crate) struct NymNodeDto {
|
||||
pub node_id: i64,
|
||||
pub node_id: i32,
|
||||
pub ed25519_identity_pubkey: String,
|
||||
pub total_stake: i64,
|
||||
pub ip_addresses: serde_json::Value,
|
||||
pub mix_port: i64,
|
||||
pub mix_port: i32,
|
||||
pub x25519_sphinx_pubkey: String,
|
||||
pub node_role: serde_json::Value,
|
||||
pub supported_roles: serde_json::Value,
|
||||
@@ -429,11 +430,11 @@ pub(crate) struct NymNodeDto {
|
||||
#[allow(dead_code)] // it's not dead code but clippy doesn't detect usage in sqlx macros
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct NymNodeInsertRecord {
|
||||
pub node_id: i64,
|
||||
pub node_id: i32,
|
||||
pub ed25519_identity_pubkey: String,
|
||||
pub total_stake: i64,
|
||||
pub ip_addresses: serde_json::Value,
|
||||
pub mix_port: i64,
|
||||
pub mix_port: i32,
|
||||
pub x25519_sphinx_pubkey: String,
|
||||
pub node_role: serde_json::Value,
|
||||
pub supported_roles: serde_json::Value,
|
||||
@@ -441,7 +442,7 @@ pub(crate) struct NymNodeInsertRecord {
|
||||
pub entry: Option<serde_json::Value>,
|
||||
pub self_described: Option<serde_json::Value>,
|
||||
pub bond_info: Option<serde_json::Value>,
|
||||
pub last_updated_utc: String,
|
||||
pub last_updated_utc: i64,
|
||||
}
|
||||
|
||||
impl NymNodeInsertRecord {
|
||||
@@ -450,7 +451,7 @@ impl NymNodeInsertRecord {
|
||||
bond_info: Option<&NymNodeDetails>,
|
||||
self_described: Option<&NymNodeDescription>,
|
||||
) -> anyhow::Result<Self> {
|
||||
let now = OffsetDateTime::now_utc().to_string();
|
||||
let now = OffsetDateTime::now_utc().unix_timestamp();
|
||||
|
||||
// if bond info is missing, set stake to 0
|
||||
let total_stake = bond_info
|
||||
@@ -461,11 +462,11 @@ impl NymNodeInsertRecord {
|
||||
let self_described = serialize_opt_to_value!(self_described)?;
|
||||
|
||||
let record = Self {
|
||||
node_id: skimmed_node.node_id.into(),
|
||||
node_id: skimmed_node.node_id as i32,
|
||||
ed25519_identity_pubkey: skimmed_node.ed25519_identity_pubkey.to_base58_string(),
|
||||
total_stake,
|
||||
ip_addresses: serde_json::to_value(&skimmed_node.ip_addresses)?,
|
||||
mix_port: skimmed_node.mix_port as i64,
|
||||
mix_port: skimmed_node.mix_port as i32,
|
||||
x25519_sphinx_pubkey: skimmed_node.x25519_sphinx_pubkey.to_base58_string(),
|
||||
node_role: serde_json::to_value(&skimmed_node.role)?,
|
||||
supported_roles: serde_json::to_value(skimmed_node.supported_roles)?,
|
||||
@@ -514,11 +515,11 @@ impl TryFrom<NymNodeDto> for SkimmedNode {
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug, Serialize, Deserialize, sqlx::Decode)]
|
||||
#[derive(Debug, Serialize, Deserialize, sqlx::Decode, FromRow)]
|
||||
pub struct NodeStats {
|
||||
pub packets_received: i64,
|
||||
pub packets_sent: i64,
|
||||
pub packets_dropped: i64,
|
||||
pub packets_received: i32,
|
||||
pub packets_sent: i32,
|
||||
pub packets_dropped: i32,
|
||||
}
|
||||
|
||||
pub struct InsertStatsRecord {
|
||||
|
||||
@@ -3,32 +3,32 @@ use std::collections::HashSet;
|
||||
use crate::{
|
||||
db::{
|
||||
models::{GatewayDto, GatewayInsertRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
http::models::Gateway,
|
||||
node_scraper::helpers::NodeDescriptionResponse,
|
||||
};
|
||||
use futures_util::TryStreamExt;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use tracing::error;
|
||||
|
||||
pub(crate) async fn select_gateway_identity(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
gateway_pk: i64,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i32,
|
||||
) -> anyhow::Result<String> {
|
||||
let record = sqlx::query!(
|
||||
let record = crate::db::query(
|
||||
r#"SELECT
|
||||
gateway_identity_key
|
||||
FROM
|
||||
gateways
|
||||
WHERE
|
||||
id = ?"#,
|
||||
gateway_pk
|
||||
)
|
||||
.bind(gateway_pk)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(record.gateway_identity_key)
|
||||
Ok(record.try_get("gateway_identity_key")?)
|
||||
}
|
||||
|
||||
pub(crate) async fn update_bonded_gateways(
|
||||
@@ -37,7 +37,7 @@ pub(crate) async fn update_bonded_gateways(
|
||||
) -> anyhow::Result<()> {
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"UPDATE
|
||||
gateways
|
||||
SET
|
||||
@@ -48,7 +48,7 @@ pub(crate) async fn update_bonded_gateways(
|
||||
.await?;
|
||||
|
||||
for record in gateways {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO gateways
|
||||
(gateway_identity_key, bonded,
|
||||
self_described, explorer_pretty_bond,
|
||||
@@ -60,13 +60,13 @@ pub(crate) async fn update_bonded_gateways(
|
||||
explorer_pretty_bond=excluded.explorer_pretty_bond,
|
||||
last_updated_utc=excluded.last_updated_utc,
|
||||
performance = excluded.performance;",
|
||||
record.identity_key,
|
||||
record.bonded,
|
||||
record.self_described,
|
||||
record.explorer_pretty_bond,
|
||||
record.last_updated_utc,
|
||||
record.performance
|
||||
)
|
||||
.bind(record.identity_key)
|
||||
.bind(record.bonded)
|
||||
.bind(record.self_described)
|
||||
.bind(record.explorer_pretty_bond)
|
||||
.bind(record.last_updated_utc)
|
||||
.bind(record.performance as i32)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
@@ -78,22 +78,21 @@ pub(crate) async fn update_bonded_gateways(
|
||||
|
||||
pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gateway>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as!(
|
||||
GatewayDto,
|
||||
let items = crate::db::query_as::<GatewayDto>(
|
||||
r#"SELECT
|
||||
gw.gateway_identity_key as "gateway_identity_key!",
|
||||
gw.bonded as "bonded: bool",
|
||||
gw.performance as "performance!",
|
||||
gw.self_described as "self_described?",
|
||||
gw.explorer_pretty_bond as "explorer_pretty_bond?",
|
||||
gw.last_probe_result as "last_probe_result?",
|
||||
gw.last_probe_log as "last_probe_log?",
|
||||
gw.last_testrun_utc as "last_testrun_utc?",
|
||||
gw.last_updated_utc as "last_updated_utc!",
|
||||
COALESCE(gd.moniker, "NA") as "moniker!",
|
||||
COALESCE(gd.website, "NA") as "website!",
|
||||
COALESCE(gd.security_contact, "NA") as "security_contact!",
|
||||
COALESCE(gd.details, "NA") as "details!"
|
||||
gw.gateway_identity_key,
|
||||
gw.bonded,
|
||||
gw.performance,
|
||||
gw.self_described,
|
||||
gw.explorer_pretty_bond,
|
||||
gw.last_probe_result,
|
||||
gw.last_probe_log,
|
||||
gw.last_testrun_utc,
|
||||
gw.last_updated_utc,
|
||||
COALESCE(gd.moniker, 'NA') as moniker,
|
||||
COALESCE(gd.website, 'NA') as website,
|
||||
COALESCE(gd.security_contact, 'NA') as security_contact,
|
||||
COALESCE(gd.details, 'NA') as details
|
||||
FROM gateways gw
|
||||
LEFT JOIN gateway_description gd
|
||||
ON gw.gateway_identity_key = gd.gateway_identity_key
|
||||
@@ -114,29 +113,29 @@ pub(crate) async fn get_all_gateways(pool: &DbPool) -> anyhow::Result<Vec<Gatewa
|
||||
|
||||
pub(crate) async fn get_bonded_gateway_id_keys(pool: &DbPool) -> anyhow::Result<HashSet<String>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query!(
|
||||
let items = crate::db::query(
|
||||
r#"
|
||||
SELECT gateway_identity_key
|
||||
FROM gateways
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|record| record.gateway_identity_key)
|
||||
.map(|record| record.try_get::<String, _>("gateway_identity_key").unwrap())
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_gateway_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
identity_key: &str,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
identity_key: String,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO gateway_description (
|
||||
gateway_identity_key,
|
||||
@@ -153,15 +152,54 @@ pub(crate) async fn insert_gateway_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
identity_key,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(identity_key)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_or_create_gateway(
|
||||
conn: &mut DbConnection,
|
||||
gateway_identity_key: &str,
|
||||
) -> anyhow::Result<i32> {
|
||||
// Try to find existing gateway
|
||||
let existing = crate::db::query("SELECT id FROM gateways WHERE gateway_identity_key = ?")
|
||||
.bind(gateway_identity_key.to_string())
|
||||
.fetch_optional(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
if let Some(row) = existing {
|
||||
return Ok(row.try_get("id")?);
|
||||
}
|
||||
|
||||
// Create new gateway
|
||||
tracing::info!("Creating new gateway record for {}", gateway_identity_key);
|
||||
let now = crate::utils::now_utc().unix_timestamp();
|
||||
|
||||
let result = crate::db::query(
|
||||
r#"INSERT INTO gateways (
|
||||
gateway_identity_key,
|
||||
bonded,
|
||||
performance,
|
||||
self_described,
|
||||
last_updated_utc
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
RETURNING id"#,
|
||||
)
|
||||
.bind(gateway_identity_key.to_string())
|
||||
.bind(true) // Assume bonded since being tested
|
||||
.bind(0) // Initial performance
|
||||
.bind("null")
|
||||
.bind(now)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(result.try_get("id")?)
|
||||
}
|
||||
|
||||
@@ -6,6 +6,7 @@ use futures_util::TryStreamExt;
|
||||
use time::Date;
|
||||
use tracing::error;
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) async fn insert_session_records(
|
||||
pool: &DbPool,
|
||||
records: Vec<GatewaySessionsRecord>,
|
||||
@@ -36,6 +37,38 @@ pub(crate) async fn insert_session_records(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) async fn insert_session_records(
|
||||
pool: &DbPool,
|
||||
records: Vec<GatewaySessionsRecord>,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut tx = pool.begin().await?;
|
||||
for record in records {
|
||||
sqlx::query!(
|
||||
"INSERT INTO gateway_session_stats
|
||||
(gateway_identity_key, node_id, day,
|
||||
unique_active_clients, session_started, users_hashes,
|
||||
vpn_sessions, mixnet_sessions, unknown_sessions)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
ON CONFLICT DO NOTHING",
|
||||
record.gateway_identity_key,
|
||||
record.node_id,
|
||||
record.day,
|
||||
record.unique_active_clients,
|
||||
record.session_started,
|
||||
record.users_hashes,
|
||||
record.vpn_sessions,
|
||||
record.mixnet_sessions,
|
||||
record.unknown_sessions,
|
||||
)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
tx.commit().await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<SessionStats>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as(
|
||||
@@ -68,7 +101,8 @@ pub(crate) async fn get_sessions_stats(pool: &DbPool) -> anyhow::Result<Vec<Sess
|
||||
|
||||
pub(crate) async fn delete_old_records(pool: &DbPool, cut_off: Date) -> anyhow::Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
sqlx::query!("DELETE FROM gateway_session_stats WHERE day <= ?", cut_off)
|
||||
crate::db::query("DELETE FROM gateway_session_stats WHERE day <= ?")
|
||||
.bind(cut_off)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
Ok(())
|
||||
|
||||
@@ -6,8 +6,8 @@ use crate::db::{models::NetworkSummary, DbPool};
|
||||
/// `daily_summary`
|
||||
pub(crate) async fn insert_summaries(
|
||||
pool: &DbPool,
|
||||
summaries: &[(&str, usize)],
|
||||
summary: &NetworkSummary,
|
||||
summaries: Vec<(String, usize)>,
|
||||
summary: NetworkSummary,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
insert_summary(pool, summaries, last_updated).await?;
|
||||
@@ -19,7 +19,7 @@ pub(crate) async fn insert_summaries(
|
||||
|
||||
async fn insert_summary(
|
||||
pool: &DbPool,
|
||||
summaries: &[(&str, usize)],
|
||||
summaries: Vec<(String, usize)>,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
let timestamp = last_updated.unix_timestamp();
|
||||
@@ -27,17 +27,17 @@ async fn insert_summary(
|
||||
|
||||
for (kind, value) in summaries {
|
||||
let value = value.to_string();
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO summary
|
||||
(key, value_json, last_updated_utc)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(key) DO UPDATE SET
|
||||
value_json=excluded.value_json,
|
||||
last_updated_utc=excluded.last_updated_utc;",
|
||||
kind,
|
||||
value,
|
||||
timestamp
|
||||
)
|
||||
.bind(kind.clone())
|
||||
.bind(value)
|
||||
.bind(timestamp)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.map_err(|err| {
|
||||
@@ -60,7 +60,7 @@ async fn insert_summary(
|
||||
/// This is not aggregate data, it's a set of latest data points
|
||||
async fn insert_summary_history(
|
||||
pool: &DbPool,
|
||||
summary: &NetworkSummary,
|
||||
summary: NetworkSummary,
|
||||
last_updated: UtcDateTime,
|
||||
) -> anyhow::Result<()> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
@@ -70,17 +70,17 @@ async fn insert_summary_history(
|
||||
|
||||
let date = datetime_to_only_date_str(last_updated);
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO summary_history
|
||||
(date, timestamp_utc, value_json)
|
||||
VALUES (?, ?, ?)
|
||||
ON CONFLICT(date) DO UPDATE SET
|
||||
timestamp_utc=excluded.timestamp_utc,
|
||||
value_json=excluded.value_json;",
|
||||
date,
|
||||
timestamp,
|
||||
value_json
|
||||
)
|
||||
.bind(date)
|
||||
.bind(timestamp)
|
||||
.bind(value_json)
|
||||
.execute(&mut *conn)
|
||||
.await?;
|
||||
|
||||
|
||||
@@ -1,13 +1,13 @@
|
||||
use std::collections::HashSet;
|
||||
|
||||
use futures_util::TryStreamExt;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use tracing::error;
|
||||
|
||||
use crate::{
|
||||
db::{
|
||||
models::{MixnodeDto, MixnodeRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
http::models::{DailyStats, Mixnode},
|
||||
node_scraper::helpers::NodeDescriptionResponse,
|
||||
@@ -20,7 +20,7 @@ pub(crate) async fn update_mixnodes(
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
// mark all as unbonded
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"UPDATE
|
||||
mixnodes
|
||||
SET
|
||||
@@ -31,9 +31,9 @@ pub(crate) async fn update_mixnodes(
|
||||
.await?;
|
||||
|
||||
// existing nodes get updated on insert
|
||||
for record in mixnodes.iter() {
|
||||
for record in mixnodes.into_iter() {
|
||||
// https://www.sqlite.org/lang_upsert.html
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO mixnodes
|
||||
(mix_id, identity_key, bonded, total_stake,
|
||||
host, http_api_port, full_details,
|
||||
@@ -46,17 +46,17 @@ pub(crate) async fn update_mixnodes(
|
||||
full_details=excluded.full_details,self_described=excluded.self_described,
|
||||
last_updated_utc=excluded.last_updated_utc,
|
||||
is_dp_delegatee = excluded.is_dp_delegatee;",
|
||||
record.mix_id,
|
||||
record.identity_key,
|
||||
record.bonded,
|
||||
record.total_stake,
|
||||
record.host,
|
||||
record.http_port,
|
||||
record.full_details,
|
||||
record.self_described,
|
||||
record.last_updated_utc,
|
||||
record.is_dp_delegatee
|
||||
)
|
||||
.bind(record.mix_id as i64)
|
||||
.bind(record.identity_key)
|
||||
.bind(record.bonded)
|
||||
.bind(record.total_stake)
|
||||
.bind(record.host)
|
||||
.bind(record.http_port as i32)
|
||||
.bind(record.full_details)
|
||||
.bind(record.self_described)
|
||||
.bind(record.last_updated_utc)
|
||||
.bind(record.is_dp_delegatee)
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
}
|
||||
@@ -78,10 +78,10 @@ pub(crate) async fn get_all_mixnodes(pool: &DbPool) -> anyhow::Result<Vec<Mixnod
|
||||
mn.full_details as "full_details!",
|
||||
mn.self_described as "self_described",
|
||||
mn.last_updated_utc as "last_updated_utc!",
|
||||
COALESCE(md.moniker, "NA") as "moniker!",
|
||||
COALESCE(md.website, "NA") as "website!",
|
||||
COALESCE(md.security_contact, "NA") as "security_contact!",
|
||||
COALESCE(md.details, "NA") as "details!"
|
||||
md.moniker as "moniker!",
|
||||
md.website as "website!",
|
||||
md.security_contact as "security_contact!",
|
||||
md.details as "details!"
|
||||
FROM mixnodes mn
|
||||
LEFT JOIN mixnode_description md ON mn.mix_id = md.mix_id
|
||||
ORDER BY mn.mix_id"#
|
||||
@@ -144,29 +144,29 @@ pub(crate) async fn get_daily_stats(pool: &DbPool) -> anyhow::Result<Vec<DailySt
|
||||
|
||||
pub(crate) async fn get_bonded_mix_ids(pool: &DbPool) -> anyhow::Result<HashSet<i64>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query!(
|
||||
let items = crate::db::query(
|
||||
r#"
|
||||
SELECT mix_id
|
||||
FROM mixnodes
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?
|
||||
.into_iter()
|
||||
.map(|record| record.mix_id)
|
||||
.map(|record| record.try_get::<i64, _>("mix_id").unwrap())
|
||||
.collect::<HashSet<_>>();
|
||||
|
||||
Ok(items)
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_mixnode_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
mix_id: &i64,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
mix_id: i64,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_description (
|
||||
mix_id, moniker, website, security_contact, details, last_updated_utc
|
||||
@@ -178,13 +178,13 @@ pub(crate) async fn insert_mixnode_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
mix_id,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
|
||||
@@ -9,7 +9,8 @@ mod summary;
|
||||
pub(crate) mod testruns;
|
||||
|
||||
pub(crate) use gateways::{
|
||||
get_all_gateways, get_bonded_gateway_id_keys, select_gateway_identity, update_bonded_gateways,
|
||||
get_all_gateways, get_bonded_gateway_id_keys, get_or_create_gateway, select_gateway_identity,
|
||||
update_bonded_gateways,
|
||||
};
|
||||
pub(crate) use gateways_stats::{delete_old_records, get_sessions_stats, insert_session_records};
|
||||
pub(crate) use misc::insert_summaries;
|
||||
|
||||
@@ -4,14 +4,14 @@ use nym_validator_client::{
|
||||
client::{NodeId, NymNodeDetails},
|
||||
models::NymNodeDescription,
|
||||
};
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use std::collections::HashMap;
|
||||
use tracing::instrument;
|
||||
|
||||
use crate::{
|
||||
db::{
|
||||
models::{NymNodeDto, NymNodeInsertRecord},
|
||||
DbPool,
|
||||
DbConnection, DbPool,
|
||||
},
|
||||
node_scraper::helpers::NodeDescriptionResponse,
|
||||
};
|
||||
@@ -19,21 +19,20 @@ use crate::{
|
||||
pub(crate) async fn get_all_nym_nodes(pool: &DbPool) -> anyhow::Result<Vec<NymNodeDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query_as!(
|
||||
NymNodeDto,
|
||||
crate::db::query_as::<NymNodeDto>(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
ip_addresses as "ip_addresses!: serde_json::Value",
|
||||
ip_addresses,
|
||||
mix_port,
|
||||
x25519_sphinx_pubkey,
|
||||
node_role as "node_role: serde_json::Value",
|
||||
supported_roles as "supported_roles: serde_json::Value",
|
||||
entry as "entry: serde_json::Value",
|
||||
node_role,
|
||||
supported_roles,
|
||||
entry,
|
||||
performance,
|
||||
self_described as "self_described: serde_json::Value",
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
self_described,
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
ORDER BY
|
||||
@@ -56,21 +55,20 @@ pub(crate) async fn get_described_bonded_nym_nodes(
|
||||
) -> anyhow::Result<Vec<NymNodeDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query_as!(
|
||||
NymNodeDto,
|
||||
crate::db::query_as::<NymNodeDto>(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
ip_addresses as "ip_addresses!: serde_json::Value",
|
||||
ip_addresses,
|
||||
mix_port,
|
||||
x25519_sphinx_pubkey,
|
||||
node_role as "node_role: serde_json::Value",
|
||||
supported_roles as "supported_roles: serde_json::Value",
|
||||
entry as "entry: serde_json::Value",
|
||||
node_role,
|
||||
supported_roles,
|
||||
entry,
|
||||
performance,
|
||||
self_described as "self_described: serde_json::Value",
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
self_described,
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -92,7 +90,7 @@ pub(crate) async fn update_nym_nodes(
|
||||
) -> anyhow::Result<usize> {
|
||||
let mut tx = pool.begin().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"UPDATE nym_nodes
|
||||
SET
|
||||
self_described = NULL,
|
||||
@@ -104,7 +102,7 @@ pub(crate) async fn update_nym_nodes(
|
||||
let inserted = node_records.len();
|
||||
for record in node_records {
|
||||
// https://www.sqlite.org/lang_upsert.html
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
"INSERT INTO nym_nodes
|
||||
(node_id, ed25519_identity_pubkey,
|
||||
total_stake,
|
||||
@@ -129,20 +127,20 @@ pub(crate) async fn update_nym_nodes(
|
||||
performance=excluded.performance,
|
||||
last_updated_utc=excluded.last_updated_utc
|
||||
;",
|
||||
record.node_id,
|
||||
record.ed25519_identity_pubkey,
|
||||
record.total_stake,
|
||||
record.ip_addresses,
|
||||
record.mix_port,
|
||||
record.x25519_sphinx_pubkey,
|
||||
record.node_role,
|
||||
record.supported_roles,
|
||||
record.entry,
|
||||
record.self_described,
|
||||
record.bond_info,
|
||||
record.performance,
|
||||
record.last_updated_utc,
|
||||
)
|
||||
.bind(record.node_id)
|
||||
.bind(record.ed25519_identity_pubkey)
|
||||
.bind(record.total_stake)
|
||||
.bind(record.ip_addresses)
|
||||
.bind(record.mix_port)
|
||||
.bind(record.x25519_sphinx_pubkey)
|
||||
.bind(record.node_role)
|
||||
.bind(record.supported_roles)
|
||||
.bind(record.entry)
|
||||
.bind(record.self_described)
|
||||
.bind(record.bond_info)
|
||||
.bind(record.performance)
|
||||
.bind(record.last_updated_utc)
|
||||
.execute(&mut *tx)
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Failed to INSERT node_id={}: {}", record.node_id, e))?;
|
||||
@@ -150,6 +148,10 @@ pub(crate) async fn update_nym_nodes(
|
||||
|
||||
tx.commit().await?;
|
||||
|
||||
tracing::debug!(
|
||||
"Successfully inserted/updated {} nym_nodes records",
|
||||
inserted
|
||||
);
|
||||
Ok(inserted)
|
||||
}
|
||||
|
||||
@@ -158,10 +160,10 @@ pub(crate) async fn get_described_node_bond_info(
|
||||
) -> anyhow::Result<HashMap<NodeId, NymNodeDetails>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
bond_info as "bond_info: serde_json::Value"
|
||||
bond_info
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -176,11 +178,11 @@ pub(crate) async fn get_described_node_bond_info(
|
||||
records
|
||||
.into_iter()
|
||||
.filter_map(|record| {
|
||||
record
|
||||
.bond_info
|
||||
// only return details for nodes which have details stored
|
||||
.and_then(|bond_info| serde_json::from_value::<NymNodeDetails>(bond_info).ok())
|
||||
.map(|res| (record.node_id as NodeId, res))
|
||||
let node_id: i32 = record.try_get("node_id").ok()?;
|
||||
let bond_info: serde_json::Value = record.try_get("bond_info").ok()?;
|
||||
serde_json::from_value::<NymNodeDetails>(bond_info)
|
||||
.ok()
|
||||
.map(|res| (node_id as i64 as NodeId, res))
|
||||
})
|
||||
.collect::<HashMap<_, _>>()
|
||||
})
|
||||
@@ -192,10 +194,10 @@ pub(crate) async fn get_node_self_description(
|
||||
) -> anyhow::Result<HashMap<NodeId, NymNodeDescription>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
node_id,
|
||||
self_described as "self_described: serde_json::Value"
|
||||
self_described
|
||||
FROM
|
||||
nym_nodes
|
||||
WHERE
|
||||
@@ -210,13 +212,11 @@ pub(crate) async fn get_node_self_description(
|
||||
records
|
||||
.into_iter()
|
||||
.filter_map(|record| {
|
||||
record
|
||||
.self_described
|
||||
// only return details for nodes which have details stored
|
||||
.and_then(|description| {
|
||||
serde_json::from_value::<NymNodeDescription>(description).ok()
|
||||
})
|
||||
.map(|res| (record.node_id as NodeId, res))
|
||||
let node_id: i32 = record.try_get("node_id").ok()?;
|
||||
let self_described: serde_json::Value = record.try_get("self_described").ok()?;
|
||||
serde_json::from_value::<NymNodeDescription>(self_described)
|
||||
.ok()
|
||||
.map(|res| (node_id as i64 as NodeId, res))
|
||||
})
|
||||
.collect::<HashMap<_, _>>()
|
||||
})
|
||||
@@ -228,7 +228,7 @@ pub(crate) async fn get_bonded_node_description(
|
||||
) -> anyhow::Result<HashMap<NodeId, NodeDescription>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"SELECT
|
||||
nd.node_id,
|
||||
moniker,
|
||||
@@ -238,7 +238,7 @@ pub(crate) async fn get_bonded_node_description(
|
||||
FROM
|
||||
nym_node_descriptions nd
|
||||
INNER JOIN
|
||||
nym_nodes
|
||||
nym_nodes nn on nd.node_id = nn.node_id
|
||||
WHERE
|
||||
bond_info IS NOT NULL
|
||||
"#,
|
||||
@@ -249,14 +249,15 @@ pub(crate) async fn get_bonded_node_description(
|
||||
records
|
||||
.into_iter()
|
||||
.map(|elem| {
|
||||
let node_id: NodeId = elem.node_id.try_into().unwrap_or_default();
|
||||
let node_id: i64 = elem.try_get("node_id").unwrap_or(0);
|
||||
let node_id: NodeId = node_id.try_into().unwrap_or_default();
|
||||
(
|
||||
node_id,
|
||||
NodeDescription {
|
||||
moniker: elem.moniker.unwrap_or_default(),
|
||||
website: elem.website.unwrap_or_default(),
|
||||
security_contact: elem.security_contact.unwrap_or_default(),
|
||||
details: elem.details.unwrap_or_default(),
|
||||
moniker: elem.try_get("moniker").unwrap_or_default(),
|
||||
website: elem.try_get("website").unwrap_or_default(),
|
||||
security_contact: elem.try_get("security_contact").unwrap_or_default(),
|
||||
details: elem.try_get("details").unwrap_or_default(),
|
||||
},
|
||||
)
|
||||
})
|
||||
@@ -266,12 +267,12 @@ pub(crate) async fn get_bonded_node_description(
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_nym_node_description(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
node_id: &i64,
|
||||
description: &NodeDescriptionResponse,
|
||||
conn: &mut DbConnection,
|
||||
node_id: i64,
|
||||
description: NodeDescriptionResponse,
|
||||
timestamp: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
crate::db::query(
|
||||
r#"
|
||||
INSERT INTO nym_node_descriptions (
|
||||
node_id, moniker, website, security_contact, details, last_updated_utc
|
||||
@@ -283,13 +284,13 @@ pub(crate) async fn insert_nym_node_description(
|
||||
details = excluded.details,
|
||||
last_updated_utc = excluded.last_updated_utc
|
||||
"#,
|
||||
node_id,
|
||||
description.moniker,
|
||||
description.website,
|
||||
description.security_contact,
|
||||
description.details,
|
||||
timestamp,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(description.moniker)
|
||||
.bind(description.website)
|
||||
.bind(description.security_contact)
|
||||
.bind(description.details)
|
||||
.bind(timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
|
||||
@@ -59,7 +59,8 @@ pub(crate) async fn batch_store_packet_stats(
|
||||
.map_err(|err| anyhow::anyhow!("Failed to commit: {err}"))
|
||||
}
|
||||
|
||||
async fn insert_node_packet_stats_uncommitted(
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) async fn insert_node_packet_stats_uncommitted(
|
||||
tx: &mut Transaction<'static, sqlx::Sqlite>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
stats: &NodeStats,
|
||||
@@ -67,35 +68,35 @@ async fn insert_node_packet_stats_uncommitted(
|
||||
) -> Result<()> {
|
||||
match node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query!(
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_packet_stats_raw (
|
||||
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
"#,
|
||||
mix_id,
|
||||
timestamp_utc,
|
||||
stats.packets_received,
|
||||
stats.packets_sent,
|
||||
stats.packets_dropped,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query!(
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO nym_nodes_packet_stats_raw (
|
||||
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
"#,
|
||||
node_id,
|
||||
timestamp_utc,
|
||||
stats.packets_received,
|
||||
stats.packets_sent,
|
||||
stats.packets_dropped,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
@@ -104,6 +105,53 @@ async fn insert_node_packet_stats_uncommitted(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) async fn insert_node_packet_stats_uncommitted(
|
||||
tx: &mut Transaction<'static, sqlx::Postgres>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
stats: &NodeStats,
|
||||
timestamp_utc: i64,
|
||||
) -> Result<()> {
|
||||
match node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_packet_stats_raw (
|
||||
mix_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES ($1, $2, $3, $4, $5)
|
||||
"#,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO nym_nodes_packet_stats_raw (
|
||||
node_id, timestamp_utc, packets_received, packets_sent, packets_dropped
|
||||
) VALUES ($1, $2, $3, $4, $5)
|
||||
"#,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(timestamp_utc)
|
||||
.bind(stats.packets_received)
|
||||
.bind(stats.packets_sent)
|
||||
.bind(stats.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) async fn get_raw_node_stats(
|
||||
tx: &mut Transaction<'static, sqlx::Sqlite>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
@@ -112,39 +160,37 @@ pub(crate) async fn get_raw_node_stats(
|
||||
// if no packets are found, it's fine to assume 0 because that's also
|
||||
// SQL default value if none provided
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query_as!(
|
||||
NodeStats,
|
||||
sqlx::query_as::<_, NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as "packets_received!: _",
|
||||
COALESCE(packets_sent, 0) as "packets_sent!: _",
|
||||
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM mixnode_packet_stats_raw
|
||||
WHERE mix_id = ?
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
mix_id
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_optional(tx.as_mut())
|
||||
.await?
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query_as!(
|
||||
NodeStats,
|
||||
sqlx::query_as::<_, NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as "packets_received!: _",
|
||||
COALESCE(packets_sent, 0) as "packets_sent!: _",
|
||||
COALESCE(packets_dropped, 0) as "packets_dropped!: _"
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM nym_nodes_packet_stats_raw
|
||||
WHERE node_id = ?
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
node_id
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_optional(tx.as_mut())
|
||||
.await?
|
||||
}
|
||||
@@ -153,6 +199,55 @@ pub(crate) async fn get_raw_node_stats(
|
||||
Ok(packets)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) async fn get_raw_node_stats(
|
||||
tx: &mut Transaction<'static, sqlx::Postgres>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
) -> Result<Option<NodeStats>> {
|
||||
let packets = match node_kind {
|
||||
// if no packets are found, it's fine to assume 0 because that's also
|
||||
// SQL default value if none provided
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
sqlx::query_as::<_, NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM mixnode_packet_stats_raw
|
||||
WHERE mix_id = $1
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_optional(tx.as_mut())
|
||||
.await?
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
sqlx::query_as::<_, NodeStats>(
|
||||
r#"
|
||||
SELECT
|
||||
COALESCE(packets_received, 0) as packets_received,
|
||||
COALESCE(packets_sent, 0) as packets_sent,
|
||||
COALESCE(packets_dropped, 0) as packets_dropped
|
||||
FROM nym_nodes_packet_stats_raw
|
||||
WHERE node_id = $1
|
||||
ORDER BY timestamp_utc DESC
|
||||
LIMIT 1 OFFSET 1
|
||||
"#,
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_optional(tx.as_mut())
|
||||
.await?
|
||||
}
|
||||
};
|
||||
|
||||
Ok(packets)
|
||||
}
|
||||
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub(crate) async fn insert_daily_node_stats_uncommitted(
|
||||
tx: &mut Transaction<'static, sqlx::Sqlite>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
@@ -161,19 +256,19 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
|
||||
) -> Result<()> {
|
||||
match node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
let total_stake = sqlx::query_scalar!(
|
||||
let total_stake = sqlx::query_scalar::<_, i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM mixnodes
|
||||
WHERE mix_id = ?
|
||||
"#,
|
||||
mix_id
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_one(tx.as_mut())
|
||||
.await?;
|
||||
|
||||
sqlx::query!(
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_daily_stats (
|
||||
mix_id, date_utc,
|
||||
@@ -186,31 +281,31 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
|
||||
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
mix_id,
|
||||
date_utc,
|
||||
total_stake,
|
||||
packets.packets_received,
|
||||
packets.packets_sent,
|
||||
packets.packets_dropped,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
let total_stake = sqlx::query_scalar!(
|
||||
let total_stake = sqlx::query_scalar::<_, i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM nym_nodes
|
||||
WHERE node_id = ?
|
||||
"#,
|
||||
node_id
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_one(tx.as_mut())
|
||||
.await?;
|
||||
|
||||
sqlx::query!(
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO nym_node_daily_mixing_stats (
|
||||
node_id, date_utc,
|
||||
@@ -223,13 +318,99 @@ pub(crate) async fn insert_daily_node_stats_uncommitted(
|
||||
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
node_id,
|
||||
date_utc,
|
||||
total_stake,
|
||||
packets.packets_received,
|
||||
packets.packets_sent,
|
||||
packets.packets_dropped,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub(crate) async fn insert_daily_node_stats_uncommitted(
|
||||
tx: &mut Transaction<'static, sqlx::Postgres>,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
date_utc: &str,
|
||||
packets: NodeStats,
|
||||
) -> Result<()> {
|
||||
match node_kind {
|
||||
ScrapeNodeKind::LegacyMixnode { mix_id } => {
|
||||
let total_stake = sqlx::query_scalar::<_, i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM mixnodes
|
||||
WHERE mix_id = $1
|
||||
"#,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.fetch_one(tx.as_mut())
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO mixnode_daily_stats (
|
||||
mix_id, date_utc,
|
||||
total_stake, packets_received,
|
||||
packets_sent, packets_dropped
|
||||
) VALUES ($1, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT(mix_id, date_utc) DO UPDATE SET
|
||||
total_stake = excluded.total_stake,
|
||||
packets_received = mixnode_daily_stats.packets_received + excluded.packets_received,
|
||||
packets_sent = mixnode_daily_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = mixnode_daily_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
)
|
||||
.bind(mix_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
ScrapeNodeKind::MixingNymNode { node_id }
|
||||
| ScrapeNodeKind::EntryExitNymNode { node_id, .. } => {
|
||||
let total_stake = sqlx::query_scalar::<_, i64>(
|
||||
r#"
|
||||
SELECT
|
||||
total_stake
|
||||
FROM nym_nodes
|
||||
WHERE node_id = $1
|
||||
"#,
|
||||
)
|
||||
.bind(node_id)
|
||||
.fetch_one(tx.as_mut())
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
r#"
|
||||
INSERT INTO nym_node_daily_mixing_stats (
|
||||
node_id, date_utc,
|
||||
total_stake, packets_received,
|
||||
packets_sent, packets_dropped
|
||||
) VALUES ($1, $2, $3, $4, $5, $6)
|
||||
ON CONFLICT(node_id, date_utc) DO UPDATE SET
|
||||
total_stake = excluded.total_stake,
|
||||
packets_received = nym_node_daily_mixing_stats.packets_received + excluded.packets_received,
|
||||
packets_sent = nym_node_daily_mixing_stats.packets_sent + excluded.packets_sent,
|
||||
packets_dropped = nym_node_daily_mixing_stats.packets_dropped + excluded.packets_dropped
|
||||
"#,
|
||||
)
|
||||
.bind(node_id)
|
||||
.bind(date_utc)
|
||||
.bind(total_stake)
|
||||
.bind(packets.packets_received)
|
||||
.bind(packets.packets_sent)
|
||||
.bind(packets.packets_dropped)
|
||||
.execute(tx.as_mut())
|
||||
.await?;
|
||||
}
|
||||
|
||||
@@ -12,6 +12,7 @@ use crate::{
|
||||
};
|
||||
use anyhow::Result;
|
||||
use nym_validator_client::nym_api::SkimmedNode;
|
||||
use sqlx::Row;
|
||||
|
||||
pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperNodeInfo>> {
|
||||
let mut nodes_to_scrape = Vec::new();
|
||||
@@ -68,12 +69,12 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
tracing::debug!("Fetched {} 🚪 entry/exit nodes", entry_exit_nodes);
|
||||
|
||||
let mut conn = pool.acquire().await?;
|
||||
let mixnodes = sqlx::query!(
|
||||
let mixnodes = crate::db::query(
|
||||
r#"
|
||||
SELECT mix_id as node_id, host, http_api_port
|
||||
FROM mixnodes
|
||||
WHERE bonded = true
|
||||
"#
|
||||
"#,
|
||||
)
|
||||
.fetch_all(&mut *conn)
|
||||
.await?;
|
||||
@@ -85,18 +86,20 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
let mut legacy_not_in_nym_node_list = 0;
|
||||
let total_legacy_mixnodes = mixnodes.len();
|
||||
for mixnode in mixnodes {
|
||||
let node_id: i64 = mixnode.try_get("node_id")?;
|
||||
let host: String = mixnode.try_get("host")?;
|
||||
let http_api_port: i64 = mixnode.try_get("http_api_port")?;
|
||||
|
||||
if nodes_to_scrape
|
||||
.iter()
|
||||
.all(|node| node.node_id() != &mixnode.node_id)
|
||||
.all(|node| node.node_id() != &node_id)
|
||||
{
|
||||
// in case polyfilling on Nym API gets removed, this part ensures
|
||||
// mixnodes are added to the final list of nodes to scrape
|
||||
nodes_to_scrape.push(ScraperNodeInfo {
|
||||
node_kind: ScrapeNodeKind::LegacyMixnode {
|
||||
mix_id: mixnode.node_id,
|
||||
},
|
||||
hosts: vec![mixnode.host],
|
||||
http_api_port: mixnode.http_api_port,
|
||||
node_kind: ScrapeNodeKind::LegacyMixnode { mix_id: node_id },
|
||||
hosts: vec![host],
|
||||
http_api_port,
|
||||
});
|
||||
|
||||
legacy_not_in_nym_node_list += 1;
|
||||
@@ -121,8 +124,8 @@ pub(crate) async fn get_nodes_for_scraping(pool: &DbPool) -> Result<Vec<ScraperN
|
||||
|
||||
pub(crate) async fn insert_scraped_node_description(
|
||||
pool: &DbPool,
|
||||
node_kind: &ScrapeNodeKind,
|
||||
description: &NodeDescriptionResponse,
|
||||
node_kind: ScrapeNodeKind,
|
||||
description: NodeDescriptionResponse,
|
||||
) -> Result<()> {
|
||||
let timestamp = now_utc().unix_timestamp();
|
||||
let mut conn = pool.acquire().await?;
|
||||
@@ -138,7 +141,7 @@ pub(crate) async fn insert_scraped_node_description(
|
||||
node_id,
|
||||
identity_key,
|
||||
} => {
|
||||
insert_nym_node_description(&mut conn, node_id, description, timestamp).await?;
|
||||
insert_nym_node_description(&mut conn, node_id, description.clone(), timestamp).await?;
|
||||
// for historic reasons (/gateways API), store this info into gateways table as well
|
||||
insert_gateway_description(&mut conn, identity_key, description, timestamp).await?;
|
||||
}
|
||||
|
||||
@@ -23,13 +23,12 @@ use crate::{
|
||||
|
||||
pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<SummaryHistory>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
let items = sqlx::query_as!(
|
||||
SummaryHistoryDto,
|
||||
let items = crate::db::query_as::<SummaryHistoryDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
date as "date!",
|
||||
timestamp_utc as "timestamp_utc!",
|
||||
value_json as "value_json!"
|
||||
id,
|
||||
date,
|
||||
timestamp_utc,
|
||||
value_json
|
||||
FROM summary_history
|
||||
ORDER BY date DESC
|
||||
LIMIT 30"#,
|
||||
@@ -51,13 +50,12 @@ pub(crate) async fn get_summary_history(pool: &DbPool) -> anyhow::Result<Vec<Sum
|
||||
|
||||
async fn get_summary_dto(pool: &DbPool) -> anyhow::Result<Vec<SummaryDto>> {
|
||||
let mut conn = pool.acquire().await?;
|
||||
Ok(sqlx::query_as!(
|
||||
SummaryDto,
|
||||
Ok(crate::db::query_as::<SummaryDto>(
|
||||
r#"SELECT
|
||||
key as "key!",
|
||||
value_json as "value_json!",
|
||||
last_updated_utc as "last_updated_utc!"
|
||||
FROM summary"#
|
||||
key,
|
||||
value_json,
|
||||
last_updated_utc
|
||||
FROM summary"#,
|
||||
)
|
||||
.fetch(&mut *conn)
|
||||
.try_collect::<Vec<_>>()
|
||||
|
||||
@@ -1,40 +1,38 @@
|
||||
use crate::db::models::{TestRunDto, TestRunStatus};
|
||||
use crate::db::DbConnection;
|
||||
use crate::db::DbPool;
|
||||
use crate::http::models::TestrunAssignment;
|
||||
use crate::utils::now_utc;
|
||||
use sqlx::{pool::PoolConnection, Sqlite};
|
||||
use sqlx::Row;
|
||||
use time::Duration;
|
||||
|
||||
pub(crate) async fn count_testruns_in_progress(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
) -> anyhow::Result<i64> {
|
||||
sqlx::query_scalar!(
|
||||
r#"SELECT
|
||||
COUNT(id) as "count: i64"
|
||||
FROM testruns
|
||||
WHERE
|
||||
status = ?
|
||||
"#,
|
||||
TestRunStatus::InProgress as i64,
|
||||
)
|
||||
pub(crate) async fn count_testruns_in_progress(conn: &mut DbConnection) -> anyhow::Result<i64> {
|
||||
#[cfg(feature = "sqlite")]
|
||||
let sql = "SELECT COUNT(id) FROM testruns WHERE status = ?";
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
let sql = "SELECT COUNT(id) FROM testruns WHERE status = $1";
|
||||
|
||||
let count: i64 = sqlx::query_scalar(sql)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map_err(anyhow::Error::from)
|
||||
.await?;
|
||||
|
||||
Ok(count)
|
||||
}
|
||||
|
||||
pub(crate) async fn get_in_progress_testrun_by_id(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
testrun_id: i64,
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i32,
|
||||
) -> anyhow::Result<TestRunDto> {
|
||||
sqlx::query_as!(
|
||||
TestRunDto,
|
||||
crate::db::query_as::<TestRunDto>(
|
||||
r#"SELECT
|
||||
id as "id!",
|
||||
gateway_id as "gateway_id!",
|
||||
status as "status!",
|
||||
created_utc as "created_utc!",
|
||||
ip_address as "ip_address!",
|
||||
log as "log!",
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
ip_address,
|
||||
log,
|
||||
last_assigned_utc
|
||||
FROM testruns
|
||||
WHERE
|
||||
@@ -43,12 +41,12 @@ pub(crate) async fn get_in_progress_testrun_by_id(
|
||||
status = ?
|
||||
ORDER BY created_utc
|
||||
LIMIT 1"#,
|
||||
testrun_id,
|
||||
TestRunStatus::InProgress as i64,
|
||||
)
|
||||
.bind(testrun_id)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map_err(|e| anyhow::anyhow!("Couldn't retrieve testrun {testrun_id}: {e}"))
|
||||
.map_err(|e| anyhow::anyhow!("Failed to retrieve in-progress testrun {testrun_id}: {e}"))
|
||||
}
|
||||
|
||||
pub(crate) async fn update_testruns_assigned_before(
|
||||
@@ -59,7 +57,7 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
let previous_run = now_utc() - max_age;
|
||||
let cutoff_timestamp = previous_run.unix_timestamp();
|
||||
|
||||
let res = sqlx::query!(
|
||||
let res = crate::db::query(
|
||||
r#"UPDATE
|
||||
testruns
|
||||
SET
|
||||
@@ -69,10 +67,10 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
AND
|
||||
last_assigned_utc < ?
|
||||
"#,
|
||||
TestRunStatus::Queued as i64,
|
||||
TestRunStatus::InProgress as i64,
|
||||
cutoff_timestamp
|
||||
)
|
||||
.bind(TestRunStatus::Queued as i32)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.bind(cutoff_timestamp)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
@@ -89,36 +87,36 @@ pub(crate) async fn update_testruns_assigned_before(
|
||||
}
|
||||
|
||||
pub(crate) async fn assign_oldest_testrun(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
conn: &mut DbConnection,
|
||||
) -> anyhow::Result<Option<TestrunAssignment>> {
|
||||
let now = now_utc().unix_timestamp();
|
||||
// find & mark as "In progress" in the same transaction to avoid race conditions
|
||||
let returning = sqlx::query!(
|
||||
let returning = crate::db::query(
|
||||
r#"UPDATE testruns
|
||||
SET
|
||||
status = ?,
|
||||
last_assigned_utc = ?
|
||||
WHERE rowid =
|
||||
WHERE id =
|
||||
(
|
||||
SELECT rowid
|
||||
SELECT id
|
||||
FROM testruns
|
||||
WHERE status = ?
|
||||
ORDER BY created_utc asc
|
||||
LIMIT 1
|
||||
)
|
||||
RETURNING
|
||||
id as "id!",
|
||||
id,
|
||||
gateway_id
|
||||
"#,
|
||||
TestRunStatus::InProgress as i64,
|
||||
now,
|
||||
TestRunStatus::Queued as i64,
|
||||
)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.bind(now)
|
||||
.bind(TestRunStatus::Queued as i32)
|
||||
.fetch_optional(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
if let Some(testrun) = returning {
|
||||
let gw_identity = sqlx::query!(
|
||||
let gw_identity = crate::db::query(
|
||||
r#"
|
||||
SELECT
|
||||
id,
|
||||
@@ -126,14 +124,14 @@ pub(crate) async fn assign_oldest_testrun(
|
||||
FROM gateways
|
||||
WHERE id = ?
|
||||
LIMIT 1"#,
|
||||
testrun.gateway_id
|
||||
)
|
||||
.bind(testrun.try_get::<i32, _>("gateway_id")?)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(Some(TestrunAssignment {
|
||||
testrun_id: testrun.id,
|
||||
gateway_identity_key: gw_identity.gateway_identity_key,
|
||||
testrun_id: testrun.try_get("id")?,
|
||||
gateway_identity_key: gw_identity.try_get("gateway_identity_key")?,
|
||||
assigned_at_utc: now,
|
||||
}))
|
||||
} else {
|
||||
@@ -142,16 +140,14 @@ pub(crate) async fn assign_oldest_testrun(
|
||||
}
|
||||
|
||||
pub(crate) async fn update_testrun_status(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
testrun_id: i64,
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i32,
|
||||
status: TestRunStatus,
|
||||
) -> anyhow::Result<()> {
|
||||
let status = status as u32;
|
||||
sqlx::query!(
|
||||
"UPDATE testruns SET status = ? WHERE id = ?",
|
||||
status,
|
||||
testrun_id
|
||||
)
|
||||
let status = status as i32;
|
||||
crate::db::query("UPDATE testruns SET status = ? WHERE id = ?")
|
||||
.bind(status)
|
||||
.bind(testrun_id)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
@@ -159,50 +155,131 @@ pub(crate) async fn update_testrun_status(
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_last_probe_log(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
gateway_pk: i64,
|
||||
log: &str,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i32,
|
||||
log: String,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_probe_log = ? WHERE id = ?",
|
||||
log,
|
||||
gateway_pk
|
||||
)
|
||||
crate::db::query("UPDATE gateways SET last_probe_log = ? WHERE id = ?")
|
||||
.bind(log)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
.map_err(|e| {
|
||||
anyhow::anyhow!(
|
||||
"Failed to update probe log for gateway {}: {}",
|
||||
gateway_pk,
|
||||
e
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_last_probe_result(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
gateway_pk: i64,
|
||||
result: &str,
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i32,
|
||||
result: String,
|
||||
) -> anyhow::Result<()> {
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_probe_result = ? WHERE id = ?",
|
||||
result,
|
||||
gateway_pk
|
||||
crate::db::query("UPDATE gateways SET last_probe_result = ? WHERE id = ?")
|
||||
.bind(result)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(|e| {
|
||||
anyhow::anyhow!(
|
||||
"Failed to update probe result for gateway {}: {}",
|
||||
gateway_pk,
|
||||
e
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_score(
|
||||
conn: &mut DbConnection,
|
||||
gateway_pk: i32,
|
||||
) -> anyhow::Result<()> {
|
||||
let now = now_utc().unix_timestamp();
|
||||
crate::db::query("UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?")
|
||||
.bind(now)
|
||||
.bind(now)
|
||||
.bind(gateway_pk)
|
||||
.execute(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
}
|
||||
|
||||
pub(crate) async fn update_gateway_score(
|
||||
conn: &mut PoolConnection<Sqlite>,
|
||||
gateway_pk: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
let now = now_utc().unix_timestamp();
|
||||
sqlx::query!(
|
||||
"UPDATE gateways SET last_testrun_utc = ?, last_updated_utc = ? WHERE id = ?",
|
||||
now,
|
||||
now,
|
||||
gateway_pk
|
||||
pub(crate) async fn get_testrun_by_id(
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i32,
|
||||
) -> anyhow::Result<TestRunDto> {
|
||||
crate::db::query_as::<TestRunDto>(
|
||||
r#"SELECT
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
ip_address,
|
||||
log,
|
||||
last_assigned_utc
|
||||
FROM testruns
|
||||
WHERE id = ?"#,
|
||||
)
|
||||
.execute(conn.as_mut())
|
||||
.bind(testrun_id)
|
||||
.fetch_one(conn.as_mut())
|
||||
.await
|
||||
.map(drop)
|
||||
.map_err(From::from)
|
||||
.map_err(|e| anyhow::anyhow!("Testrun {} not found: {}", testrun_id, e))
|
||||
}
|
||||
|
||||
pub(crate) async fn insert_external_testrun(
|
||||
conn: &mut DbConnection,
|
||||
testrun_id: i32,
|
||||
gateway_id: i32,
|
||||
assigned_at_utc: i64,
|
||||
) -> anyhow::Result<()> {
|
||||
let now = crate::utils::now_utc().unix_timestamp();
|
||||
|
||||
crate::db::query(
|
||||
r#"INSERT INTO testruns (
|
||||
id,
|
||||
gateway_id,
|
||||
status,
|
||||
created_utc,
|
||||
last_assigned_utc,
|
||||
ip_address,
|
||||
log
|
||||
) VALUES (?, ?, ?, ?, ?, ?, ?)"#,
|
||||
)
|
||||
.bind(testrun_id)
|
||||
.bind(gateway_id)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.bind(now)
|
||||
.bind(assigned_at_utc)
|
||||
.bind("external") // Marker for external origin
|
||||
.bind("") // Empty initial log
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
tracing::debug!(
|
||||
"Created external testrun {} for gateway {}",
|
||||
testrun_id,
|
||||
gateway_id
|
||||
);
|
||||
Ok(())
|
||||
}
|
||||
|
||||
pub(crate) async fn update_testrun_status_by_gateway(
|
||||
conn: &mut DbConnection,
|
||||
gateway_id: i32,
|
||||
status: TestRunStatus,
|
||||
) -> anyhow::Result<()> {
|
||||
let status = status as i32;
|
||||
crate::db::query("UPDATE testruns SET status = ? WHERE gateway_id = ? AND status = ?")
|
||||
.bind(status)
|
||||
.bind(gateway_id)
|
||||
.bind(TestRunStatus::InProgress as i32)
|
||||
.execute(conn.as_mut())
|
||||
.await?;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
use sqlx::Database;
|
||||
|
||||
/// Converts SQLite-style ? placeholders to PostgreSQL $N format
|
||||
#[cfg(feature = "pg")]
|
||||
fn convert_placeholders(query: &str) -> String {
|
||||
let mut result = String::with_capacity(query.len() + 10);
|
||||
let mut placeholder_count = 0;
|
||||
let mut chars = query.chars();
|
||||
let mut in_string: Option<char> = None;
|
||||
let mut escape_next = false;
|
||||
|
||||
#[allow(clippy::while_let_on_iterator)]
|
||||
while let Some(ch) = chars.next() {
|
||||
if escape_next {
|
||||
result.push(ch);
|
||||
escape_next = false;
|
||||
continue;
|
||||
}
|
||||
|
||||
if let Some(quote_char) = in_string {
|
||||
result.push(ch);
|
||||
if ch == quote_char {
|
||||
in_string = None;
|
||||
} else if ch == '\\' {
|
||||
escape_next = true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
match ch {
|
||||
'\\' => {
|
||||
result.push(ch);
|
||||
escape_next = true;
|
||||
}
|
||||
'\'' | '"' => {
|
||||
result.push(ch);
|
||||
in_string = Some(ch);
|
||||
}
|
||||
'?' => {
|
||||
placeholder_count += 1;
|
||||
result.push_str(&format!("${placeholder_count}"));
|
||||
}
|
||||
_ => {
|
||||
result.push(ch);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
result
|
||||
}
|
||||
|
||||
/// Creates a query that automatically handles placeholder conversion
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub fn query(
|
||||
sql: &str,
|
||||
) -> sqlx::query::Query<'_, sqlx::Sqlite, <sqlx::Sqlite as Database>::Arguments<'_>> {
|
||||
sqlx::query(sql)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub fn query(
|
||||
sql: &str,
|
||||
) -> sqlx::query::Query<'static, sqlx::Postgres, <sqlx::Postgres as Database>::Arguments<'static>> {
|
||||
let converted = convert_placeholders(sql);
|
||||
sqlx::query(Box::leak(converted.into_boxed_str()))
|
||||
}
|
||||
|
||||
/// Creates a query_as that automatically handles placeholder conversion
|
||||
#[cfg(feature = "sqlite")]
|
||||
pub fn query_as<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryAs<'_, sqlx::Sqlite, O, <sqlx::Sqlite as Database>::Arguments<'_>>
|
||||
where
|
||||
O: for<'r> sqlx::FromRow<'r, <sqlx::Sqlite as Database>::Row>,
|
||||
{
|
||||
sqlx::query_as(sql)
|
||||
}
|
||||
|
||||
#[cfg(feature = "pg")]
|
||||
pub fn query_as<O>(
|
||||
sql: &str,
|
||||
) -> sqlx::query::QueryAs<
|
||||
'static,
|
||||
sqlx::Postgres,
|
||||
O,
|
||||
<sqlx::Postgres as Database>::Arguments<'static>,
|
||||
>
|
||||
where
|
||||
O: for<'r> sqlx::FromRow<'r, <sqlx::Postgres as Database>::Row>,
|
||||
{
|
||||
let converted = convert_placeholders(sql);
|
||||
sqlx::query_as(Box::leak(converted.into_boxed_str()))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
#[cfg(feature = "pg")]
|
||||
fn test_convert_placeholders() {
|
||||
// Basic conversion
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE id = ?"),
|
||||
r"SELECT * FROM table WHERE id = $1"
|
||||
);
|
||||
|
||||
// Multiple placeholders
|
||||
assert_eq!(
|
||||
convert_placeholders(r"INSERT INTO table (a, b, c) VALUES (?, ?, ?)"),
|
||||
r"INSERT INTO table (a, b, c) VALUES ($1, $2, $3)"
|
||||
);
|
||||
|
||||
// Placeholder inside string literal should be ignored
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE name = 'test?' AND id = ?"),
|
||||
r"SELECT * FROM table WHERE name = 'test?' AND id = $1"
|
||||
);
|
||||
|
||||
// Update statement
|
||||
assert_eq!(
|
||||
convert_placeholders(r"UPDATE table SET a = ?, b = ? WHERE id = ?"),
|
||||
r"UPDATE table SET a = $1, b = $2 WHERE id = $3"
|
||||
);
|
||||
|
||||
// Test with 10 placeholders (like in update_mixnodes)
|
||||
assert_eq!(
|
||||
convert_placeholders(r"INSERT INTO t VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)"),
|
||||
r"INSERT INTO t VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10)"
|
||||
);
|
||||
|
||||
// No placeholders
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table"),
|
||||
r"SELECT * FROM table"
|
||||
);
|
||||
|
||||
// Placeholder at the beginning
|
||||
assert_eq!(convert_placeholders(r"? AND ?"), r"$1 AND $2");
|
||||
|
||||
// Placeholder at the end
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE id = ?"),
|
||||
r"SELECT * FROM table WHERE id = $1"
|
||||
);
|
||||
|
||||
// Adjacent placeholders
|
||||
assert_eq!(
|
||||
convert_placeholders(r"VALUES(?,? ,?)"),
|
||||
r"VALUES($1,$2 ,$3)"
|
||||
);
|
||||
|
||||
// Escaped single quote
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM foo WHERE bar = 'it\'s a test' AND baz = ?"),
|
||||
r"SELECT * FROM foo WHERE bar = 'it\'s a test' AND baz = $1"
|
||||
);
|
||||
|
||||
// Double quotes
|
||||
assert_eq!(
|
||||
convert_placeholders(r#"SELECT * FROM "table" WHERE "column" = ? AND name = "test?""#),
|
||||
r#"SELECT * FROM "table" WHERE "column" = $1 AND name = "test?""#
|
||||
);
|
||||
|
||||
// Mixed quotes
|
||||
assert_eq!(
|
||||
convert_placeholders(
|
||||
r#"SELECT * FROM table WHERE a = 'single?' AND b = "double?" AND c = ?"#
|
||||
),
|
||||
r#"SELECT * FROM table WHERE a = 'single?' AND b = "double?" AND c = $1"#
|
||||
);
|
||||
|
||||
// Escaped backslash before quote
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE path = 'C:\\?' AND id = ?"),
|
||||
r"SELECT * FROM table WHERE path = 'C:\\?' AND id = $1"
|
||||
);
|
||||
|
||||
// Multiple escaped quotes
|
||||
assert_eq!(
|
||||
convert_placeholders(
|
||||
r#"INSERT INTO table (msg) VALUES ('it\'s "complex" test') WHERE id = ?"#
|
||||
),
|
||||
r#"INSERT INTO table (msg) VALUES ('it\'s "complex" test') WHERE id = $1"#
|
||||
);
|
||||
|
||||
// Very long query with many placeholders
|
||||
let long_query = r"INSERT INTO very_long_table_name (col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14, col15) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
|
||||
let expected = r"INSERT INTO very_long_table_name (col1, col2, col3, col4, col5, col6, col7, col8, col9, col10, col11, col12, col13, col14, col15) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15)";
|
||||
assert_eq!(convert_placeholders(long_query), expected);
|
||||
|
||||
// Query with comments (question marks in comments are also converted)
|
||||
assert_eq!(
|
||||
convert_placeholders(
|
||||
r"-- This is a comment with ?
|
||||
SELECT * FROM table WHERE id = ? -- another comment ?"
|
||||
),
|
||||
r"-- This is a comment with $1
|
||||
SELECT * FROM table WHERE id = $2 -- another comment $3"
|
||||
);
|
||||
|
||||
// Multiline strings
|
||||
assert_eq!(
|
||||
convert_placeholders(
|
||||
r"SELECT * FROM table
|
||||
WHERE description = 'This is a
|
||||
multiline string with ?'
|
||||
AND id = ?"
|
||||
),
|
||||
r"SELECT * FROM table
|
||||
WHERE description = 'This is a
|
||||
multiline string with ?'
|
||||
AND id = $1"
|
||||
);
|
||||
|
||||
// Complex nested quotes
|
||||
assert_eq!(
|
||||
convert_placeholders(
|
||||
r#"SELECT json_extract(data, '$.items[?(@.name=="test?")]') FROM table WHERE id = ?"#
|
||||
),
|
||||
r#"SELECT json_extract(data, '$.items[?(@.name=="test?")]') FROM table WHERE id = $1"#
|
||||
);
|
||||
|
||||
// Empty string
|
||||
assert_eq!(convert_placeholders(""), "");
|
||||
|
||||
// Only placeholders
|
||||
assert_eq!(convert_placeholders("???"), "$1$2$3");
|
||||
|
||||
// Unicode in strings
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE name = '测试?' AND id = ?"),
|
||||
r"SELECT * FROM table WHERE name = '测试?' AND id = $1"
|
||||
);
|
||||
|
||||
// Test case with backslash at end of string
|
||||
assert_eq!(
|
||||
convert_placeholders(r"SELECT * FROM table WHERE path LIKE '%\\' AND id = ?"),
|
||||
r"SELECT * FROM table WHERE path LIKE '%\\' AND id = $1"
|
||||
);
|
||||
|
||||
// Mismatched quotes
|
||||
assert_eq!(
|
||||
convert_placeholders(r#"SELECT * FROM foo WHERE bar = "'" AND baz = ?"#),
|
||||
r#"SELECT * FROM foo WHERE bar = "'" AND baz = $1"#
|
||||
);
|
||||
|
||||
// Unmatched quote
|
||||
assert_eq!(convert_placeholders(r"SELECT 'oops?"), r"SELECT 'oops?");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,461 @@
|
||||
#[cfg(test)]
|
||||
mod db_tests {
|
||||
|
||||
#[test]
|
||||
fn test_gateway_dto_try_from() {
|
||||
let gateway_dto = crate::db::models::GatewayDto {
|
||||
gateway_identity_key: "test_identity".to_string(),
|
||||
bonded: true,
|
||||
performance: 100,
|
||||
self_described: Some("{\"key\":\"value\"}".to_string()),
|
||||
explorer_pretty_bond: Some("{\"key\":\"value\"}".to_string()),
|
||||
last_probe_result: Some("{\"key\":\"value\"}".to_string()),
|
||||
last_probe_log: Some("log".to_string()),
|
||||
last_testrun_utc: Some(1672531200),
|
||||
last_updated_utc: 1672531200,
|
||||
moniker: "moniker".to_string(),
|
||||
security_contact: "contact".to_string(),
|
||||
details: "details".to_string(),
|
||||
website: "website".to_string(),
|
||||
};
|
||||
|
||||
let http_gateway: crate::http::models::Gateway = gateway_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(http_gateway.gateway_identity_key, "test_identity");
|
||||
assert!(http_gateway.bonded);
|
||||
assert_eq!(http_gateway.performance, 100);
|
||||
assert!(http_gateway.self_described.is_some());
|
||||
assert!(http_gateway.explorer_pretty_bond.is_some());
|
||||
assert!(http_gateway.last_probe_result.is_some());
|
||||
assert_eq!(http_gateway.last_probe_log, Some("log".to_string()));
|
||||
assert!(http_gateway.last_testrun_utc.is_some());
|
||||
assert!(!http_gateway.last_updated_utc.is_empty());
|
||||
assert_eq!(http_gateway.description.moniker, "moniker");
|
||||
assert_eq!(http_gateway.description.website, "website");
|
||||
assert_eq!(http_gateway.description.security_contact, "contact");
|
||||
assert_eq!(http_gateway.description.details, "details");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mixnode_dto_try_from() {
|
||||
let mixnode_dto = crate::db::models::MixnodeDto {
|
||||
mix_id: 1,
|
||||
bonded: true,
|
||||
is_dp_delegatee: false,
|
||||
total_stake: 1000000,
|
||||
full_details: "{\"key\":\"value\"}".to_string(),
|
||||
self_described: Some("{\"key\":\"value\"}".to_string()),
|
||||
last_updated_utc: 1672531200,
|
||||
moniker: "moniker".to_string(),
|
||||
website: "website".to_string(),
|
||||
security_contact: "contact".to_string(),
|
||||
details: "details".to_string(),
|
||||
};
|
||||
|
||||
let http_mixnode: crate::http::models::Mixnode = mixnode_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(http_mixnode.mix_id, 1);
|
||||
assert!(http_mixnode.bonded);
|
||||
assert!(!http_mixnode.is_dp_delegatee);
|
||||
assert_eq!(http_mixnode.total_stake, 1000000);
|
||||
assert!(http_mixnode.full_details.is_some());
|
||||
assert!(http_mixnode.self_described.is_some());
|
||||
assert!(!http_mixnode.last_updated_utc.is_empty());
|
||||
assert_eq!(http_mixnode.description.moniker, "moniker");
|
||||
assert_eq!(http_mixnode.description.website, "website");
|
||||
assert_eq!(http_mixnode.description.security_contact, "contact");
|
||||
assert_eq!(http_mixnode.description.details, "details");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_summary_history_dto_try_from() {
|
||||
let summary_history_dto = crate::db::models::SummaryHistoryDto {
|
||||
id: 1,
|
||||
date: "2023-01-01".to_string(),
|
||||
value_json: "{\"key\":\"value\"}".to_string(),
|
||||
timestamp_utc: 1672531200,
|
||||
};
|
||||
|
||||
let summary_history: crate::http::models::SummaryHistory =
|
||||
summary_history_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(summary_history.date, "2023-01-01");
|
||||
assert!(summary_history.value_json.is_object());
|
||||
assert!(!summary_history.timestamp_utc.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_gateway_sessions_record_try_from() {
|
||||
let gateway_sessions_record = crate::db::models::GatewaySessionsRecord {
|
||||
gateway_identity_key: "test_identity".to_string(),
|
||||
node_id: 1,
|
||||
day: time::macros::date!(2023 - 01 - 01),
|
||||
unique_active_clients: 10,
|
||||
session_started: 100,
|
||||
users_hashes: Some("{\"key\":\"value\"}".to_string()),
|
||||
vpn_sessions: Some("{\"key\":\"value\"}".to_string()),
|
||||
mixnet_sessions: Some("{\"key\":\"value\"}".to_string()),
|
||||
unknown_sessions: Some("{\"key\":\"value\"}".to_string()),
|
||||
};
|
||||
|
||||
let session_stats: crate::http::models::SessionStats =
|
||||
gateway_sessions_record.try_into().unwrap();
|
||||
|
||||
assert_eq!(session_stats.gateway_identity_key, "test_identity");
|
||||
assert_eq!(session_stats.node_id, 1);
|
||||
assert_eq!(session_stats.day, time::macros::date!(2023 - 01 - 01));
|
||||
assert_eq!(session_stats.unique_active_clients, 10);
|
||||
assert_eq!(session_stats.session_started, 100);
|
||||
assert!(session_stats.users_hashes.is_some());
|
||||
assert!(session_stats.vpn_sessions.is_some());
|
||||
assert!(session_stats.mixnet_sessions.is_some());
|
||||
assert!(session_stats.unknown_sessions.is_some());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_nym_node_dto_try_from() {
|
||||
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
|
||||
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
|
||||
|
||||
let nym_node_dto = crate::db::models::NymNodeDto {
|
||||
node_id: 1,
|
||||
ed25519_identity_pubkey: ed25519_pk.to_base58_string(),
|
||||
total_stake: 1000000,
|
||||
ip_addresses: serde_json::json!(["1.1.1.1"]),
|
||||
mix_port: 1789,
|
||||
x25519_sphinx_pubkey: x25519_pk.to_base58_string(),
|
||||
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
|
||||
layer: 1
|
||||
}),
|
||||
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
|
||||
entry: false,
|
||||
mixnode: true,
|
||||
exit_nr: false,
|
||||
exit_ipr: false,
|
||||
}),
|
||||
entry: None,
|
||||
performance: "1.0".to_string(),
|
||||
self_described: None,
|
||||
bond_info: None,
|
||||
};
|
||||
|
||||
let skimmed_node: nym_validator_client::nym_api::SkimmedNode =
|
||||
nym_node_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(skimmed_node.node_id, 1);
|
||||
assert_eq!(skimmed_node.ed25519_identity_pubkey, ed25519_pk);
|
||||
assert_eq!(
|
||||
skimmed_node.ip_addresses,
|
||||
vec!["1.1.1.1".parse::<std::net::IpAddr>().unwrap()]
|
||||
);
|
||||
assert_eq!(skimmed_node.mix_port, 1789);
|
||||
assert_eq!(skimmed_node.x25519_sphinx_pubkey, x25519_pk);
|
||||
|
||||
match skimmed_node.role {
|
||||
nym_validator_client::nym_nodes::NodeRole::Mixnode { layer } => assert_eq!(layer, 1),
|
||||
_ => panic!("Unexpected node role"),
|
||||
}
|
||||
assert!(!skimmed_node.supported_roles.entry);
|
||||
assert!(skimmed_node.supported_roles.mixnode);
|
||||
assert!(!skimmed_node.supported_roles.exit_nr);
|
||||
assert!(!skimmed_node.supported_roles.exit_ipr);
|
||||
assert!(skimmed_node.entry.is_none());
|
||||
assert_eq!(
|
||||
skimmed_node.performance,
|
||||
nym_contracts_common::Percent::from_percentage_value(100).unwrap()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_nym_node_insert_record_new() {
|
||||
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
|
||||
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
|
||||
|
||||
let skimmed_node = nym_validator_client::nym_api::SkimmedNode {
|
||||
node_id: 1,
|
||||
ed25519_identity_pubkey: ed25519_pk,
|
||||
ip_addresses: vec!["1.1.1.1".parse().unwrap()],
|
||||
mix_port: 1789,
|
||||
x25519_sphinx_pubkey: x25519_pk,
|
||||
role: nym_validator_client::nym_nodes::NodeRole::Mixnode { layer: 1 },
|
||||
supported_roles: nym_validator_client::models::DeclaredRoles {
|
||||
entry: false,
|
||||
mixnode: true,
|
||||
exit_nr: false,
|
||||
exit_ipr: false,
|
||||
},
|
||||
entry: None,
|
||||
performance: nym_contracts_common::Percent::from_percentage_value(100).unwrap(),
|
||||
};
|
||||
|
||||
let record = crate::db::models::NymNodeInsertRecord::new(skimmed_node, None, None).unwrap();
|
||||
|
||||
assert_eq!(record.node_id, 1);
|
||||
assert_eq!(
|
||||
record.ed25519_identity_pubkey,
|
||||
ed25519_pk.to_base58_string()
|
||||
);
|
||||
assert_eq!(record.total_stake, 0);
|
||||
assert_eq!(record.ip_addresses, serde_json::json!(["1.1.1.1"]));
|
||||
assert_eq!(record.mix_port, 1789);
|
||||
assert_eq!(record.x25519_sphinx_pubkey, x25519_pk.to_base58_string());
|
||||
assert_eq!(
|
||||
record.node_role,
|
||||
serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode { layer: 1 })
|
||||
);
|
||||
assert_eq!(
|
||||
record.supported_roles,
|
||||
serde_json::json!(nym_validator_client::models::DeclaredRoles {
|
||||
entry: false,
|
||||
mixnode: true,
|
||||
exit_nr: false,
|
||||
exit_ipr: false,
|
||||
})
|
||||
);
|
||||
assert_eq!(record.performance, "1");
|
||||
assert!(record.entry.is_none());
|
||||
assert!(record.self_described.is_none());
|
||||
assert!(record.bond_info.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_nym_node_insert_record_with_entry() {
|
||||
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
|
||||
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
|
||||
|
||||
let skimmed_node = nym_validator_client::nym_api::SkimmedNode {
|
||||
node_id: 1,
|
||||
ed25519_identity_pubkey: ed25519_pk,
|
||||
ip_addresses: vec!["1.1.1.1".parse().unwrap()],
|
||||
mix_port: 1789,
|
||||
x25519_sphinx_pubkey: x25519_pk,
|
||||
role: nym_validator_client::nym_nodes::NodeRole::EntryGateway,
|
||||
supported_roles: nym_validator_client::models::DeclaredRoles {
|
||||
entry: true,
|
||||
mixnode: false,
|
||||
exit_nr: true,
|
||||
exit_ipr: false,
|
||||
},
|
||||
entry: Some(nym_validator_client::nym_nodes::BasicEntryInformation {
|
||||
hostname: Some("gateway.example.com".to_string()),
|
||||
ws_port: 9001,
|
||||
wss_port: Some(9002),
|
||||
}),
|
||||
performance: nym_contracts_common::Percent::from_percentage_value(99).unwrap(),
|
||||
};
|
||||
|
||||
let record = crate::db::models::NymNodeInsertRecord::new(skimmed_node, None, None).unwrap();
|
||||
|
||||
assert_eq!(record.node_id, 1);
|
||||
assert_eq!(record.total_stake, 0); // No bond info provided
|
||||
assert!(record.entry.is_some());
|
||||
assert!(record.self_described.is_none());
|
||||
assert!(record.bond_info.is_none());
|
||||
assert!(record.last_updated_utc > 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_gateway_dto_with_null_values() {
|
||||
let gateway_dto = crate::db::models::GatewayDto {
|
||||
gateway_identity_key: "test_identity".to_string(),
|
||||
bonded: false,
|
||||
performance: 0,
|
||||
self_described: None,
|
||||
explorer_pretty_bond: None,
|
||||
last_probe_result: None,
|
||||
last_probe_log: None,
|
||||
last_testrun_utc: None,
|
||||
last_updated_utc: 0,
|
||||
moniker: "".to_string(),
|
||||
security_contact: "".to_string(),
|
||||
details: "".to_string(),
|
||||
website: "".to_string(),
|
||||
};
|
||||
|
||||
let http_gateway: crate::http::models::Gateway = gateway_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(http_gateway.gateway_identity_key, "test_identity");
|
||||
assert!(!http_gateway.bonded);
|
||||
assert_eq!(http_gateway.performance, 0);
|
||||
assert!(http_gateway.self_described.is_none());
|
||||
assert!(http_gateway.explorer_pretty_bond.is_none());
|
||||
assert!(http_gateway.last_probe_result.is_none());
|
||||
assert!(http_gateway.last_probe_log.is_none());
|
||||
assert!(http_gateway.last_testrun_utc.is_none());
|
||||
assert_eq!(http_gateway.last_updated_utc, "1970-01-01T00:00:00Z");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_mixnode_dto_with_invalid_json() {
|
||||
let mixnode_dto = crate::db::models::MixnodeDto {
|
||||
mix_id: 1,
|
||||
bonded: true,
|
||||
is_dp_delegatee: false,
|
||||
total_stake: 1000000,
|
||||
full_details: "invalid json".to_string(),
|
||||
self_described: Some("also invalid".to_string()),
|
||||
last_updated_utc: 1672531200,
|
||||
moniker: "moniker".to_string(),
|
||||
website: "website".to_string(),
|
||||
security_contact: "contact".to_string(),
|
||||
details: "details".to_string(),
|
||||
};
|
||||
|
||||
let http_mixnode: crate::http::models::Mixnode = mixnode_dto.try_into().unwrap();
|
||||
|
||||
// Invalid JSON should result in None
|
||||
assert!(http_mixnode.full_details.is_none());
|
||||
assert_eq!(http_mixnode.self_described, Some(serde_json::Value::Null));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_summary_history_dto_with_invalid_json() {
|
||||
let summary_history_dto = crate::db::models::SummaryHistoryDto {
|
||||
id: 1,
|
||||
date: "2023-01-01".to_string(),
|
||||
value_json: "not valid json".to_string(),
|
||||
timestamp_utc: 1672531200,
|
||||
};
|
||||
|
||||
let summary_history: crate::http::models::SummaryHistory =
|
||||
summary_history_dto.try_into().unwrap();
|
||||
|
||||
assert_eq!(summary_history.date, "2023-01-01");
|
||||
// Invalid JSON should result in default (null)
|
||||
assert!(summary_history.value_json.is_null());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_gateway_sessions_record_with_all_none() {
|
||||
let gateway_sessions_record = crate::db::models::GatewaySessionsRecord {
|
||||
gateway_identity_key: "test_identity".to_string(),
|
||||
node_id: 1,
|
||||
day: time::macros::date!(2023 - 01 - 01),
|
||||
unique_active_clients: 0,
|
||||
session_started: 0,
|
||||
users_hashes: None,
|
||||
vpn_sessions: None,
|
||||
mixnet_sessions: None,
|
||||
unknown_sessions: None,
|
||||
};
|
||||
|
||||
let session_stats: crate::http::models::SessionStats =
|
||||
gateway_sessions_record.try_into().unwrap();
|
||||
|
||||
assert_eq!(session_stats.gateway_identity_key, "test_identity");
|
||||
assert_eq!(session_stats.node_id, 1);
|
||||
assert_eq!(session_stats.unique_active_clients, 0);
|
||||
assert_eq!(session_stats.session_started, 0);
|
||||
assert!(session_stats.users_hashes.is_none());
|
||||
assert!(session_stats.vpn_sessions.is_none());
|
||||
assert!(session_stats.mixnet_sessions.is_none());
|
||||
assert!(session_stats.unknown_sessions.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scraper_node_info_contact_addresses() {
|
||||
use crate::db::models::{ScrapeNodeKind, ScraperNodeInfo};
|
||||
|
||||
let node_info = ScraperNodeInfo {
|
||||
node_kind: ScrapeNodeKind::MixingNymNode { node_id: 123 },
|
||||
hosts: vec!["1.1.1.1".to_string(), "example.com".to_string()],
|
||||
http_api_port: 8080,
|
||||
};
|
||||
|
||||
let addresses = node_info.contact_addresses();
|
||||
|
||||
// Should generate multiple URLs for each host
|
||||
// Custom port (8080) should be inserted at the beginning
|
||||
assert!(addresses.contains(&"http://1.1.1.1:8080".to_string()));
|
||||
assert!(addresses.contains(&"http://example.com:8080".to_string()));
|
||||
assert!(addresses.contains(&"http://1.1.1.1:8000".to_string()));
|
||||
assert!(addresses.contains(&"https://1.1.1.1".to_string()));
|
||||
assert!(addresses.contains(&"http://example.com:8000".to_string()));
|
||||
// Check that URLs follow the expected pattern
|
||||
assert!(addresses.len() >= 8); // At least 4 URLs per host
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_scrape_node_kind_node_id() {
|
||||
use crate::db::models::ScrapeNodeKind;
|
||||
|
||||
let legacy = ScrapeNodeKind::LegacyMixnode { mix_id: 42 };
|
||||
assert_eq!(*legacy.node_id(), 42);
|
||||
|
||||
let mixing = ScrapeNodeKind::MixingNymNode { node_id: 123 };
|
||||
assert_eq!(*mixing.node_id(), 123);
|
||||
|
||||
let entry_exit = ScrapeNodeKind::EntryExitNymNode {
|
||||
node_id: 456,
|
||||
identity_key: "key123".to_string(),
|
||||
};
|
||||
assert_eq!(*entry_exit.node_id(), 456);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_nym_node_dto_with_invalid_keys() {
|
||||
let nym_node_dto = crate::db::models::NymNodeDto {
|
||||
node_id: 1,
|
||||
ed25519_identity_pubkey: "invalid_base58".to_string(),
|
||||
total_stake: 1000000,
|
||||
ip_addresses: serde_json::json!(["1.1.1.1"]),
|
||||
mix_port: 1789,
|
||||
x25519_sphinx_pubkey: "also_invalid".to_string(),
|
||||
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
|
||||
layer: 1
|
||||
}),
|
||||
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
|
||||
entry: false,
|
||||
mixnode: true,
|
||||
exit_nr: false,
|
||||
exit_ipr: false,
|
||||
}),
|
||||
entry: None,
|
||||
performance: "1.0".to_string(),
|
||||
self_described: None,
|
||||
bond_info: None,
|
||||
};
|
||||
|
||||
let result: Result<nym_validator_client::nym_api::SkimmedNode, _> = nym_node_dto.try_into();
|
||||
assert!(result.is_err());
|
||||
assert!(result
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("ed25519_identity_pubkey"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_nym_node_dto_with_invalid_performance() {
|
||||
let ed25519_pk = nym_crypto::asymmetric::ed25519::PublicKey::from_bytes(&[1; 32]).unwrap();
|
||||
let x25519_pk = nym_crypto::asymmetric::x25519::PublicKey::from_bytes(&[2; 32]).unwrap();
|
||||
|
||||
let nym_node_dto = crate::db::models::NymNodeDto {
|
||||
node_id: 1,
|
||||
ed25519_identity_pubkey: ed25519_pk.to_base58_string(),
|
||||
total_stake: 1000000,
|
||||
ip_addresses: serde_json::json!(["1.1.1.1"]),
|
||||
mix_port: 1789,
|
||||
x25519_sphinx_pubkey: x25519_pk.to_base58_string(),
|
||||
node_role: serde_json::json!(nym_validator_client::nym_nodes::NodeRole::Mixnode {
|
||||
layer: 1
|
||||
}),
|
||||
supported_roles: serde_json::json!(nym_validator_client::models::DeclaredRoles {
|
||||
entry: false,
|
||||
mixnode: true,
|
||||
exit_nr: false,
|
||||
exit_ipr: false,
|
||||
}),
|
||||
entry: None,
|
||||
performance: "invalid_percent".to_string(),
|
||||
self_described: None,
|
||||
bond_info: None,
|
||||
};
|
||||
|
||||
let result: Result<nym_validator_client::nym_api::SkimmedNode, _> = nym_node_dto.try_into();
|
||||
assert!(result.is_err());
|
||||
assert!(result
|
||||
.unwrap_err()
|
||||
.to_string()
|
||||
.contains("can't parse Percent"));
|
||||
}
|
||||
@@ -65,3 +65,36 @@ pub async fn dvpn_gateways(
|
||||
.await,
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_routes_construction() {
|
||||
let router = routes();
|
||||
// Verify the router builds without panic
|
||||
let _routes = router;
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_min_node_version_query_deserialization() {
|
||||
// Test with version
|
||||
let json = r#"{"min_node_version": "1.2.3"}"#;
|
||||
let query: MinNodeVersionQuery = serde_json::from_str(json).unwrap();
|
||||
assert_eq!(query.min_node_version, Some("1.2.3".to_string()));
|
||||
|
||||
// Test without version
|
||||
let json_empty = r#"{}"#;
|
||||
let query_empty: MinNodeVersionQuery = serde_json::from_str(json_empty).unwrap();
|
||||
assert_eq!(query_empty.min_node_version, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_min_supported_version() {
|
||||
// Test that the lazy static initializes correctly
|
||||
assert_eq!(MIN_SUPPORTED_VERSION.major, 1);
|
||||
assert_eq!(MIN_SUPPORTED_VERSION.minor, 6);
|
||||
assert_eq!(MIN_SUPPORTED_VERSION.patch, 2);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -57,21 +57,7 @@ async fn gateways_skinny(
|
||||
) -> HttpResult<Json<PagedResult<GatewaySkinny>>> {
|
||||
let db = state.db_pool();
|
||||
let res = state.cache().get_gateway_list(db).await;
|
||||
let res: Vec<GatewaySkinny> = res
|
||||
.iter()
|
||||
.filter(|g| g.bonded)
|
||||
.map(|g| GatewaySkinny {
|
||||
gateway_identity_key: g.gateway_identity_key.clone(),
|
||||
self_described: g.self_described.clone(),
|
||||
performance: g.performance,
|
||||
explorer_pretty_bond: g.explorer_pretty_bond.clone(),
|
||||
last_probe_result: g.last_probe_result.clone(),
|
||||
last_testrun_utc: g.last_testrun_utc.clone(),
|
||||
last_updated_utc: g.last_updated_utc.clone(),
|
||||
routing_score: g.routing_score,
|
||||
config_score: g.config_score,
|
||||
})
|
||||
.collect();
|
||||
let res: Vec<GatewaySkinny> = filter_bonded_gateways_to_skinny(res);
|
||||
|
||||
Ok(Json(PagedResult::paginate(pagination, res)))
|
||||
}
|
||||
@@ -108,3 +94,126 @@ async fn get_gateway(
|
||||
None => Err(HttpError::invalid_input(identity_key)),
|
||||
}
|
||||
}
|
||||
|
||||
// Extract filtering logic for testing
|
||||
fn filter_bonded_gateways_to_skinny(gateways: Vec<Gateway>) -> Vec<GatewaySkinny> {
|
||||
gateways
|
||||
.iter()
|
||||
.filter(|g| g.bonded)
|
||||
.map(|g| GatewaySkinny {
|
||||
gateway_identity_key: g.gateway_identity_key.clone(),
|
||||
self_described: g.self_described.clone(),
|
||||
performance: g.performance,
|
||||
explorer_pretty_bond: g.explorer_pretty_bond.clone(),
|
||||
last_probe_result: g.last_probe_result.clone(),
|
||||
last_testrun_utc: g.last_testrun_utc.clone(),
|
||||
last_updated_utc: g.last_updated_utc.clone(),
|
||||
routing_score: g.routing_score,
|
||||
config_score: g.config_score,
|
||||
})
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use crate::http::models::Gateway;
|
||||
use nym_node_requests::api::v1::node::models::NodeDescription;
|
||||
|
||||
fn create_test_gateway(identity_key: &str, bonded: bool, performance: u8) -> Gateway {
|
||||
Gateway {
|
||||
gateway_identity_key: identity_key.to_string(),
|
||||
bonded,
|
||||
performance,
|
||||
self_described: Some(serde_json::json!({"test": "data"})),
|
||||
explorer_pretty_bond: Some(serde_json::json!({"bond": "info"})),
|
||||
description: NodeDescription {
|
||||
moniker: "Test Gateway".to_string(),
|
||||
website: "".to_string(),
|
||||
security_contact: "".to_string(),
|
||||
details: "".to_string(),
|
||||
},
|
||||
last_probe_result: Some(serde_json::json!({"result": "ok"})),
|
||||
last_probe_log: None,
|
||||
last_testrun_utc: Some("2024-01-20T10:00:00Z".to_string()),
|
||||
last_updated_utc: "2024-01-20T11:00:00Z".to_string(),
|
||||
routing_score: 0.95,
|
||||
config_score: 100,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filter_bonded_gateways_to_skinny_empty_list() {
|
||||
let gateways = vec![];
|
||||
let result = filter_bonded_gateways_to_skinny(gateways);
|
||||
assert_eq!(result.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filter_bonded_gateways_to_skinny_all_bonded() {
|
||||
let gateways = vec![
|
||||
create_test_gateway("gw1", true, 90),
|
||||
create_test_gateway("gw2", true, 95),
|
||||
create_test_gateway("gw3", true, 85),
|
||||
];
|
||||
|
||||
let result = filter_bonded_gateways_to_skinny(gateways);
|
||||
assert_eq!(result.len(), 3);
|
||||
assert_eq!(result[0].gateway_identity_key, "gw1");
|
||||
assert_eq!(result[1].gateway_identity_key, "gw2");
|
||||
assert_eq!(result[2].gateway_identity_key, "gw3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filter_bonded_gateways_to_skinny_mixed() {
|
||||
let gateways = vec![
|
||||
create_test_gateway("gw1", true, 90),
|
||||
create_test_gateway("gw2", false, 95),
|
||||
create_test_gateway("gw3", true, 85),
|
||||
create_test_gateway("gw4", false, 100),
|
||||
];
|
||||
|
||||
let result = filter_bonded_gateways_to_skinny(gateways);
|
||||
assert_eq!(result.len(), 2);
|
||||
assert_eq!(result[0].gateway_identity_key, "gw1");
|
||||
assert_eq!(result[1].gateway_identity_key, "gw3");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_filter_bonded_gateways_to_skinny_none_bonded() {
|
||||
let gateways = vec![
|
||||
create_test_gateway("gw1", false, 90),
|
||||
create_test_gateway("gw2", false, 95),
|
||||
];
|
||||
|
||||
let result = filter_bonded_gateways_to_skinny(gateways);
|
||||
assert_eq!(result.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_gateway_to_skinny_conversion() {
|
||||
let gateway = create_test_gateway("test_gw", true, 98);
|
||||
let gateways = vec![gateway.clone()];
|
||||
|
||||
let result = filter_bonded_gateways_to_skinny(gateways);
|
||||
assert_eq!(result.len(), 1);
|
||||
|
||||
let skinny = &result[0];
|
||||
assert_eq!(skinny.gateway_identity_key, gateway.gateway_identity_key);
|
||||
assert_eq!(skinny.performance, gateway.performance);
|
||||
assert_eq!(skinny.self_described, gateway.self_described);
|
||||
assert_eq!(skinny.explorer_pretty_bond, gateway.explorer_pretty_bond);
|
||||
assert_eq!(skinny.last_probe_result, gateway.last_probe_result);
|
||||
assert_eq!(skinny.last_testrun_utc, gateway.last_testrun_utc);
|
||||
assert_eq!(skinny.last_updated_utc, gateway.last_updated_utc);
|
||||
assert_eq!(skinny.routing_score, gateway.routing_score);
|
||||
assert_eq!(skinny.config_score, gateway.config_score);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_identity_key_param_deserialization() {
|
||||
let json = r#"{"identity_key": "test_key_123"}"#;
|
||||
let param: IdentityKeyParam = serde_json::from_str(json).unwrap();
|
||||
assert_eq!(param.identity_key, "test_key_123");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -8,3 +8,15 @@ pub(crate) fn routes() -> Router<AppState> {
|
||||
Router::new().nest("/sessions", sessions::routes())
|
||||
//eventually add other metrics type
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
#[test]
|
||||
fn test_routes_construction() {
|
||||
let router = routes();
|
||||
// Verify the router builds without panic
|
||||
let _routes = router;
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user