Implements automatic cleanup of route monitoring results to prevent
unbounded storage growth while maintaining data for performance analysis.
- Add purge_old_routes() method to StorageManager following existing patterns
- Integrate route cleanup into purge_old_statuses() wrapper function
- Route data now purged every epoch with 48-hour retention, to facilitate comparisons with legacy data
- Update logging to reflect cleanup of both node statuses and routes
This commit addresses critical N+1 query performance issues identified in
the reward simulation system. The optimizations significantly reduce database
round trips and improve performance when processing large datasets.
**Key Optimizations:**
1. **Batch Identity Key Lookups**
- Added `get_mixnode_identity_keys_batch()` and `get_gateway_identity_keys_batch()`
- Updated simulation performance conversion to use batch operations
- Reduced from N individual queries to 2 batch queries
2. **Batch Node Classification**
- Added `classify_nodes_batch()` method for mixnode/gateway determination
- Updated reliability calculation methods to use batch classification
- Reduced from N individual lookups to 2 batch queries
3. **Batch Epoch Metadata Enhancement**
- Added `count_simulated_node_performance_for_epochs_batch()`
- Added `get_available_calculation_methods_for_epochs_batch()`
- Updated API handlers to use batch operations for metadata enhancement
- Reduced from 2N queries to 2 batch queries for epoch data
4. **Bulk Insert Optimizations**
- Converted individual INSERT operations to use `sqlx::QueryBuilder::push_values()`
- Optimized simulation data insertion methods
- Eliminated transaction overhead from individual inserts
**Performance Impact:**
- Before: N+2N database queries for N nodes/epochs
- After: 2+2 batch queries regardless of dataset size
- Significant performance improvement for large simulation datasets
All changes maintain backward compatibility while providing substantial
performance benefits for the reward simulation system.
Refactors the simulation system to focus on performance methodology comparison
rather than reward amounts, enabling robust analysis of old vs new calculation
methods. Key improvements:
- Replace simulated_rewards table with performance_comparisons for better metrics
- Add performance_rankings table for ranking analysis across methodologies
- Enhance database schema with additional performance tracking fields
- Update simulation coordinator to use performance-focused data structures
- Add comprehensive performance ranking calculations
- Improve API models and handlers for performance comparison workflows
- Update SQLx query cache with new database schema changes
This provides a foundation for data-driven performance methodology evaluation
while maintaining separation from actual reward calculations.
This completes Phase 3 of the simulation system implementation:
- Add comprehensive REST API endpoints for simulation data access
- Implement /v1/simulation/* routes with full CRUD operations
- Support JSON/CSV export for external analysis
- Add statistical comparison between old vs new methods
- Provide node performance history tracking
- Include proper error handling and response formatting
- Simplify simulation coordinator to remove unused complex return types
- Clean up dead code while maintaining all functionality
- Pass clippy with no warnings
The simulation API provides complete access to:
- Simulation epoch listing and details
- Method comparison analytics (old 24h vs new 1h)
- Node performance analysis across epochs
- Route reliability statistics
- Export capabilities for further analysis
All simulation data is persisted and accessible via REST endpoints.
Add complete simulation engine that compares old (24h cache-based) vs new (1h route-based)
reward calculation methodologies with full integration into epoch operations.
Core Simulation Engine:
- Add SimulationCoordinator with configurable time windows and comparison settings
- Implement dual calculation methods with proper Performance type conversions
- Add comprehensive error handling with DatabaseError variant in RewardingError
- Store simulation results in database with proper relationship constraints
Old Method Implementation (24h Cache-Based):
- Wrap existing reliability calculation using get_all_avg_mix_reliability_in_last_24hr()
- Convert reliability percentages to Performance types using from_percentage_value()
- Maintain exact same logic as production for accurate baseline comparison
- Generate simulation data structures with proper metadata
New Method Implementation (1h Route-Based):
- Leverage calculate_corrected_node_reliabilities_for_interval() for route analysis
- Support configurable time windows (default 1 hour vs 24 hours)
- Provide detailed route statistics including success rates and failure analysis
- Convert route reliability data to Performance types with naive_try_from_f64()
Epoch Operations Integration:
- Extend EpochAdvancer struct with optional SimulationConfig field
- Update constructor and start method to accept simulation configuration
- Add simulation trigger in perform_epoch_operations() before real rewarding
- Ensure simulation failures don't break epoch advancement process
CLI Integration:
- Update run.rs to handle both --enable-rewarding and --simulate-rewarding modes
- Create SimulationConfig from rewarding.debug configuration settings
- Implement mutual exclusivity between real rewarding and simulation mode
- Skip permission checks for simulation-only mode (no blockchain transactions)
The simulation system runs in parallel with epoch operations, storing comparative
data for analysis without affecting production reward distribution.
Implement foundation for simulated reward calculations to compare old (24h cache-based)
vs new (1h route-based) methodologies without blockchain transactions.
Database Changes:
- Add migration with 4 new tables for simulation system
- simulated_reward_epochs: tracks each simulation run
- simulated_node_performance: stores performance calculations
- simulated_rewards: stores reward calculation results
- simulated_route_analysis: metadata for route analysis
- Add comprehensive indexes for efficient querying
Configuration Changes:
- Add simulation_mode flag to rewarding configuration
- Add CLI flag --simulate-rewarding with proper dependencies/conflicts
- Add validation for simulation-specific settings
- Add time window configuration for new method (default: 1 hour)
Storage Layer:
- Add model structs with SQLx FromRow derives for all simulation tables
- Add comprehensive CRUD methods for simulation data management
- Add proper type annotations to fix SQLx compile issues
- Maintain separation between simulation and real rewarding logic
The simulation mode is mutually exclusive with real rewarding and does not
require mnemonic since no blockchain transactions are performed.
* refactor FormattedResponse to allow attaching additional headers
* helper method for including expiration headers
* add expires header for /active nodes responses
* added additional 'with_expires_header_delta' builder to FormattedResponse to allow setting expiration header with time delta
* introduce 'Bincode' variant for FormattedResponse
* allow nym-api to return responses in bincode (and also yaml)
* client parsing support
* cargo fmt
* missing changes to nym-api tests
* fixed node status api build + adjusted NymApiClient construction
* NMv2 fixes + further api changes
* feature-locking http-api-common to fix wasm build
* Add /account/{address}
* Don't query vesting info
* Don't query rewards
* Remove unused code
* Fix clippy
* Fix build.rs build on Windows
* Addressing PR feedback
- not cloning nym nodes from cache
- reduced number of nym nodes kept in memory
- reduced number of iterations to read all data
- removed some fields
* Fix total_delegations
* Optimize nym_nodes hashmap
* Split flow into functions
* Remove vesting info
* Add caching for endpoint
* Cache optimizations
* Return early if balance is 0
* Refactor state cloning shenanigans
* Improve swagger definitions
* Sort data in DB
* Improve logging
* Store gw description to nym nodes table
* Move explorer related path to /explorer
* Bump package version
* removed support for legacy packet types from NymCodec
I think nodes had plenty of time to upgrade given versioned variant was introduced in 2022
* temp: use local sphinx packet for development
* introduce new messages that use more efficient reply surbs encoding
* checks for incorrect encoding
* generate correct message depending on config value
* fixed current packet version
* made packet type selection configurable
* updated sphinx packet crate to the published version
* fixed wasm build
* fixes in outfox due to sphinx api changes
* additional tests
* clippy
* fixed log/tracing import
* change ticketbook data request to allow for pagination
* implemented api endpoints on nym-api side
* auxiliary nym-api queries for number of issued ticketbooks
* ensure that challenged issuers support new queries
* addeed persistent identity to the rewarder
* clippy
* stupid chrono feature workaround
* clippy
* debugging issuance verification
* remove redundant closure
* added a minimum issuance threshold
durch