* rename nyxd-scraper to sqlite
wip: made storage mostly generic minus modules
changed error types to make modules dyn compatible
implemented traits for sqlite instance
using sqlite instance for rewarder and chain watcher
psql scaffolding
initial postgres support - missing some proto -> json parsing
use postgres in chain scraper
added message registry to block processor
message content parsing in psql
involved addresses
adding null value for logs
Revert "use postgres in chain scraper"
This reverts commit 83c84bfd2d.
using SignerInfo proto definitions for db serialisation
added ibc messages to MessageRegistry
* add the data observatory
* add the data observatory
* move message parsing and change webhook
* handle wasm messages in a module
* formatting and clippy
* copy shared migrations and add comments to ignore file to explain
* update offline queries
* change to clap args and use url::Url to parse args
* tidy up README, startup info, typos
* tidy up validator rewarder
* lock file
* change webhook module from msg to tx handler
* ignore profiler output
* add missing things and make clippy happy
* updated cosmrs version used by the nym wallet
* add glob to workspace dependencies
* rename migration files
* remove copying from shared migrations
* duplicate shared migrations to keep things simple
* add check for manual migration sync that will fail on `cargo build` in CI
* build.rs checks data observatory migrations have content of all shared scraper migrations and errors on changes or new files
* update runner
* add reset target to make file
* process events and logs
* migrations - remove unnecessary columns
* update offline queries
* chore: run cargo fmt
* fix up: inpsect_err instead of map_err
---------
Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
Co-authored-by: benedettadavico <benedetta.davico@gmail.com>
* vpn client report v2
* report v2 support in nym-stats API
* version bump
* CI fix while we're at it
* more CI fix
* needed the dind after all
* PR comments
dont build netstack in CI
additional rust 2024 fixes
fixes
removed temp.rs
first round of cleanup
removed duplicated NS types
moved gateway probe to the monorepo
* Debug logging
* Yield based logging
* Reintroduce non-dummy task manager, try add counting for
BatchMessageSender, a couple of compiler target introductions on use
statements.
* Fixed time runtime err
* Uncomment forgetme/rememberme
* remove diffs from debug
* missed commented out forgetme
* yet more forgetme comments
* * Added missing clientreqestsender clone to wasm client to stop
premature drop & busyloop
* Removed hacky mem::forget fix
* Remove debug panic_hook
* Conditional import + use of wasm_utils::console_log
* add wasm_util dep
* Commenting out or removing debug logging
* Remove missed comment
* cleanup gitignore
* clippy
* update go version in ci
* removed unused deps
* add clippy ignore
* remove mixfetch from ci build
* add minifetch fix
* comment out unused ts builds
* stop contract clients killing ci for the moment
* wasm target locking for imports
* Either remove console_log! macro or introduce cfg(debug_assertions)
* downgrade netlink
* debug assertions for console_log import
* modify config logging (debug -> normal)
* remove clone for client_request_sender + grab directly in struct
creation
* reintroduce debug print for config in debug mode
* remove ood / unused custom topology from worker example file
* clippy
* clippy - ignore todo() tests
* modified humantime test in line with new parsing rules
* ns-api: remove sqlite support
ns-api: add env var to skip migrations for local dev
ns-api: tidy up imports
ns-api: fix deserialisation fo node descriptions
update dockerfile
update README
fix up README and example env
ns-api: bump major version to 4
ns-api: add more geoip data and new performance field in dvpn responses
* ability to import partial ticketbooks
* wip: adding common ecash state to NS API
* buffering ticketbooks
* wip
* distribute tickets when getting testrun assignment
* passing ticketbook data to gateway probe
* wrapped around storage tx
* ticketbook query fixes
* clippy
* modified testrun assignment to always return tickets
* Update version
* Update push-node-status-agent.yaml
* Update Cargo.toml
* add entrypoint for ns agents
* sqlx prepare and cargo fmt
* clippy fixes
* Update ci-check-ns-api-version.yml
---------
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
Co-authored-by: benedetta davico <46782255+benedettadavico@users.noreply.github.com>
Co-authored-by: benedettadavico <benedetta.davico@gmail.com>
* ns-api: remove sqlite support
ns-api: add env var to skip migrations for local dev
ns-api: tidy up imports
ns-api: fix deserialisation fo node descriptions
update dockerfile
update README
fix up README and example env
ns-api: bump major version to 4
ns-api: add more geoip data and new performance field in dvpn responses
* ns-api: polyfill dVPN probe outcomes to make compatible with existing clients
* Use explicit transaction for testrun status change (#6046)
* Use explicit transaction for testrun status change
* Improve run scripts
* Skip locked rows
* bump version 4.0.2
* Fix build.rs
* Fix up .sqlx queries
* Bump agent version and change dockerfile to run the agent in a loop
* Make time between agents configurable by env var SLEEP_TIME
* Update entrypoint.sh
* Update Dockerfile with full path
* Force bigint to avoid postgres numeric cast
* Add override args to agent entry point, bump agent version and NS API version
---------
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
Co-authored-by: dynco-nym <173912580+dynco-nym@users.noreply.github.com>
* conditionally enable console-subscriber within nym-node
* Update ci-build-upload-binaries.yml
* Update ci-build-upload-binaries.yml
add features console
* updated feature name
* fixed filtering on tracing layers
* add track_caller when spawning futures for better tokio-console support
* allow [client] tasks to specify their names when used within tokio console
* clippy
* pre-emptively fix wasm clippy
---------
Co-authored-by: Tommy Verrall <60836166+tommyv1987@users.noreply.github.com>
* feat(db): add SQL query wrapper for PostgreSQL placeholder conversion
- Created query_wrapper module with functions to automatically convert
SQLite ? placeholders to PostgreSQL $1, $2, ... format
- Updated build.rs to handle mutually exclusive feature flags
- Modified one query in mixnodes.rs as proof of concept
- Added type conversions for PostgreSQL compatibility (u32->i64, u16->i32)
This is a checkpoint commit before converting all queries to use the wrapper.
* feat(nym-node-status-api): add PostgreSQL database support via feature flags
Implement dual database support for SQLite and PostgreSQL through Cargo feature flags.
The implementation uses a query wrapper that automatically converts SQLite-style ?
placeholders to PostgreSQL-style $1, $2, ... placeholders at runtime.
Key changes:
- Add query wrapper functions that handle placeholder conversion
- Convert all sqlx::query\! macros to use wrapper functions
- Handle type conversions between databases (i64 vs i32)
- Add feature-gated implementations for database-specific SQL syntax
- Update Makefile with clippy targets for both database features
- Document database support in README
* feat(nym-node-status-agent): add multi-API support with random selection
Agents can now connect to multiple APIs and randomly select one for each testrun:
- Accept multiple --server arguments in format "address:port:auth_key"
- Randomly shuffle server list before attempting connections
- Try each server until a testrun is obtained
- Submit results back only to the API that provided the testrun
- Continue to next server if one is down or has no testruns available
* feat(nym-node-status): implement primary/secondary server architecture
- Agent now requests testruns only from primary server (first in list)
- Results are submitted to all configured servers in parallel
- Secondary servers accept external testruns via new v2 endpoint
- Added auto-creation of gateway and testrun records on secondary servers
- New database queries: get_or_create_gateway, insert_external_testrun
- Client library enhanced with submit_results_with_context method
* Bump Node status API version
* Fix build workdir
* Bump to 3.1.4
* Fix types and queries
* 3.1.6
* Fix gateway perf, bump 3.1.7
* NodeId -> i32, 3.1.8
* Bump agent version
* i64 -> i32
* Use image yq
* Migration and more types
* Update remaining JSONB columns
* Simplify server config
* Update build path
* Change delimiter
* bump agent
* Split up pg and sqlite builds
* More typing fixes, build-and-push script
* Fix Dockerfile-pg
* Bump node-status-api
* TYping
* Agent build script
* More logging around testruns
* Fail loudly on read errors
* Cleanup
* Debug get gateways query
* Fix get_gateways query
* Use pg cert, 3.1.16
* Submit regular results to primary server
* Bump freshenss cutoff
* Update Cargo.lock
* fix: resolve rebase conflicts and compilation errors
After rebasing onto develop, fixed several issues:
- Fixed borrowed data escapes error by using sqlx::query directly in transaction functions
- Removed unused imports and cleaned up code
- Maintained database-specific implementations for transaction functions
* fmt
* Make PG default to make lives easier
* Performance improvements for Explorer v2
* Fix sqlite build
* Fix PG migration
* Tests round 1
* DB tests
* More tests
* And some more tests
* And some more, more tests
* cargo fmt
* Fix some failing lints
* Fix lioness version problems
* Clippy in tests
---------
Co-authored-by: dynco-nym <173912580+dynco-nym@users.noreply.github.com>
Mark Sinclair