Mark Sinclair
f2091cc9d6
* rename nyxd-scraper to sqlite
wip: made storage mostly generic minus modules
changed error types to make modules dyn compatible
implemented traits for sqlite instance
using sqlite instance for rewarder and chain watcher
psql scaffolding
initial postgres support - missing some proto -> json parsing
use postgres in chain scraper
added message registry to block processor
message content parsing in psql
involved addresses
adding null value for logs
Revert "use postgres in chain scraper"
This reverts commit 83c84bfd2d.
using SignerInfo proto definitions for db serialisation
added ibc messages to MessageRegistry
* add the data observatory
* add the data observatory
* move message parsing and change webhook
* handle wasm messages in a module
* formatting and clippy
* copy shared migrations and add comments to ignore file to explain
* update offline queries
* change to clap args and use url::Url to parse args
* tidy up README, startup info, typos
* tidy up validator rewarder
* lock file
* change webhook module from msg to tx handler
* ignore profiler output
* add missing things and make clippy happy
* updated cosmrs version used by the nym wallet
* add glob to workspace dependencies
* rename migration files
* remove copying from shared migrations
* duplicate shared migrations to keep things simple
* add check for manual migration sync that will fail on `cargo build` in CI
* build.rs checks data observatory migrations have content of all shared scraper migrations and errors on changes or new files
* update runner
* add reset target to make file
* process events and logs
* migrations - remove unnecessary columns
* update offline queries
* chore: run cargo fmt
* fix up: inpsect_err instead of map_err
---------
Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
Co-authored-by: benedettadavico <benedetta.davico@gmail.com>
2.8 KiB
2.8 KiB
Nym Data Observatory
Collects data about the Nym network including:
- Chain scraper - that parses blocks, transactions and messages on the Nyx chain
- Price scraper - to get the NYM/USD token price from CoinGecko
- Webhooks - trigger on messages or all messages to call with details
Running locally
1. Install Prerequisites
# Install sqlx-cli if not already installed
make sqlx-cli
2. Prepare PostgreSQL for Development
# This will:
# - Start PostgreSQL in Docker
# - Run migrations
# - Generate SQLx offline query cache
# - Stop the database
make prepare-pg
3. Build
make build-pg
4. Run with PostgreSQL
# Start PostgreSQL for development (keeps running)
make test-db-up
# In another terminal, run the application
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test \
NYM_DATA_OBSERVATORY_WEBHOOK_URL="https://webhook.site" \
NYM_DATA_OBSERVATORY_WEBHOOK_AUTH=1234 \
cargo run -- run
To start from a block add the env var: NYXD_SCRAPER_START_HEIGHT=19266184.
Deploying
Connect with psql to your local database:
CREATE USER nym_data_observatory WITH PASSWORD 'data-data-data';
CREATE DATABASE nym_data_observatory_data;
GRANT ALL ON DATABASE nym_data_observatory_data TO nym_data_observatory;
Then run:
cargo run -- init --db_url postgres://testuser:testpass@localhost:5433/nym_data_observatory_test
and then:
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test \
NYM_DATA_OBSERVATORY_WEBHOOK_URL="https://webhook.site" \
NYM_DATA_OBSERVATORY_WEBHOOK_AUTH=1234 \
cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net
or just:
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net
If you want to watch for cosmwasm messages and send to a webhook:
NYM_DATA_OBSERVATORY_WEBHOOK_URL=https://webhook.site \
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test\
cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net --start-block-height 20966360 --watch-for-chain-message-types "/cosmwasm.wasm.v1.MsgExecuteContract"
Troubleshooting
SQLx Offline Mode
If you see "no cached data for this query" errors:
- Ensure PostgreSQL is running:
make dev-db - Run:
make test-db-prepare
Also see README_SQLX.md.
Connection Refused
If you see "Connection refused" errors:
- Check Docker is running:
docker ps - Check PostgreSQL container: `docker ps | grep nym_data_observatory
- Restart database:
make test-db-down && make dev-db