Mark Sinclair
f2091cc9d6
* rename nyxd-scraper to sqlite
wip: made storage mostly generic minus modules
changed error types to make modules dyn compatible
implemented traits for sqlite instance
using sqlite instance for rewarder and chain watcher
psql scaffolding
initial postgres support - missing some proto -> json parsing
use postgres in chain scraper
added message registry to block processor
message content parsing in psql
involved addresses
adding null value for logs
Revert "use postgres in chain scraper"
This reverts commit 83c84bfd2d.
using SignerInfo proto definitions for db serialisation
added ibc messages to MessageRegistry
* add the data observatory
* add the data observatory
* move message parsing and change webhook
* handle wasm messages in a module
* formatting and clippy
* copy shared migrations and add comments to ignore file to explain
* update offline queries
* change to clap args and use url::Url to parse args
* tidy up README, startup info, typos
* tidy up validator rewarder
* lock file
* change webhook module from msg to tx handler
* ignore profiler output
* add missing things and make clippy happy
* updated cosmrs version used by the nym wallet
* add glob to workspace dependencies
* rename migration files
* remove copying from shared migrations
* duplicate shared migrations to keep things simple
* add check for manual migration sync that will fail on `cargo build` in CI
* build.rs checks data observatory migrations have content of all shared scraper migrations and errors on changes or new files
* update runner
* add reset target to make file
* process events and logs
* migrations - remove unnecessary columns
* update offline queries
* chore: run cargo fmt
* fix up: inpsect_err instead of map_err
---------
Co-authored-by: Jędrzej Stuczyński <jedrzej.stuczynski@gmail.com>
Co-authored-by: Mark Sinclair <mmsinclair@users.noreply.github.com>
Co-authored-by: benedettadavico <benedetta.davico@gmail.com>
111 lines
2.8 KiB
Markdown
111 lines
2.8 KiB
Markdown
# Nym Data Observatory
|
|
|
|
Collects data about the Nym network including:
|
|
|
|
- **Chain scraper** - that parses blocks, transactions and messages on the Nyx chain
|
|
- **Price scraper** - to get the NYM/USD token price from CoinGecko
|
|
- **Webhooks** - trigger on messages or all messages to call with details
|
|
|
|
## Running locally
|
|
|
|
### 1. Install Prerequisites
|
|
|
|
```bash
|
|
# Install sqlx-cli if not already installed
|
|
make sqlx-cli
|
|
```
|
|
|
|
### 2. Prepare PostgreSQL for Development
|
|
|
|
```bash
|
|
# This will:
|
|
# - Start PostgreSQL in Docker
|
|
# - Run migrations
|
|
# - Generate SQLx offline query cache
|
|
# - Stop the database
|
|
make prepare-pg
|
|
```
|
|
|
|
### 3. Build
|
|
|
|
```bash
|
|
make build-pg
|
|
```
|
|
|
|
### 4. Run with PostgreSQL
|
|
|
|
```bash
|
|
# Start PostgreSQL for development (keeps running)
|
|
make test-db-up
|
|
|
|
# In another terminal, run the application
|
|
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test \
|
|
NYM_DATA_OBSERVATORY_WEBHOOK_URL="https://webhook.site" \
|
|
NYM_DATA_OBSERVATORY_WEBHOOK_AUTH=1234 \
|
|
cargo run -- run
|
|
```
|
|
|
|
To start from a block add the env var: `NYXD_SCRAPER_START_HEIGHT=19266184`.
|
|
|
|
## Deploying
|
|
|
|
Connect with `psql` to your local database:
|
|
|
|
```sql
|
|
CREATE USER nym_data_observatory WITH PASSWORD 'data-data-data';
|
|
|
|
CREATE DATABASE nym_data_observatory_data;
|
|
GRANT ALL ON DATABASE nym_data_observatory_data TO nym_data_observatory;
|
|
```
|
|
|
|
Then run:
|
|
|
|
```
|
|
cargo run -- init --db_url postgres://testuser:testpass@localhost:5433/nym_data_observatory_test
|
|
```
|
|
|
|
and then:
|
|
|
|
```
|
|
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test \
|
|
NYM_DATA_OBSERVATORY_WEBHOOK_URL="https://webhook.site" \
|
|
NYM_DATA_OBSERVATORY_WEBHOOK_AUTH=1234 \
|
|
cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net
|
|
```
|
|
|
|
or just:
|
|
|
|
```
|
|
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net
|
|
```
|
|
|
|
If you want to watch for cosmwasm messages and send to a webhook:
|
|
|
|
```
|
|
NYM_DATA_OBSERVATORY_WEBHOOK_URL=https://webhook.site \
|
|
NYM_DATA_OBSERVATORY_DB_URL=postgres://testuser:testpass@localhost:5433/nym_data_observatory_test\
|
|
cargo run -- run --websocket-url wss://rpc.nymtech.net/websocket --rpc-url https://rpc.nymtech.net --start-block-height 20966360 --watch-for-chain-message-types "/cosmwasm.wasm.v1.MsgExecuteContract"
|
|
```
|
|
|
|
## Troubleshooting
|
|
|
|
### SQLx Offline Mode
|
|
|
|
If you see "no cached data for this query" errors:
|
|
|
|
1. Ensure PostgreSQL is running: `make dev-db`
|
|
2. Run: `make test-db-prepare`
|
|
|
|
Also see [README_SQLX.md](../nyx-chain-watcher/README_SQLX.md).
|
|
|
|
### Connection Refused
|
|
|
|
If you see "Connection refused" errors:
|
|
|
|
1. Check Docker is running: `docker ps`
|
|
2. Check PostgreSQL container: `docker ps | grep nym_data_observatory
|
|
3. Restart database: `make test-db-down && make dev-db`
|
|
|
|
|
|
|