* db: migrate from lmdb-zero to heed
* fix: check resizing operation and wait to avoid crash with multiple batches, fix exists check at batch
* build: fix missing deps at Cargo.lock
* lmdb: single environment, migrate existing databases with provided non-default environment name
* fix: revert chunk size to 128mb
* lmdb: ability to use multiple shared environments
* build: remove unused dependency
* fix: resize to have correct multiplier of the system page size
* lmdb: speed up prefix iter by storing keys
* lmdb: default env name
* lmdb: wait db resize before read, reduce timeout before resizing
* lmdb: use static reader for iterator, count existing batches for stable resize
* fix: check batches count on resize waiting
* lmdb: use separate databases instead of prefixes, use default db for values without prefixes, migrate old environment
* fix: pop pos key
* lmdb: count all open transactions to finish before resizing
* lmdb: immediate resize if there are no open transactions
* lmdb: remove env state when there are no more stores
* lmdb: use atomic for resize and resize checking flags
* lmdb: sleep 10ms when waiting all opened txs to be closed
* lmdb: use atomic open txs and stores count
* lmdb: use index to detect separator, ignore unknown db key to not have a panic
* lmdb: store max 10k keys in the iterator
* lmdb: check iter result on getting total
* lmdb: handle errors at iterator
* lmdb: handle an error when db with provided key not found
* lmdb: fix iterate over 10k keys
* lmdb: document migration resize safety
* lmdb: fix iter test
* lmdb: clear new db after unsuccessful migration, handle read error on migration to interrupt process
* store: bring back old key methods to reproduce data migration
* lmdb: return an error on unsuccessful migration
* lmdb: migration test, clean data after allocate test
* lmdb: info migration log
* fix: move iterator before handling an error to allow skip bad value
* lmdb: return an error if removal of old DB file failed after migration
* lmdb: lifetime for iterator, use write transaction at batch iterator
* lmdb: migration progress
* fix: tests
* lmdb: immediately set resizing flag, ignore resizing flag while there are more than 0 opened txs to avoid stuck, optimize tx counter for some operations
* lmdb: key for successful migration
* lmdb: fix put database creation at separate block to avoid lifetime issues when returning an error on migration
* tui: show server initialization status and error
* fix: compilation issues
* fix: add documenting to status, remove unused imports
* fix: do not empty server value
* fix: server ref
* tui: do not quit on q when another dialog is showing (progress or error)
* fix: server stop on tui shutdown
* fix: stop server if tui was stopped after start
* server: panic on error at non-tui mode like before with unwrap
* fix: pop dialog
* fix: do not return result on tx after server start
* tui: close current dialog before quit
* tui: pass stop state to server creation after tui quit
* tui: exit code 1 after error, also for non-tui
* tui: better exit code
* peer: unknown state for new peers, check peers state on every monitor (128 healthy non-connected + 128 defuncts + 128 unknown), mark peer as defunct when ping not passed, do not crash on toml parse with dns failure
* p2p: cleanup before selection at monitor, add outbound to connected list only when there is not enough peers + disconnect extra peer immediately, reconnect to seeds at monitor to avoid stuck, update only defunct state to unknown when received existing peer address
* p2p: reduced amount of total peers to check at monitor
* p2p: do not check healthy and defunct peers more often than once per hour, store last connection attempt, do not ask for more peers when there is enough outbound
* peer: update last_attempt when changing peer state to other than Banned
* fix: log of peers amount to check
* Update versioning on master to 5.4.0-alpha.0
* updates for 1.80 and other accumulated warnings
* further warning cleanups
* move dead code tag to function defn rather than module
* [PIBD_IMPL] Introduce PIBD state into sync workflow (#3685)
* experimental addition of pibd download state for testnet only
* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap
* attempt to call a test message
* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain
* fixes to state sync
* add pibd receive messages to network, and basic calls to desegmenter from each (#3686)
* [PIBD_IMPL] PIBD Desegmenter State (#3688)
* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested
* add segmentidentifier type to id requested segments uniquely
* make a call on where to keep track of which PIBD segments have been requested
* move segmenttype definition, add functions to manipulate peer segment list
* remove desegmenter state enum
* change chain desegmenter function to provide rwlock
* trace, warning cleanup
* udpate to test compliation
* [PIBD_IMPL] Bitmap accumulator reconstruction + TxHashset set reconstruction (#3689)
* application of received bitmap segments to local accumulator
* add all required elements to send/receive output segment requests and responses
* testing of output sync
* add special cases to pmmr segment request
* [PIBD_IMPL] PMMR Reassembly from Segments (#3690)
* update pibd copy test to use new desgmenter structure
* begin reconstruction of output pmmr
* clean up hash/leaf insertion logic
* push pruned subtree appears to be working, now also calculates left hand hashes correctly
* factor out ordering of segment/hash order array
* refactor for pmmr application code
* test of chain copy appears to be working
* add rangeproof functions to desegmenter
* add kernel functions, attempt refactor
* small test cleanup, reconstruction of live chain working in manual copy test
* [PIBD_IMPL] PIBD tree sync via network and kill/resume functionality (#3691)
* add functions to determing latest verifiable block height for the given pibd state
* attempting to allow for pibd to resume after killing process
* fix to ensure prune list is properly flushed during pibd sync
* removal of unneeded code
* ignore test for now (fix before full merge)
* [PIBD_IMPL] Finalize PIBD download and move state to chain validation (#3692)
* investigations as to why a slight rewind is needed on startup during PIBD
* move validation code into desegmenter validation thread (for now)
* ensure genesis entries in pmmrs are removed if they're removed in the first segment
* validation all working except for verifying kernel sums
* remove unneeded pmmr rollbacks on resume now root cause was found
* updates to remove unpruned leaves from leaf set when rebuilding pmmr
* remove + 1 to segment traversal iter length
* [PIBD_IMPL] PIBD Stats + Retry on validation errors (#3694)
* start to add stats and reset chain state after errors detected
* add functions to reset prune list when resetting chain pibd state
* debug statement
* remove test function
* [PIBD_IMPL] Update number of simultaneous peer requests for segments (#3696)
* cleanup of segment request list
* allow for more simultaneous requests during state sync
* up number of simultaneous peer requests for segments
* [PIBD_IMPL] Thread simplification + More TUI Updates + Stop State Propagation (#3698)
* change pibd stat display to show progress as a percentage of downloaded leaves
* attempt some inline rp validation
* propagate shutdown state through kernel validation
* change validation loop timing
* simplify validator threading
* add more detailed tracking of kernel history validation to tui, allow stop state during
* adding more stop state + tui progress indication
* remove progressive validate
* test fix
* revert to previous method of applying segments (#3699)
* fix for deadlock issue (#3700)
* update Cargo.lock for next release
* [PIBD_IMPL] Catch-Up functionality + Fixes based on testing (#3702)
* ensure desegmenter attempts to apply correct block after a resumte
* ensure txhashset's committed implementation takes into account output bitmap for summing purposes
* remove check to de-apply outputs during segment application
* return removal of spent outputs during pibd
* remove unneeded status
* remove uneeded change to rewind function
* documentation updates + todo fixes (#3703)
* add pibd abort timeout case (#3704)
* [PIBD_IMPL] BitmapAccumulator Serialization Fix (#3705)
* fix for writing / calculating incorrect length for negative indices
* update capabilities with new version of PIBD hist
* remove incorrect comment
* fix capabilities flag, trace output
* test fix
* Merge DNSSeed scope changes into pibd impl branch (#3708)
* update Cargo.lock for next release
* visibility scope tweaks to aid seed test utilities (#3707)
* move all PIBD-related constants into pibd_params modules (#3711)
* remove potential double read lock during compaction
* Minor update for foreign api docs. Peers seems to be no longer needed as an arg in the grin_api::Foreign::new.
* This changes foreign_rpc and owner_rpc modules to public so the helper modules for the rpc clients can be accessible in external projects.
* Updated test seeds.
* Added functions that override the one time globals. This is needed for the grin-gui so it can change the global contexts when it needs to switch from testnet to mainnet.
* Refactor
* Fix panick at 'attempt to subtract with overflow' when grin-wallet sends in a start_index of 0 for new wallets.
* Fix overflow panic using saturating_sub(1).
* WIP remove failure from all `Cargo.toml`
* WIP remove `extern crate failure_derive`
* Use `thiserror` to fix all errors
* StoreErr is still a tuple
* Remove another set of unnecessary `.into()`s
* update fuzz tests
* update pool/fuzz dependencies in cargo.lock
* small changes based on feedback
Co-authored-by: trevyn <trevyn-git@protonmail.com>
* add parameter fast_validation to the api
* arg `verify-chain` on node client
* arg verify chain + read_timeout parameter
* change timeout to 2hours
* update with tromp review
* 2nd comment from tromp
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* wip - "reset_head" via owner api functionality
* jsonrpc pass hash in as a string
* sort of works
* not a reorg if we simply accept several blocks at once
* remember to reset header MMR separately
as it is readonly when interacting with txhashset extension
* basic client integration
needs error handling etc.
* reset sync status when reset chain head
* track "denylist" (todo) and validate headers against this via the ctx
* track denylist (header hashes) in chain itself
* header denylist in play
* expose invalidateheader as client cmd
* rework reset_chain_head - rewind txhashset then header MMR
* fix for comparing scaled vs unscaled difficulty
fix mix use of scaled and unscaled, add additional comments to clarify which values are which
improved consistency - scaled and unscaled difficulty
* combine from_proof_unscaled() and to_unscaled_difficulty{}
* code review - rust syntax
* track sync_head on header_sync sync status
follow header fork as appropriate
* track highest_diff alongside highest_height on HeaderSync status
* cleanup
* not not
* wip - body sync for full archive
* allow chain compaction during sync
* placeholder for logic to ensure archive nodes sync from archive nodes
* body sync from archival peers
* allow chain compaction during sync
* placeholder for logic to ensure archive nodes sync from archive nodes
* add column ordering, set mining workers table to sort the connected workers to the top by default
* add column ordering, set mining workers table to sort the connected workers to the top by default
* Mining TUI fixes and updates
* add FeeFields type
* use FeeFields with ::zero and try_into().unwrap()
* fixed tests
* avoid 0 accept_base_fee
* add aggregate_fee_fields method for transaction
* implement std::fmt::Display trait for FeeFields
* make base_fee argument non-optional in libtx::mod::tx_fee
* add global and thread local accept_fee_base; use to simplify tests
* set unusually high fee base for a change
* revert to optional base fee argument; default coming from either grin-{server,wallet}.toml
* remove optional base fee argument; can be set with global::set_local_accept_fee_base instead
* define constant global::DEFAULT_ACCEPT_FEE_BASE and set global value
* add Transaction::accept_fee() method and use
* Manual (de)ser impl on FeeFields
* fix comment bug
* Serialize FeeFields as int in tx
* allow feefields: u32:into() for tests
* try adding height args everywhere
* make FeeFields shift/fee methods height dependent
* prior to hf4 feefield testing
* rename selected fee_fields back to fee for serialization compatibility
* fix test_fee_fields test, merge conflict, and doctest use of obsolete fee_fields
* make accept_fee height dependent
* Accept any u64 in FeeFields deser
Co-authored-by: Jasper van der Maarel <j@sper.dev>
ardocrat