* first pass compilation of croaring update
* cargo.lock
* add roaring arch flag into CI build scripts
* revert CI to use windows 2019 image
* add more debug
* more debug info
* update range arguments to bitmap remove_range function calls
* remove unnecessary cast
* [PIBD_IMPL] Introduce PIBD state into sync workflow (#3685)
* experimental addition of pibd download state for testnet only
* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap
* attempt to call a test message
* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain
* fixes to state sync
* add pibd receive messages to network, and basic calls to desegmenter from each (#3686)
* [PIBD_IMPL] PIBD Desegmenter State (#3688)
* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested
* add segmentidentifier type to id requested segments uniquely
* make a call on where to keep track of which PIBD segments have been requested
* move segmenttype definition, add functions to manipulate peer segment list
* remove desegmenter state enum
* change chain desegmenter function to provide rwlock
* trace, warning cleanup
* udpate to test compliation
* [PIBD_IMPL] Bitmap accumulator reconstruction + TxHashset set reconstruction (#3689)
* application of received bitmap segments to local accumulator
* add all required elements to send/receive output segment requests and responses
* testing of output sync
* add special cases to pmmr segment request
* [PIBD_IMPL] PMMR Reassembly from Segments (#3690)
* update pibd copy test to use new desgmenter structure
* begin reconstruction of output pmmr
* clean up hash/leaf insertion logic
* push pruned subtree appears to be working, now also calculates left hand hashes correctly
* factor out ordering of segment/hash order array
* refactor for pmmr application code
* test of chain copy appears to be working
* add rangeproof functions to desegmenter
* add kernel functions, attempt refactor
* small test cleanup, reconstruction of live chain working in manual copy test
* [PIBD_IMPL] PIBD tree sync via network and kill/resume functionality (#3691)
* add functions to determing latest verifiable block height for the given pibd state
* attempting to allow for pibd to resume after killing process
* fix to ensure prune list is properly flushed during pibd sync
* removal of unneeded code
* ignore test for now (fix before full merge)
* [PIBD_IMPL] Finalize PIBD download and move state to chain validation (#3692)
* investigations as to why a slight rewind is needed on startup during PIBD
* move validation code into desegmenter validation thread (for now)
* ensure genesis entries in pmmrs are removed if they're removed in the first segment
* validation all working except for verifying kernel sums
* remove unneeded pmmr rollbacks on resume now root cause was found
* updates to remove unpruned leaves from leaf set when rebuilding pmmr
* remove + 1 to segment traversal iter length
* [PIBD_IMPL] PIBD Stats + Retry on validation errors (#3694)
* start to add stats and reset chain state after errors detected
* add functions to reset prune list when resetting chain pibd state
* debug statement
* remove test function
* [PIBD_IMPL] Update number of simultaneous peer requests for segments (#3696)
* cleanup of segment request list
* allow for more simultaneous requests during state sync
* up number of simultaneous peer requests for segments
* [PIBD_IMPL] Thread simplification + More TUI Updates + Stop State Propagation (#3698)
* change pibd stat display to show progress as a percentage of downloaded leaves
* attempt some inline rp validation
* propagate shutdown state through kernel validation
* change validation loop timing
* simplify validator threading
* add more detailed tracking of kernel history validation to tui, allow stop state during
* adding more stop state + tui progress indication
* remove progressive validate
* test fix
* revert to previous method of applying segments (#3699)
* fix for deadlock issue (#3700)
* update Cargo.lock for next release
* [PIBD_IMPL] Catch-Up functionality + Fixes based on testing (#3702)
* ensure desegmenter attempts to apply correct block after a resumte
* ensure txhashset's committed implementation takes into account output bitmap for summing purposes
* remove check to de-apply outputs during segment application
* return removal of spent outputs during pibd
* remove unneeded status
* remove uneeded change to rewind function
* documentation updates + todo fixes (#3703)
* add pibd abort timeout case (#3704)
* [PIBD_IMPL] BitmapAccumulator Serialization Fix (#3705)
* fix for writing / calculating incorrect length for negative indices
* update capabilities with new version of PIBD hist
* remove incorrect comment
* fix capabilities flag, trace output
* test fix
* Merge DNSSeed scope changes into pibd impl branch (#3708)
* update Cargo.lock for next release
* visibility scope tweaks to aid seed test utilities (#3707)
* move all PIBD-related constants into pibd_params modules (#3711)
* remove potential double read lock during compaction
* WIP remove failure from all `Cargo.toml`
* WIP remove `extern crate failure_derive`
* Use `thiserror` to fix all errors
* StoreErr is still a tuple
* Remove another set of unnecessary `.into()`s
* update fuzz tests
* update pool/fuzz dependencies in cargo.lock
* small changes based on feedback
Co-authored-by: trevyn <trevyn-git@protonmail.com>
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* initial commit of WIP pibd explorations
* correct calling for obtaining and validating first segment
* update test to properly iterate through each segment of the test pmmrs, validating each segment as it goes
* updated test to fully segment and validate PMMRs from compacted and uncompacted sample data. Also contains method of running test againt live chain data
* remove logger change
* change test file name
* change test file name
* change directory reference in test for CI
* add initial (experimental) structure for PIBD desegmenting
* move bitmap desegmentation logic into desegmenter
* added txhashset methods to apply pibd segments (note this only works for fully unpruned trees atm)
* change last_pos to mmr_size
* fix to pmmr::peaks call
* don't verify POW when copying headers
* prepare for commit of work thus far'
* update test paths
* few updates based on early review
* use 0-based positions in methods pmmr_leaf_to_insertion_index and bintree_postorder_height; add round_up_to_leaf_pos method
* use 0-based positions in method insertion_to_pmmr_index
* use 0-based positions in method is_leaf
* use 0-based positions in method family()
* use 0-based positions in method is_left_sibling
* use 0-based positions in method family_branch
* use 0-based positions in methods bintree_{left,right}most
* use 0-based positions in method bintree_pos_iter
* use 0-based positions in method bintree_range
* use 0-based positions in method bintree_leaf_pos_iter
* rename last_pos in MMR related structs to size
* use 0-based positions in method prune
* use 0-based positions in method push and apply_output return value
* use 0-based position argument of method merkle_proof
* use 0-based outputs in method pmmr::peaks
* fix peaks() code comments
* refix peaks() code comments
* use 0-based positions in method get_peak_from_file
* use 0-based positions in methods get_data_from_file
* use 0-based positions in methods get_from_file
* use 0-based positions in methods get_data
* use 0-based positions in methods get_hash
* use 0-based positions in method peak_path
* use 0-based positions in method bag_the_rhs
* use 0-based positions in method Backend::remove
* use 0-based positions in method leaf_pos_iter
* use 0-based positions in method self.LeafSet::includes
* use 0-based positions in methods self.LeafSet::{add,remove}
* use 0-based positions in methods is_pruned,is_pruned_root,is_compacted
* use 0-based positions in methods PruneList::append
* use 0-based positions in methods append_pruned_subtree
* use 0-based positions in method calculate_next_leaf_shift
* use 0-based positions in method append_single
* use 0-based positions in method calculate_next_shift
* use 0-based positions in method segment_pos_range
* use 0-based positions in method reconstruct_root
* use 0-based positions in method validate_with
* use 0-based positions in method validate
* rename size (formerly last_pos) to mmr_size
* use 0-based positions in Segment's hash_pos and leaf_pos
* minimize use of saturating_sub(1) and rename some pos/idx to size
* use 0-based positions in methods get_output_pos
* use 0-based positions in method get_unspent_output_at
* use 0-based positions in method get_header_hash
* use 0-based positions in methods MerkleProof::verify{,_consume}
* use 0-based positions in method cleanup_subtree
* don't allow 0 in prunelist bitmap
* use 0-based positions in methods get_{,leaf_}shift
* rename some 1-based pos to pos1; identify TODO
* Address yeastplume's PR review comments
* initial commit of WIP pibd explorations
* correct calling for obtaining and validating first segment
* update test to properly iterate through each segment of the test pmmrs, validating each segment as it goes
* updated test to fully segment and validate PMMRs from compacted and uncompacted sample data. Also contains method of running test againt live chain data
* remove logger change
* change test file name
* change test file name
* change directory reference in test for CI
* remove +1 from segment count
* prediction comment change
* add ignore of full-chain test
* wip - "reset_head" via owner api functionality
* jsonrpc pass hash in as a string
* sort of works
* not a reorg if we simply accept several blocks at once
* remember to reset header MMR separately
as it is readonly when interacting with txhashset extension
* basic client integration
needs error handling etc.
* reset sync status when reset chain head
* track "denylist" (todo) and validate headers against this via the ctx
* track denylist (header hashes) in chain itself
* header denylist in play
* expose invalidateheader as client cmd
* rework reset_chain_head - rewind txhashset then header MMR
* track sync_head on header_sync sync status
follow header fork as appropriate
* track highest_diff alongside highest_height on HeaderSync status
* cleanup
* not not
* More efficient serialization for bitmap segments
* Rename a const
* Correctly count number of chunks in a segment
* Enum for BitmapBlock (de)ser mode
* Add more segments in test
* Fix duplicate function
* add segmenter for generating segments from txhashset with consistent rewind
* rework segmenter to take a txhashset wrapped in rwlock
rework our rewindable pmmr so we can convert to readonly easily
* placeholder code for rewinding readonly txhashset extension to build a rangeproof segment
* segment creation for outputs/rangeproofs/kernels/bitmaps
* placeholder segment impl
* commit
* rework segmenter to use a cached bitmap (rewind is expensive)
* cache segmenter instance based on current archive header
* integrate the real segment and segment identifier with our segmenter
* exercise the segmenter code on chain init
* wrap accumulator in an arc, no need to clone each time
* Introduce CommitOnly variant of Inputs.
Introduce CommitWrapper so we can sort commit only inputs correctly.
* rememebr to resort if converting
* write inputs based on variant and protocol version
* read and write protocol version specific inputs
* store full blocks in local db in v3
convert to v2 when relaying to v2 peers
* add debug version_str for inputs
* no assumptions about spent index sort order
* add additional version debug logs
* fix ser/deser tests for proto v3
* cleanup coinbase maturity
* rework pool to better handle v2 conversion robustly
* cleanup txpool add_to_pool
* fix nrd kernel test
* move init conversion earlier
* cleanup
* cleanup based on PR feedback
* refactor output to have internal output identifier
refactor to use AsRef for output identifier
make the output MMR explicit in terms of output identifiers
* put the serde deser back for rangeproof
* add json test for transactions
* make sure header PMMR is init correctly based on header_head from db
* fix to ensure header PMMR is consistent with header_head in db on chain init
* change order of operations - validate header earlier
and avoid applying header to header PMMR before validation
as this potentially leaves the PMMR in a tricky state to rewind from
* variants for output_pos linked list entries (head/tail/middle/unique)
next and prev and Vec<u8> lmdb keys
* get_pos on enum
* break list and list entries out into separate enums
* track output features in the new output_pos index, so we can determine coinbase maturity
* push entry impl for none and unique
* some test coverage for output_pos_list
* commit
* wip - FooListEntry
* use instance of the index
* linked list of output_pos and commit_pos both now supported
* linked_list
* cleanup and rename
* rename
* peek_pos
* push some, peek some, pop some
* cleanup
* commit pos
cleanup
* split list and entry out into separate db prefixes
* cleanup and add placeholder for pop_back
* pop_pos_back (for popping off the back of the linked list)
test coverage for pop_pos_back
* wip
* placeholder for prune via a trait
pos must always increase in the index
* rewind kernel_pos_idx when calling rewind_single_block
* RewindableListIndex with rewind support.
* test coverage for rewindable list index
* test coverage for rewind back to 0
* rewind past end of list
* add tests for kernel_pos_idx with multiple commits
* commit
* cleanup
* hook NRD relative lock height validation into block processing and tx validation
* cleanup
* set local chain type for kernel_idx tests
* add test coverage for NRD rules in block processing
* NRD test coverage and cleanup
* NRD relative height 1 test
* test coverage for NRD kernels in block processing
* cleanup
* start of test coverage for txpool NRD kernel rules
* wip
* rework pool tests to use real chain (was mock chain) to better reflect reality (tx/block validation rules etc.)
* cleanup
* cleanup pruneable trait for kernel pos index
* add clear() to kernel_pos idx and test coverage
* hook kernel_pos rebuild into node startup, compaction and fast sync
* verify full NRD history on fast sync
* return early if nrd disabled
* fix header sync issue
Currently Writable accepts trait Write as a type parameter but Readable
takes Read as a trait object, which is not symmetrical and also less performant. This PR changes Readable trait and all places where it's used
* Node API: don't error on missing output
* Propagate errors from get_output*
* Rename is_unspent and small refactor
* Forgot to rename function in tests
* Change Batch get_output_pos_height type signature
* no need to rehash with index to compare output with input spending it
* compare output identifier when checking is_unspent()
* output identifier from cleanup
* first pass at rewind_single_block
and reworking rewind to simply iterate over blocks, rewinding each incrementally
* commit
* commit
* cleanup
* add test coverage for output_pos index transactional semantics during rewind
* commit
* do not store commitments in spent_index
just use the order of the inputs in the block
* compare key with commitment when cleaning output_pos index
* remove unused OutputPos struct
ardocrat