* [PIBD_IMPL] Introduce PIBD state into sync workflow (#3685)
* experimental addition of pibd download state for testnet only
* fixes to bitmap number of segments calculation + conversion of bitmap accumulator to bitmap
* attempt to call a test message
* add p2p methods for receiving bitmap segment and applying to desegmenter associated with chain
* fixes to state sync
* add pibd receive messages to network, and basic calls to desegmenter from each (#3686)
* [PIBD_IMPL] PIBD Desegmenter State (#3688)
* add functions to desegmenter to report next desired segments, begin to add state to determine which segments have been requested
* add segmentidentifier type to id requested segments uniquely
* make a call on where to keep track of which PIBD segments have been requested
* move segmenttype definition, add functions to manipulate peer segment list
* remove desegmenter state enum
* change chain desegmenter function to provide rwlock
* trace, warning cleanup
* udpate to test compliation
* [PIBD_IMPL] Bitmap accumulator reconstruction + TxHashset set reconstruction (#3689)
* application of received bitmap segments to local accumulator
* add all required elements to send/receive output segment requests and responses
* testing of output sync
* add special cases to pmmr segment request
* [PIBD_IMPL] PMMR Reassembly from Segments (#3690)
* update pibd copy test to use new desgmenter structure
* begin reconstruction of output pmmr
* clean up hash/leaf insertion logic
* push pruned subtree appears to be working, now also calculates left hand hashes correctly
* factor out ordering of segment/hash order array
* refactor for pmmr application code
* test of chain copy appears to be working
* add rangeproof functions to desegmenter
* add kernel functions, attempt refactor
* small test cleanup, reconstruction of live chain working in manual copy test
* [PIBD_IMPL] PIBD tree sync via network and kill/resume functionality (#3691)
* add functions to determing latest verifiable block height for the given pibd state
* attempting to allow for pibd to resume after killing process
* fix to ensure prune list is properly flushed during pibd sync
* removal of unneeded code
* ignore test for now (fix before full merge)
* [PIBD_IMPL] Finalize PIBD download and move state to chain validation (#3692)
* investigations as to why a slight rewind is needed on startup during PIBD
* move validation code into desegmenter validation thread (for now)
* ensure genesis entries in pmmrs are removed if they're removed in the first segment
* validation all working except for verifying kernel sums
* remove unneeded pmmr rollbacks on resume now root cause was found
* updates to remove unpruned leaves from leaf set when rebuilding pmmr
* remove + 1 to segment traversal iter length
* [PIBD_IMPL] PIBD Stats + Retry on validation errors (#3694)
* start to add stats and reset chain state after errors detected
* add functions to reset prune list when resetting chain pibd state
* debug statement
* remove test function
* [PIBD_IMPL] Update number of simultaneous peer requests for segments (#3696)
* cleanup of segment request list
* allow for more simultaneous requests during state sync
* up number of simultaneous peer requests for segments
* [PIBD_IMPL] Thread simplification + More TUI Updates + Stop State Propagation (#3698)
* change pibd stat display to show progress as a percentage of downloaded leaves
* attempt some inline rp validation
* propagate shutdown state through kernel validation
* change validation loop timing
* simplify validator threading
* add more detailed tracking of kernel history validation to tui, allow stop state during
* adding more stop state + tui progress indication
* remove progressive validate
* test fix
* revert to previous method of applying segments (#3699)
* fix for deadlock issue (#3700)
* update Cargo.lock for next release
* [PIBD_IMPL] Catch-Up functionality + Fixes based on testing (#3702)
* ensure desegmenter attempts to apply correct block after a resumte
* ensure txhashset's committed implementation takes into account output bitmap for summing purposes
* remove check to de-apply outputs during segment application
* return removal of spent outputs during pibd
* remove unneeded status
* remove uneeded change to rewind function
* documentation updates + todo fixes (#3703)
* add pibd abort timeout case (#3704)
* [PIBD_IMPL] BitmapAccumulator Serialization Fix (#3705)
* fix for writing / calculating incorrect length for negative indices
* update capabilities with new version of PIBD hist
* remove incorrect comment
* fix capabilities flag, trace output
* test fix
* Merge DNSSeed scope changes into pibd impl branch (#3708)
* update Cargo.lock for next release
* visibility scope tweaks to aid seed test utilities (#3707)
* move all PIBD-related constants into pibd_params modules (#3711)
* remove potential double read lock during compaction
* Minor update for foreign api docs. Peers seems to be no longer needed as an arg in the grin_api::Foreign::new.
* This changes foreign_rpc and owner_rpc modules to public so the helper modules for the rpc clients can be accessible in external projects.
* Updated test seeds.
* Added functions that override the one time globals. This is needed for the grin-gui so it can change the global contexts when it needs to switch from testnet to mainnet.
* Refactor
* Fix panick at 'attempt to subtract with overflow' when grin-wallet sends in a start_index of 0 for new wallets.
* Fix overflow panic using saturating_sub(1).
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* replace bitvec with more efficient bitpack algorithm
* optimise proof_unpack_len
* move proof pack length calculation
* small refactor
* first pass attempt at not deserializing proof nonces in difficulty iter
* another 10 seconds gained by not deserialising the proof from the difficulty iterator
* add new deser parameters to tests where needed
* add skip_proof variants to store
* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
* add FeeFields type
* use FeeFields with ::zero and try_into().unwrap()
* fixed tests
* avoid 0 accept_base_fee
* add aggregate_fee_fields method for transaction
* implement std::fmt::Display trait for FeeFields
* make base_fee argument non-optional in libtx::mod::tx_fee
* add global and thread local accept_fee_base; use to simplify tests
* set unusually high fee base for a change
* revert to optional base fee argument; default coming from either grin-{server,wallet}.toml
* remove optional base fee argument; can be set with global::set_local_accept_fee_base instead
* define constant global::DEFAULT_ACCEPT_FEE_BASE and set global value
* add Transaction::accept_fee() method and use
* Manual (de)ser impl on FeeFields
* fix comment bug
* Serialize FeeFields as int in tx
* allow feefields: u32:into() for tests
* try adding height args everywhere
* make FeeFields shift/fee methods height dependent
* prior to hf4 feefield testing
* rename selected fee_fields back to fee for serialization compatibility
* fix test_fee_fields test, merge conflict, and doctest use of obsolete fee_fields
* make accept_fee height dependent
* Accept any u64 in FeeFields deser
Co-authored-by: Jasper van der Maarel <j@sper.dev>
* make min_wtema difficulty chain type dependent
* fix bad graph weights for automated and user testing
* only mainnet can get wtema difficulty unstuck at 59s blocks
* HardFork4 initial commit
* HF4 obsoletes secondary_scale, which becomes an nonce extension 8->12 bytes
* fix TESTNET_FOURTH_HARD_FORK height
* improve predicted height for testnet fork on dec 8
* move local variable as suggested by antiochp
* Introduce CommitOnly variant of Inputs.
Introduce CommitWrapper so we can sort commit only inputs correctly.
* rememebr to resort if converting
* write inputs based on variant and protocol version
* read and write protocol version specific inputs
* store full blocks in local db in v3
convert to v2 when relaying to v2 peers
* add debug version_str for inputs
* no assumptions about spent index sort order
* add additional version debug logs
* fix ser/deser tests for proto v3
* cleanup coinbase maturity
* rework pool to better handle v2 conversion robustly
* cleanup txpool add_to_pool
* fix nrd kernel test
* move init conversion earlier
* cleanup
* cleanup based on PR feedback
* use mmr sizes in header to validate block weight given header only
* add global weight validaton to headers when reading them untrusted off the network
* fixup tests, one pending test
* add test for deserializing untrusted block header exceeeding weight limits
* validate header mmr sizes, must grow for each block
* remove redundant height check
* fix tests after rebase
block specific TooHeavy error
* wip
* commit
* add block level validation rule around NRD kernels and HF3 block height
* wip
* test coverage for kernel ser/deser for NRD kernels
* add some type safety around ser/deser of nrd relative height
* cleanup
* cleanup tx sig handling and add tests around NRD kernel sig
* test coverage for chain apply block containing NRD kernel
* verify kernel variants when adding tx to pool
NRD kernels will not be accepted until HF3
* add txpool test for NRD kernel handling
* fix merge
* fix api for NRD kernels
* commit
* wip - feature flag for NRD kernel support
* wip
* global config and thread local feature flag for NRD kernel support
* add feature flag support for NRD kernels
default to false when starting up node
allow it to be set to true in context of individual tests
* feature flag
* explicit testing of NRD kernel via feature flag
* add chain_type and feature flag logging on startup
* PR feedback and cleanup
* PR feedback
* Introduce GLOBAL_CHAIN_TYPE and make CHAIN_TYPE thread_local.
This makes testing more explicit and significantly more robust.
* set_local_chain_type() in tests
* cleanup - weird
* get pool tests working with explicit local chain_type config
* core tests working with explicit local chain_type
* p2p tests working with explicit local chain_type
* store tests working
* cleanup, feedback
* add 2nd HF and cuckaroom
* add cuckaroom and hardfork tests
* remove all traces of later phaseouts
* refactor header_version from valid_header_version
We use crate renaming to remove grin prefix, as result imports for util and keychain crates start with crate::
which looks like a part of the current crate. This PR fixes it.
Also some small improvements were made.
If this approach works I'll replicate it to other crates.
* wip
* exhaustive match
* write with fixed v1 strategy when writing for hashing
* local protocol version is 2
* cleanup "size" tests that exercise v1 vs v2 vs default protocol versions
* add proto version to Connected! log msg
* cleanup docs
* negotiate protocol version min(local, peer) when doing hand/shake
* Optimize POW read
This functionality is used to deserialize header (from network or from
DB), it was taking up to 40% cpu time during initial header sync. This
PR brings it down to 3-4%. Function read_number would look better as
closure, unfortunately the compliler doesn't inline it in this case, so
it would be 2x slower.
* Remove unused code
* introduce protocol version to deserialize and read
* thread protocol version through our reader
* cleanup
* cleanup
* streaming_reader cleanup
* Pass protocol version into BinWriter to allow for version specific serialization rules.
* rustfmt
* read and write now protocol version specific
* allow version 2 blocks for next 6 months
* add cuckarood.rs with working tests
* switch cuckaroo to cuckarood at right heights
* reorder to reduce conditions
* remove _ prefix on used args; fix typo
* Make Valid Header Version dependant on ChainType
* Rustfmt
* Add tests, uncomment header v2
* Rustfmt
* Add FLOONET_FIRST_HARD_FORK height and simplify logic
* assume floonet stays closer to avg 60s block time
* move floonet hf forward by half a day
* update version in new block when previous no longer valid
* my next commit:-)
* micro optimization
* rework bucket txs
* introduce max_block_weight to global (based on chain_type)
not yet in use anywhere...
* now using global::max_block_weight() everywhere
* testing max_block_weight needs to be big enough to cover existing chain tests...
* add some test coverage for prepare mineable transactions at the block weight limit
introduce AsLimitedTx{max_weight} so we can build artifically small blocks (per mining config)
* cleanup
* default_mineable_max_weight is just the default max_block_weight
we do not need to account for coinbase reward here (tx vs block)
* 75 change outputs in a test is not valid now that we have a low block weight limit...
* Fix secondary scaling bugs; rename is_testnet -> is_floonet (#2215)
* add global::is_mainnet()
* use it to change pre-genesis pow type
* rename is_testnet -> is_floonet
* Support multiple chain configurations (#2217)
* Support multiple chain configurations
Supports generating the proper configuration for each chain type
(mainnet, floonet, usernet). Will run them by default under
their respective root directory (~/.grin/main, ~/.grin/floo, etc).
Assigned default ports for mainnet, overriding them to keep Floonet
ports unchanged.
For now, starting on mainnet will abort.
* Fixed usernet command line help message. Fixes#2217
* Differing magic numbers for each chain type (#2208)
* stick to e=H(R|P|m) when use schnorr signature (#2200)
* stick to e=H(R|P|m) when use schnorr signature
* (1)add verify_slate_messages for wallet receive (2)log the message content
* remove debug log on verify_slate_messages
* verify the sender's message signature when receive_tx in wallet listen
* Revert "remove debug log on verify_slate_messages"
This reverts commit 65ea32a407bfd57d02bf169803f1483ba611962e.
* Revert "rustfmt"
This reverts commit c380ab91856344b73595bb04eef1fc087dedd84d.
* Revert "(1)add verify_slate_messages for wallet receive (2)log the message content"
This reverts commit 9584ca7a893b22a768dea061039140033c07e8eb.
* [re-floonet] Keychain Floonet BIP32 version/network option (#2235)
* add 'is_floonet' property to keychain
* fix hex encoding and tests
* Fix couple floonet loose ends (#2230)
* Fix couple floonet loose ends. Fixes#2216
* Doc fix for sig message
* Refuse unkown kernel features (#2244)
* Minor: magic number change for re-floonet
* Set pre genesis is_secondary to true (#2247)
* Minor: tx validation error display underlying
* New floonet genesis
* genesis rustfmt
* Use chain-specific config for wallet toml gen
* Fix default wallet_listener_url
* New more reasonable genesis block, bumped version
* genesis rustfmt
* Couple minor fixes to genesis generation script
* replace is_mainnet() with !is_testnet()
makes testing "mainnet" code significantly easier (its the default)
* fix next_target_adjustment test based on tromp analysis
* rustfmt
* cleanup wallet db files and add to gitignore
* fix
* Cuckatoo size shift upgrade schedule
* Move the schedule into graph_weight instead of messing with min edge bits
* Cleanup and fixes now that we have an agreed upon schedule
* PoW context is now properly picked depending on the chain type,
edge bits and block height. Height const for T4 hard fork leaving
a couple weeks to have miners in place. Removed now unused Cuckoo context.
* Simplified block siphash
* Fix servers crate compilation
* Tiny bit cleaner block siphash. Maybe.
* Cuckatoo min edge bits update for T4 and mainnet
* Fix header size tests, Cuckatoo31 default means one more bit per edge
* Remove redundant param from verify_size
* split horizon into two explicit values for cut through and txhashset request
* let node which has 2-7 days of history be able to handle forks larger than 2 days
* add test simulate_long_fork
* add pause/resume feature on p2p for tests
* refactor the state_sync
* ignore the test case simulate_long_fork for normal Travis-CI
* refactor function check_txhashset_needed to be shared with body_sync
* fix: state TxHashsetDone should allow header sync
Yeastplume