* default to full chain validation on processing each block (except during sync)
* add sample config for chain_validation_mode
* document chain_validation_mode
* Initial Dandelion Commit
* Changed stem_tx_pool to tx_stempool
* Introduction of stem memory pool and stem pool config
* Pool push now send to stem memory pool
* Add stem transaction functions
* Add stem transaction pool
* Drastically simplified code structure
* Add monitor transactions
* Add Dandelion monitor and remove transactions from stempool
* Add peer relay monitor
* Reconcile block with stempool
* Fix total size bug
* Add fluff option for pool push
* Added details on dandelion monitor
* Fix issue with missing parent
* Child transaction with stempool parent are now forced stem
* Update Dandelion Relay from outgoing peers
* Fix missing pool reconciliation
* Add the ability to fluff a transaction directly
* Fix tests for Dandelion
* Missing send_stem_transaction method...
* Add fluff handler for wallet
* Add logger when successfully updated Dandelion relay
* Launch transaction monitor last
* Fix dandelion relay misplaced
* Add logging and updating for stempool
* Additionnal check for stem transaction
* Fix 2 Locks in a row
* move from storing per block kernel_offset in the block header
to storing the total accumulated kernel offset on each block header
* rustfmt
* cleanup
* wip
* failing test for being too eager when pruning a sibling
* commit
* rustfmt
* [WIP] modified get_shift and get_leaf_shift to account for leaving "pruned but not compacted" leaves in place
Note: this currently breaks check_compact as nothing else is aware of the modified behavior
* rustfmt
* commit
* rustfmt
* basic prune/compact/shift working
* rustfmt
* commit
* rustfmt
* next_pruned_idx working (I think)
* commit
* horizon test uncovered some subtle issues - wip
* rustfmt
* cleanup
* rustfmt
* commit
* cleanup
* cleanup
* commit
* rustfmt
* contains -> binary_search
* rustfmt
* no need for height==0 special case
* wip - works for single compact, 2nd one breaks the mmr hashes
* commit
* rustfmt
* fixed it (needs a lot of cleanup)
we were not traversing all the way up to the peak if we pruned an entire tree
so rm_log and prune list were inconsistent
* multiple compact steps are working
data file not being copmacted currently (still to investigate)
* cleanup store tests
* cleanup
* cleanup up debug
* rustfmt
* take kernel offsets into account when summing kernels and outputs for full txhashset validation
validate chain state pre and post compaction
* rustfmt
* fix wallet refresh (we need block height to be refreshed on non-coinbase outputs)
otherwise we cannot spend them...
* rustfmt
* family_branch() to recursively call family() up the branch
todo
- we hit a peak, then we need to get to the root somehow
- actually get the hashes to build the proof
* wip
* some additional testing around merkle tree branches
* track left/right branch for each sibling as we build the merkle path up
* MerkleProof and basic (incomplete) verify fn
* I think a MerkleProof verifies correctly now
need to test on test case with multiple peaks
* basic pmmr merkle proof working
* MerkleProof now serializable/deserializable
* coinbase maturity via merkle proof basically working
* ser/deser merkle proof into hex in api and wallet.dat
* cleanup
* wip - temporarily saving merkle proofs to the commit index
* assert merkle proof in store matches the rewound version
there are cases where it does not...
* commit
* commit
* can successfully rewind the output PMMR and generate a Merkle proof
need to fix the tests up now
and cleanup the code
and add docs for functions etc.
* core tests passing
* fixup chain tests using merkle proofs
* pool tests working with merkle proofs
* api tests working with merkle proof
* fix the broken comapct block hashing behavior
made nonce for short_ids explicit to help with this
* cleanup and comment as necessary
* cleanup variety of TODOs
Due to occasional bad network conditions, a node can miss a block.
When the next one is received and detected orphaned, we should at least try a request for the
previous block, assuming it's not already an orphan in itself and we're not syncing already.
No additinal tests were implemented, test like `simulate_full_sync` test this functionality because
sync starts later. It seems to be an issue with sync test coverage. As fix I'd suggest to start sync
as soon as we get a peer connection.
[https://github.com/mimblewimble/grin/issues/705]
* wip - basic tx pool lookup for kernel short ids (compact block hydration)
* use the nonce in the compact_block to correctly generate short_ids for lookup
* query the tx pool based on kernel short_ids
* tests passing
* cleanup some logging
* cleanup logging
* Clean server shutdown, generalizes usage of `Weak`. Introduces 2 main changes:
* A shared `AtomicBool` that signals server shutdown. All server
threads regularly check it to break out of their main loops when
it changes to `true`.
* Breaking of circular `Arc` references, which can never be
destroyed, by downgrading to `Weak` instead. Only the main server
keeps the `Arc` while all other components get the `Weak` variant.
Both of these are required for all long-living structs to be
cleanly destroyed. Note that in Rust this is fairly important as
most resource-freeing logic is associated with `drop`,
which is only called when said struct is free of scope or `Arc`
references.
Should address most of #536 (only need the stop hook to call
`Server` shutdown).
* Test for fast sync, followed by restart, followed by re-fast-sync
* P2P test fix
* Double sync taking too long for Travis, commenting out for now
* Util to zip and unzip directories
* First pass at sumtree request/response. Add message types, implement the exchange in the protocol, zip up the sumtree directory and stream the file over, with necessary adapter hooks.
* Implement the sumtree archive receive logicGets the sumtree archive data stream from the network and write it to a file. Unzip the file, place it at the right spot and reconstruct the sumtree data structure, rewinding where to the right spot.
* Sumtree hash structure validation
* Simplify sumtree backend buffering logic. The backend for a sumtree has to implement some in-memory buffering logic to provide a commit/rollback interface. The backend itself is an aggregate of 3 underlying storages (an append only file, a remove log and a skip list). The buffering was previously implemented both by the backend and some of the underlying storages. Now pushing back all buffering logic to the storages to keep the backend simpler.
* Add kernel append only store file to sumtrees. The chain sumtrees structure now also saves all kernels to a dedicated file. As that storage is implemented by the append only file wrapper, it's also rewind-aware.
* Full state validation. Checks that:
- MMRs are sane (hash and sum each node)
- Tree roots match the corresponding header
- Kernel signatures are valid
- Sum of all kernel excesses equals the sum of UTXO commitments
minus the supply
* Fast sync handoff to body sync. Once the fast-sync state is fully setup, get bacj in body sync
mode to get the full bodies of the last blocks we're missing.
* First fully working fast sync
* Facility in p2p conn to deal with attachments (raw binary after message).
* Re-introduced sumtree send and receive message handling using the above.
* Fixed test and finished updating all required db state after sumtree validation.
* Massaged a little bit the pipeline orphan check to still work after the new sumtrees have been setup.
* Various cleanup. Consolidated fast sync and full sync into a single function as they're very similar. Proper conditions to trigger a sumtree request and some checks on receiving it.
* we can now hydrate empty compact blocks and process them
* add some tests to verify size of
various serialized blocks and compact blocks
* add_test_hydrate_empty_block
* fix broken test
* first pass at allow/deny lists for hard-coded peers (not just seeds)
* commit
* add peers_allow and peers_deny examples and comments to grin.toml
* always ask for compact block
always fail to hydrate
always fallback to requesting full block
* decide to send full (empty) block over compact (empty) block
* add some randomness to the decision around broadcasting an empty block
as a block or as a compact block (so we can exercise more code paths easily)
* [wip] header first propagation
successfully propagating headers (unless we mined the block itself)
not yet asking for the block if we receive a header
* call request_block after successful processing header
* cleanup and skip asking for block if header is an orphan
* comments around error handling in receive_header
* experiment with lock_heights on outputs
* playing around with lock_height as part of the switch commitment hash
* cleanup
* include features in the switch commit hash key
* commit
* rebase off master
* commit
* cleanup
* missing docs
* rework coinbase maturity test to build valid tx
* pool and chain tests passing (inputs have switch commitments)
* commit
* cleanup
* check inputs spending coinbase outputs have valid lock_heights
* wip - got it building (tests still failing)
* use zero key for non coinbase switch commit hash
* fees and height wrong order...
* send output lock_height over to wallet via api
* no more header by height index
workaround this for wallet refresh and wallet restore
* refresh heights for unspent wallet outputs where missing
* TODO - might be slow?
* simplify - do not pass around lock_height for non coinbase outputs
* commit
* fix tests after merge
* build input vs coinbase_input
switch commit hash key encodes lock_height
cleanup output by commit index (currently broken...)
* is_unspent and get_unspent cleanup - we have no outputs, only switch_commit_hashes
* separate concept of utxo vs output in the api
utxos come from the sumtrees (and only the sumtrees, limited info)
outputs come from blocks (and we need to look them up via block height)
* cleanup
* better api support for block outputs with range proofs
* basic wallet operations appear to work
restore is not working fully
refresh refreshes heights correctly (at least appears to)
* wallet refresh and wallet restore appear to be working now
* fix core tests
* fix some mine_simple_chain tests
* fixup chain tests
* rework so pool tests pass
* wallet restore now safely habndles duplicate commitments (reused wallet keys)
for coinbase outputs where lock_height is _very_ important
* wip
* validate_coinbase_maturity
got things building
tests are failing
* lite vs full versions of is_unspent
* builds and working locally
zero-conf - what to do here?
* handle zero-conf edge case (use latest block)
* introduce OutputIdentifier, avoid leaking SumCommit everywhere
* fix the bad merge
* pool verifies coinbase maturity via is_matured
this uses sumtree in a consistent way
* cleanup
* add docs, cleanup build warnings
* fix core tests
* fix chain tests
* fix pool tests
* cleanup debug logging that we no longer need
* make out_block optional on an input (only care about it for spending coinbase outputs)
* cleanup
* bump the build
Moved handling to the peer map out of the p2p server and into
its own struct. Allowed factoring code from the net adapter and
simplification of some interactions. Also removes the need for
the adapter to reference the p2p server or peers.
Fixes#430, #453 and #456
When generating the locator, we just looked for the first block
in the locator we had locally. However that block could be on a
losing fork, in which case we replied with blocks from that losing
fork, making things worse. Now building the locator only based on
blocks in our winning fork.
* Very quick peer banning endpoint, helps with #406
* Ping heights (#407)
* add height to ping/ping
* reformat output
* fix p2p test
* Fix orphan handling, not related to current head. Fixes#412
* Check before borrow, fixes#267
* Not finding an output commit in pos is an AlreadySpent
* Fix race condition, sending before conn is ready
* Explicit error for unknown pos of a forked block
* Remove config outdated tests. Fix#333
* Check ref and try before borrow, fix#400
* We do not want to sync with old peers anyway
* Hide cargo compiler warning for unused NoopAdapter and unused test code. Add TODOs
* introduce cpu_pool to read_msg
* cleanup and less unwrap() in handler
* expose cpu_pool as part of net_adapter (reuse across all peer connections and handlers)
* fix DummyAdaptor for test usage
* add new() to DummyAdapter for convenience
* simplify sync process further - one thread, try_read on peer for robustness
* add most_work_peers() for convenience, add most_work_peers count to "monitoring peers" log msg
Added support for peer banning on the p2p server. The peer status
is changed and the peer is disconnected. A banned peer won't be
able to reconnect as well.
Tracking of chain errors due to a block that's intrinsically bad
and banning of the peer that sent it. If we're syncing, resetting
the header chain to the same as the main chain to force
backtracking.
* port simple_sync across from testnet1, sync head & improved fork handling
* introduce new sync_head for tracking header chain during sync
* add debug logging in body_sync for state of various heads
* update header_head during sync if header is now the one with most work, pass in sync and head contexts to pipe
* port across sync_head changes from master, put the 1s sleep back in...
* let sync run even if the full 512 headers are all known, give sync a chance to sync against a fork that forked a long time ago
* handle height jumping during a reorg
block validaton should not check for height+1 based on head
no assumptions should be made about height indices
* quick(er) check for previous block in process_block (we cannot check height against ctx head)
* make body_sync a _lot_ faster by finding the forked block more efficiently...
* fix monitoring peers log msg
* fix chain tests
* fix grin tests - we were using the wrong genesis hash (wrong chain type)
* apparently needs setting in both places...
* body -> header -> sync (#414)
* port over body -> header -> sync changes from testnet1
* port across "simple_sync" changes from testnet1 branch
Cleanup direct refs to peer map or peer store
P2P server acts as a facade, handling the list of connected peers
and the storage of their information. Everything else goes through
the p2p server instead of having a peer map reference or going
straight to the store.
Fix p2p tests
* fix "monitoring peers" log msg (use connected_peers)
And error in the sync run loop would just exit it silently. Abort
with a panic instead.
Peer errors (like disconnect) on locator request was bubling up in
the loop. Just logging instead.
* add total_diff to ping/pong msgs
debug log for total_diff on each ping/pong
* expose peer addr to the handle_payload fn
so we know where it came from
* fix p2p tests for ping
* default to 0 if we cannot read total_difficulty
* updating a connected peer in place
* actually update peer info diff
* fixup p2p tests
Fixes#281 by doing:
- "Starting validation pipeline for " -> "Processing "
- stdout logging default = Warning
- ERR -> INFO for "Transaction rejected: Already in pool" -- only for netadapter transaction_received, which (hopefully!) only gets tx from peers, and lots of those will be duplicates, that's good and not an error.
* Downgrade ERR even down to DEBUG.
Also let Transaction rejected show tx hash,
so users can check if such a rejection is for the
payment they expected, or just any random peer
sending in some old and already known tx.
Renamed mining parameter mode to chain type, with existing types
of CI testing, user testing, testnet1 or mainnet. The public
chains (testnet1 and mainnet) come with their fully pre-defined
genesis block.
Still need to set the nonce and cycle for testnet1 genesis.
* bad case of trying to rewind to block header height 0
* rewind_to_genesis appears to work
* do not assume genesis block at height 0 is empty, pass full block in to rewind, check for last output and kernel, use index 0 if block is empty
* use HashMap internally for tracking connected peers (to avoid duplicates)
* reuse the handshake on the server so we can track our own nonce to avoid self connections correctly
* make sure we start up the clean_peers loop even in seedless mode
* logging in monitoring peers loop
* simplify monitoring for no seeds
* fixup and cleanup simulnet tests (real seeds in places)
* only start the sync if we have either seeds or peers that we know about, exit syncer safely if we have no connected peers
Moved the HTTP APIs away from the REST endpoint abstraction and
to simpler Hyper handlers. Re-established all routes as v1.
Changed wallet receiver port to 13415 to avoid a gap in port
numbers.
Finally, rustfmt seems to have ignored specific files arguments,
running on everything.
* Fix for the chain pipeline partly relying on an outdated head,
leading to not properly recognizing a fork and inconsistent sum
tree state.
* Do not drop block requests during sync that don't get satisfied,
retry enough time to get them and avoid stall.
* Always validate header, even in sync where we may have validated
it already. We don't want a block coming from a peer that could
squeeze through with an invalid header.
* When syncing, do not mark blocks that were errored by the chain
as received (typical case: orphan). Keep retrying.
* Improved chain state dump for debugging.
* Do not add to orphans blocks too far in the future.
* Better error reporting on db errors.
* Related sync test fixes.
TODO figure out why syncing peers timeout so often, very useful
to test but not that great for a fast sync experience.
Ignotus Peverell