Commit Graph

101 Commits

Author SHA1 Message Date
Yeastplume 63c65605bb [SYNC PERFORMANCE] Adjust DifficultyIterator to no longer deserialize PoW proof nonces (#3671)
* replace bitvec with more efficient bitpack algorithm

* optimise proof_unpack_len

* move proof pack length calculation

* small refactor

* first pass attempt at not deserializing proof nonces in difficulty iter

* another 10 seconds gained by not deserialising the proof from the difficulty iterator

* add new deser parameters to tests where needed

* add skip_proof variants to store

* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo

* replace bitvec with more efficient bitpack algorithm

* optimise proof_unpack_len

* move proof pack length calculation

* small refactor

* first pass attempt at not deserializing proof nonces in difficulty iter

* another 10 seconds gained by not deserialising the proof from the difficulty iterator

* add new deser parameters to tests where needed

* add skip_proof variants to store

* remove hash from difficulty iterator struct, rename HeaderInfo to HeaderDifficultyInfo
2021-12-10 11:39:39 +00:00
Quentin Le Sceller 0259ed23ea Update copyright year to 2021 (#3592)
* Update copyright year to 2021
2021-03-10 10:14:48 -05:00
Quentin Le Sceller e023739658 Update and do not use serve private export (#3585)
* Update serde, serde_derive, syn, failure and failure_derive

* Do not use serde private export
2021-03-06 15:25:26 -05:00
jaspervdm defc714f6e Refactor p2p reader (#3433)
* Refactor p2p reader

* Rename Output enum

* Consume enum takes owned values instead of references

* Deserialization in codec, remove Consume enum

* Calculate block header size

* Read headers in batches

* Remove headers type from deserializer
2020-09-28 14:43:37 +01:00
Antioch Peverell 7dc94576bd Introduce CommitOnly variant of Inputs (#3419)
* Introduce CommitOnly variant of Inputs.
Introduce CommitWrapper so we can sort commit only inputs correctly.

* rememebr to resort if converting

* write inputs based on variant and protocol version

* read and write protocol version specific inputs

* store full blocks in local db in v3
convert to v2 when relaying to v2 peers

* add debug version_str for inputs

* no assumptions about spent index sort order

* add additional version debug logs

* fix ser/deser tests for proto v3

* cleanup coinbase maturity

* rework pool to better handle v2 conversion robustly

* cleanup txpool add_to_pool

* fix nrd kernel test

* move init conversion earlier

* cleanup

* cleanup based on PR feedback
2020-09-07 16:58:41 +01:00
Antioch Peverell 133089e985 Refactor Output Identifiers (#3371)
* refactor output to have internal output identifier

refactor to use AsRef for output identifier

make the output MMR explicit in terms of output identifiers

* put the serde deser back for rangeproof

* add json test for transactions
2020-09-03 10:59:54 +01:00
Antioch Peverell 988a05f023 Enable NRD kernel support (noop) feature flagged (#3303)
* wip

* commit

* add block level validation rule around NRD kernels and HF3 block height

* wip

* test coverage for kernel ser/deser for NRD kernels

* add some type safety around ser/deser of nrd relative height

* cleanup

* cleanup tx sig handling and add tests around NRD kernel sig

* test coverage for chain apply block containing NRD kernel

* verify kernel variants when adding tx to pool
NRD kernels will not be accepted until HF3

* add txpool test for NRD kernel handling

* fix merge

* fix api for NRD kernels

* commit

* wip - feature flag for NRD kernel support

* wip

* global config and thread local feature flag for NRD kernel support

* add feature flag support for NRD kernels
default to false when starting up node
allow it to be set to true in context of individual tests

* feature flag

* explicit testing of NRD kernel via feature flag

* add chain_type and feature flag logging on startup

* PR feedback and cleanup

* PR feedback
2020-05-29 09:56:24 +01:00
hashmap a82041d0ed Refactor Readable trait (#3309)
Currently Writable accepts trait Write as a type parameter but Readable
takes Read as a trait object, which is not symmetrical and also less performant. This PR changes Readable trait and all places where it's used
2020-04-30 17:42:19 +02:00
Quentin Le Sceller 04a0123752 Less cloning and additional pattern simplifications (#3223)
* API Cleanup

* Chain Cleanup

* Core Cleanup

* Keychain Cleanup

* P2P Cleanup

* Pool Cleanup

* Store Cleanup

* Util Cleanup

* Cleanup clone_from_slice

* Address jasper comments
2020-02-12 19:35:33 +01:00
Quentin Le Sceller c4e69717ab Less cloning and pattern simplifications (#3216)
* Less cloning and pattern simplifications

* Revert inclusive range and remove unecessary Error:From
2020-02-05 11:02:07 -05:00
Antioch Peverell 5e1fe44bed remove writeable impl [u8; 4] (#3132) 2020-01-29 14:56:06 +00:00
Antioch Peverell 616dad43fd Cleanup redundant AsFixedBytes and FixedLength traits (#3131)
* no need for AsFixedBytes we can just use AsRef<[u8]>

* cleanup FixedLength trait

* revert this change for now

* fix store tests

* cleanup and fix tests after rebase

* fix tests

* update based on PR review
less hard-coded values now

* cleanup
2020-01-29 13:41:50 +00:00
Quentin Le Sceller 6e5afe496b Update License to 2020 (#3196) 2020-01-20 11:40:58 +00:00
hashmap ba9cdd323e Revert "Remove error serialization/deserialization (#3110)" (#3151)
This reverts commit 76ac9419b8.
2019-11-29 12:37:50 +00:00
hashmap 43bd5a56ff Make imports in core crate more Rust 2018 idiomatic (#3112)
We use crate renaming to remove grin prefix, as  result imports for util and keychain crates start with crate::
which looks like a part of the current crate. This PR fixes it.
Also some small improvements were made.
If this approach works I'll replicate it to other crates.
2019-11-14 10:27:30 -05:00
hashmap 76ac9419b8 Remove error serialization/deserialization (#3110)
it's not used
2019-11-13 21:08:20 +01:00
Quentin Le Sceller 6be6391225 Change 2018 to 2019 in copyright files (#3072) 2019-10-02 09:40:20 +01:00
Antioch Peverell bc6108cf12 Kernels v2 (variable size) (#3034)
* wip

* exhaustive match

* write with fixed v1 strategy when writing for hashing

* local protocol version is 2

* cleanup "size" tests that exercise v1 vs v2 vs default protocol versions

* add proto version to Connected! log msg

* cleanup docs

* negotiate protocol version min(local, peer) when doing hand/shake
2019-09-19 14:31:46 +01:00
Antioch Peverell 6036b671e8 fee and lock_height maintained in kernel features variants (#2859)
* fee and lock_height now maintained in kernel features enum variants

* sum fees via explicit enum variants

* cleanup semantics around with_fee and with_lock_height for tx builders

* document the kernel feature variants

* serialize kernel features correctly for api json

* cleanup

* commit

* bump to unstick azure
2019-08-19 14:28:02 +01:00
hashmap d3dbafa80b [2.x.x] Use blocking IO in P2P to reduce CPU load (#2855)
* Use blocking IO in P2P to reduce CPU load
2019-07-12 23:17:38 +02:00
Antioch Peverell d284d8f6de [2.x.x] Writeable protocol version aware (#2856)
* introduce protocol version to deserialize and read

* thread protocol version through our reader

* cleanup

* cleanup

* streaming_reader cleanup

* Pass protocol version into BinWriter to allow for version specific serialization rules.

* rustfmt

* read and write now protocol version specific
2019-07-06 15:51:03 +01:00
Antioch Peverell 5aaf2d058d [2.x.x] deserialization now protocol version aware (#2824)
* introduce protocol version to deserialize and read

* thread protocol version through our reader

* example protocol version access in kernel read

* fix our StreamingReader impl (WouldBlock woes)

* debug log progress of txhashset download
2019-06-27 17:19:41 +01:00
Mike Dallas 4b88a9b5bd impl Readable/Writable for PublicKey (#2841)
* impl Readable/Writable for PublicKey

* rustfmt
2019-05-27 10:19:24 +01:00
Antioch Peverell 4ba7b0a46a make sure we use BigEndian when reading numbers via StreamingReader (#2821) 2019-05-13 15:04:14 -07:00
Antioch Peverell fabff51dd3 Header version cleanup (#2809)
* introduce HeaderVersion (was u16) for type safety
cleanup pow ser/deser (version unused)

* fixup tests for HeaderVersion

* validate header version during header deserialization
2019-05-08 21:10:42 +01:00
Antioch Peverell 3d817f6cd0 [1.1.0] Support variable size PMMR (optional size_file) (#2734)
* Introduce optional size_file.
Support fixed size data file via an optional elmt_size.
Support variable size data file via optional size_file.

* remember to release the size_file

* fix scoping for windows support
2019-04-12 15:41:21 -07:00
Andrew Dirksen f30e59adc5 make error types serializable (#2659) 2019-03-07 19:08:29 +00:00
Jeremy Rubin 99494c6fa6 get rid of default implementation of Hashed trait for writable things (#2573) 2019-02-15 13:41:19 +00:00
hashmap aedac483f5 Convert to Rust 2018 edition (#2084)
* Convert to Rust 2018 edition

* Update gitignore
2018-12-08 00:59:40 +01:00
Ignotus Peverell 54a034c40a Fix txpool double spend edge case (#2081)
Sorting validation on inputs, outputs and kernels now also checks for uniqueness.
2018-12-05 09:44:32 -08:00
Antioch Peverell 8d8f533b8e store pmmr cleanup (#2075)
* introduce DataFile<T> to hide the LEN handling
use slices rather than vec<u8> where possible

* cleanup

* use DataFile<Hash> and not HashFile

* fix core tests
2018-12-05 13:02:24 +00:00
antiochp fd41b4918e rustfmt 2018-12-05 10:28:21 +00:00
antiochp bd771ecbc5 fix txpool double spend edge case 2018-12-05 10:27:46 +00:00
Antioch Peverell 8bad93c8c9 Reduce amount of clone() going on when applying data to MMRs (#2046)
* less cloning when pushing to PMMR

* pass a ref to data when applying to the MMR and appending to the backend

* cleanup

* rustfmt

* fixup tests
2018-12-02 19:59:24 +00:00
Antioch Peverell 79d540cf86 Header MMR Refactor (#2028)
* refactor header MMR to use regular backend
this lets us store header hashes in the mmr data file
the hashes in the hash file are "hash with index"

* rustfmt

* get_header_hash(pos) on header extension

* rustfmt

* cleanup

* cleanup

* comments

* cleanup

* fixup testelem so tests work
2018-11-27 12:32:39 +00:00
Antioch Peverell f0fa410273 Streaming headers (#1989)
* headers msg is now "streamed" off the tcp stream

* rustfmt

* cleanup

* move StreamingReader into ser.rs
extract read_exact out into util crate

* rustfmt
2018-11-16 11:00:39 +00:00
Antioch Peverell 2e421191e2 implement read_multi with an internal iterating reader (#1981)
* rework read_multi to use an "iterating reader" internally
we are going to use this for reading headers in batches

* rustfmt
2018-11-14 14:52:43 +00:00
Antioch Peverell c631b45ab6 cleanup how we deal with flexible rproof size in ser/deser (#1965) 2018-11-13 09:30:40 +00:00
Antioch Peverell a3c1c6d603 Do not use mem::size_of() when calculating serialized size of data (#1967)
* use FixedLength to define serialized size in bytes of various structs
replace usages of mem::size_of() with ::LEN so we correctly calculate serialized sizes in bytes

* rustfmt
2018-11-13 09:30:02 +00:00
Antioch Peverell d23dec73d0 PMMRable cleanup (#1910)
* cleanup pmmrable and len()
introduce FixedLength trait with a const LEN
make Hash impl FixedLength for consistency

* rustfmt

* store tests cleanup

* rustfmt

* whats going on with those comments and rustfmt?
2018-11-01 20:14:46 +00:00
hashmap 3efa7bdac9 Small style changes in core crate (#1816) 2018-10-23 21:09:16 +02:00
Ignotus Peverell 9cc7aed713 Dual proof of work, first step (#1534)
* Move Proof and Difficulty types to pow module. Difficulty scaling calculation.
* Diffculty scaling and size shift for 2nd PoW
* Backport e7eb26ee
* Test compilation fixes
* Scaling only need to go one way, as @tromp pointed out
2018-09-18 15:12:57 -07:00
Ignotus Peverell a83404b22e Simplify mining header serialization (#1509) 2018-09-11 20:31:05 -07:00
Antioch Peverell 7dfca6077c Refactor compact block body (#1394)
* introduce CompactBlockBody

* rustfmt

* implement From<Block> for CompactBlock

* rustfmt

* remove debug logging

* wip

* rustfmt
2018-08-22 20:19:37 +01:00
Yeastplume d2a84b7600 Error handling improvements (particularly in chain) (#1208)
* update error handling in chain and other modules to use error/errorkind

* sizeshift errorkind
2018-06-30 23:36:38 +01:00
Ignotus Peverell 0967a5302b Kernel sum and MMR sizes in block header (#1163)
* Add kernel commitments sum and kernel and output MMR sizes to block header
* Sum a block without including previous sums, cleanup. Blocks are now summed and validated based on their own totals and not the totals since genesis. This allows to get rid of BlockSum and simplified the setting of a new block's roots, kernel sum and MMR sizes. Fixes #116
* Additional kernel MMR validation to check all prior header roots successively
* Wallet tests fix
2018-06-21 02:30:22 +01:00
Quentin Le Sceller a30ee88236 Cleanup imports (#1161)
Reorganize imports
2018-06-14 08:16:14 -04:00
Quentin Le Sceller 663733e72b Replace try! for ? (#1134)
* Replace try for ?
2018-06-04 12:05:49 -04:00
Quentin Le Sceller ec3c2e89a4 Store and communicate ban reason (#1085)
* Add read_i32 and write_i32
* Store and send reason for ban
* Client corruption ban reason
* Forgotten max message size
2018-05-29 03:45:31 +01:00
Antioch Peverell 388beafb65 Standardize and cleanup SerializationMode (#856) 2018-03-23 23:34:50 +00:00