[1.1.0] Merge master into 1.1.0 (#2720)

* cleanup legacy "3 dot" check (#2625)

* Allow to peers behind NAT to get up to preferred_max connections (#2543)

Allow to peers behind NAT to get up to preffered_max connections

If peer has only outbound connections it's mot likely behind NAT and we should not stop it from getting more outbound connections

* Reduce usage of unwrap in p2p crate (#2627)

Also change store crate a bit

* Simplify (and fix) output_pos cleanup during chain compaction (#2609)

* expose leaf pos iterator
use it for various things in txhashset when iterating over outputs

* fix

* cleanup

* rebuild output_pos index (and clear it out first) when compacting the chain

* fixup tests

* refactor to match on (output, proof) tuple

* add comments to compact() to explain what is going on.

* get rid of some boxing around the leaf_set iterator

* cleanup

* [docs] Add switch commitment documentation (#2526)

* remove references to no-longer existing switch commitment hash

(as switch commitments were removed in ca8447f3bd
and moved into the blinding factor of the Pedersen Commitment)

* some rewording (points vs curves) and fix of small formatting issues

* Add switch commitment documentation

* [docs] Documents in grin repo had translated in Korean.  (#2604)

*  Start to M/W intro translate in Korean
*  translate in Korean
*  add korean translation  on intro
* table_of_content.md translate in Korean.
*  table_of_content_KR.md finish translate in Korean, start to translate State_KR.md
*  add state_KR.md & commit some translation in State_KR.md
*  WIP stat_KR.md translation
*  add build_KR.md && stratum_KR.md
*  finish translate stratum_KR.md & table_of_content_KR.md
*  rename intro.KR.md to intro_KR.md
*  add intro_KR.md file path each language's  intro.md
*  add Korean translation file path to stratum.md & table_of_contents.md
*  fix difference with grin/master

* Fix TxHashSet file filter for Windows. (#2641)

* Fix TxHashSet file filter for Windows.

* rustfmt

* Updating regexp

* Adding in test case

* Display the current download rate rather than the average when syncing the chain (#2633)

* When syncing the chain, calculate the displayed download speed using the current rate from the most recent iteration, rather than the average download speed from the entire syncing process.

* Replace the explicitly ignored variables in the pattern with an implicit ignore

* remove root = true from editorconfig (#2655)

* Add Medium post to intro (#2654)

Spoke to @yeastplume who agreed it makes sense to add the "Grin Transactions Explained, Step-by-Step" Medium post to intro.md

Open for suggestions on a better location.

* add a new configure item for log_max_files (#2601)

* add a new configure item for log_max_files

* rustfmt

* use a constant instead of multiple 32

* rustfmt

* Fix the build warning of deprecated trim_right_matches (#2662)

* [DOC] state.md, build.md and chain directory documents translate in Korean.  (#2649)

*  add md files for translation.

*  start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md

* add dandelion_KR.md && simulation_KR.md for Korean translation.

*  add md files for translation.

*  start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md

* add dandelion_KR.md && simulation_KR.md for Korean translation.

* remove some useless md files for translation. this is rearrange set up translation order.

*  add dot end of sentence & translate build.md in korean

*  remove fast-sync_KR.md

*  finish build_KR.md translation

*  finish build_KR.md translation

*  finish translation state_KR.md & add phrase in state.md to move other language md file

* translate blocks_and_headers.md && chain_sync.md in Korean

*  add . in chain_sync.md , translation finished in doc/chain dir.

* fix some miss typos

* Api documentation fixes (#2646)

* Fix the API documentation for Chain Validate (v1/chain/validate).  It was documented as a POST, but it is actually a GET request, which can be seen in its handler ChainValidationHandler
* Update the API V1 route list response to include the headers and merkleproof routes.  Also clarify that for the chain/outputs route you must specify either byids or byheight to select outputs.

* refactor(ci): reorganize CI related code (#2658)

Break-down the CI related code into smaller more maintainable pieces.

* Specify grin or nanogrins in API docs where applicable (#2642)

* Set Content-Type in API client (#2680)

* Reduce number of unwraps in chain crate (#2679)

* fix: the restart of state sync doesn't work sometimes (#2687)

* let check_txhashset_needed return true on abnormal case (#2684)

*  Reduce number of unwwaps in api crate  (#2681)

* Reduce number of unwwaps in api crate

* Format use section

* Small QoL improvements for wallet developers (#2651)

* Small changes for wallet devs

* Move create_nonce into Keychain trait

* Replace match by map_err

* Add flag to Slate to skip fee check

* Fix secp dependency

* Remove check_fee flag in Slate

* Add Japanese edition of build.md (#2697)

* catch the panic to avoid peer thread quit early (#2686)

* catch the panic to avoid peer thread quit before taking the chance to ban
* move catch wrapper logic down into the util crate
* log the panic info
* keep txhashset.rs untouched
* remove a warning

* [DOC] dandelion.md, simulation.md ,fast-sync.md and pruning.md documents translate in Korean. (#2678)

* Show response code in API client error message (#2683)

It's hard to investigate what happens when an API client error is
printed out

* Add some better logging for get_outputs_by_id failure states (#2705)

* Switch commitment doc fixes (#2645)

Fix some typos and remove the use of parentheses in a
couple of places to make the reading flow a bit better.

* docs: update/add new README.md badges (#2708)

Replace existing badges with SVG counterparts and add a bunch of new ones.

* Update intro.md (#2702)

Add mention of censoring attack prevented by range proofs

* use sandbox folder for txhashset validation on state sync (#2685)

* use sandbox folder for txhashset validation on state sync

* rustfmt

* use temp directory as the sandbox instead actual db_root txhashset dir

* rustfmt

* move txhashset overwrite to the end of full validation

* fix travis-ci test

* rustfmt

* fix: hashset have 2 folders including txhashset and header

* rustfmt

* 
(1)switch to rebuild_header_mmr instead of copy the sandbox header mmr 
(2)lock txhashset when overwriting and opening and rebuild

* minor improve on sandbox_dir

* add Japanese edition of state.md (#2703)

* Attempt to fix broken TUI locale (#2713)

Can confirm that on the same machine 1.0.2 TUI looks great and is broken on
the current master. Bump of `cursive` version fixed it for me.
Fixes #2676

* clean the header folder in sandbox (#2716)

* forgot to clean the header folder in sandbox in #2685

* Reduce number of unwraps in servers crate (#2707)

It doesn't include stratum server which is sufficiently changed in 1.1
branch and adapters, which is big enough for a separate PR.

* rustfmt

* change version to beta

store/src/leaf_set.rs

@@ -199,4 +199,9 @@ impl LeafSet {
 	pub fn is_empty(&self) -> bool {
 		self.len() == 0
 	}
+
+	/// Iterator over positionns in the leaf_set (all leaf positions).
+	pub fn iter(&self) -> impl Iterator<Item = u64> + '_ {
+		self.bitmap.iter().map(|x| x as u64)
+	}
 }

store/src/lmdb.rs

@@ -248,8 +248,8 @@ impl Store {
 		res.to_opt().map(|r| r.is_some()).map_err(From::from)
 	}
 
-	/// Produces an iterator of `Readable` types moving forward from the
-	/// provided key.
+	/// Produces an iterator of (key, value) pairs, where values are `Readable` types
+	/// moving forward from the provided key.
 	pub fn iter<T: ser::Readable>(&self, from: &[u8]) -> Result<SerIterator<T>, Error> {
 		let db = self.db.read();
 		let tx = Arc::new(lmdb::ReadTransaction::new(self.env.clone())?);
@@ -367,9 +367,9 @@ impl<T> Iterator for SerIterator<T>
 where
 	T: ser::Readable,
 {
-	type Item = T;
+	type Item = (Vec<u8>, T);
 
-	fn next(&mut self) -> Option<T> {
+	fn next(&mut self) -> Option<(Vec<u8>, T)> {
 		let access = self.tx.access();
 		let kv = if self.seek {
 			Arc::get_mut(&mut self.cursor).unwrap().next(&access)
@@ -379,7 +379,10 @@ where
 				.unwrap()
 				.seek_range_k(&access, &self.prefix[..])
 		};
-		self.deser_if_prefix_match(kv)
+		match kv {
+			Ok((k, v)) => self.deser_if_prefix_match(k, v),
+			Err(_) => None,
+		}
 	}
 }
 
@@ -387,17 +390,16 @@ impl<T> SerIterator<T>
 where
 	T: ser::Readable,
 {
-	fn deser_if_prefix_match(&self, kv: Result<(&[u8], &[u8]), lmdb::Error>) -> Option<T> {
-		match kv {
-			Ok((k, v)) => {
-				let plen = self.prefix.len();
-				if plen == 0 || k[0..plen] == self.prefix[..] {
-					ser::deserialize(&mut &v[..]).ok()
-				} else {
-					None
-				}
+	fn deser_if_prefix_match(&self, key: &[u8], value: &[u8]) -> Option<(Vec<u8>, T)> {
+		let plen = self.prefix.len();
+		if plen == 0 || key[0..plen] == self.prefix[..] {
+			if let Ok(value) = ser::deserialize(&mut &value[..]) {
+				Some((key.to_vec(), value))
+			} else {
+				None
 			}
-			Err(_) => None,
+		} else {
+			None
 		}
 	}
 }

store/src/pmmr.rs

@@ -21,7 +21,7 @@ use crate::core::core::BlockHeader;
 use crate::core::ser::PMMRable;
 use crate::leaf_set::LeafSet;
 use crate::prune_list::PruneList;
-use crate::types::{prune_noop, DataFile};
+use crate::types::DataFile;
 use croaring::Bitmap;
 use std::path::{Path, PathBuf};
 
@@ -121,6 +121,17 @@ impl<T: PMMRable> Backend<T> for PMMRBackend<T> {
 		self.get_data_from_file(pos)
 	}
 
+	/// Returns an iterator over all the leaf positions.
+	/// for a prunable PMMR this is an iterator over the leaf_set bitmap.
+	/// For a non-prunable PMMR this is *all* leaves (this is not yet implemented).
+	fn leaf_pos_iter(&self) -> Box<Iterator<Item = u64> + '_> {
+		if self.prunable {
+			Box::new(self.leaf_set.iter())
+		} else {
+			panic!("leaf_pos_iter not implemented for non-prunable PMMR")
+		}
+	}
+
 	/// Rewind the PMMR backend to the given position.
 	fn rewind(&mut self, position: u64, rewind_rm_pos: &Bitmap) -> Result<(), String> {
 		// First rewind the leaf_set with the necessary added and removed positions.
@@ -199,13 +210,7 @@ impl<T: PMMRable> PMMRBackend<T> {
 				data_dir.join(PMMR_LEAF_FILE).to_str().unwrap(),
 				header.hash()
 			);
-			// Check for a ... (3 dot) ending version of the file - could probably be removed after mainnet
-			let compatible_snapshot_path = PathBuf::from(leaf_snapshot_path.clone() + "...");
-			if compatible_snapshot_path.exists() {
-				LeafSet::copy_snapshot(&leaf_set_path, &compatible_snapshot_path)?;
-			} else {
-				LeafSet::copy_snapshot(&leaf_set_path, &PathBuf::from(leaf_snapshot_path))?;
-			}
+			LeafSet::copy_snapshot(&leaf_set_path, &PathBuf::from(leaf_snapshot_path))?;
 		}
 
 		let leaf_set = LeafSet::open(&leaf_set_path)?;
@@ -284,15 +289,7 @@ impl<T: PMMRable> PMMRBackend<T> {
 	/// aligned. The block_marker in the db/index for the particular block
 	/// will have a suitable output_pos. This is used to enforce a horizon
 	/// after which the local node should have all the data to allow rewinding.
-	pub fn check_compact<P>(
-		&mut self,
-		cutoff_pos: u64,
-		rewind_rm_pos: &Bitmap,
-		prune_cb: P,
-	) -> io::Result<bool>
-	where
-		P: Fn(&[u8]),
-	{
+	pub fn check_compact(&mut self, cutoff_pos: u64, rewind_rm_pos: &Bitmap) -> io::Result<bool> {
 		assert!(self.prunable, "Trying to compact a non-prunable PMMR");
 
 		// Paths for tmp hash and data files.
@@ -312,7 +309,7 @@ impl<T: PMMRable> PMMRBackend<T> {
 			});
 
 			self.hash_file
-				.save_prune(&tmp_prune_file_hash, &off_to_rm, &prune_noop)?;
+				.save_prune(&tmp_prune_file_hash, &off_to_rm)?;
 		}
 
 		// 2. Save compact copy of the data file, skipping removed leaves.
@@ -330,7 +327,7 @@ impl<T: PMMRable> PMMRBackend<T> {
 			});
 
 			self.data_file
-				.save_prune(&tmp_prune_file_data, &off_to_rm, prune_cb)?;
+				.save_prune(&tmp_prune_file_data, &off_to_rm)?;
 		}
 
 		// 3. Update the prune list and write to disk.

store/src/types.rs

@@ -20,9 +20,6 @@ use std::io::{self, BufWriter, ErrorKind, Read, Write};
 use std::marker;
 use std::path::{Path, PathBuf};
 
-/// A no-op function for doing nothing with some pruned data.
-pub fn prune_noop(_pruned_data: &[u8]) {}
-
 /// Data file (MMR) wrapper around an append only file.
 pub struct DataFile<T> {
 	file: AppendOnlyFile,
@@ -113,16 +110,13 @@ where
 	}
 
 	/// Write the file out to disk, pruning removed elements.
-	pub fn save_prune<F>(&self, target: &str, prune_offs: &[u64], prune_cb: F) -> io::Result<()>
-	where
-		F: Fn(&[u8]),
-	{
+	pub fn save_prune(&self, target: &str, prune_offs: &[u64]) -> io::Result<()> {
 		let prune_offs = prune_offs
 			.iter()
 			.map(|x| x * T::LEN as u64)
 			.collect::<Vec<_>>();
 		self.file
-			.save_prune(target, prune_offs.as_slice(), T::LEN as u64, prune_cb)
+			.save_prune(target, prune_offs.as_slice(), T::LEN as u64)
 	}
 }
 
@@ -294,15 +288,8 @@ impl AppendOnlyFile {
 
 	/// Saves a copy of the current file content, skipping data at the provided
 	/// prune indices. The prune Vec must be ordered.
-	pub fn save_prune<T, P>(
-		&self,
-		target: P,
-		prune_offs: &[u64],
-		prune_len: u64,
-		prune_cb: T,
-	) -> io::Result<()>
+	pub fn save_prune<P>(&self, target: P, prune_offs: &[u64], prune_len: u64) -> io::Result<()>
 	where
-		T: Fn(&[u8]),
 		P: AsRef<Path>,
 	{
 		if prune_offs.is_empty() {
@@ -332,8 +319,6 @@ impl AppendOnlyFile {
 					let prune_at = (prune_offs[prune_pos] - read) as usize;
 					if prune_at != buf_start {
 						writer.write_all(&buf[buf_start..prune_at])?;
-					} else {
-						prune_cb(&buf[buf_start..prune_at]);
 					}
 					buf_start = prune_at + (prune_len as usize);
 					if prune_offs.len() > prune_pos + 1 {