[1.1.0] Merge master into 1.1.0 (#2720)

* cleanup legacy "3 dot" check (#2625)

* Allow to peers behind NAT to get up to preferred_max connections (#2543)

Allow to peers behind NAT to get up to preffered_max connections

If peer has only outbound connections it's mot likely behind NAT and we should not stop it from getting more outbound connections

* Reduce usage of unwrap in p2p crate (#2627)

Also change store crate a bit

* Simplify (and fix) output_pos cleanup during chain compaction (#2609)

* expose leaf pos iterator
use it for various things in txhashset when iterating over outputs

* fix

* cleanup

* rebuild output_pos index (and clear it out first) when compacting the chain

* fixup tests

* refactor to match on (output, proof) tuple

* add comments to compact() to explain what is going on.

* get rid of some boxing around the leaf_set iterator

* cleanup

* [docs] Add switch commitment documentation (#2526)

* remove references to no-longer existing switch commitment hash

(as switch commitments were removed in ca8447f3bd
and moved into the blinding factor of the Pedersen Commitment)

* some rewording (points vs curves) and fix of small formatting issues

* Add switch commitment documentation

* [docs] Documents in grin repo had translated in Korean.  (#2604)

*  Start to M/W intro translate in Korean
*  translate in Korean
*  add korean translation  on intro
* table_of_content.md translate in Korean.
*  table_of_content_KR.md finish translate in Korean, start to translate State_KR.md
*  add state_KR.md & commit some translation in State_KR.md
*  WIP stat_KR.md translation
*  add build_KR.md && stratum_KR.md
*  finish translate stratum_KR.md & table_of_content_KR.md
*  rename intro.KR.md to intro_KR.md
*  add intro_KR.md file path each language's  intro.md
*  add Korean translation file path to stratum.md & table_of_contents.md
*  fix difference with grin/master

* Fix TxHashSet file filter for Windows. (#2641)

* Fix TxHashSet file filter for Windows.

* rustfmt

* Updating regexp

* Adding in test case

* Display the current download rate rather than the average when syncing the chain (#2633)

* When syncing the chain, calculate the displayed download speed using the current rate from the most recent iteration, rather than the average download speed from the entire syncing process.

* Replace the explicitly ignored variables in the pattern with an implicit ignore

* remove root = true from editorconfig (#2655)

* Add Medium post to intro (#2654)

Spoke to @yeastplume who agreed it makes sense to add the "Grin Transactions Explained, Step-by-Step" Medium post to intro.md

Open for suggestions on a better location.

* add a new configure item for log_max_files (#2601)

* add a new configure item for log_max_files

* rustfmt

* use a constant instead of multiple 32

* rustfmt

* Fix the build warning of deprecated trim_right_matches (#2662)

* [DOC] state.md, build.md and chain directory documents translate in Korean.  (#2649)

*  add md files for translation.

*  start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md

* add dandelion_KR.md && simulation_KR.md for Korean translation.

*  add md files for translation.

*  start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md

* add dandelion_KR.md && simulation_KR.md for Korean translation.

* remove some useless md files for translation. this is rearrange set up translation order.

*  add dot end of sentence & translate build.md in korean

*  remove fast-sync_KR.md

*  finish build_KR.md translation

*  finish build_KR.md translation

*  finish translation state_KR.md & add phrase in state.md to move other language md file

* translate blocks_and_headers.md && chain_sync.md in Korean

*  add . in chain_sync.md , translation finished in doc/chain dir.

* fix some miss typos

* Api documentation fixes (#2646)

* Fix the API documentation for Chain Validate (v1/chain/validate).  It was documented as a POST, but it is actually a GET request, which can be seen in its handler ChainValidationHandler
* Update the API V1 route list response to include the headers and merkleproof routes.  Also clarify that for the chain/outputs route you must specify either byids or byheight to select outputs.

* refactor(ci): reorganize CI related code (#2658)

Break-down the CI related code into smaller more maintainable pieces.

* Specify grin or nanogrins in API docs where applicable (#2642)

* Set Content-Type in API client (#2680)

* Reduce number of unwraps in chain crate (#2679)

* fix: the restart of state sync doesn't work sometimes (#2687)

* let check_txhashset_needed return true on abnormal case (#2684)

*  Reduce number of unwwaps in api crate  (#2681)

* Reduce number of unwwaps in api crate

* Format use section

* Small QoL improvements for wallet developers (#2651)

* Small changes for wallet devs

* Move create_nonce into Keychain trait

* Replace match by map_err

* Add flag to Slate to skip fee check

* Fix secp dependency

* Remove check_fee flag in Slate

* Add Japanese edition of build.md (#2697)

* catch the panic to avoid peer thread quit early (#2686)

* catch the panic to avoid peer thread quit before taking the chance to ban
* move catch wrapper logic down into the util crate
* log the panic info
* keep txhashset.rs untouched
* remove a warning

* [DOC] dandelion.md, simulation.md ,fast-sync.md and pruning.md documents translate in Korean. (#2678)

* Show response code in API client error message (#2683)

It's hard to investigate what happens when an API client error is
printed out

* Add some better logging for get_outputs_by_id failure states (#2705)

* Switch commitment doc fixes (#2645)

Fix some typos and remove the use of parentheses in a
couple of places to make the reading flow a bit better.

* docs: update/add new README.md badges (#2708)

Replace existing badges with SVG counterparts and add a bunch of new ones.

* Update intro.md (#2702)

Add mention of censoring attack prevented by range proofs

* use sandbox folder for txhashset validation on state sync (#2685)

* use sandbox folder for txhashset validation on state sync

* rustfmt

* use temp directory as the sandbox instead actual db_root txhashset dir

* rustfmt

* move txhashset overwrite to the end of full validation

* fix travis-ci test

* rustfmt

* fix: hashset have 2 folders including txhashset and header

* rustfmt

* 
(1)switch to rebuild_header_mmr instead of copy the sandbox header mmr 
(2)lock txhashset when overwriting and opening and rebuild

* minor improve on sandbox_dir

* add Japanese edition of state.md (#2703)

* Attempt to fix broken TUI locale (#2713)

Can confirm that on the same machine 1.0.2 TUI looks great and is broken on
the current master. Bump of `cursive` version fixed it for me.
Fixes #2676

* clean the header folder in sandbox (#2716)

* forgot to clean the header folder in sandbox in #2685

* Reduce number of unwraps in servers crate (#2707)

It doesn't include stratum server which is sufficiently changed in 1.1
branch and adapters, which is big enough for a separate PR.

* rustfmt

* change version to beta

chain/src/txhashset/txhashset.rs

@@ -32,13 +32,12 @@ use crate::util::secp::pedersen::{Commitment, RangeProof};
 use crate::util::{file, secp_static, zip};
 use croaring::Bitmap;
 use grin_store;
-use grin_store::pmmr::{clean_files_by_prefix, PMMRBackend, PMMR_FILES};
-use grin_store::types::prune_noop;
+use grin_store::pmmr::{PMMRBackend, PMMR_FILES};
 use std::collections::HashSet;
 use std::fs::{self, File};
 use std::path::{Path, PathBuf};
 use std::sync::Arc;
-use std::time::Instant;
+use std::time::{Instant, SystemTime, UNIX_EPOCH};
 
 const HEADERHASHSET_SUBDIR: &'static str = "header";
 const TXHASHSET_SUBDIR: &'static str = "txhashset";
@@ -67,7 +66,10 @@ impl<T: PMMRable> PMMRHandle<T> {
 	) -> Result<PMMRHandle<T>, Error> {
 		let path = Path::new(root_dir).join(sub_dir).join(file_name);
 		fs::create_dir_all(path.clone())?;
-		let backend = PMMRBackend::new(path.to_str().unwrap().to_string(), prunable, header)?;
+		let path_str = path.to_str().ok_or(Error::from(ErrorKind::Other(
+			"invalid file path".to_owned(),
+		)))?;
+		let backend = PMMRBackend::new(path_str.to_string(), prunable, header)?;
 		let last_pos = backend.unpruned_size();
 		Ok(PMMRHandle { backend, last_pos })
 	}
@@ -206,21 +208,27 @@ impl TxHashSet {
 			.get_last_n_insertions(distance)
 	}
 
-	/// Get the header at the specified height based on the current state of the txhashset.
-	/// Derives the MMR pos from the height (insertion index) and retrieves the header hash.
-	/// Looks the header up in the db by hash.
-	pub fn get_header_by_height(&self, height: u64) -> Result<BlockHeader, Error> {
+	/// Get the header hash at the specified height based on the current state of the txhashset.
+	pub fn get_header_hash_by_height(&self, height: u64) -> Result<Hash, Error> {
 		let pos = pmmr::insertion_to_pmmr_index(height + 1);
 		let header_pmmr =
 			ReadonlyPMMR::at(&self.header_pmmr_h.backend, self.header_pmmr_h.last_pos);
 		if let Some(entry) = header_pmmr.get_data(pos) {
-			let header = self.commit_index.get_block_header(&entry.hash())?;
-			Ok(header)
+			Ok(entry.hash())
 		} else {
-			Err(ErrorKind::Other(format!("get header by height")).into())
+			Err(ErrorKind::Other(format!("get header hash by height")).into())
 		}
 	}
 
+	/// Get the header at the specified height based on the current state of the txhashset.
+	/// Derives the MMR pos from the height (insertion index) and retrieves the header hash.
+	/// Looks the header up in the db by hash.
+	pub fn get_header_by_height(&self, height: u64) -> Result<BlockHeader, Error> {
+		let hash = self.get_header_hash_by_height(height)?;
+		let header = self.commit_index.get_block_header(&hash)?;
+		Ok(header)
+	}
+
 	/// returns outputs from the given insertion (leaf) index up to the
 	/// specified limit. Also returns the last index actually populated
 	pub fn outputs_by_insertion_index(
@@ -280,39 +288,30 @@ impl TxHashSet {
 	}
 
 	/// Compact the MMR data files and flush the rm logs
-	pub fn compact(&mut self) -> Result<(), Error> {
-		let commit_index = self.commit_index.clone();
-		let head_header = commit_index.head_header()?;
+	pub fn compact(&mut self, batch: &mut Batch<'_>) -> Result<(), Error> {
+		debug!("txhashset: starting compaction...");
+
+		let head_header = batch.head_header()?;
 		let current_height = head_header.height;
 
 		// horizon for compacting is based on current_height
-		let horizon = current_height.saturating_sub(global::cut_through_horizon().into());
-		let horizon_header = self.get_header_by_height(horizon)?;
+		let horizon_height = current_height.saturating_sub(global::cut_through_horizon().into());
+		let horizon_hash = self.get_header_hash_by_height(horizon_height)?;
+		let horizon_header = batch.get_block_header(&horizon_hash)?;
 
-		let batch = self.commit_index.batch()?;
+		let rewind_rm_pos = input_pos_to_rewind(&horizon_header, &head_header, batch)?;
 
-		let rewind_rm_pos = input_pos_to_rewind(&horizon_header, &head_header, &batch)?;
+		debug!("txhashset: check_compact output mmr backend...");
+		self.output_pmmr_h
+			.backend
+			.check_compact(horizon_header.output_mmr_size, &rewind_rm_pos)?;
 
-		{
-			let clean_output_index = |commit: &[u8]| {
-				let _ = batch.delete_output_pos(commit);
-			};
+		debug!("txhashset: check_compact rangeproof mmr backend...");
+		self.rproof_pmmr_h
+			.backend
+			.check_compact(horizon_header.output_mmr_size, &rewind_rm_pos)?;
 
-			self.output_pmmr_h.backend.check_compact(
-				horizon_header.output_mmr_size,
-				&rewind_rm_pos,
-				clean_output_index,
-			)?;
-
-			self.rproof_pmmr_h.backend.check_compact(
-				horizon_header.output_mmr_size,
-				&rewind_rm_pos,
-				&prune_noop,
-			)?;
-		}
-
-		// Finally commit the batch, saving everything to the db.
-		batch.commit()?;
+		debug!("txhashset: ... compaction finished");
 
 		Ok(())
 	}
@@ -797,11 +796,9 @@ impl<'a> Committed for Extension<'a> {
 
 	fn outputs_committed(&self) -> Vec<Commitment> {
 		let mut commitments = vec![];
-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			if pmmr::is_leaf(n) {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					commitments.push(out.commit);
-				}
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			if let Some(out) = self.output_pmmr.get_data(pos) {
+				commitments.push(out.commit);
 			}
 		}
 		commitments
@@ -1259,20 +1256,18 @@ impl<'a> Extension<'a> {
 	pub fn rebuild_index(&self) -> Result<(), Error> {
 		let now = Instant::now();
 
-		let mut count = 0;
+		self.batch.clear_output_pos()?;
 
-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			// non-pruned leaves only
-			if pmmr::bintree_postorder_height(n) == 0 {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					self.batch.save_output_pos(&out.commit, n)?;
-					count += 1;
-				}
+		let mut count = 0;
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			if let Some(out) = self.output_pmmr.get_data(pos) {
+				self.batch.save_output_pos(&out.commit, pos)?;
+				count += 1;
 			}
 		}
 
 		debug!(
-			"txhashset: rebuild_index ({} UTXOs), took {}s",
+			"txhashset: rebuild_index: {} UTXOs, took {}s",
 			count,
 			now.elapsed().as_secs(),
 		);
@@ -1325,13 +1320,23 @@ impl<'a> Extension<'a> {
 		let total_kernels = pmmr::n_leaves(self.kernel_pmmr.unpruned_size());
 		for n in 1..self.kernel_pmmr.unpruned_size() + 1 {
 			if pmmr::is_leaf(n) {
-				if let Some(kernel) = self.kernel_pmmr.get_data(n) {
-					kernel.verify()?;
-					kern_count += 1;
+				let kernel = self
+					.kernel_pmmr
+					.get_data(n)
+					.ok_or::<Error>(ErrorKind::TxKernelNotFound.into())?;
+
+				kernel.verify()?;
+				kern_count += 1;
+
+				if kern_count % 20 == 0 {
+					status.on_validation(kern_count, total_kernels, 0, 0);
+				}
+				if kern_count % 1_000 == 0 {
+					debug!(
+						"txhashset: verify_kernel_signatures: verified {} signatures",
+						kern_count,
+					);
 				}
-			}
-			if n % 20 == 0 {
-				status.on_validation(kern_count, total_kernels, 0, 0);
 			}
 		}
 
@@ -1353,30 +1358,34 @@ impl<'a> Extension<'a> {
 
 		let mut proof_count = 0;
 		let total_rproofs = pmmr::n_leaves(self.output_pmmr.unpruned_size());
-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			if pmmr::is_leaf(n) {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					if let Some(rp) = self.rproof_pmmr.get_data(n) {
-						commits.push(out.commit);
-						proofs.push(rp);
-					} else {
-						// TODO - rangeproof not found
-						return Err(ErrorKind::OutputNotFound.into());
-					}
-					proof_count += 1;
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			let output = self.output_pmmr.get_data(pos);
+			let proof = self.rproof_pmmr.get_data(pos);
 
-					if proofs.len() >= 1000 {
-						Output::batch_verify_proofs(&commits, &proofs)?;
-						commits.clear();
-						proofs.clear();
-						debug!(
-							"txhashset: verify_rangeproofs: verified {} rangeproofs",
-							proof_count,
-						);
-					}
+			// Output and corresponding rangeproof *must* exist.
+			// It is invalid for either to be missing and we fail immediately in this case.
+			match (output, proof) {
+				(None, _) => return Err(ErrorKind::OutputNotFound.into()),
+				(_, None) => return Err(ErrorKind::RangeproofNotFound.into()),
+				(Some(output), Some(proof)) => {
+					commits.push(output.commit);
+					proofs.push(proof);
 				}
 			}
-			if n % 20 == 0 {
+
+			proof_count += 1;
+
+			if proofs.len() >= 1_000 {
+				Output::batch_verify_proofs(&commits, &proofs)?;
+				commits.clear();
+				proofs.clear();
+				debug!(
+					"txhashset: verify_rangeproofs: verified {} rangeproofs",
+					proof_count,
+				);
+			}
+
+			if proof_count % 20 == 0 {
 				status.on_validation(0, 0, proof_count, total_rproofs);
 			}
 		}
@@ -1404,38 +1413,22 @@ impl<'a> Extension<'a> {
 
 /// Packages the txhashset data files into a zip and returns a Read to the
 /// resulting file
-pub fn zip_read(root_dir: String, header: &BlockHeader) -> Result<File, Error> {
-	let txhashset_zip = format!("{}_{}.zip", TXHASHSET_ZIP, header.hash().to_string());
+pub fn zip_read(root_dir: String, header: &BlockHeader, rand: Option<u32>) -> Result<File, Error> {
+	let ts = if let None = rand {
+		let now = SystemTime::now();
+		now.duration_since(UNIX_EPOCH).unwrap().subsec_micros()
+	} else {
+		rand.unwrap()
+	};
+	let txhashset_zip = format!("{}_{}.zip", TXHASHSET_ZIP, ts);
 
 	let txhashset_path = Path::new(&root_dir).join(TXHASHSET_SUBDIR);
 	let zip_path = Path::new(&root_dir).join(txhashset_zip);
-
-	// if file exist, just re-use it
-	let zip_file = File::open(zip_path.clone());
-	if let Ok(zip) = zip_file {
-		return Ok(zip);
-	} else {
-		// clean up old zips.
-		// Theoretically, we only need clean-up those zip files older than STATE_SYNC_THRESHOLD.
-		// But practically, these zip files are not small ones, we just keep the zips in last one hour
-		let data_dir = Path::new(&root_dir);
-		let pattern = format!("{}_", TXHASHSET_ZIP);
-		if let Ok(n) = clean_files_by_prefix(data_dir.clone(), &pattern, 60 * 60) {
-			debug!(
-				"{} zip files have been clean up in folder: {:?}",
-				n, data_dir
-			);
-		}
-	}
-
-	// otherwise, create the zip archive
-	let path_to_be_cleanup = {
+	// create the zip archive
+	{
 		// Temp txhashset directory
-		let temp_txhashset_path = Path::new(&root_dir).join(format!(
-			"{}_zip_{}",
-			TXHASHSET_SUBDIR,
-			header.hash().to_string()
-		));
+		let temp_txhashset_path =
+			Path::new(&root_dir).join(format!("{}_zip_{}", TXHASHSET_SUBDIR, ts));
 		// Remove temp dir if it exist
 		if temp_txhashset_path.exists() {
 			fs::remove_dir_all(&temp_txhashset_path)?;
@@ -1447,38 +1440,70 @@ pub fn zip_read(root_dir: String, header: &BlockHeader) -> Result<File, Error> {
 		// Compress zip
 		zip::compress(&temp_txhashset_path, &File::create(zip_path.clone())?)
 			.map_err(|ze| ErrorKind::Other(ze.to_string()))?;
-
-		temp_txhashset_path
-	};
+	}
 
 	// open it again to read it back
-	let zip_file = File::open(zip_path.clone())?;
-
-	// clean-up temp txhashset directory.
-	if let Err(e) = fs::remove_dir_all(&path_to_be_cleanup) {
-		warn!(
-			"txhashset zip file: {:?} fail to remove, err: {}",
-			zip_path.to_str(),
-			e
-		);
-	}
+	let zip_file = File::open(zip_path)?;
 	Ok(zip_file)
 }
 
 /// Extract the txhashset data from a zip file and writes the content into the
 /// txhashset storage dir
 pub fn zip_write(
-	root_dir: String,
+	root_dir: PathBuf,
 	txhashset_data: File,
 	header: &BlockHeader,
 ) -> Result<(), Error> {
-	let txhashset_path = Path::new(&root_dir).join(TXHASHSET_SUBDIR);
+	debug!("zip_write on path: {:?}", root_dir);
+	let txhashset_path = root_dir.clone().join(TXHASHSET_SUBDIR);
 	fs::create_dir_all(txhashset_path.clone())?;
 	zip::decompress(txhashset_data, &txhashset_path, expected_file)
 		.map_err(|ze| ErrorKind::Other(ze.to_string()))?;
 	check_and_remove_files(&txhashset_path, header)
 }
 
+/// Overwrite txhashset folders in "to" folder with "from" folder
+pub fn txhashset_replace(from: PathBuf, to: PathBuf) -> Result<(), Error> {
+	debug!("txhashset_replace: move from {:?} to {:?}", from, to);
+
+	// clean the 'to' folder firstly
+	clean_txhashset_folder(&to);
+
+	// rename the 'from' folder as the 'to' folder
+	if let Err(e) = fs::rename(
+		from.clone().join(TXHASHSET_SUBDIR),
+		to.clone().join(TXHASHSET_SUBDIR),
+	) {
+		error!("hashset_replace fail on {}. err: {}", TXHASHSET_SUBDIR, e);
+		Err(ErrorKind::TxHashSetErr(format!("txhashset replacing fail")).into())
+	} else {
+		Ok(())
+	}
+}
+
+/// Clean the txhashset folder
+pub fn clean_txhashset_folder(root_dir: &PathBuf) {
+	let txhashset_path = root_dir.clone().join(TXHASHSET_SUBDIR);
+	if txhashset_path.exists() {
+		if let Err(e) = fs::remove_dir_all(txhashset_path.clone()) {
+			warn!(
+				"clean_txhashset_folder: fail on {:?}. err: {}",
+				txhashset_path, e
+			);
+		}
+	}
+}
+
+/// Clean the header folder
+pub fn clean_header_folder(root_dir: &PathBuf) {
+	let header_path = root_dir.clone().join(HEADERHASHSET_SUBDIR);
+	if header_path.exists() {
+		if let Err(e) = fs::remove_dir_all(header_path.clone()) {
+			warn!("clean_header_folder: fail on {:?}. err: {}", header_path, e);
+		}
+	}
+}
+
 fn expected_file(path: &Path) -> bool {
 	use lazy_static::lazy_static;
 	use regex::Regex;
@@ -1491,7 +1516,7 @@ fn expected_file(path: &Path) -> bool {
 			)
 			.as_str()
 		)
-		.unwrap();
+		.expect("invalid txhashset regular expression");
 	}
 	RE.is_match(&s_path)
 }
@@ -1530,7 +1555,7 @@ fn check_and_remove_files(txhashset_path: &PathBuf, header: &BlockHeader) -> Res
 	}
 
 	// Then compare the files found in the subdirectories
-	let mut pmmr_files_expected: HashSet<_> = PMMR_FILES
+	let pmmr_files_expected: HashSet<_> = PMMR_FILES
 		.iter()
 		.cloned()
 		.map(|s| {
@@ -1541,8 +1566,6 @@ fn check_and_remove_files(txhashset_path: &PathBuf, header: &BlockHeader) -> Res
 			}
 		})
 		.collect();
-	// prevent checker from deleting 3 dot file, could be removed after mainnet
-	pmmr_files_expected.insert(format!("pmmr_leaf.bin.{}...", header.hash()));
 
 	let subdirectories = fs::read_dir(txhashset_path)?;
 	for subdirectory in subdirectories {