[1.1.0] Merge master into 1.1.0 (#2720)

* cleanup legacy "3 dot" check (#2625) * Allow to peers behind NAT to get up to preferred_max connections (#2543) Allow to peers behind NAT to get up to preffered_max connections If peer has only outbound connections it's mot likely behind NAT and we should not stop it from getting more outbound connections * Reduce usage of unwrap in p2p crate (#2627) Also change store crate a bit * Simplify (and fix) output_pos cleanup during chain compaction (#2609) * expose leaf pos iterator use it for various things in txhashset when iterating over outputs * fix * cleanup * rebuild output_pos index (and clear it out first) when compacting the chain * fixup tests * refactor to match on (output, proof) tuple * add comments to compact() to explain what is going on. * get rid of some boxing around the leaf_set iterator * cleanup * [docs] Add switch commitment documentation (#2526) * remove references to no-longer existing switch commitment hash (as switch commitments were removed in ca8447f3bd and moved into the blinding factor of the Pedersen Commitment) * some rewording (points vs curves) and fix of small formatting issues * Add switch commitment documentation * [docs] Documents in grin repo had translated in Korean. (#2604) * Start to M/W intro translate in Korean * translate in Korean * add korean translation on intro * table_of_content.md translate in Korean. * table_of_content_KR.md finish translate in Korean, start to translate State_KR.md * add state_KR.md & commit some translation in State_KR.md * WIP stat_KR.md translation * add build_KR.md && stratum_KR.md * finish translate stratum_KR.md & table_of_content_KR.md * rename intro.KR.md to intro_KR.md * add intro_KR.md file path each language's intro.md * add Korean translation file path to stratum.md & table_of_contents.md * fix difference with grin/master * Fix TxHashSet file filter for Windows. (#2641) * Fix TxHashSet file filter for Windows. * rustfmt * Updating regexp * Adding in test case * Display the current download rate rather than the average when syncing the chain (#2633) * When syncing the chain, calculate the displayed download speed using the current rate from the most recent iteration, rather than the average download speed from the entire syncing process. * Replace the explicitly ignored variables in the pattern with an implicit ignore * remove root = true from editorconfig (#2655) * Add Medium post to intro (#2654) Spoke to @yeastplume who agreed it makes sense to add the "Grin Transactions Explained, Step-by-Step" Medium post to intro.md Open for suggestions on a better location. * add a new configure item for log_max_files (#2601) * add a new configure item for log_max_files * rustfmt * use a constant instead of multiple 32 * rustfmt * Fix the build warning of deprecated trim_right_matches (#2662) * [DOC] state.md, build.md and chain directory documents translate in Korean. (#2649) * add md files for translation. * start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md * add dandelion_KR.md && simulation_KR.md for Korean translation. * add md files for translation. * start to translation fast-sync, code_structure. add file build_KR.md, states_KR.md * add dandelion_KR.md && simulation_KR.md for Korean translation. * remove some useless md files for translation. this is rearrange set up translation order. * add dot end of sentence & translate build.md in korean * remove fast-sync_KR.md * finish build_KR.md translation * finish build_KR.md translation * finish translation state_KR.md & add phrase in state.md to move other language md file * translate blocks_and_headers.md && chain_sync.md in Korean * add . in chain_sync.md , translation finished in doc/chain dir. * fix some miss typos * Api documentation fixes (#2646) * Fix the API documentation for Chain Validate (v1/chain/validate). It was documented as a POST, but it is actually a GET request, which can be seen in its handler ChainValidationHandler * Update the API V1 route list response to include the headers and merkleproof routes. Also clarify that for the chain/outputs route you must specify either byids or byheight to select outputs. * refactor(ci): reorganize CI related code (#2658) Break-down the CI related code into smaller more maintainable pieces. * Specify grin or nanogrins in API docs where applicable (#2642) * Set Content-Type in API client (#2680) * Reduce number of unwraps in chain crate (#2679) * fix: the restart of state sync doesn't work sometimes (#2687) * let check_txhashset_needed return true on abnormal case (#2684) * Reduce number of unwwaps in api crate (#2681) * Reduce number of unwwaps in api crate * Format use section * Small QoL improvements for wallet developers (#2651) * Small changes for wallet devs * Move create_nonce into Keychain trait * Replace match by map_err * Add flag to Slate to skip fee check * Fix secp dependency * Remove check_fee flag in Slate * Add Japanese edition of build.md (#2697) * catch the panic to avoid peer thread quit early (#2686) * catch the panic to avoid peer thread quit before taking the chance to ban * move catch wrapper logic down into the util crate * log the panic info * keep txhashset.rs untouched * remove a warning * [DOC] dandelion.md, simulation.md ,fast-sync.md and pruning.md documents translate in Korean. (#2678) * Show response code in API client error message (#2683) It's hard to investigate what happens when an API client error is printed out * Add some better logging for get_outputs_by_id failure states (#2705) * Switch commitment doc fixes (#2645) Fix some typos and remove the use of parentheses in a couple of places to make the reading flow a bit better. * docs: update/add new README.md badges (#2708) Replace existing badges with SVG counterparts and add a bunch of new ones. * Update intro.md (#2702) Add mention of censoring attack prevented by range proofs * use sandbox folder for txhashset validation on state sync (#2685) * use sandbox folder for txhashset validation on state sync * rustfmt * use temp directory as the sandbox instead actual db_root txhashset dir * rustfmt * move txhashset overwrite to the end of full validation * fix travis-ci test * rustfmt * fix: hashset have 2 folders including txhashset and header * rustfmt * (1)switch to rebuild_header_mmr instead of copy the sandbox header mmr (2)lock txhashset when overwriting and opening and rebuild * minor improve on sandbox_dir * add Japanese edition of state.md (#2703) * Attempt to fix broken TUI locale (#2713) Can confirm that on the same machine 1.0.2 TUI looks great and is broken on the current master. Bump of `cursive` version fixed it for me. Fixes #2676 * clean the header folder in sandbox (#2716) * forgot to clean the header folder in sandbox in #2685 * Reduce number of unwraps in servers crate (#2707) It doesn't include stratum server which is sufficiently changed in 1.1 branch and adapters, which is big enough for a separate PR. * rustfmt * change version to beta
2019-04-01 11:47:48 +01:00
parent bd6c73417d
commit 5cb8025ddd
115 changed files with 3593 additions and 1370 deletions
@@ -35,7 +35,9 @@ use crate::util::secp::pedersen::{Commitment, RangeProof};
 use crate::util::{Mutex, RwLock, StopState};
 use grin_store::Error::NotFoundErr;
 use std::collections::HashMap;
+use std::env;
 use std::fs::File;
+use std::path::PathBuf;
 use std::sync::atomic::{AtomicUsize, Ordering};
 use std::sync::Arc;
 use std::time::{Duration, Instant};
@@ -688,7 +690,7 @@ impl Chain {
 		}

 		// prepares the zip and return the corresponding Read
-		let txhashset_reader = txhashset::zip_read(self.db_root.clone(), &header)?;
+		let txhashset_reader = txhashset::zip_read(self.db_root.clone(), &header, None)?;
 		Ok((
 			header.output_mmr_size,
 			header.kernel_mmr_size,
@@ -763,11 +765,15 @@ impl Chain {
 	}

 	/// Check chain status whether a txhashset downloading is needed
-	pub fn check_txhashset_needed(&self, caller: String, hashes: &mut Option<Vec<Hash>>) -> bool {
+	pub fn check_txhashset_needed(
+		&self,
+		caller: String,
+		hashes: &mut Option<Vec<Hash>>,
+	) -> Result<bool, Error> {
 		let horizon = global::cut_through_horizon() as u64;
-		let body_head = self.head().unwrap();
-		let header_head = self.header_head().unwrap();
-		let sync_head = self.get_sync_head().unwrap();
+		let body_head = self.head()?;
+		let header_head = self.header_head()?;
+		let sync_head = self.get_sync_head()?;

 		debug!(
 			"{}: body_head - {}, {}, header_head - {}, {}, sync_head - {}, {}",
@@ -785,7 +791,7 @@ impl Chain {
 				"{}: no need txhashset. header_head.total_difficulty: {} <= body_head.total_difficulty: {}",
 				caller, header_head.total_difficulty, body_head.total_difficulty,
 			);
-			return false;
+			return Ok(false);
 		}

 		let mut oldest_height = 0;
@@ -797,6 +803,7 @@ impl Chain {
 				"{}: header_head not found in chain db: {} at {}",
 				caller, header_head.last_block_h, header_head.height,
 			);
+			return Ok(false);
 		}

 		//
@@ -822,17 +829,30 @@ impl Chain {

 		if oldest_height < header_head.height.saturating_sub(horizon) {
 			if oldest_height > 0 {
+				// this is the normal case. for example:
+				// body head height is 1 (and not a fork), oldest_height will be 2
+				// body head height is 0 (a typical fresh node), oldest_height will be 1
+				// body head height is 10,001 (but at a fork), oldest_height will be 10,001
+				// body head height is 10,005 (but at a fork with depth 5), oldest_height will be 10,001
 				debug!(
 					"{}: need a state sync for txhashset. oldest block which is not on local chain: {} at {}",
 					caller, oldest_hash, oldest_height,
 				);
-				return true;
 			} else {
-				error!("{}: something is wrong! oldest_height is 0", caller);
-				return false;
-			};
+				// this is the abnormal case, when is_on_current_chain() already return Err, and even for genesis block.
+				error!("{}: corrupted storage? oldest_height is 0 when check_txhashset_needed. state sync is needed", caller);
+			}
+			Ok(true)
+		} else {
+			Ok(false)
 		}
-		return false;
+	}
+
+	/// Clean the temporary sandbox folder
+	pub fn clean_txhashset_sandbox(&self) {
+		let sandbox_dir = env::temp_dir();
+		txhashset::clean_txhashset_folder(&sandbox_dir);
+		txhashset::clean_header_folder(&sandbox_dir);
 	}

 	/// Writes a reading view on a txhashset state that's been provided to us.
@@ -849,24 +869,27 @@ impl Chain {

 		// Initial check whether this txhashset is needed or not
 		let mut hashes: Option<Vec<Hash>> = None;
-		if !self.check_txhashset_needed("txhashset_write".to_owned(), &mut hashes) {
+		if !self.check_txhashset_needed("txhashset_write".to_owned(), &mut hashes)? {
 			warn!("txhashset_write: txhashset received but it's not needed! ignored.");
 			return Err(ErrorKind::InvalidTxHashSet("not needed".to_owned()).into());
 		}

 		let header = self.get_block_header(&h)?;

-		{
-			let mut txhashset_ref = self.txhashset.write();
-			// Drop file handles in underlying txhashset
-			txhashset_ref.release_backend_files();
-		}
+		// Write txhashset to sandbox (in the os temporary directory)
+		let sandbox_dir = env::temp_dir();
+		txhashset::clean_txhashset_folder(&sandbox_dir);
+		txhashset::clean_header_folder(&sandbox_dir);
+		txhashset::zip_write(sandbox_dir.clone(), txhashset_data.try_clone()?, &header)?;

-		// Rewrite hashset
-		txhashset::zip_write(self.db_root.clone(), txhashset_data, &header)?;
-
-		let mut txhashset =
-			txhashset::TxHashSet::open(self.db_root.clone(), self.store.clone(), Some(&header))?;
+		let mut txhashset = txhashset::TxHashSet::open(
+			sandbox_dir
+				.to_str()
+				.expect("invalid sandbox folder")
+				.to_owned(),
+			self.store.clone(),
+			Some(&header),
+		)?;

 		// The txhashset.zip contains the output, rangeproof and kernel MMRs.
 		// We must rebuild the header MMR ourselves based on the headers in our db.
@@ -918,9 +941,28 @@ impl Chain {

 		debug!("txhashset_write: finished committing the batch (head etc.)");

-		// Replace the chain txhashset with the newly built one.
+		// Sandbox full validation ok, go to overwrite txhashset on db root
 		{
 			let mut txhashset_ref = self.txhashset.write();
+
+			// Before overwriting, drop file handlers in underlying txhashset
+			txhashset_ref.release_backend_files();
+
+			// Move sandbox to overwrite
+			txhashset.release_backend_files();
+			txhashset::txhashset_replace(sandbox_dir.clone(), PathBuf::from(self.db_root.clone()))?;
+
+			// Re-open on db root dir
+			txhashset = txhashset::TxHashSet::open(
+				self.db_root.clone(),
+				self.store.clone(),
+				Some(&header),
+			)?;
+
+			self.rebuild_header_mmr(&Tip::from_header(&header), &mut txhashset)?;
+			txhashset::clean_header_folder(&sandbox_dir);
+
+			// Replace the chain txhashset with the newly built one.
 			*txhashset_ref = txhashset;
 		}

@@ -933,35 +975,22 @@ impl Chain {
 		Ok(())
 	}

-	fn compact_txhashset(&self) -> Result<(), Error> {
-		debug!("Starting txhashset compaction...");
-		{
-			// Note: We take a lock on the stop_state here and do not release it until
-			// we have finished processing this chain compaction operation.
-			let stop_lock = self.stop_state.lock();
-			if stop_lock.is_stopped() {
-				return Err(ErrorKind::Stopped.into());
-			}
-
-			let mut txhashset = self.txhashset.write();
-			txhashset.compact()?;
-		}
-		debug!("... finished txhashset compaction.");
-		Ok(())
-	}
-
 	/// Cleanup old blocks from the db.
 	/// Determine the cutoff height from the horizon and the current block height.
 	/// *Only* runs if we are not in archive mode.
-	fn compact_blocks_db(&self) -> Result<(), Error> {
+	fn remove_historical_blocks(
+		&self,
+		txhashset: &txhashset::TxHashSet,
+		batch: &mut store::Batch<'_>,
+	) -> Result<(), Error> {
 		if self.archive_mode {
 			return Ok(());
 		}

 		let horizon = global::cut_through_horizon() as u64;
-		let head = self.head()?;
+		let head = batch.head()?;

-		let tail = match self.tail() {
+		let tail = match batch.tail() {
 			Ok(tail) => tail,
 			Err(_) => Tip::from_header(&self.genesis),
 		};
@@ -969,7 +998,7 @@ impl Chain {
 		let cutoff = head.height.saturating_sub(horizon);

 		debug!(
-			"compact_blocks_db: head height: {}, tail height: {}, horizon: {}, cutoff: {}",
+			"remove_historical_blocks: head height: {}, tail height: {}, horizon: {}, cutoff: {}",
 			head.height, tail.height, horizon, cutoff,
 		);

@@ -979,10 +1008,11 @@ impl Chain {

 		let mut count = 0;

-		let tail = self.get_header_by_height(head.height - horizon)?;
-		let mut current = self.get_header_by_height(head.height - horizon - 1)?;
+		let tail_hash = txhashset.get_header_hash_by_height(head.height - horizon)?;
+		let tail = batch.get_block_header(&tail_hash)?;

-		let batch = self.store.batch()?;
+		let current_hash = txhashset.get_header_hash_by_height(head.height - horizon - 1)?;
+		let mut current = batch.get_block_header(&current_hash)?;

 		loop {
 			// Go to the store directly so we can handle NotFoundErr robustly.
@@ -1010,11 +1040,12 @@ impl Chain {
 			}
 		}
 		batch.save_body_tail(&Tip::from_header(&tail))?;
-		batch.commit()?;
+
 		debug!(
-			"compact_blocks_db: removed {} blocks. tail height: {}",
+			"remove_historical_blocks: removed {} blocks. tail height: {}",
 			count, tail.height
 		);
+
 		Ok(())
 	}

@@ -1024,12 +1055,35 @@ impl Chain {
 	/// * removes historical blocks and associated data from the db (unless archive mode)
 	///
 	pub fn compact(&self) -> Result<(), Error> {
-		self.compact_txhashset()?;
-
-		if !self.archive_mode {
-			self.compact_blocks_db()?;
+		// Note: We take a lock on the stop_state here and do not release it until
+		// we have finished processing this chain compaction operation.
+		// We want to avoid shutting the node down in the middle of compacting the data.
+		let stop_lock = self.stop_state.lock();
+		if stop_lock.is_stopped() {
+			return Err(ErrorKind::Stopped.into());
 		}

+		// Take a write lock on the txhashet and start a new writeable db batch.
+		let mut txhashset = self.txhashset.write();
+		let mut batch = self.store.batch()?;
+
+		// Compact the txhashset itself (rewriting the pruned backend files).
+		txhashset.compact(&mut batch)?;
+
+		// Rebuild our output_pos index in the db based on current UTXO set.
+		txhashset::extending(&mut txhashset, &mut batch, |extension| {
+			extension.rebuild_index()?;
+			Ok(())
+		})?;
+
+		// If we are not in archival mode remove historical blocks from the db.
+		if !self.archive_mode {
+			self.remove_historical_blocks(&txhashset, &mut batch)?;
+		}
+
+		// Commit all the above db changes.
+		batch.commit()?;
+
 		Ok(())
 	}

@@ -1142,12 +1196,20 @@ impl Chain {
 	}

 	/// Gets the block header at the provided height.
-	/// Note: This takes a read lock on the txhashset.
+	/// Note: Takes a read lock on the txhashset.
 	/// Take care not to call this repeatedly in a tight loop.
 	pub fn get_header_by_height(&self, height: u64) -> Result<BlockHeader, Error> {
+		let hash = self.get_header_hash_by_height(height)?;
+		self.get_block_header(&hash)
+	}
+
+	/// Gets the header hash at the provided height.
+	/// Note: Takes a read lock on the txhashset.
+	/// Take care not to call this repeatedly in a tight loop.
+	fn get_header_hash_by_height(&self, height: u64) -> Result<Hash, Error> {
 		let txhashset = self.txhashset.read();
-		let header = txhashset.get_header_by_height(height)?;
-		Ok(header)
+		let hash = txhashset.get_header_hash_by_height(height)?;
+		Ok(hash)
 	}

 	/// Gets the block header in which a given output appears in the txhashset.
@@ -1208,10 +1270,10 @@ impl Chain {
 	/// Builds an iterator on blocks starting from the current chain head and
 	/// running backward. Specialized to return information pertaining to block
 	/// difficulty calculation (timestamp and previous difficulties).
-	pub fn difficulty_iter(&self) -> store::DifficultyIter<'_> {
-		let head = self.head().unwrap();
+	pub fn difficulty_iter(&self) -> Result<store::DifficultyIter<'_>, Error> {
+		let head = self.head()?;
 		let store = self.store.clone();
-		store::DifficultyIter::from(head.last_block_h, store)
+		Ok(store::DifficultyIter::from(head.last_block_h, store))
 	}

 	/// Check whether we have a block without reading it
@@ -89,9 +89,15 @@ pub enum ErrorKind {
 	/// Error validating a Merkle proof (coinbase output)
 	#[fail(display = "Error validating merkle proof")]
 	MerkleProof,
-	/// output not found
+	/// Output not found
 	#[fail(display = "Output not found")]
 	OutputNotFound,
+	/// Rangeproof not found
+	#[fail(display = "Rangeproof not found")]
+	RangeproofNotFound,
+	/// Tx kernel not found
+	#[fail(display = "Tx kernel not found")]
+	TxKernelNotFound,
 	/// output spent
 	#[fail(display = "Output is spent")]
 	OutputSpent,
@@ -183,16 +183,21 @@ pub fn sync_block_headers(
 	headers: &[BlockHeader],
 	ctx: &mut BlockContext<'_>,
 ) -> Result<Option<Tip>, Error> {
-	if let Some(header) = headers.first() {
-		debug!(
-			"pipe: sync_block_headers: {} headers from {} at {}",
-			headers.len(),
-			header.hash(),
-			header.height,
-		);
-	} else {
-		return Ok(None);
-	}
+	let first_header = match headers.first() {
+		Some(header) => {
+			debug!(
+				"pipe: sync_block_headers: {} headers from {} at {}",
+				headers.len(),
+				header.hash(),
+				header.height,
+			);
+			header
+		}
+		None => {
+			error!("failed to get the first header");
+			return Ok(None);
+		}
+	};

 	let all_known = if let Some(last_header) = headers.last() {
 		ctx.batch.get_block_header(&last_header.hash()).is_ok()
@@ -201,7 +206,6 @@ pub fn sync_block_headers(
 	};

 	if !all_known {
-		let first_header = headers.first().unwrap();
 		let prev_header = ctx.batch.get_previous_header(&first_header)?;
 		txhashset::sync_extending(&mut ctx.txhashset, &mut ctx.batch, |extension| {
 			extension.rewind(&prev_header)?;
@@ -50,12 +50,13 @@ impl ChainStore {
 	}
 }

-#[allow(missing_docs)]
 impl ChainStore {
+	/// The current chain head.
 	pub fn head(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![HEAD_PREFIX]), "HEAD")
 	}

+	/// The current chain "tail" (earliest block in the store).
 	pub fn tail(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![TAIL_PREFIX]), "TAIL")
 	}
@@ -70,10 +71,12 @@ impl ChainStore {
 		option_to_not_found(self.db.get_ser(&vec![HEADER_HEAD_PREFIX]), "HEADER_HEAD")
 	}

+	/// The "sync" head.
 	pub fn get_sync_head(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![SYNC_HEAD_PREFIX]), "SYNC_HEAD")
 	}

+	/// Get full block.
 	pub fn get_block(&self, h: &Hash) -> Result<Block, Error> {
 		option_to_not_found(
 			self.db.get_ser(&to_key(BLOCK_PREFIX, &mut h.to_vec())),
@@ -81,10 +84,12 @@ impl ChainStore {
 		)
 	}

+	/// Does this full block exist?
 	pub fn block_exists(&self, h: &Hash) -> Result<bool, Error> {
 		self.db.exists(&to_key(BLOCK_PREFIX, &mut h.to_vec()))
 	}

+	/// Get block_sums for the block hash.
 	pub fn get_block_sums(&self, h: &Hash) -> Result<BlockSums, Error> {
 		option_to_not_found(
 			self.db.get_ser(&to_key(BLOCK_SUMS_PREFIX, &mut h.to_vec())),
@@ -92,10 +97,12 @@ impl ChainStore {
 		)
 	}

+	/// Get previous header.
 	pub fn get_previous_header(&self, header: &BlockHeader) -> Result<BlockHeader, Error> {
 		self.get_block_header(&header.prev_hash)
 	}

+	/// Get block header.
 	pub fn get_block_header(&self, h: &Hash) -> Result<BlockHeader, Error> {
 		option_to_not_found(
 			self.db
@@ -104,6 +111,7 @@ impl ChainStore {
 		)
 	}

+	/// Get PMMR pos for the given output commitment.
 	pub fn get_output_pos(&self, commit: &Commitment) -> Result<u64, Error> {
 		option_to_not_found(
 			self.db
@@ -126,12 +134,13 @@ pub struct Batch<'a> {
 	db: store::Batch<'a>,
 }

-#[allow(missing_docs)]
 impl<'a> Batch<'a> {
+	/// The head.
 	pub fn head(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![HEAD_PREFIX]), "HEAD")
 	}

+	/// The tail.
 	pub fn tail(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![TAIL_PREFIX]), "TAIL")
 	}
@@ -146,27 +155,33 @@ impl<'a> Batch<'a> {
 		option_to_not_found(self.db.get_ser(&vec![HEADER_HEAD_PREFIX]), "HEADER_HEAD")
 	}

+	/// Get "sync" head.
 	pub fn get_sync_head(&self) -> Result<Tip, Error> {
 		option_to_not_found(self.db.get_ser(&vec![SYNC_HEAD_PREFIX]), "SYNC_HEAD")
 	}

+	/// Save head to db.
 	pub fn save_head(&self, t: &Tip) -> Result<(), Error> {
 		self.db.put_ser(&vec![HEAD_PREFIX], t)?;
 		self.db.put_ser(&vec![HEADER_HEAD_PREFIX], t)
 	}

+	/// Save body head to db.
 	pub fn save_body_head(&self, t: &Tip) -> Result<(), Error> {
 		self.db.put_ser(&vec![HEAD_PREFIX], t)
 	}

+	/// Save body "tail" to db.
 	pub fn save_body_tail(&self, t: &Tip) -> Result<(), Error> {
 		self.db.put_ser(&vec![TAIL_PREFIX], t)
 	}

+	/// Save header_head to db.
 	pub fn save_header_head(&self, t: &Tip) -> Result<(), Error> {
 		self.db.put_ser(&vec![HEADER_HEAD_PREFIX], t)
 	}

+	/// Save "sync" head to db.
 	pub fn save_sync_head(&self, t: &Tip) -> Result<(), Error> {
 		self.db.put_ser(&vec![SYNC_HEAD_PREFIX], t)
 	}
@@ -191,6 +206,7 @@ impl<'a> Batch<'a> {
 		)
 	}

+	/// Does the block exist?
 	pub fn block_exists(&self, h: &Hash) -> Result<bool, Error> {
 		self.db.exists(&to_key(BLOCK_PREFIX, &mut h.to_vec()))
 	}
@@ -224,6 +240,7 @@ impl<'a> Batch<'a> {
 		Ok(())
 	}

+	/// Save block header to db.
 	pub fn save_block_header(&self, header: &BlockHeader) -> Result<(), Error> {
 		let hash = header.hash();

@@ -234,6 +251,7 @@ impl<'a> Batch<'a> {
 		Ok(())
 	}

+	/// Save output_pos to index.
 	pub fn save_output_pos(&self, commit: &Commitment, pos: u64) -> Result<(), Error> {
 		self.db.put_ser(
 			&to_key(COMMIT_POS_PREFIX, &mut commit.as_ref().to_vec())[..],
@@ -241,6 +259,7 @@ impl<'a> Batch<'a> {
 		)
 	}

+	/// Get output_pos from index.
 	pub fn get_output_pos(&self, commit: &Commitment) -> Result<u64, Error> {
 		option_to_not_found(
 			self.db
@@ -249,15 +268,21 @@ impl<'a> Batch<'a> {
 		)
 	}

-	pub fn delete_output_pos(&self, commit: &[u8]) -> Result<(), Error> {
-		self.db
-			.delete(&to_key(COMMIT_POS_PREFIX, &mut commit.to_vec()))
+	/// Clear all entries from the output_pos index (must be rebuilt after).
+	pub fn clear_output_pos(&self) -> Result<(), Error> {
+		let key = to_key(COMMIT_POS_PREFIX, &mut "".to_string().into_bytes());
+		for (k, _) in self.db.iter::<u64>(&key)? {
+			self.db.delete(&k)?;
+		}
+		Ok(())
 	}

+	/// Get the previous header.
 	pub fn get_previous_header(&self, header: &BlockHeader) -> Result<BlockHeader, Error> {
 		self.get_block_header(&header.prev_hash)
 	}

+	/// Get block header.
 	pub fn get_block_header(&self, h: &Hash) -> Result<BlockHeader, Error> {
 		option_to_not_found(
 			self.db
@@ -266,6 +291,7 @@ impl<'a> Batch<'a> {
 		)
 	}

+	/// Save the input bitmap for the block.
 	fn save_block_input_bitmap(&self, bh: &Hash, bm: &Bitmap) -> Result<(), Error> {
 		self.db.put(
 			&to_key(BLOCK_INPUT_BITMAP_PREFIX, &mut bh.to_vec())[..],
@@ -273,16 +299,19 @@ impl<'a> Batch<'a> {
 		)
 	}

+	/// Delete the block input bitmap.
 	fn delete_block_input_bitmap(&self, bh: &Hash) -> Result<(), Error> {
 		self.db
 			.delete(&to_key(BLOCK_INPUT_BITMAP_PREFIX, &mut bh.to_vec()))
 	}

+	/// Save block_sums for the block.
 	pub fn save_block_sums(&self, h: &Hash, sums: &BlockSums) -> Result<(), Error> {
 		self.db
 			.put_ser(&to_key(BLOCK_SUMS_PREFIX, &mut h.to_vec())[..], &sums)
 	}

+	/// Get block_sums for the block.
 	pub fn get_block_sums(&self, h: &Hash) -> Result<BlockSums, Error> {
 		option_to_not_found(
 			self.db.get_ser(&to_key(BLOCK_SUMS_PREFIX, &mut h.to_vec())),
@@ -290,10 +319,12 @@ impl<'a> Batch<'a> {
 		)
 	}

+	/// Delete the block_sums for the block.
 	fn delete_block_sums(&self, bh: &Hash) -> Result<(), Error> {
 		self.db.delete(&to_key(BLOCK_SUMS_PREFIX, &mut bh.to_vec()))
 	}

+	/// Build the input bitmap for the given block.
 	fn build_block_input_bitmap(&self, block: &Block) -> Result<Bitmap, Error> {
 		let bitmap = block
 			.inputs()
@@ -304,6 +335,7 @@ impl<'a> Batch<'a> {
 		Ok(bitmap)
 	}

+	/// Build and store the input bitmap for the given block.
 	fn build_and_store_block_input_bitmap(&self, block: &Block) -> Result<Bitmap, Error> {
 		// Build the bitmap.
 		let bitmap = self.build_block_input_bitmap(block)?;
@@ -314,8 +346,8 @@ impl<'a> Batch<'a> {
 		Ok(bitmap)
 	}

-	// Get the block input bitmap from the db or build the bitmap from
-	// the full block from the db (if the block is found).
+	/// Get the block input bitmap from the db or build the bitmap from
+	/// the full block from the db (if the block is found).
 	pub fn get_block_input_bitmap(&self, bh: &Hash) -> Result<Bitmap, Error> {
 		if let Ok(Some(bytes)) = self
 			.db
@@ -32,13 +32,12 @@ use crate::util::secp::pedersen::{Commitment, RangeProof};
 use crate::util::{file, secp_static, zip};
 use croaring::Bitmap;
 use grin_store;
-use grin_store::pmmr::{clean_files_by_prefix, PMMRBackend, PMMR_FILES};
-use grin_store::types::prune_noop;
+use grin_store::pmmr::{PMMRBackend, PMMR_FILES};
 use std::collections::HashSet;
 use std::fs::{self, File};
 use std::path::{Path, PathBuf};
 use std::sync::Arc;
-use std::time::Instant;
+use std::time::{Instant, SystemTime, UNIX_EPOCH};

 const HEADERHASHSET_SUBDIR: &'static str = "header";
 const TXHASHSET_SUBDIR: &'static str = "txhashset";
@@ -67,7 +66,10 @@ impl<T: PMMRable> PMMRHandle<T> {
 	) -> Result<PMMRHandle<T>, Error> {
 		let path = Path::new(root_dir).join(sub_dir).join(file_name);
 		fs::create_dir_all(path.clone())?;
-		let backend = PMMRBackend::new(path.to_str().unwrap().to_string(), prunable, header)?;
+		let path_str = path.to_str().ok_or(Error::from(ErrorKind::Other(
+			"invalid file path".to_owned(),
+		)))?;
+		let backend = PMMRBackend::new(path_str.to_string(), prunable, header)?;
 		let last_pos = backend.unpruned_size();
 		Ok(PMMRHandle { backend, last_pos })
 	}
@@ -206,21 +208,27 @@ impl TxHashSet {
 			.get_last_n_insertions(distance)
 	}

-	/// Get the header at the specified height based on the current state of the txhashset.
-	/// Derives the MMR pos from the height (insertion index) and retrieves the header hash.
-	/// Looks the header up in the db by hash.
-	pub fn get_header_by_height(&self, height: u64) -> Result<BlockHeader, Error> {
+	/// Get the header hash at the specified height based on the current state of the txhashset.
+	pub fn get_header_hash_by_height(&self, height: u64) -> Result<Hash, Error> {
 		let pos = pmmr::insertion_to_pmmr_index(height + 1);
 		let header_pmmr =
 			ReadonlyPMMR::at(&self.header_pmmr_h.backend, self.header_pmmr_h.last_pos);
 		if let Some(entry) = header_pmmr.get_data(pos) {
-			let header = self.commit_index.get_block_header(&entry.hash())?;
-			Ok(header)
+			Ok(entry.hash())
 		} else {
-			Err(ErrorKind::Other(format!("get header by height")).into())
+			Err(ErrorKind::Other(format!("get header hash by height")).into())
 		}
 	}

+	/// Get the header at the specified height based on the current state of the txhashset.
+	/// Derives the MMR pos from the height (insertion index) and retrieves the header hash.
+	/// Looks the header up in the db by hash.
+	pub fn get_header_by_height(&self, height: u64) -> Result<BlockHeader, Error> {
+		let hash = self.get_header_hash_by_height(height)?;
+		let header = self.commit_index.get_block_header(&hash)?;
+		Ok(header)
+	}
+
 	/// returns outputs from the given insertion (leaf) index up to the
 	/// specified limit. Also returns the last index actually populated
 	pub fn outputs_by_insertion_index(
@@ -280,39 +288,30 @@ impl TxHashSet {
 	}

 	/// Compact the MMR data files and flush the rm logs
-	pub fn compact(&mut self) -> Result<(), Error> {
-		let commit_index = self.commit_index.clone();
-		let head_header = commit_index.head_header()?;
+	pub fn compact(&mut self, batch: &mut Batch<'_>) -> Result<(), Error> {
+		debug!("txhashset: starting compaction...");
+
+		let head_header = batch.head_header()?;
 		let current_height = head_header.height;

 		// horizon for compacting is based on current_height
-		let horizon = current_height.saturating_sub(global::cut_through_horizon().into());
-		let horizon_header = self.get_header_by_height(horizon)?;
+		let horizon_height = current_height.saturating_sub(global::cut_through_horizon().into());
+		let horizon_hash = self.get_header_hash_by_height(horizon_height)?;
+		let horizon_header = batch.get_block_header(&horizon_hash)?;

-		let batch = self.commit_index.batch()?;
+		let rewind_rm_pos = input_pos_to_rewind(&horizon_header, &head_header, batch)?;

-		let rewind_rm_pos = input_pos_to_rewind(&horizon_header, &head_header, &batch)?;
+		debug!("txhashset: check_compact output mmr backend...");
+		self.output_pmmr_h
+			.backend
+			.check_compact(horizon_header.output_mmr_size, &rewind_rm_pos)?;

-		{
-			let clean_output_index = |commit: &[u8]| {
-				let _ = batch.delete_output_pos(commit);
-			};
+		debug!("txhashset: check_compact rangeproof mmr backend...");
+		self.rproof_pmmr_h
+			.backend
+			.check_compact(horizon_header.output_mmr_size, &rewind_rm_pos)?;

-			self.output_pmmr_h.backend.check_compact(
-				horizon_header.output_mmr_size,
-				&rewind_rm_pos,
-				clean_output_index,
-			)?;
-
-			self.rproof_pmmr_h.backend.check_compact(
-				horizon_header.output_mmr_size,
-				&rewind_rm_pos,
-				&prune_noop,
-			)?;
-		}
-
-		// Finally commit the batch, saving everything to the db.
-		batch.commit()?;
+		debug!("txhashset: ... compaction finished");

 		Ok(())
 	}
@@ -797,11 +796,9 @@ impl<'a> Committed for Extension<'a> {

 	fn outputs_committed(&self) -> Vec<Commitment> {
 		let mut commitments = vec![];
-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			if pmmr::is_leaf(n) {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					commitments.push(out.commit);
-				}
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			if let Some(out) = self.output_pmmr.get_data(pos) {
+				commitments.push(out.commit);
 			}
 		}
 		commitments
@@ -1259,20 +1256,18 @@ impl<'a> Extension<'a> {
 	pub fn rebuild_index(&self) -> Result<(), Error> {
 		let now = Instant::now();

-		let mut count = 0;
+		self.batch.clear_output_pos()?;

-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			// non-pruned leaves only
-			if pmmr::bintree_postorder_height(n) == 0 {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					self.batch.save_output_pos(&out.commit, n)?;
-					count += 1;
-				}
+		let mut count = 0;
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			if let Some(out) = self.output_pmmr.get_data(pos) {
+				self.batch.save_output_pos(&out.commit, pos)?;
+				count += 1;
 			}
 		}

 		debug!(
-			"txhashset: rebuild_index ({} UTXOs), took {}s",
+			"txhashset: rebuild_index: {} UTXOs, took {}s",
 			count,
 			now.elapsed().as_secs(),
 		);
@@ -1325,13 +1320,23 @@ impl<'a> Extension<'a> {
 		let total_kernels = pmmr::n_leaves(self.kernel_pmmr.unpruned_size());
 		for n in 1..self.kernel_pmmr.unpruned_size() + 1 {
 			if pmmr::is_leaf(n) {
-				if let Some(kernel) = self.kernel_pmmr.get_data(n) {
-					kernel.verify()?;
-					kern_count += 1;
+				let kernel = self
+					.kernel_pmmr
+					.get_data(n)
+					.ok_or::<Error>(ErrorKind::TxKernelNotFound.into())?;
+
+				kernel.verify()?;
+				kern_count += 1;
+
+				if kern_count % 20 == 0 {
+					status.on_validation(kern_count, total_kernels, 0, 0);
+				}
+				if kern_count % 1_000 == 0 {
+					debug!(
+						"txhashset: verify_kernel_signatures: verified {} signatures",
+						kern_count,
+					);
 				}
-			}
-			if n % 20 == 0 {
-				status.on_validation(kern_count, total_kernels, 0, 0);
 			}
 		}

@@ -1353,30 +1358,34 @@ impl<'a> Extension<'a> {

 		let mut proof_count = 0;
 		let total_rproofs = pmmr::n_leaves(self.output_pmmr.unpruned_size());
-		for n in 1..self.output_pmmr.unpruned_size() + 1 {
-			if pmmr::is_leaf(n) {
-				if let Some(out) = self.output_pmmr.get_data(n) {
-					if let Some(rp) = self.rproof_pmmr.get_data(n) {
-						commits.push(out.commit);
-						proofs.push(rp);
-					} else {
-						// TODO - rangeproof not found
-						return Err(ErrorKind::OutputNotFound.into());
-					}
-					proof_count += 1;
+		for pos in self.output_pmmr.leaf_pos_iter() {
+			let output = self.output_pmmr.get_data(pos);
+			let proof = self.rproof_pmmr.get_data(pos);

-					if proofs.len() >= 1000 {
-						Output::batch_verify_proofs(&commits, &proofs)?;
-						commits.clear();
-						proofs.clear();
-						debug!(
-							"txhashset: verify_rangeproofs: verified {} rangeproofs",
-							proof_count,
-						);
-					}
+			// Output and corresponding rangeproof *must* exist.
+			// It is invalid for either to be missing and we fail immediately in this case.
+			match (output, proof) {
+				(None, _) => return Err(ErrorKind::OutputNotFound.into()),
+				(_, None) => return Err(ErrorKind::RangeproofNotFound.into()),
+				(Some(output), Some(proof)) => {
+					commits.push(output.commit);
+					proofs.push(proof);
 				}
 			}
-			if n % 20 == 0 {
+
+			proof_count += 1;
+
+			if proofs.len() >= 1_000 {
+				Output::batch_verify_proofs(&commits, &proofs)?;
+				commits.clear();
+				proofs.clear();
+				debug!(
+					"txhashset: verify_rangeproofs: verified {} rangeproofs",
+					proof_count,
+				);
+			}
+
+			if proof_count % 20 == 0 {
 				status.on_validation(0, 0, proof_count, total_rproofs);
 			}
 		}
@@ -1404,38 +1413,22 @@ impl<'a> Extension<'a> {

 /// Packages the txhashset data files into a zip and returns a Read to the
 /// resulting file
-pub fn zip_read(root_dir: String, header: &BlockHeader) -> Result<File, Error> {
-	let txhashset_zip = format!("{}_{}.zip", TXHASHSET_ZIP, header.hash().to_string());
+pub fn zip_read(root_dir: String, header: &BlockHeader, rand: Option<u32>) -> Result<File, Error> {
+	let ts = if let None = rand {
+		let now = SystemTime::now();
+		now.duration_since(UNIX_EPOCH).unwrap().subsec_micros()
+	} else {
+		rand.unwrap()
+	};
+	let txhashset_zip = format!("{}_{}.zip", TXHASHSET_ZIP, ts);

 	let txhashset_path = Path::new(&root_dir).join(TXHASHSET_SUBDIR);
 	let zip_path = Path::new(&root_dir).join(txhashset_zip);
-
-	// if file exist, just re-use it
-	let zip_file = File::open(zip_path.clone());
-	if let Ok(zip) = zip_file {
-		return Ok(zip);
-	} else {
-		// clean up old zips.
-		// Theoretically, we only need clean-up those zip files older than STATE_SYNC_THRESHOLD.
-		// But practically, these zip files are not small ones, we just keep the zips in last one hour
-		let data_dir = Path::new(&root_dir);
-		let pattern = format!("{}_", TXHASHSET_ZIP);
-		if let Ok(n) = clean_files_by_prefix(data_dir.clone(), &pattern, 60 * 60) {
-			debug!(
-				"{} zip files have been clean up in folder: {:?}",
-				n, data_dir
-			);
-		}
-	}
-
-	// otherwise, create the zip archive
-	let path_to_be_cleanup = {
+	// create the zip archive
+	{
 		// Temp txhashset directory
-		let temp_txhashset_path = Path::new(&root_dir).join(format!(
-			"{}_zip_{}",
-			TXHASHSET_SUBDIR,
-			header.hash().to_string()
-		));
+		let temp_txhashset_path =
+			Path::new(&root_dir).join(format!("{}_zip_{}", TXHASHSET_SUBDIR, ts));
 		// Remove temp dir if it exist
 		if temp_txhashset_path.exists() {
 			fs::remove_dir_all(&temp_txhashset_path)?;
@@ -1447,38 +1440,70 @@ pub fn zip_read(root_dir: String, header: &BlockHeader) -> Result<File, Error> {
 		// Compress zip
 		zip::compress(&temp_txhashset_path, &File::create(zip_path.clone())?)
 			.map_err(|ze| ErrorKind::Other(ze.to_string()))?;
-
-		temp_txhashset_path
-	};
+	}

 	// open it again to read it back
-	let zip_file = File::open(zip_path.clone())?;
-
-	// clean-up temp txhashset directory.
-	if let Err(e) = fs::remove_dir_all(&path_to_be_cleanup) {
-		warn!(
-			"txhashset zip file: {:?} fail to remove, err: {}",
-			zip_path.to_str(),
-			e
-		);
-	}
+	let zip_file = File::open(zip_path)?;
 	Ok(zip_file)
 }

 /// Extract the txhashset data from a zip file and writes the content into the
 /// txhashset storage dir
 pub fn zip_write(
-	root_dir: String,
+	root_dir: PathBuf,
 	txhashset_data: File,
 	header: &BlockHeader,
 ) -> Result<(), Error> {
-	let txhashset_path = Path::new(&root_dir).join(TXHASHSET_SUBDIR);
+	debug!("zip_write on path: {:?}", root_dir);
+	let txhashset_path = root_dir.clone().join(TXHASHSET_SUBDIR);
 	fs::create_dir_all(txhashset_path.clone())?;
 	zip::decompress(txhashset_data, &txhashset_path, expected_file)
 		.map_err(|ze| ErrorKind::Other(ze.to_string()))?;
 	check_and_remove_files(&txhashset_path, header)
 }

+/// Overwrite txhashset folders in "to" folder with "from" folder
+pub fn txhashset_replace(from: PathBuf, to: PathBuf) -> Result<(), Error> {
+	debug!("txhashset_replace: move from {:?} to {:?}", from, to);
+
+	// clean the 'to' folder firstly
+	clean_txhashset_folder(&to);
+
+	// rename the 'from' folder as the 'to' folder
+	if let Err(e) = fs::rename(
+		from.clone().join(TXHASHSET_SUBDIR),
+		to.clone().join(TXHASHSET_SUBDIR),
+	) {
+		error!("hashset_replace fail on {}. err: {}", TXHASHSET_SUBDIR, e);
+		Err(ErrorKind::TxHashSetErr(format!("txhashset replacing fail")).into())
+	} else {
+		Ok(())
+	}
+}
+
+/// Clean the txhashset folder
+pub fn clean_txhashset_folder(root_dir: &PathBuf) {
+	let txhashset_path = root_dir.clone().join(TXHASHSET_SUBDIR);
+	if txhashset_path.exists() {
+		if let Err(e) = fs::remove_dir_all(txhashset_path.clone()) {
+			warn!(
+				"clean_txhashset_folder: fail on {:?}. err: {}",
+				txhashset_path, e
+			);
+		}
+	}
+}
+
+/// Clean the header folder
+pub fn clean_header_folder(root_dir: &PathBuf) {
+	let header_path = root_dir.clone().join(HEADERHASHSET_SUBDIR);
+	if header_path.exists() {
+		if let Err(e) = fs::remove_dir_all(header_path.clone()) {
+			warn!("clean_header_folder: fail on {:?}. err: {}", header_path, e);
+		}
+	}
+}
+
 fn expected_file(path: &Path) -> bool {
 	use lazy_static::lazy_static;
 	use regex::Regex;
@@ -1491,7 +1516,7 @@ fn expected_file(path: &Path) -> bool {
 			)
 			.as_str()
 		)
-		.unwrap();
+		.expect("invalid txhashset regular expression");
 	}
 	RE.is_match(&s_path)
 }
@@ -1530,7 +1555,7 @@ fn check_and_remove_files(txhashset_path: &PathBuf, header: &BlockHeader) -> Res
 	}

 	// Then compare the files found in the subdirectories
-	let mut pmmr_files_expected: HashSet<_> = PMMR_FILES
+	let pmmr_files_expected: HashSet<_> = PMMR_FILES
 		.iter()
 		.cloned()
 		.map(|s| {
@@ -1541,8 +1566,6 @@ fn check_and_remove_files(txhashset_path: &PathBuf, header: &BlockHeader) -> Res
 			}
 		})
 		.collect();
-	// prevent checker from deleting 3 dot file, could be removed after mainnet
-	pmmr_files_expected.insert(format!("pmmr_leaf.bin.{}...", header.hash()));

 	let subdirectories = fs::read_dir(txhashset_path)?;
 	for subdirectory in subdirectories {