From a002ddda8d9003e0ef20710c7be0fc97ec11cbfa Mon Sep 17 00:00:00 2001 From: serinko <97586125+serinko@users.noreply.github.com> Date: Thu, 27 Jul 2023 12:41:22 +0200 Subject: [PATCH] operators-guide:troubleshooting.md done - content ready to go out --- documentation/operators/src/faq.md | 3 +- .../operators/src/nodes/maintenance.md | 41 ++- .../operators/src/nodes/mix-node-setup.md | 10 +- .../operators/src/nodes/troubleshooting.md | 325 +++++++++++++++++- 4 files changed, 365 insertions(+), 14 deletions(-) diff --git a/documentation/operators/src/faq.md b/documentation/operators/src/faq.md index e09a78c21c..88836f17f2 100644 --- a/documentation/operators/src/faq.md +++ b/documentation/operators/src/faq.md @@ -3,7 +3,8 @@ TODO - [ ] Explain active set and needed metrics - [ ] List friendly VPS providers - [ ] Explain difficulty running a node on a self-hosted machine -- [ ] Explain uNYM vs NYM. Why, ratio, etc +- [ ] Explain uNYM vs NYM/uNYX vs NYX. Why, ratio, etc +- [ ] Explain why validators set is whitelisted Some community questions: Hi, do you know if it's on the roadmap for network nodes (requesters, socks5 client, etc.) to reconnect to another gateway if the current gateway's connection drops or has issues? I have to reset my requester and socks5 clients to connect them to a new gateway almost every day...๐Ÿ˜• having to delete the entire folders associated with them diff --git a/documentation/operators/src/nodes/maintenance.md b/documentation/operators/src/nodes/maintenance.md index d4884bad84..d809027150 100644 --- a/documentation/operators/src/nodes/maintenance.md +++ b/documentation/operators/src/nodes/maintenance.md @@ -3,6 +3,11 @@ ## Useful commands @@ -107,6 +112,32 @@ Running the command `df -H` will return the size of the various partitions of yo If the `/dev/sda` partition is almost full, try pruning some of the `.gz` syslog archives and restart your validator process. +## Moving a node + +In case of a need to move a node from one machine to another and avoiding to lose the delegation, here are few steps how to do it. + +The following examples transfers a mixnode (in case of other nodes, change the `mixnodes` in the command for the `` of your desire. + +* Pause your node process. + +Assuming both machines are remote VPS. + +* Make sure your `~/.ssh/.pub` is in both of the machines `~/.ssh/authorized_keys` file +* Create a `mixnodes` folder in the target VPS. Ssh in from your terminal and run: +``` +# in case none of the nym configs was created previously +mkdir ~/.nym + +#in case no nym mixnode was initialized previously +mkdir ~/.nym/mixnodes +``` +* Move the node data (keys) and config file to the new machine by opening a local terminal (as that one's ssh key is authorized in both of the machines) and running: +``` +scp -r @:~/.nym/mixnodes/ @:~/.nym/mixnodes/ +``` +* Re-run init (remember that init doesn't overwrite existing keys) to generate a config with the new listening address etc. +* Change the node smart contract info via the wallet interface. Otherwise the keys will point to the old IP address in the smart contract, and the node will not be able to be connected, and it will fail up-time checks. +* Re-run the node from the new location. ## VPS Setup and Automation ### Configure your firewall @@ -149,10 +180,18 @@ sudo ufw status For more information about your node's port configuration, check the [port reference table](https://nymtech.net/docs/nodes/gateway-setup.html#ports) below. -### Automating your node with tmux and systemd +### Automating your node with nohup, tmux and systemd Although itโ€™s not totally necessary, it's useful to have the mix node automatically start at system boot time. +#### nohup + +`nohup` is a command with which your terminal is told to ignore the `HUP` or 'hangup' signal. This will stop the node process ending if you kill your session. + +``` +nohup ./ run --id # where `` is the id you set during the `init` command and depends on which node you starting +``` + #### tmux One way is to use `tmux` shell on top of your current VPS terminal. Tmux is a terminal multiplexer, it allows you to create several terminal windows and panes from a single terminal. Processes started in `tmux` keep running after closing the terminal as long as the given `tmux` window was not terminated. diff --git a/documentation/operators/src/nodes/mix-node-setup.md b/documentation/operators/src/nodes/mix-node-setup.md index 8797c13056..7e1cafa64c 100644 --- a/documentation/operators/src/nodes/mix-node-setup.md +++ b/documentation/operators/src/nodes/mix-node-setup.md @@ -8,13 +8,7 @@ TODO - [x] Add step 2/3 in the wallet bonding - [x] Better picture alignment. - [x] Add second console output to Initialising your mixnode manually as the automatized is not working -- [ ] Point users to understand the requirements for becoming a part of the active set. -- [ ] Include how to move a mix node from one VPS to another while keeping the delegation etc (ie init a new mix node and move `/home/nym/.nym/mixnodes/data/*` there ?) - - copy that dir over to the new vps - - re-run init - remember that init doesn't overwrite existing keys - to generate a config with the new listening address etc - - change the node info via the wallet - - Since otherwise their keys will point to the old IP address in the smart contract, and their node will (a) not be able to be connected to, and (b) fail uptime checks -- [ ] Explain more about self hosted nodes in the 6.1 not only as a troubleshooting +- [x] Explain more about self hosted nodes in the 6.1 not only as a troubleshooting - [x] Move the upgrade, automation, ulimit, vps, API and port parts to maintenance.md ---> @@ -327,7 +321,7 @@ There are also 2 community explorers which have been created by [Nodes Guru](htt For more details see [Troubleshooting FAQ](../nodes/troubleshooting.md) - + ## Maintenance diff --git a/documentation/operators/src/nodes/troubleshooting.md b/documentation/operators/src/nodes/troubleshooting.md index b27a417063..e5333bcfe4 100644 --- a/documentation/operators/src/nodes/troubleshooting.md +++ b/documentation/operators/src/nodes/troubleshooting.md @@ -1,13 +1,323 @@ - - # Troubleshooting +## Binary Build Problems + +### I am trying to build from the GitHub archive files and the build fails + +GitHub automatically includes .zip and tar.gz files of the Nym repository in its release. You cannot extract these and build - you'll see something like this: + +``` + process didn't exit successfully: `/build/nym/src/nym-0.12.1/target/release/build/nym-socks5-client-c1d0f76a8c7d7e9a/build-script-build` (exit status: 101) + --- stderr + thread 'main' panicked at 'failed to extract build metadata: could not find repository from '/build/nym/src/nym-0.12.1/clients/socks5'; class=Repository (6); code=NotFound (-3)', clients/socks5/build.rs:7:31 + note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace +warning: build failed, waiting for other jobs to finish... +error: build failed +``` + +Why does this happen? + +We have scripts which automatically include the Git commit hash and Git tag in the binary for easier debugging later. If you download a .zip and try building from that, it's not a Git repository and build will fail as above. + +What to do? + +* Open terminal in the directory where you want to have a git repository +* To get Nym repository for the first time, run: +``` +git clone https://github.com/nymtech/nym.git +``` +* Follow the instructions to build the platform +* To upgrade, pause your nodes, in the same terminal window run `git pull`, follow the upgrade instructions and re-start your nodes. + +## General Node Config + +### Where can I find my private and public keys and config? + +All config and keys files are stored in a directory named after your `id` which you chose during the `init` process, and can be found at the following PATH: `$HOME/.nym//` where `$HOME` is a home directory of the user (your current user in this case) that launched the node or client. + +The directory structure for each node will be roughly as follows: + +``` +bob@nym:~$ tree /home/nym/.nym/mixnodes/ +/home/nym/.nym/mixnodes/ +|-- nym010 +| |-- config +| | `-- config.toml +| `-- data +| |-- private_identity.pem +| |-- private_sphinx.pem +| |-- public_identity.pem +| `-- public_sphinx.pem +``` + +> If you `cat` the `public_sphinx.pem` key, the output will be different from the public key you will see on Nym [dashboard](https://sandbox-explorer.nymtech.net/). The reason for this is that `.pem` files are encoded in **base64**, however on the web they are in **base58**. Don't be confused if your keys look different. They are the same keys, just with different encoding :) + + +## Mix Nodes + +### How can I tell my node is up and running and mixing traffic? + +First of all check the 'Mixnodes' section of either the Nym Network Explorers: +* [Mainnet](https://explorer.nymtech.net/) +* [Sandbox testnet](https://sandbox-explorer.nymtech.net/) + +Enter your **identity key** to find your node. Check the contents of the `Mixnode stats` and `Routing score` sections. + +There are 2 community explorers currently, which have been created by [Nodes Guru](https://nodes.guru): +* [Mainnet](https://mixnet.explorers.guru/) +* [Sandbox testnet](https://sandbox.mixnet.explorers.guru/) + +If you want more information, or if your node isn't showing up on the explorer of your choice and you want to double-check, here are some examples on how to check if the node is configured properly. + +#### Check from your VPS + +Additional details can be obtained via various methods after you connect to your VPS: + +##### Socket statistics with `ss` + +``` +sudo ss -s -t | grep 1789 # if you have specified a different port in your mixnode config, change accordingly +``` + +This command should return a lot of data containing `ESTAB`. This command should work on every unix based system. + +##### List open files and reliant processes with `lsof` + +``` +# check if lsof is installed: +lsof -v +# install if not installed +sudo apt install lsof +# run against mixnode port +sudo lsof -i TCP:1789 # if you have specified a different port in your mixnode config, change accordingly +``` + +This command should return something like this: + +``` +nym-mixno 103349 root 53u IPv6 1333229972 0t0 TCP [2a03:b0c0:3:d0::ff3:f001]:57844->[2a01:4f9:c011:38ae::5]:1789 (ESTABLISHED) +nym-mixno 103349 root 54u IPv4 1333229973 0t0 TCP nym:57104->194.5.78.73:1789 (ESTABLISHED) +nym-mixno 103349 root 55u IPv4 1333229974 0t0 TCP nym:48130->static.236.109.119.168.clients.your-server.de:1789 (ESTABLISHED) +nym-mixno 103349 root 56u IPv4 1333229975 0t0 TCP nym:52548->vmi572614.contaboserver.net:1789 (ESTABLISHED) +nym-mixno 103349 root 57u IPv6 1333229976 0t0 TCP [2a03:b0c0:3:d0::ff3:f001]:43244->[2600:1f18:1031:2401:c04b:2f25:ca79:fef3]:1789 (ESTABLISHED) +``` + +##### Query `systemd` journal with `journalctl` + +``` +sudo journalctl -u nym-mixnode -o cat | grep "Since startup mixed" +``` + +If you have created `nym-mixnode.service` file (i.e. you are running your mixnode via `systemd`) then this command shows you how many packets have you mixed so far, and should return a list of messages like this: + +``` +2021-05-18T12:35:24.057Z INFO nym_mixnode::node::metrics > Since startup mixed 233639 packets! +2021-05-18T12:38:02.178Z INFO nym_mixnode::node::metrics > Since startup mixed 233739 packets! +2021-05-18T12:40:32.344Z INFO nym_mixnode::node::metrics > Since startup mixed 233837 packets! +2021-05-18T12:46:08.549Z INFO nym_mixnode::node::metrics > Since startup mixed 234081 packets! +2021-05-18T12:56:57.129Z INFO nym_mixnode::node::metrics > Since startup mixed 234491 packets! +``` + +You can add ` | tail` to the end of the command to watch for new entries in real time if needed. + +##### build-info + +A `build-info` command prints the build information like commit hash, rust version, binary version just like what command `--version` does. However, you can also specify an `--output=json` flag that will format the whole output as a json, making it an order of magnitude easier to parse. + +For example `./target/debug/nym-network-requester --no-banner build-info --output json` will return: + +``` +{"binary_name":"nym-network-requester","build_timestamp":"2023-07-24T15:38:37.00657Z","build_version":"1.1.23","commit_sha":"c70149400206dce24cf20babb1e64f22202672dd","commit_timestamp":"2023-07-24T14:45:45Z","commit_branch":"feature/simplify-cli-parsing","rustc_version":"1.71.0","rustc_channel":"stable","cargo_profile":"debug"} +``` + +#### Check from your local machine + +##### Scan ports with `nmap`: + +``` +nmap -p 1789 -Pn +``` + +If your mixnode is configured properly it should output something like this: + +``` +bob@desktop:~$ nmap -p 1789 95.296.134.220 -Pn + +Host is up (0.053s latency). + +PORT STATE SERVICE +1789/tcp open hello +``` + +##### Query online nodes: + +``` +curl --location --request GET 'https://validator.nymtech.net/api/v1/mixnodes/' +``` + +Will return a list all nodes currently online. + +You can query gateways by replacing `mixnodes` with `gateways` in the above command, and can query for the mixnodes and gateways on the Sandbox testnet by replacing `validator` with `sandbox-validator`. + + +#### Check with Network API + +We currently have an API set up returning our metrics tests of the network. There are two endpoints to ping for information about your mixnode, `report` and `history`. Find more information about this in the [Mixnodes metrics documentation](./maintenance.md#metrics--api-endpoints). + +### Why is my node not mixing any packets? + +If you are still unable to see your node on the dashboard, or your node is declaring it has not mixed any packets, there are several potential issues: + +- The firewall on your host machine is not configured properly. Checkout the [instructions](./maintenance.md#configure-your-firewall). +- You provided incorrect information when bonding your node. +- You are running your mixnode from a VPS without IPv6 support. +- You did not use the `--announce-host` flag while running the mixnode from your local machine behind NAT. +- You did not configure your router firewall while running the mixnode from your local machine behind NAT, or you are lacking IPv6 support. +- Your mixnode is not running at all, it has either exited / panicked or you closed the session without making the node persistent. Check out the [instructions](./maintenance.md#automating-your-node-with-tmux-and-systemd). + +```admonish caution +Your mixnode **must speak both IPv4 and IPv6** in order to cooperate with other nodes and route traffic. This is a common reason behind many errors we are seeing among node operators, so check with your provider that your VPS is able to do this! +``` + +#### Incorrect bonding information + +Check that you have provided the correct information when bonding your mixnode in the web wallet [interface](https://sandbox-wallet.nymtech.net/). When in doubt, un-bond and then re-bond your node! + +> All delegated stake will be lost when un-bonding! However the mixnode must be operational in the first place for the delegation to have any effect. + +#### Missing `announce-host` flag + +On certain cloud providers such as AWS and Google Cloud, you need to do some additional configuration of your firewall and use `--host` with your **local ip** and `--announce-host` with the **public ip** of your mixnode host. + +If the difference between the two is unclear, contact the help desk of your VPS provider. + +#### No IPv6 connectivity + +Make sure that your VPS has IPv6 connectivity available with whatever provider you are using. + +To get all ip addresses of your host, try following commands: + +``` +hostname -i +``` + +Will return your **local ip** address. + +``` +hostname -I +``` + +Will return all of the ip addresses of your host. This output should look something like this: + +``` +bob@nym:~$ hostname -I +88.36.11.23 172.18.0.1 2a01:28:ca:102::1:641 +``` + +- The first **ipv4** is the public ip you need to use for the `--announce-host` flag. +- The second **ipv4** is the local ip you need to use for the `--host` flag. +- The 3rd output should confirm if your machine has ipv6 available. + +### Running on a local machine behind NAT with no fixed IP address + +Your ISP has to be IPv6 ready if you want to run a mixnode on your local machine. Sadly, in 2020, most of them are not and you won't get an IPv6 address by default from your ISP. Usually it is a extra paid service or they simply don't offer it. + +Before you begin, check if you have IPv6 [here](https://test-ipv6.cz/) or by running command explained in the [section above](./troubleshooting.md#no-ipv6-connectivity). If not, then don't waste your time to run a node which won't ever be able to mix any packet due to this limitation. Call your ISP and ask for IPv6, there is a plenty of it for everyone! + +If all goes well and you have IPv6 available, then you will need to `init` the mixnode with an extra flag, `--announce-host`. You will also need to edit your `config.toml` file each time your IPv4 address changes, that could be a few days or a few weeks. Check the your IPv4 in the [section above](./troubleshooting.md#no-ipv6-connectivity). + +Additional configuration on your router might also be needed to allow traffic in and out to port 1789 and IPv6 support. + +Here is a sample of the `init` command example to create the mixnode config. + +``` +./nym-mixnode init --id --host 0.0.0.0 --announce-host 85.160.12.13 +``` + +- `--host 0.0.0.0` should work every time even if your local machine IPv4 address changes. For example on Monday your router gives your machine an address `192.168.0.13` and on Wednesday, the DHCP lease will end and you will be asigned `192.168.0.14`. Using `0.0.0.0` should avoid this without having to set any static ip in your router`s configuration. + +- you can get your current IPv4 address by either using `curl ipinfo.io` if you're on MacOS or Linux or visiting [whatsmyip site](https://www.whatsmyip.org/). Simply copy it and use it as `--anounce-host` address. + +Make sure you check if your node is really mixing. We are aiming to improve the setup for operators running locally, however you may need a bit of patience to set this up from your home behind NAT. + +### Accidentally killing your node process on exiting session + +When you close your current terminal session, you need to make sure you don't kill the mixnode process! There are multiple ways on how to make it persistent even after exiting your ssh session, the easiest solution is to use `tmux` or `nohup`, and the more elegant solution is to run the node with `systemd`. Read the automation manual [here](./maintenance.md#automating-your-node-with-tmux-and-systemd). + +### Common errors and warnings + +Most of the `ERROR` and `WARN` messages in your node logs are benign - as long as your node outputs `since startup mixed X packets!` (`X` bust be > 0) in your logs (and this number increases over time), your node is mixing packets. If you want to be sure, check the Nym [dashboard](https://sandbox-explorer.nymtech.net/) or see other ways on how to check if your node is mixing properly as outlined in the section [**How can I tell my node is up and running and mixing traffic?**](./troubleshooting.md#how-can-i-tell-my-node-is-up-and-running-and-mixing-traffic?) above. + +More specific errors and warnings are covered below. + +#### `tokio runtime worker` error + +If you are running into issues with an error including the following: + +``` +thread 'tokio-runtime-worker' panicked at 'Failed to create TCP listener: Os { code: 99, kind: AddrNotAvailable, message: "Cannot assign requested address" }' +``` + +Then you need to `--announce-host ` and `--host ` on startup. This issue is addressed [above](./troubleshooting.md#missing-`announce-host`-flag) + + +### Can I use a port other than 1789 ? + +Yes! Here is what you will need to do: + +Assuming you would like to use port `1337` for your mixnode, you need to open the new port (and close the old one): + +``` +sudo ufw allow 1337 +sudo ufw deny 1789 +``` + +And then edit the mixnode's config. + +> If you want to change the port for an already running node, you need to stop the process before editing your config file. + +The config file is located at `~/.nym/mixnodes//config/config.toml`. + +For example, assuming `` was chosen to be `alice-node`: + +``` +nano ~/.nym/mixnodes/alice-node/config/config.toml +``` + +You will need to edit two parts of the file. `announce_address` and `listening_address` in the config.toml file. Simply replace `:1789` (the default port) with `:1337` (your new port) after your IP address. + +Finally, restart your node. You should see if the mixnode is using the port you have changed in the config.toml file right after you run the node. + +### What is `verloc` and do I have to configure my mixnode to implement it? + +`verloc` is short for _verifiable location_. Mixnodes and gateways now measure speed-of-light distances to each other, in an attempt to verify how far apart they are. In later releases, this will allow us to algorithmically verify node locations in a non-fakeable and trustworthy manner. + +You don't have to do any additional configuration for your node to implement this, it is a passive process that runs in the background of the mixnet from version `0.10.1` onward. + +## Validators + ### Common reasons for your validator being jailed The most common reason for your validator being jailed is that your validator is out of memory because of bloated syslogs. @@ -16,3 +326,10 @@ Running the command `df -H` will return the size of the various partitions of yo If the `/dev/sda` partition is almost full, try pruning some of the `.gz` syslog archives and restart your validator process. + +## Where can I get more help? + +The fastest way to reach one of us or get a help from the community, visit our [Telegram Node Setup Help Chat](https://t.me/nymchan_help_chat) or head to our [Discord](https://Discord.gg/nym). + +For more tech heavy question join our [Matrix core community channel](https://matrix.to/#/#general:nymtech.chat), where you can meet other builders and Nym core team members. +