nym/documentation/docs/pages/operators/orchestration/ansible.mdx

import { Callout } from 'nextra/components';
import { Tabs } from 'nextra/components';
import { MyTab } from 'components/generic-tabs.tsx';
import { Steps } from 'nextra/components';
import { RunTabs } from 'components/operators/nodes/node-run-command-tabs';
import { VarInfo } from 'components/variable-info.tsx';
import { AccordionTemplate } from 'components/accordion-template.tsx';

# Orchestrating Nym Nodes with Ansible

<VarInfo />

[Ansible](https://docs.ansible.com/) is an open-source automation engine that can perform IT tasks and remove complexity from workflows. Ansible ensures that your environment is exactly as you describe it. You can automate any command with Ansible to make your system maintenance very efficient. **For `nym-node` operators Ansible is particularly useful as it can scale infinitely the amount of nodes operators can setup, bond, upgrade, maintain and re-configure from their local shell, removing the complexity and required time when managing many nodes one by one.**


<Callout type="warning" emoji="⚠️">
**This setup should be used only by operators who understand `nym-node` administration and [requirements](/operators/nodes#minimum-requirements)**

**Ansible is more suitable for skilled power users managing multiple nodes at the same time!** 
</Callout>

If you are not familiar with Ansible, operating Nym nodes may be a good motivation to learn something new and improve your admin skills, it's worth the time. 

Start by reading through [Ansible documentation pages](https://docs.ansible.com)


## Installation

### Ansible installation

For anything regarding the installation and management of Ansible itself, the best is to refer to their documentation. On [this page](https://docs.ansible.com/projects/ansible/latest/installation_guide/intro_installation.html#latest-release-via-dnf-or-yum) you can see the installation guide. 

If you are confident and want to start right away, install Ansible on your machine using one of these two ways:

1. `apt` repository:
```sh
sudo apt-get update
sudo apt-get install ansible
```
2. `pip` or `pipx` - recommended by Ansible community:
```sh
pip install ansible
# or
pipx install ansible
```

### Nym Node Playbook Installation

Nym Node Ansible playbook template is located in our monorepo [`nymtech/nym/ansible/nym-node`](https://github.com/nymtech/nym/tree/develop/ansible/nym-node)

<Steps>
###### 1. Get `nym/ansible/nym-node` playbook:

The easiest way is to use `git` to `clone` or `pull` the repository:

```sh
git clone https://github.com/nymtech/nym.git

# or navigate where you already have the repo and run

git checkout develop
git pull origin develop
```

###### 2. Save the template to your location:

You may want to create a directory outside of the repository and move the template there so it can be modified without risking that your configuration will be accidentally shared when working with the repository in the future. 

- Navigate to any location and create a directory for your Ansible `nym-node` playbook:
```sh
cd <PATH>
mkdir `ansible`
cd ansible
```

- Copy the template to the newly created location:
```sh
cp -r <PATH>/nym/ansible/nym-node ./
```

</ Steps>

Now you have the template of Ansible playbook for `nym-node` remote administration. To make it work, there are a few variables requiring your attention. 

## Configuration

After [getting the ansible Nym node playbook](#ansible-installation) to your location, it's time to configure it for your own needs. 

> Mind that *idempotency* is an essential character when dealing with orchestration. A playbook, even when run many times should ensure that state of your targeted system will not change from what you intended. Therefore, it is important to make sure that all tasks in your playbook do not change the system in any way if the change you required has already been applied.

<Callout type="warning" emoji="⚠️">
Before starting Ansible, ensure that your `A` and `AAAA` records are pointed to your server IPs and propagated. Good test is to be able to ping them or use them for ssh into the server.
</ Callout>

**Open your local copy of the playbook in your favourite text editor and begin with these steps:**

<Steps>
###### 1. Configure global variables:
- Open `playbooks/group_vars/all.yml`
- Define all values under the section labeled as `## MANDATORY - uncomment & define` - these values will be propagated on all your nodes globally
- Optionally define values of your choice under the section `## OPTIONAL - uncomment & define`
- Note that in the next step we will be setting up a node inventory, where each of the variable can be configured per node, taking priority over the global ones.
- Setup a correct path for your SSH kety to `ansible_ssh_private_key_file:`, alternatively export your SSH key as an env var and use this:
```sh
ansible_ssh_private_key_file: "{{ lookup('env', '<YOUR_ANSIBLE SSH_KEY_ENV_VAR>') }}"
```
- Keep `hostname=""` as a fallback for nodes without a hostname

###### 2. Create node inventory:
- Open `playbooks/inventory/all`
- Make an entry for each of your node:
```sh
node1 ansible_host=<YOUR_SERVER_IP> ansible_user=<USER> hostname=<HOSTNAME> location=<LOCATION> email=<EMAIL> mode=<MODE> wireguard_enabled=<true/false> moniker=<MONIKER> description=<DESCRIPTION>
```
- Mandatory values specific for each node - must be defined in the inventory:
    - `ansible_host`: IPv4 host address
    - `hostname`: node domain, otherwise fallbacks to `""` for nodes without domain
    - `location`: node server location
- Mandatory values which can be setup per node or in `group_vars/all` globally:
    - `ansible_user`
    - `email`
    - `website`
    - `moniker`
    - `description`
    - `mode`
    - `wireguard_enabled`
    - `accept_operator_terms` - **Make sure to read [Nym Operators Terms & Conditions](/operators/nodes/nym-node/setup#terms--conditions) first!**

###### 3. Test your setup
Run this command to check if everything is configured correctly in your inventory:
```sh
cd playbooks
ansible-inventory --graph
```

###### 4. Optional: remove some of `nym-node run` command arguments
These variables are read by the main task for `nym-node` installation: `roles/nym/tasks/config.yaml`

Open `roles/nym/defaults/main.yml` and have a look on the variables used:

###### 5. Optional: Configure `deploy.yml` playbook
Open `playbooks/deploy.yml` and comment out `tunnel` and `quic` roles in case of running your playbook for nodes in a mode `mixnode`.

Save all the files and test with:
```sh
cd playbooks
ansible-inventory --graph
```

Right now you should be ready to go.
</Steps>

## Flow & Usage

This chapter describes fundamental commands for using Ansible playbooks in relation to orchestrating multiple servers running a `nym-node`. For a full understanding of Ansible usage, read [Ansible documentation pages](https://docs.ansible.com).

### Logic

<AccordionTemplate name="See Ansible logic description">

The main logic of the playbook flow when running with a basic command and playbook like this:
```sh
ansible-playbook <PLAYBOOK>.yml
```
  <Steps>
###### 1. Read inventory
Ansible parses `inventory/all` and performs the playbook on all entries in it, unless specified otherwise

###### 2. Read global vars
Ansible parses `group_vars/all.yml` and asigns global variables to all inventory entries, unless they were defined in the inventory.

**Variables defined in the inventory per entry take highest priority!**

###### 3. Follow roles in the playbook
Ansible reads the roles defined in `<PLAYBOOK>.yml` passed with the command and executes the tasks defined under each role

  </ Steps>
</AccordionTemplate>

### Usage

The simplest way is to run `ansible-playbook` binary with a provided playbook as a command. That will do the defined roles on all entries in the inventory. In Nym we currently have these playbooks:

<Steps>

###### 1. Deploy

A playbook to deploy server and `nym-node` from scratch, configuring networking, routing, firewall, systemd, bridges, reverse proxy, exit policy and all required tasks.  

This playbook will run roles on all the inventory entries in parallel by default.

```sh
cd playbooks
ansible-playbook deploy.yml
```

###### 2. Bond

<Callout type="warning" emoji="⚠️">
Anyone having acces to your account mnemonic can take all your funds and manage manage your node, be careful where you store it!
</Callout>

Bonding can be managed via two playbooks:

1. `bond.yml`: an interactive way, requiring operator to use own wallet (desktop or CLI)
2. `auto-bond.yml`: automatic bonding flow requiring operator to prepare `nodes.csv` and have `nym-cli` installed

<div>
  <Tabs items={[
    <code>bond.yml</code>,
    <code>auto-bond.yml</code>,
    ]} defaultIndex="1">
    <MyTab>

A playbook to *interactively* register your nodes to Nym network by bonding it to Nyx blockchain accounts.
This playbook is intercative as it prompts user for data from Nym wallet to sign a message. It will run roles on one inventory entry at a time by default.

**Requirements**

- Nym Wallet or `nym-cli` to be used as a CLI wallet
- An account per each node
- At least 101 NYM per account

**Usage**

1. Sign in to the wallet per node
2. Follow steps in `Bond` section
3. Run the playbook on a side and follow the prompts

```sh
cd playbooks
ansible-playbook bond.yml
```

Your nodes are bonded and will show in the network in the next epoch (max 60min).

    </MyTab>
    <MyTab>

A playbook to *automatically* register your nodes to Nym Network by bonding it to Nyx blockchain accounts.
This automatic flow is slightly harder to setup in the beginning and it's recommended for operators bonding many nodes, as the initial work is worth it by saving the time of bonding a node at a time.

**Requirements**

- Installed [`nym-cli`](/developers/tools/nym-cli)
- Python3
- Nym repository with directory `scripts/nym-node-setup/auto-bond/`, containing:
  - Python program [`auto_bond_all.py`](https://github.com/nymtech/nym/tree/develop/scripts/nym-node-setup/auto-bond/auto_bond_all.py)
  - `nodes.csv.example` with correct data

**Usage**

1. Copy `nodes.csv.example` to your prefered location without `.example` suffix
2. Fill correctly the csv columns for each node you want to bond:
  - `inventory_node_id`: Your Ansible inventory node ID (ie `node1`)
  - `hostname`: same like in your `playbooks/inventory/all` (without `https://` !)
  - `ip`: same like `ansible_host` value in your Ansible inventory
  - `account`: Nyx account to bond this node with
  - `mnemonic`: Your nyx acount mnemonic
  - `identity_key`: node identity key - the easiest way to get it is to navigate to `playbooks/` and run:

```sh
ansible all -i inventory/all -a "/root/nym-binaries/nym-node bonding-information"
```

  - `amount`: Amount to bond in `uNYM` (1 NYM = 1 000 000 uNYM), Make sure to leave extra 1 NYM (1 000 000 uNYM) for fees
  - `operator_cost`: [Operator cost](/operators/tokenomics/mixnet-rewards#rewards-distribution) in `uNYM` (1 NYM = 1 000 000 uNYM)

3. Save the csv
4. Run `auto_bond_all.py` with all needed arguments.

- To see help menu:

```sh
python3 ./auto_bond_all.py --help
```

- To test your paths run with `--dry-run`

- Argument usage:

```sh
--ansible-repo ANSIBLE_REPO
                      Path to ansible playbooks directory (contains auto-bond.yml and inventory/)
--cli-dir CLI_DIR     Directory containing the nym-cli binary
--dry-run             Print commands without executing
```

- Example (note that the `nodes.csv` has no flag as it's a required argument):

```sh
python ./auto_bond_all.py \
  --ansible-repo ~/admin/nym-nodes/nym-nodes-ansible/playbooks \
  --cli-dir ~/repos/nymtech/nym/target/release \
  ~/admin/nym-nodes/nodes.csv
```

5. Your nodes should be bonded and come up in the next epoch (max 60min)

**Additional scripts**

Your `nodes.csv` can be used for other operations:
- [`show_balances.py`](https://github.com/nymtech/nym/tree/develop/scripts/nym-node-setup/auto-bond/auto_bond_all.py): Shows all accounts balances if provided with Nyx accounts (`account` column)
- [`unbond_all.py`](https://github.com/nymtech/nym/tree/develop/scripts/nym-node-setup/auto-bond/unbond_all.py): Unbond all nodes in the csv if provided with mnemonics (`mnemonic` column)
<br/>

    </MyTab>
  </Tabs>
</div>


A playbook to interactively register your node to Nym network by bonding it to Nyx blockchain account.

This playbook is interactive as it prompts user for data from Nym wallet to sign a message. It will run roles on one inventory entry at a time by default.

```sh
cd playbooks
ansible-playbook bond.yml
```

###### 3. Upgrade

A playbook to upgrade `nym-node` binary to the *Latest* by default. Operators can hard code a specific binary version in `roles/upgrade/defaults/main.yml` by un-commenting the `nym-version` line and providing their desired version. 

This playbook will run roles on all the inventory entries in parallel by default.

```sh
cd playbooks
ansible-playbook upgrade.yml
```

###### 4. System Maintenance

A playbook to de-clogg and cleanup the servers (VMs) hosting `nym-node`. This playbook should be used for operators having issues with their servers filling up disk space too fast. For operators hosting many VMs on a hypervisor dedicated / bare metal server - run this playbook alongside [the guide to clean up the hypervisor](/operators/troubleshooting/vps-isp#on-the-hypervisor-host-machine).

Maintaining the VMs using this playbook and reclaiming space on hypervisor will prevent your servers from:

- Sudden *disk full* crashes
- `journald` runaway growth
- Duplicate `rsyslog` logs
- `nym-node` log growth
- `qcow2` like allocation growth inside VPS storage

The playbook is located in `roles/upgrade/defaults/system-maintenance.yml`, but the most important part is in `playbooks/group_vars/all.yml`, starting with a header `## SYSTEM MAINTENANCE PLAYBOOK KNOBS`. This commented list of adjustable knobs serves as a dashboard to tweak values of the playbook. Any overwrite will superseed the default ones in the playbook itself. 

This playbook will run roles on all the inventory entries in parallel by default.

```sh
cd playbooks
ansible-playbook system-maintenance.yml
```

###### 5. Mitigate kernel CVE

This playbook is to mitigate some of the Kernel issues found in April and May 2026. 

This playbook will run roles on all the inventory entries in parallel by default.

```sh
cd playbooks
ansible-playbook mitigate_kernel_CVE.yml
```

</ Steps>


### Useful Commands

[Ansible](https://docs.ansible.com) has many smart ways to manage your playbooks, roles or inventory entries. 

**Here are some useful tips:**

<Steps>

###### Node limit
To test new configuration, it's advised to try it on one server at first. Of course you can comment out any entries in the inventory, but there are easier ways to do it. 

- Provide flag `-l` followed by inventory entry and Ansible will change state only of that entry:

- Some possibilities are (in example we use upgrade.yml, you can use any playbook):
```sh
# point to one entry
ansible-playbook upgrade.yml -l node1

# point to multiple entries
ansible-playbook upgrade.yml -l "node1,node2"

# use regex - ie all exit nodes
ansible-playbook upgrade.yml -l "*exit*"

# use group in inventory labeled as [group]
ansible-playbook deploy.yml -l new_nodes
```

###### Tag selection

<Callout>
To update your exit policy by using the newest [NTM](https://github.com/nymtech/nym/blob/develop/scripts/nym-node-setup/network-tunnel-manager.sh) via Ansible, just run:
```sh
ansible-playbook deploy.yml -t network_tunnel_manager
```
This will download the script from `develop`, make executable and run it with the command `complete_networking_configuration`.
</Callout>

Sometimes you may want to run just one tag at a time, for that use `-t` flag, for example:
```sh
# in case of wanting to run only quic deployment role
ansible-playbook deploy.yml -t quic

# in case of running the same on only one node
ansible-playbook deploy.yml -l node2 -t quic
```

To list all tags, run:
```sh
ansible-playbook <PLAYBOOK>.yml --list-tags

# for example
ansible-playbook deploy.yml --list-tags
```

###### Arbitrary command output

You can use ansible to read a `STDOUT` from any command, using this logic:
```sh
ansible all -i inventory/all -a "<COMMAND>>"

# for example to get all node ID keys
ansible all -i inventory/all -a "/root/nym-binaries/nym-node bonding-information"
```

- Note that the command gets also run, be mindful what you executing 

- This logic can be combined with the arguments above, for example to limit the range of nodes

###### nocows

Yes, by default there is a cow printed under each task, you can turn it off by opening `playbooks/ansible.cfg` and un-commenting the `nocows` line:

```cfg
nocows = 1
```
</ Steps>