Starting a Validator
Configure Velas CLI
The Velas cli includes get
and set
configuration commands to automatically
set the --url
argument for cli commands. For example:
velas config set --url https://explorer.testnet.velas.com/rpc
While this section demonstrates how to connect to the Devnet cluster, the steps are similar for the other Velas Clusters.
Confirm The Cluster Is Reachable
Before attaching a validator node, sanity check that the cluster is accessible to your machine by fetching the transaction count:
velas transaction-count
Confirm your Installation
Try running following command to join the gossip network and view all the other nodes in the cluster:
velas-gossip spy --entrypoint bootstrap.testnet.velas.com:8001
# Press ^C to exit
Enabling CUDA
If your machine has a GPU with CUDA installed (Linux-only currently), include
the --cuda
argument to velas-validator
.
When your validator is started look for the following log message to indicate
that CUDA is enabled: "[<timestamp> solana::validator] CUDA is enabled"
System Tuning
Linux
Automatic
The Velas repo includes a daemon to adjust system settings to optimize performance (namely by increasing the OS UDP buffer limits, and scheduling PoH with realtime policy).
The daemon (solana-sys-tuner
) is included in the Velas binary release. Restart
it, before restarting your validator, after each software upgrade to ensure that
the latest recommended settings are applied.
To run it:
sudo solana-sys-tuner --user $(whoami) > sys-tuner.log 2>&1 &
Manual
If you would prefer to manage system settings on your own, you may do so with the following commands.
Increase UDP buffers
sudo bash -c "cat >/etc/sysctl.d/20-solana-udp-buffers.conf <<EOF
# Increase UDP buffer size
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728
EOF"
sudo sysctl -p /etc/sysctl.d/20-solana-udp-buffers.conf
Increased memory mapped files limit
sudo bash -c "cat >/etc/sysctl.d/20-solana-mmaps.conf <<EOF
# Increase memory mapped files limit
vm.max_map_count = 500000
EOF"
sudo sysctl -p /etc/sysctl.d/20-solana-mmaps.conf
Add
LimitNOFILE=500000
to the [Service]
section of your systemd service file, if you use one,
otherwise add it to /etc/systemd/system.conf
.
sudo systemctl daemon-reload
sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
# Increase process file descriptor count limit
* - nofile 500000
EOF"
### Close all open sessions (log out then, in again) ###
Generate identity
Create an identity keypair for your validator by running:
velas-keygen new -o ~/validator-keypair.json
The identity public key can now be viewed by running:
velas-keygen pubkey ~/validator-keypair.json
Note: The "validator-keypair.json” file is also your (ed25519) private key.
Paper Wallet identity
You can create a paper wallet for your identity file instead of writing the keypair file to disk with:
velas-keygen new --no-outfile
The corresponding identity public key can now be viewed by running:
velas-keygen pubkey ASK
and then entering your seed phrase.
See Paper Wallet Usage for more info.
Vanity Keypair
You can generate a custom vanity keypair using solana-keygen. For instance:
velas-keygen grind --starts-with e1v1s
Depending on the string requested, it may take days to find a match...
Your validator identity keypair uniquely identifies your validator within the network. It is crucial to back-up this information.
If you don’t back up this information, you WILL NOT BE ABLE TO RECOVER YOUR VALIDATOR if you lose access to it. If this happens, YOU WILL LOSE YOUR ALLOCATION OF SOL TOO.
To back-up your validator identify keypair, back-up your "validator-keypair.json” file or your seed phrase to a secure location.
More Velas CLI Configuration
Now that you have a keypair, set the velas configuration to use your validator keypair for all following commands:
velas config set --keypair ~/validator-keypair.json
You should see the following output:
Wallet Config Updated: /home/solana/.config/solana/wallet/config.yml
* url: http://explorer.testnet.velas.com/rpc
* keypair: /home/solana/validator-keypair.json
Airdrop & Check Validator Balance
Airdrop yourself some VLX to get started:
velas airdrop 10 --url http://bootstrap.testnet.velas.com
Note that airdrops are only available on bootstrap node at Devnet.
To view your current balance:
velas balance
Or to see in finer detail:
velas balance --lamports
Create Vote Account
If you haven’t already done so, create a vote-account keypair and create the vote account on the network. If you have completed this step, you should see the “vote-account-keypair.json” in your Velas runtime directory:
velas-keygen new -o ~/vote-account-keypair.json
The following command can be used to create your vote account on the blockchain with all the default options:
velas create-vote-account ~/vote-account-keypair.json ~/validator-keypair.json
Read more about creating and managing a vote account.
Trusted validators
If you know and trust other validator nodes, you can specify this on the command line with the --trusted-validator <PUBKEY>
argument to velas-validator
. You can specify multiple ones by repeating the argument --trusted-validator <PUBKEY1> --trusted-validator <PUBKEY2>
.
This has two effects, one is when the validator is booting with --no-untrusted-rpc
, it will only ask that set of
trusted nodes for downloading genesis and snapshot data. Another is that in combination with the --halt-on-trusted-validator-hash-mismatch
option,
it will monitor the merkle root hash of the entire accounts state of other trusted nodes on gossip and if the hashes produce any mismatch,
the validator will halt the node to prevent the validator from voting or processing potentially incorrect state values. At the moment, the slot that
the validator publishes the hash on is tied to the snapshot interval. For the feature to be effective, all validators in the trusted
set should be set to the same snapshot interval value or multiples of the same.
It is highly recommended you use these options to prevent malicious snapshot state download or account state divergence.
Connect Your Validator
Connect to the cluster by running:
velas-validator \
--identity ~/validator-keypair.json \
--vote-account ~/vote-account-keypair.json \
--ledger ~/validator-ledger \
--rpc-port 8899 \
--entrypoint bootstrap.testnet.velas.com:8001 \
--limit-ledger-size \
--log ~/velas-validator.log
Lastly, to configure log rotation, please run the following:
# Setup log rotation
cat > logrotate.sol <<EOF
~/velas-validator.log {
rotate 7
daily
missingok
postrotate
systemctl kill -s USR1 sol.service
endscript
}
EOF
sudo cp logrotate.sol /etc/logrotate.d/sol
systemctl restart logrotate.service
To force validator logging to the console add a --log -
argument, otherwise
the validator will automatically log to a file.
Note: You can use a paper wallet seed phrase for your
--identity
and/or--vote-account
keypairs. To use these, pass the respective argument asvelas-validator --identity ASK ... --vote-account ASK ...
and you will be prompted to enter your seed phrases and optional passphrase.
Confirm your validator connected to the network by opening a new terminal and running:
velas-gossip spy --entrypoint bootstrap.testnet.velas.com:8001
If your validator is connected, its public key and IP address will appear in the list.
Controlling local network port allocation
By default the validator will dynamically select available network ports in the
8000-10000 range, and may be overridden with --dynamic-port-range
. For
example, velas-validator --dynamic-port-range 11000-11010 ...
will restrict
the validator to ports 11000-11010.
Limiting ledger size to conserve disk space
The --limit-ledger-size
argument will instruct the validator to only retain the
last couple hours of ledger. To retain the full ledger, simply remove that arg.
Systemd Unit
Running the validator as a systemd unit is one easy way to manage running in the background.
Assuming you have a user called vel
on your machine, create the file /etc/systemd/system/vel.service
with the following:
[Unit]
Description=Velas Validator
After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
Restart=always
RestartSec=1
User=vel
LimitNOFILE=1000000
LogRateLimitIntervalSec=0
Environment="PATH=/bin:/usr/bin:/home/vel/.local/share/velas/install/active_release/bin"
ExecStart=/home/vel/bin/validator.sh
[Install]
WantedBy=multi-user.target
Now create /home/vel/bin/validator.sh
to include the desired velas-validator
command-line. Ensure that the 'exec' command is used to start the validator process
(i.e. "exec velas-validator ..."). This is important because without it, logrotate will end up killing the validator every time the logs are rotated.
Ensure that running /home/vel/bin/validator.sh
manually starts the validator as expected. Don't forget to mark it executable with chmod +x /home/vel/bin/validator.sh
Start the service with:
$ sudo systemctl enable --now vel
Logging
Log output tuning
The messages that a validator emits to the log can be controlled by the RUST_LOG
environment variable. Details can by found in the documentation for the env_logger
Rust crate.
Note that if logging output is reduced, this may make it difficult to debug issues encountered later. Should support be sought from the team, any changes will need to be reverted and the issue reproduced before help can be provided.
Log rotation
The validator log file, as specified by --log ~/velas-validator.log
, can get very large over time and it's recommended that log rotation be configured.
The validator will re-open its when it receives the USR1
signal, which is the basic primitive that enables log rotation.
If the validator is being started by a wrapper shell script, it is important to launch the process with exec
(exec velas-validator ...
) when using logrotate. This will prevent the USR1
signal from being sent to the script's process instead of the validator's, which will kill them both.
Using logrotate
An example setup for the logrotate
, which assumes that the validator is running as a systemd service called vel.service
and writes a log file at /home/vel/velas-validator.log:
# Setup log rotation
cat > logrotate.vel <<EOF
/home/vel/velas-validator.log {
rotate 7
daily
missingok
postrotate
systemctl kill -s USR1 vel.service
endscript
}
EOF
sudo cp logrotate.vel /etc/logrotate.d/vel
systemctl restart logrotate.service
As mentioned earlier, be sure that if you use logrotate, any script you create which starts the velas validator process uses "exec" to do so (example: "exec velas-validator ..."); otherwise, when logrotate sends its signal to the validator, the enclosing script will die and take the validator process with it.
Disable port checks to speed up restarts
Once your validator is operating normally, you can reduce the time it takes to restart your validator by adding
the --no-port-check
flag to your velas-validator
command-line.
Using a ramdisk with spill-over into swap for the accounts database to reduce SSD wear
If your machine has plenty of RAM, a tmpfs ramdisk (tmpfs) may be used to hold the accounts database.
When using tmpfs it's essential to also configure swap on your machine as well to avoid running out of tmpfs space periodically.
A 300GB tmpfs partition is recommended, with an accompanying 250GB swap partition.
Example configuration:
sudo mkdir /mnt/velas-accounts
- Add a 300GB tmpfs parition by adding a new line containing
tmpfs /mnt/velas-accounts tmpfs rw,size=300G,user=vel 0 0
to/etc/fstab
(assuming your validator is running under the user "vel"). CAREFUL: If you incorrectly edit /etc/fstab your machine may no longer boot - Create at least 250GB of swap space
- Choose a device to use in place of
SWAPDEV
for the remainder of these instructions. Ideally select a free disk partition of 250GB or greater on a fast disk. If one is not available, create a swap file withsudo dd if=/dev/zero of=/swapfile bs=1MiB count=250KiB
, set its permissions withsudo chmod 0600 /swapfile
and use/swapfile
asSWAPDEV
for the remainder of these instructions - Format the device for usage as swap with
sudo mkswap SWAPDEV
- Add the swap file to
/etc/fstab
with a new line containingSWAPDEV swap swap defaults 0 0
- Enable swap with
sudo swapon -a
and mount the tmpfs withsudo mount /mnt/velas-accounts/
- Confirm swap is active with
free -g
and the tmpfs is mounted withmount
Now add the --accounts /mnt/velas-accounts
argument to your velas-validator
command-line arguments and restart the validator.
Account indexing
As the number of populated accounts on the cluster grows, account-data RPC requests that scan the entire account set -- like getProgramAccounts
and SPL-token-specific requests
-- may perform poorly. If your validator needs to support any of these requests, you can use the --account-index
parameter to activate one
or more in-memory account indexes that significantly improve RPC performance by indexing accounts by the key field. Currently supports the following parameter values:
program-id
: each account indexed by its owning program; used bygetProgramAccounts
spl-token-mint
: each SPL token account indexed by its token Mint; used by getTokenAccountsByDelegate, and getTokenLargestAccountsspl-token-owner
: each SPL token account indexed by the token-owner address; used by getTokenAccountsByOwner, andgetProgramAccounts
requests that include an spl-token-owner filter.