Starting a Node

Please make sure you are running telemetry

Machine Limits

We need to modify the maximum open files and unix socket buffers to increase their limits.

Open Files


sudo echo "fermah	soft	nofile	65535" >> /etc/security/limits.conf
sudo echo "fermah	hard	nofile	65535" >> /etc/security/limits.conf

Unix Socket Buffers


sudo sysctl -w net.core.wmem_max=16777216
sudo sysctl -w net.core.rmem_max=16777216

Start

Make sure the configuration file exists at ~/.fermah/config/prover-node-config.toml

fpn

This launches the prover-node binary. It connects you to our network using the provided key and waits for Proof Requests. The process also checks for background updates.

Daemonization

Systemd

Service File

Create the service file at /etc/systemd/system/fermah.service:


[Unit]
Description=Fermah Prover Node
After=network.target local-fs.target
 
[Service]
Type=simple
User=%u
Group=%g
Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock
ExecStart=/home/%u/.fermah/bin/fpn
ExecStop=/bin/kill -s TERM $MAINPID
Restart=on-failure
KillMode=control-group
ExecStartPost=/bin/sleep 0.5
TimeoutStopSec=20
StartLimitBurst=5
RestartSec=5
StandardOutput=append:/var/log/fermah/pn.log
Environment=DOCKER_HOST=unix:///run/user/1001/docker.sock
 
ExecStartPre=/bin/mkdir -p /var/log/fermah
ExecStartPre=/bin/chown %u:%g /var/log/fermah
ExecStartPre=/bin/chmod 755 /var/log/fermah
LimitNOFILE=65535
 
[Install]
WantedBy=multi-user.target

Logrotate

Create the file /etc/logrotate.d/fermah
Set permissions sudo chmod 644 /etc/logrotate.d/fermah


/var/log/fermah/pn.log {
    su root adm
    daily
    missingok
    rotate 30
    maxsize 1G
    compress
    notifempty
    postrotate
        /usr/lib/rsyslog/rsyslog-rotate
    endscript
}

Enable service


sudo systemctl daemon-reload
sudo systemctl enable fermah.service
sudo systemctl start fermah.service

Docker Compose

You can also run the prover node as a Docker container using Docker Compose.

This requires the NVIDIA Container Toolkit to be installed and configured. See Install CUDA containers toolkit.

Container Image

Build a minimal container image for the prover node:


FROM nvidia/cuda:12.9.1-runtime-ubuntu24.04
 
RUN apt-get update && \
    apt-get install -y --no-install-recommends ca-certificates tini && \
    rm -rf /var/lib/apt/lists/*
 
ENV HOME=/root
WORKDIR /root
 
ENTRYPOINT ["/usr/bin/tini", "--", "/root/.fermah/bin/fpn"]


docker build -t fermah-fpn:24.04 .

Replace the base image with nvidia/cuda:12.2.0-runtime-ubuntu22.04 if you are running CUDA 12.2.

docker-compose.yml


services:
  fpn:
    image: fermah-fpn:24.04
    restart: unless-stopped
    environment:
      NVIDIA_VISIBLE_DEVICES: 0
    volumes:
      - /home/fermah/.fermah:/root/.fermah:rw
      - /var/log/fermah:/var/log/fermah:rw
      - /var/run/docker.sock:/var/run/docker.sock
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    networks:
      - fermah-net
 
networks:
  fermah-net:
    external: true

Create the network and start:


docker network create fermah-net
docker compose up -d

Headless Password

If you are using a password for your machine keystore file, you can start the process headless with no prompts, by using the env var FERMAH_KEYSTORE_PW_FILE and setting it to the absolute path to the file containing the password.

Keep in mind these security considerations:

Set password file permissions to 600
Ensure the keystore file has 600 permissions.
Run the systemd service with a non root user and owning both keystore file and password file