Running several pAIring servers

pAIring itself is light, but one machine can still become the limit: many users, many open chats, or you want pAIring to keep running when a machine goes down. Then run several pAIring servers behind a load balancer.

This page is about pAIring. The heavy work, the models, runs on Matricxon. For that side, see Deployment layouts and Cluster.

The layout

                 users
                   |
            HAProxy (or Nginx)
          /        |        \
   pAIring 1   pAIring 2   pAIring 3
          \        |        /
     MySQL  +  shared folders (NFS)
                   |
   Matricxon server(s), Remote mode or cluster

Any server can answer any request. A user can reach a different server on every click and nothing breaks.

What every server needs

• One MySQL database. Every server uses the same DATABASE_URL. SQLite is a local file, so it can’t be shared between machines. To switch, use Admin settings → System → Database, with Copy my data to bring your users, chats and settings along. See Admin settings.

• The same SECRET_KEY. It signs the login cookie. With different keys, users are logged out each time they reach another server.

• Shared folders. Mount these from shared storage, such as NFS, on every server, and point the settings at them:

  Setting	What it holds
  KNOWLEDGE_DIR	Knowledge-base documents
  ATTACHMENTS_DIR	Files attached to chat messages
  AVATARS_DIR	Profile pictures
  IMAGES_DIR	Generated images

• Matricxon in Remote mode: a list of servers or a cluster. Set it under Admin settings → Inference engines. No pAIring server starts or stops Matricxon itself. A cluster created from this machine’s Matricxon starts in Local mode: switch it to Remote first.

Example HAProxy config

frontend pairing
    bind *:80
    default_backend pairing_servers

backend pairing_servers
    balance leastconn
    option httpchk GET /health
    timeout server 10m
    server p1 10.0.0.11:8000 check
    server p2 10.0.0.12:8000 check
    server p3 10.0.0.13:8000 check

• /health answers 503 when a server can’t reach the database or any inference engine, so HAProxy stops sending users to it.
• timeout server 10m lets long replies stream to the end. Replies on a CPU can take minutes.
• No sticky sessions are needed. Every server reads the same database.
• HTTPS: terminate TLS in HAProxy, and set SESSION_COOKIE_SECURE=true on every server.

How the servers coordinate

You don’t set any of this up. It all runs through the shared database:

• One server at a time through startup. Database migrations and first-time setup, such as the starter admin, run on one server while the others wait. Starting all servers at once after an update is safe.
• One leader for background work. Polling the engines, cleaning up old statistics and syncing the knowledge folder run on one server only, the leader. If the leader stops, another server takes over within 30 seconds.
• Replies belong to the server writing them. Restarting one server never cuts off replies on the others. If a server goes offline, the leader marks its unfinished replies as interrupted after a minute.
• Settings reach every server within about 5 seconds of a save.
• Upload progress is in the database, so the progress bar keeps working when the next poll reaches another server.

The Servers page

Cluster → Servers, under pAIring in the Cluster page’s sidebar, checks all of the above for you, on every online server:

• a checklist: shared database, the same SECRET_KEY, each of the 4 folders really shared, Matricxon in Remote mode and the same version, each with how to fix it;
• the folders on each server. A folder passes only when the server sees the marker file the leader writes into it, so the same path on two local disks does not count;
• This server: change its 4 folders and SECRET_KEY. Fill the 4 folders sets all of them from one shared folder (knowledge, attachments, avatars and generated_images inside it). They are saved in that server’s own .env and used after a restart. Do the same on every server. Generate makes a new key to copy to the others, and the current key is never shown;
• Keys and secrets: what each pAIring and Matricxon key is for, and where it must be the same.

The server list

The server list shows every server using the database. For each one it shows:

• its version
• whether it is online
• which one is the leader
• which one you are on now

An offline server can be removed from the list.

Local instances (more pAIring processes on the same machine, set under Admin settings → System) are listed under their server as a tree, and the server row shows how many processes it has. They belong to that one server: they share its folders and are updated with it.

With SQLite, which only one machine can use, the list shows this machine’s own processes, and there is no Update all servers. None of the coordination above writes to the database then.

The system statistics (CPU, memory and disk) show the leader’s machine.

Updating every server

Update all servers updates pAIring from GitHub on one server at a time. Every server uses the same source, the checkout’s own origin or the repository set under Use a different repo or branch (stored in the shared database):

1. The server downloads the update and restarts itself. On the way up, it brings the database up to date.
2. It reports back, and the next server starts.
3. The leader goes last. The rollout stops at the first error, and you can cancel it between servers.

While one server restarts, the others keep serving, so users see no downtime.

A progress panel follows the update:

• one bar for the whole update, with how long it has taken;
• each server in update order, with its result and how long it took;
• for the server updating now, its stage (Download and install, Restart, Back online) and its current step with a percent.

The panel stays on the page after the update ends, with the result: finished, stopped with the error, or cancelled.

A server can restart itself only when it was started with pAIring’s own scripts. A server run by systemd is updated and then shown as “Updated, restart it”, so you restart it yourself. A server in Docker is skipped. Update its image instead.

Containers and server names

Each server is named after its machine and port. If the machine name changes on every restart, as in some containers, set PAIRING_SERVER_ID to a fixed name on each server. Then a restarted server recognises its own unfinished replies.