I also wanted three things: nodes that scale out without per-node configuration, a crashed node's accounts picked up by the others, and no extra service to run just to decide who owns what. The answer turned out to be one table, one SQL statement and a margin of fifteen seconds.

Ownership is two columns

The account row is the thing being owned, so ownership lives on the row:

trading_accounts (
    id          bigint primary key,
    node_id     text,          -- who owns it now
    lease_until timestamptz,   -- until when, by the database's clock
    ...
)

A node owns an account while node_id is its own id and lease_until is in the future. Nobody has to keep a heartbeat table, and an owner that disappears simply stops renewing.

One lease cycle is one transaction

Every 20 seconds each node runs one cycle. In one transaction it releases accounts that were switched off, renews its own leases, claims free ones while it is below capacity, and reads back everything it now owns:

-- renew what I hold
UPDATE trading_accounts SET lease_until = now() + make_interval(secs => $2)
WHERE node_id = $1 AND platform = $3 AND enabled;

-- claim free or expired accounts, up to my free capacity
UPDATE trading_accounts SET node_id = $1, lease_until = now() + make_interval(secs => $2)
WHERE id IN (
    SELECT id FROM trading_accounts
    WHERE platform = $3 AND enabled
      AND (node_id IS NULL OR lease_until IS NULL OR lease_until < now())
    ORDER BY id
    LIMIT $4
    FOR UPDATE SKIP LOCKED);

FOR UPDATE SKIP LOCKED is what lets many nodes claim at once. Two nodes that run the claim in the same moment do not wait for each other and do not block; each locks its own rows and skips the ones the other has locked. No account can be handed to two nodes, and no node needs to know the others exist. A new node just starts, claims up to its capacity and is part of the cluster.

The node id defaults to <platform>-<hostname>, so replicas need no instance configuration at all.

The lease alone is not enough

The tempting reading is: "I renewed, so I own it until lease_until." That is not safe. The node can be cut off from the database, for example by a network problem, and then it does not know whether another node has already taken its accounts. If it keeps sending orders until the lease runs out by its own count, it can overlap with the new owner.

So the node stops earlier than the lease ends. The numbers are configuration, not something I measured:

// AccountManager.SyncAsync
var started = DateTime.UtcNow;
var result = await _db.ClaimAccountsAsync(...);
_leaseValidUntil = started + _options.Lease - _options.LeaseFence;

// AccountManager
public bool HoldsLease => DateTime.UtcNow < _leaseValidUntil;

The node trusts its ownership for 45 seconds after the last successful renewal, not 60. Two details make that work:

• started is read before the query. The database writes lease_until = now() + 60 s at some moment after that, so the lease in the database ends at least 60 s after started. The node's own deadline is started + 45 s. The gap is at least 15 s however the query was delayed.
• Only elapsed time on the node's own clock matters. The node never compares its clock with the database's or another node's. It measures "how long since I last got a confirmed renewal". Clock offsets between machines do not enter into it.

Another node can claim the account only after lease_until < now(), which is at least 60 s after the last renewal. The old owner has stopped sending 15 s before that.

The gate on every order

Every unit of work for a follower checks the lease right before it executes:

if (!_engine.HoldsLease)
{
    // Cut off from the database long enough that another node may own this slave now:
    // sending orders could duplicate its copies. The reconcile after the next lease
    // renewal (or on the new owner) catches up.
    _log.LogWarning("[slave {Id}] Lease not confirmed, {Work} not executed", ...);
    return;
}

Dropping the work looks dangerous, but the copier is designed so that a missed event is repaired rather than ignored. Each event carries an epoch and sequence number, and a gap or an unknown epoch triggers a reconcile that compares the master's positions with the follower's. A node that lost its lease and got it back runs one. A node that took over a follower runs one too.

Writes are fenced by the owner as well

Orders are not the only thing a node writes. It also reports account status, balances and snapshots, and those batched account writes carry the owner in their WHERE:

UPDATE trading_accounts t SET status = s.status, ...
FROM (...) s
WHERE t.id = s.id AND t.node_id = $1   -- simplified

A node that has lost an account cannot overwrite what the new owner just wrote, even if its write is still in a queue.

Letting go

• Clean shutdown: the node sets node_id = NULL on everything it owns, so others take over at once.
• Crash: nothing is released, and the accounts move after the lease expires, so within 60 seconds of the last renewal.
• Account switched off: released in the same cycle's transaction.
• An account the node no longer owns: the next cycle sees that it is not in the result and disconnects the broker session.

That 60 seconds is the failover time. A shorter lease means faster failover, but the fence shrinks with it, so a short stall is more likely to eat the margin. For a copier a minute is acceptable, because a gap is repaired by the reconcile and a duplicate is not.

What this does not guarantee

I would rather say this than have a reader find it.

• It is a lease with a margin, not a fencing token. A real fencing token is checked by the resource, and a broker's trade server will not check mine. If a process freezes for more than the 15 s margin between the lease check and the send (a stalled VM, for example), it could still send once. The margin covers ordinary pauses, not a frozen machine.
• A backstop exists, but I would not call it proof. Copy orders carry a per-link magic number and a comment with the master's ticket (mc<masterTicket>). A reconcile on the new owner matches what is already open against the master's positions through that marker, so a takeover does not blindly open everything again.
• I use the wall clock. DateTime.UtcNow can jump if someone steps the system time. A monotonic clock (Stopwatch) is the more careful choice for "how long since the last renewal", and it is on my list.
• There is no automated test with two real nodes racing for the same rows. The SQL has run in production, which is not the same thing.

Takeaways

1. If the thing to be owned is already a row in your database, put the ownership on the row. Two columns and one statement replaced a coordination service.
2. SKIP LOCKED gives you contention-free claiming. It does nothing about a node that is slow or cut off.
3. Make the owner stop earlier than the lease ends, and measure that time from before the renewal query.
4. Fence the writes as well as the actions. A node that lost an account should not be able to change it.
5. Assume that someone will drop work. Build the thing that repairs the gap first.

I build SnapCopier, a cloud trade copier for MT4 and MT5. The code above is from its node.