pilot-leader-election

Implement leader election protocols with automatic failover detection.

Commands

Announce candidacy

pilotctl --json publish "registry-hostname" "election:$ELECTION_GROUP" \
  --data "{\"type\":\"election\",\"candidate\":\"$AGENT_ID\",\"priority\":$PRIORITY,\"term\":$TERM}"

Declare victory as leader

pilotctl --json publish "registry-hostname" "election:$ELECTION_GROUP" \
  --data "{\"type\":\"leader\",\"leader\":\"$AGENT_ID\",\"term\":$TERM,\"elected_at\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\"}"

Send leader heartbeat

pilotctl --json publish "registry-hostname" "election:$ELECTION_GROUP" \
  --data "{\"type\":\"heartbeat\",\"leader\":\"$AGENT_ID\",\"term\":$TERM,\"timestamp\":\"$(date -u +%s)\"}"

Detect leader failure

LAST_HEARTBEAT=$(pilotctl --json inbox \
  | jq -r '[.messages[] | select(.topic == "election:'$ELECTION_GROUP'" and .payload.type == "heartbeat")] | sort_by(.payload.timestamp) | last | .payload.timestamp')

ELAPSED=$(( $(date -u +%s) - LAST_HEARTBEAT ))

if [ "$ELAPSED" -gt 10 ]; then
  echo "Leader timeout, starting election"
fi

Workflow Example

Bully algorithm with priority-based election:

#!/bin/bash
ELECTION_GROUP="task-coordinator"
AGENT_ID=$(pilotctl --json info | jq -r '.node_id')
PRIORITY=$(echo -n "$AGENT_ID" | cksum | cut -d' ' -f1)
REGISTRY_HOST="registry.example.com"

# Announce candidacy
pilotctl --json publish "$REGISTRY_HOST" "election:$ELECTION_GROUP" \
  --data "{\"type\":\"election\",\"candidate\":\"$AGENT_ID\",\"priority\":$PRIORITY,\"term\":$TERM}"

# Wait and check if highest priority
sleep 10

HIGHEST=$(pilotctl --json inbox \
  | jq -r '[.messages[] | select(.topic == "election:'$ELECTION_GROUP'" and .payload.type == "election")] | sort_by(.payload.priority) | last | .payload.candidate')

if [ "$HIGHEST" = "$AGENT_ID" ]; then
  pilotctl --json publish "$REGISTRY_HOST" "election:$ELECTION_GROUP" \
    --data "{\"type\":\"leader\",\"leader\":\"$AGENT_ID\",\"term\":$TERM}"
fi

Dependencies

Requires pilot-protocol skill, jq, and cksum.