hanarang-rails/docs/operations.md

Operations Guide

Day-to-day operations for running hanarang-rails in production.

Process management

Rails is a long-lived orchestrator. Use pm2 (recommended), systemd, or docker-compose to supervise it.

PM2

cd /path/to/hanarang-rails
pm2 start ecosystem.config.cjs
pm2 save
pm2 startup    # enable auto-start on reboot

Check status:

pm2 list
pm2 logs hanarang-rails
pm2 restart hanarang-rails

Health check

Run this on a cron or uptime monitor:

pnpm rails doctor

Exit code 0 = healthy, 1 = one or more errors.

For deeper state:

pnpm rails status              # list recent pipelines
pnpm rails status <pipeline>   # single pipeline with timeline

Common tasks

Start a pipeline from the shell

pnpm rails run my-project -r "Add Live2D avatar component"

Resume an escalated pipeline

pnpm rails status --state escalated
pnpm rails resume <pipeline-id>

Abort a runaway pipeline

pnpm rails abort <pipeline-id> -r "wrong branch"

Generate and freeze a contract

pnpm rails contract generate .plans/sprints/SPRINT-007.md -s SPRINT-007
# review the draft in .rails/contracts/<id>.sprint-contract.json
pnpm rails contract freeze <id>
pnpm rails contract validate <id>

Run QA for a sprint type

pnpm rails qa run feature -s SPRINT-007
pnpm rails qa show <artifact-id>

Troubleshooting

"No skill context found"

The enforcement hook is blocking tool calls because the rails skill context is missing or expired.

pnpm rails skill-context create --pipeline <id> --skill rails
pnpm rails skill-context show

To temporarily disable enforcement for debugging (logged to trace):

RAILS_ENFORCE=off pnpm rails run ...

Pipeline stuck in retrying

The retrying state has an always transition — it should move forward immediately. If you see it stuck in SQL dumps, check for a stale process holding a DB connection. Restart the orchestrator:

pm2 restart hanarang-rails

"Contract validator ABORT_PRECHECK"

Environment prerequisites failed. The validator output will name the missing prereq. Common causes:

• node22 prereq → upgrade Node runtime
• DATABASE_URL env var missing → check .env
• port_open → the target service is down
• http_reachable → network / firewall

xhigh thinking tier refused

Rails refuses to pass xhigh to agents because it caused indefinite waits in the legacy system. Use high or below. If an agent config still sets xhigh, grep and update:

grep -rn "thinking_tier.*xhigh" agents/

Manual QA checks always SKIPPED

By default, rails qa run marks manual checks as SKIPPED (passed=true with a skip note). To actually evaluate, plug in a resolver programmatically. A shipped LLM resolver is tracked for v0.2.0.

Logs

Rails uses pino for structured logging. Every log line is JSON with at least:

{"level":30,"time":...,"service":"hanarang-rails","module":"runner","pipelineId":"01..."}

Pipe through pino-pretty for interactive reading:

pm2 logs hanarang-rails --raw | pino-pretty

Database maintenance

Rails uses a single MariaDB schema with 5 tables: pipelines, state_transitions, actor_spawns, contracts, escalations.

Retention

By default there is no automatic retention. Add a cron:

-- Trim state_transitions older than 90 days for terminal pipelines
DELETE st FROM state_transitions st
  JOIN pipelines p ON p.id = st.pipelineId
  WHERE p.currentState IN ('done', 'aborted')
    AND p.updatedAt < NOW() - INTERVAL 90 DAY;

Backup

Standard MariaDB dump:

mysqldump hanarang_rails > backup-$(date +%F).sql

Security

• Never commit .env. Use secret management for production deployments.
• Rotate DISCORD_TOKEN periodically. Rails reads env vars on startup.
• GITEA_WEBHOOK_SECRET must be a high-entropy random string — used for HMAC verification.
• Skill enforcement trace at .rails/skill-trace.jsonl may contain tool usage history. Rotate/truncate on long-lived installs.

Versioning

Rails follows semver. Check current version:

pnpm rails --help | head -1

Upgrade:

git pull
pnpm install --prod
pnpm build
pnpm prisma migrate deploy
pm2 restart hanarang-rails

Related

• migration-guide.md — moving from legacy installs
• .plans/design/ — architecture
• .plans/failure-audit.md — F1–F6 that rails prevents