# Operations Guide

> Day-to-day operations for running `hanarang-rails` in production.

## Process management

Rails is a long-lived orchestrator. Use `pm2` (recommended), `systemd`, or `docker-compose` to supervise it.

### PM2

```bash
cd /path/to/hanarang-rails
pm2 start ecosystem.config.cjs
pm2 save
pm2 startup    # enable auto-start on reboot
```

Check status:
```bash
pm2 list
pm2 logs hanarang-rails
pm2 restart hanarang-rails
```

## Health check

Run this on a cron or uptime monitor:

```bash
pnpm rails doctor
```

Exit code 0 = healthy, 1 = one or more errors.

For deeper state:
```bash
pnpm rails status              # list recent pipelines
pnpm rails status <pipeline>   # single pipeline with timeline
```

## Common tasks

### Start a pipeline from the shell

```bash
pnpm rails run my-project -r "Add Live2D avatar component"
```

### Resume an escalated pipeline

```bash
pnpm rails status --state escalated
pnpm rails resume <pipeline-id>
```

### Abort a runaway pipeline

```bash
pnpm rails abort <pipeline-id> -r "wrong branch"
```

### Generate and freeze a contract

```bash
pnpm rails contract generate .plans/sprints/SPRINT-007.md -s SPRINT-007
# review the draft in .rails/contracts/<id>.sprint-contract.json
pnpm rails contract freeze <id>
pnpm rails contract validate <id>
```

### Run QA for a sprint type

```bash
pnpm rails qa run feature -s SPRINT-007
pnpm rails qa show <artifact-id>
```

## Troubleshooting

### "No skill context found"

The enforcement hook is blocking tool calls because the rails skill context is missing or expired.

```bash
pnpm rails skill-context create --pipeline <id> --skill rails
pnpm rails skill-context show
```

To temporarily disable enforcement for debugging (logged to trace):
```bash
RAILS_ENFORCE=off pnpm rails run ...
```

### Pipeline stuck in `retrying`

The retrying state has an `always` transition — it should move forward immediately. If you see it stuck in SQL dumps, check for a stale process holding a DB connection. Restart the orchestrator:

```bash
pm2 restart hanarang-rails
```

### "Contract validator ABORT_PRECHECK"

Environment prerequisites failed. The validator output will name the missing prereq. Common causes:

- `node22` prereq → upgrade Node runtime
- `DATABASE_URL` env var missing → check `.env`
- `port_open` → the target service is down
- `http_reachable` → network / firewall

### xhigh thinking tier refused

Rails refuses to pass `xhigh` to agents because it caused indefinite waits in the legacy system. Use `high` or below. If an agent config still sets `xhigh`, grep and update:

```bash
grep -rn "thinking_tier.*xhigh" agents/
```

### Manual QA checks always SKIPPED

By default, `rails qa run` marks manual checks as SKIPPED (passed=true with a skip note). To actually evaluate, plug in a resolver programmatically. A shipped LLM resolver is tracked for v0.2.0.

## Logs

Rails uses `pino` for structured logging. Every log line is JSON with at least:
```json
{"level":30,"time":...,"service":"hanarang-rails","module":"runner","pipelineId":"01..."}
```

Pipe through `pino-pretty` for interactive reading:
```bash
pm2 logs hanarang-rails --raw | pino-pretty
```

## Database maintenance

Rails uses a single MariaDB schema with 5 tables: `pipelines`, `state_transitions`, `actor_spawns`, `contracts`, `escalations`.

### Retention

By default there is no automatic retention. Add a cron:

```sql
-- Trim state_transitions older than 90 days for terminal pipelines
DELETE st FROM state_transitions st
  JOIN pipelines p ON p.id = st.pipelineId
  WHERE p.currentState IN ('done', 'aborted')
    AND p.updatedAt < NOW() - INTERVAL 90 DAY;
```

### Backup

Standard MariaDB dump:
```bash
mysqldump hanarang_rails > backup-$(date +%F).sql
```

## Security

- **Never commit `.env`.** Use secret management for production deployments.
- **Rotate `DISCORD_TOKEN` periodically.** Rails reads env vars on startup.
- **`GITEA_WEBHOOK_SECRET`** must be a high-entropy random string — used for HMAC verification.
- **Skill enforcement trace** at `.rails/skill-trace.jsonl` may contain tool usage history. Rotate/truncate on long-lived installs.

## Versioning

Rails follows semver. Check current version:
```bash
pnpm rails --help | head -1
```

Upgrade:
```bash
git pull
pnpm install --prod
pnpm build
pnpm prisma migrate deploy
pm2 restart hanarang-rails
```

## Related

- `migration-guide.md` — moving from legacy installs
- `.plans/design/` — architecture
- `.plans/failure-audit.md` — F1–F6 that rails prevents