HaBraid/docs/plans/PLAN.md

Generator Batch Processing Implementation Plan

For Codex: Read this plan carefully. Implement all tasks sequentially. Commit after each task.

Goal: Fix wiki generator to process 500+ drawers in room-based batches with proper LLM calls, and fix fallback to produce one merged page per room instead of per-drawer (which overwrites itself).

Architecture: Group drawers by room, send each room as one LLM batch (max 25 drawers). LLM generates consolidated wiki pages per room. If LLM fails, merge all drawers in that room into one fallback page.

Tech Stack: TypeScript, Node.js, z.ai OpenAI-compatible API, gray-matter

---

Problem Analysis

Current Issues

1. updateWiki() in generator.ts sends ALL 503 drawers in a single LLM call → token overflow → LLM fails → falls to fallback
2. buildFallbackFiles() creates one file per drawer but uses room-summary as slug → same room drawers overwrite each other → only last drawer's content survives
3. readRawDrawers() reads all 503 drawers into memory at once (fine for 503, but prompt construction is the bottleneck)

Solution

1. Group drawers by room after reading
2. Process each room group as a separate LLM call (batch within room if >25 drawers)
3. Fix fallback to merge all drawers in a room into ONE page
4. LLM timeout set to 120s per batch, 2s delay between batches

---

Files to Modify

• src/wiki/generator.ts — Main changes: batch processing, fixed fallback
• src/wiki/prompts.ts — Adjust prompt for per-room batch context
• src/wiki/llm.ts — Add timeout parameter to generate()
• src/vault/render.ts — Fix renderFallbackWikiMarkdown to accept multiple drawers

---

Task 1: Add timeout to ZaiLlmClient

Objective: Allow callers to set custom timeout per request.

Files:

• Modify: src/wiki/llm.ts

Step 1: Update generate() method signature and implementation

Add an optional timeoutMs parameter (default 60000ms) to the generate method. Pass it to fetch via AbortController.

async generate(systemPrompt: string, userPrompt: string, timeoutMs: number = 60000): Promise<string> {
  try {
    const apiKey = process.env[this.config.llm.api_key_env];
    if (!apiKey) {
      throw new LlmCallError(
        `Missing API key environment variable ${this.config.llm.api_key_env} for wiki generation.`,
      );
    }

    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), timeoutMs);

    const response = await fetch(`${this.config.llm.api_url}/chat/completions`, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${apiKey}`,
      },
      signal: controller.signal,
      body: JSON.stringify({
        model: this.config.llm.model,
        max_tokens: this.config.llm.max_tokens,
        messages: [
          { role: "system", content: systemPrompt },
          { role: "user", content: userPrompt },
        ],
      }),
    });

    clearTimeout(timeout);

    if (!response.ok) {
      const body = await response.text();
      throw new LlmCallError(`LLM request failed with ${response.status}: ${body}`);
    }

    const data = (await response.json()) as {
      choices?: Array<{ message?: { content?: string } }>;
    };
    const content = data.choices?.[0]?.message?.content;
    if (!content) {
      throw new LlmCallError("LLM response did not contain any message content.");
    }

    return content;
  } catch (error) {
    if (error instanceof LlmCallError) {
      throw error;
    }

    throw new LlmCallError("Failed to call the z.ai LLM backend.", error as Error);
  }
}

Step 2: Update LLMClient interface

generate(systemPrompt: string, userPrompt: string, timeoutMs?: number): Promise<string>;

Step 3: Commit

git add src/wiki/llm.ts
git commit -m "feat: add timeout parameter to LLM client"

---

Task 2: Fix fallback to merge drawers per room

Objective: Instead of one fallback file per drawer (which overwrites), create one merged page per room.

Files:

• Modify: src/vault/render.ts

Step 1: Add renderMergedFallbackWikiMarkdown function

Add a new function that takes an array of drawers from the same room and produces ONE wiki page:

export function renderMergedFallbackWikiMarkdown(drawers: MemPalaceDrawer[]): string {
  if (drawers.length === 0) {
    return "";
  }

  const first = drawers[0];
  const title = `${humanizeSegment(first.room)} 정리`;
  const category = inferWikiCategory(first.wing, first.room);
  const allIds = drawers.map((d) => d.id);
  const allTags = [...new Set(drawers.flatMap((d) => d.tags))];

  const frontmatter: WikiFrontmatter = {
    type: "wiki",
    category,
    title,
    created: toDateString(first.createdAt),
    updated: toDateString(new Date()),
    sources: allIds,
    tags: allTags,
    status: "draft",
    agent: first.addedBy,
  };

  const sections = drawers.map((drawer) => {
    const header = drawer.content.match(/^#\s+(.+)$/m);
    const sectionTitle = header ? header[1] : drawer.id;
    return [
      `### ${sectionTitle}`,
      "",
      drawer.content,
      "",
    ].join("\n");
  });

  const body = [
    `# ${title}`,
    "",
    "## 개요",
    `${first.wing}/${first.room} 서랍의 원본 내용을 정리한 초안입니다. 총 ${drawers.length}개 서랍.`,
    "",
    "## 핵심 내용",
    "",
    ...sections,
    "## 원본 서랍",
    ...drawers.map((d) => `- [[raw/mempalace/${d.wing}/${d.room}/${d.id}|${d.id}]]`),
  ].join("\n");

  return stringifyFrontmatter(body, frontmatter);
}

Step 2: Commit

git add src/vault/render.ts
git commit -m "feat: add merged fallback renderer for room-based grouping"

---

Task 3: Rewrite generator with batch processing

Objective: Group drawers by room, call LLM per room batch, use merged fallback on failure.

Files:

• Modify: src/wiki/generator.ts

Step 1: Replace updateWiki function

The new updateWiki should:

1. Read raw drawers
2. Group by room
3. For each room group (split into batches of 25 if needed): a. Call LLM with per-room prompt b. Parse generated files c. On LLM failure, use renderMergedFallbackWikiMarkdown for the whole room
4. Write all files
5. Update sync state and rebuild overview

const ROOM_BATCH_SIZE = 25;
const LLM_TIMEOUT_MS = 120_000;
const BATCH_DELAY_MS = 2000;

export async function updateWiki(config: WikiEngineConfig): Promise<WikiUpdateResult> {
  try {
    const vaultPath = config.vault.path;
    const state = await readSyncState(vaultPath);
    const existingWikiFiles = await listFilesRecursive(path.join(vaultPath, "wiki"), ".md");

    const sourceDrawers = readRawDrawers(vaultPath);

    if (sourceDrawers.length === 0) {
      await rebuildOverview(vaultPath);
      return { filesWritten: 0, pageSlugs: [], filePaths: [] };
    }

    // Group by room
    const roomGroups = groupDrawersByRoom(sourceDrawers);
    const llmClient = new ZaiLlmClient(config);

    const allFiles: GeneratedWikiFile[] = [];

    for (const [roomKey, roomDrawers] of roomGroups) {
      const batches = splitIntoBatches(roomDrawers, ROOM_BATCH_SIZE);

      for (let i = 0; i < batches.length; i++) {
        const batch = batches[i];

        try {
          const prompt = await buildIncrementalWikiPrompt(batch, existingWikiFiles, vaultPath);
          const response = await llmClient.generate(WIKI_SYSTEM_PROMPT, prompt, LLM_TIMEOUT_MS);
          const files = parseGeneratedWikiFiles(response);

          if (files.length > 0) {
            allFiles.push(...files);
          } else {
            // LLM returned empty — use merged fallback for this batch
            allFiles.push(buildMergedFallbackFile(batch));
          }
        } catch (error) {
          // LLM failed — use merged fallback
          allFiles.push(buildMergedFallbackFile(batch));
        }

        // Rate limit between batches
        if (i < batches.length - 1 || roomKey !== roomGroups[roomGroups.length - 1]?.[0]) {
          await sleep(BATCH_DELAY_MS);
        }
      }
    }

    // Write all files
    const writtenPaths: string[] = [];
    const pageSlugs: string[] = [];

    for (const file of allFiles) {
      const absolutePath = path.join(vaultPath, file.path);
      await writeTextFile(absolutePath, file.content);
      writtenPaths.push(absolutePath);
      pageSlugs.push(path.basename(absolutePath, ".md"));
    }

    // Update state
    const nextState: SyncState = {
      ...state,
      last_wiki_update: toIsoTimestamp(),
      wiki_pages: [...new Set([...state.wiki_pages, ...pageSlugs])],
    };

    await writeSyncState(vaultPath, nextState);
    await rebuildOverview(vaultPath);
    await updateVaultIndex(vaultPath, nextState);

    const logEntry: SyncLogEntry = {
      time: new Date().toLocaleString("sv-SE", { timeZone: config.sync.timezone }).replace("T", " "),
      action: "wiki",
      target: "raw -> wiki",
      result: `+${writtenPaths.length} pages`,
    };
    await appendSyncLog(vaultPath, [logEntry]);

    return { filesWritten: writtenPaths.length, pageSlugs, filePaths: writtenPaths };
  } catch (error) {
    throw new LlmCallError("Wiki generation failed.", error as Error);
  }
}

Step 2: Add helper functions

/** Groups drawers by their room field. Returns entries sorted by drawer count (largest first). */
function groupDrawersByRoom(drawers: MemPalaceDrawer[]): [string, MemPalaceDrawer[]][] {
  const groups = new Map<string, MemPalaceDrawer[]>();
  for (const drawer of drawers) {
    const key = `${drawer.wing}/${drawer.room}`;
    if (!groups.has(key)) {
      groups.set(key, []);
    }
    groups.get(key)!.push(drawer);
  }
  return [...groups.entries()].sort((a, b) => b[1].length - a[1].length);
}

/** Splits an array into batches of given size. */
function splitIntoBatches<T>(items: T[], batchSize: number): T[][] {
  const batches: T[][] = [];
  for (let i = 0; i < items.length; i += batchSize) {
    batches.push(items.slice(i, i + batchSize));
  }
  return batches;
}

/** Builds a single merged fallback file for a batch of drawers. */
function buildMergedFallbackFile(batch: MemPalaceDrawer[]): GeneratedWikiFile {
  const first = batch[0];
  const content = renderMergedFallbackWikiMarkdown(batch);
  const slug = toKebabCase(`${first.room}-summary`);
  const category = inferWikiCategory(first.wing, first.room);
  return {
    path: path.join("wiki", category, `${slug}.md`),
    content,
  };
}

/** Simple promise-based sleep. */
function sleep(ms: number): Promise<void> {
  return new Promise((resolve) => setTimeout(resolve, ms));
}

Step 3: Remove old buildFallbackFiles function

Delete the old function that creates one file per drawer.

Step 4: Add required imports

Add at the top of generator.ts:

import { renderMergedFallbackWikiMarkdown, inferWikiCategory } from "../vault/render.js";

Remove the old import of renderFallbackWikiMarkdown if it exists.

Step 5: Commit

git add src/wiki/generator.ts
git commit -m "feat: room-based batch processing for wiki generation"

---

Task 4: Clean up temporary files

Objective: Remove the temporary cli-generate.ts script.

Files:

• Delete: src/cli-generate.ts

Step 1: Delete the file

rm src/cli-generate.ts
git add -A
git commit -m "chore: remove temporary cli-generate script"

---

Task 5: Verify TypeScript compilation

Step 1: Run type check

npx tsc --noEmit

Expected: No errors. If there are type errors, fix them.

Step 2: Commit any fixes

git add -A
git commit -m "fix: resolve type errors"

---

Verification

After all tasks, run:

cd ~/.habraid/app
source ~/.hermes/.env
npx tsx src/index.ts generate

Expected behavior:

• Drawers grouped by room (7 rooms: memory, general, agents, workflows, state, security, protocol, diary)
• Each room processed as separate LLM batch
• If LLM succeeds: 2-5 wiki pages per room
• If LLM fails: 1 merged fallback page per room (not per drawer)
• Total: ~10-30 wiki pages instead of 503 identical overwrites