feat: HaBraid v0.1.0 — host-following memory visualization engine for Obsidian
- Hybrid BM25 + HNSW vector search with context enrichment - Knowledge graph with entities, relations, and community detection - Host-following LLM route with fallback backends - 3-tier lint system (static + HNSW dup + contradiction detection) - Q&A Synthesis (Karpathy LLM Wiki pattern) - 16 MCP tools for agent-driven workflows - Incremental wiki generation with checkpointing - SQLite-backed item store with FTS5 + vector indexes
This commit is contained in:
Contributor
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@@ -0,0 +1,7 @@
|
||||
node_modules/
|
||||
dist/
|
||||
.sync-state.json
|
||||
*.db
|
||||
.env
|
||||
|
||||
.hermes/
|
||||
132
AGENTS.md
Normal file
132
AGENTS.md
Normal file
@@ -0,0 +1,132 @@
|
||||
# AGENTS.md — habraid
|
||||
|
||||
> Codex 에이전트를 위한 프로젝트 지도. 상세한 내용은 docs/ 참조.
|
||||
|
||||
## 작업 시작 전 필수 읽기
|
||||
|
||||
1. `docs/specs/host-following-llm.md` (해당 변경 작업일 때)
|
||||
2. `docs/architecture.md`
|
||||
3. `docs/config-reference.md`
|
||||
4. 관련 `PLAN*.md`
|
||||
|
||||
## 작업 규칙
|
||||
|
||||
- 구현 전에 관련 `docs/specs/*.md`를 먼저 읽고, 없으면 먼저 spec부터 작성한다.
|
||||
- 프롬프트/요구사항 정리는 `Goal / Context / Constraints / Done when` 구조를 우선한다.
|
||||
- 큰 변경은 탐색 → 설계 → 구현 → 검증 순서로 진행한다.
|
||||
- HaBraid는 memory backend를 wiki 표현층으로 바꾸는 orchestration layer다.
|
||||
- 모델 선택은 기본적으로 host가 담당하고, habraid는 host-following을 기본값으로 지향한다.
|
||||
- standalone LLM backend는 fallback/독립 실행 용도로만 유지한다.
|
||||
|
||||
## 프로젝트 개요
|
||||
|
||||
**habraid**: Obsidian 마크다운 위키 엔진. 지식 아이템을 수집/검색/위키화.
|
||||
MemPalace가 있으면 연동하고, 없어도 자체 SQLite DB로 완전 동작.
|
||||
|
||||
## 빠른 시작
|
||||
|
||||
```bash
|
||||
npm install
|
||||
npx tsx src/index.ts --help # 실행
|
||||
npx tsx src/index.ts init # 볼트 초기화
|
||||
npx tsx src/index.ts sync # 전체 동기화
|
||||
```
|
||||
|
||||
## 아키텍처
|
||||
|
||||
```
|
||||
src/
|
||||
├── index.ts # CLI 진입점 (commander)
|
||||
├── mcp-server.ts # MCP 서버 (stdio)
|
||||
├── types.ts # 공통 타입
|
||||
├── config.ts # 설정 로드 (JSON)
|
||||
├── errors.ts # 커스텀 에러
|
||||
├── utils.ts # 유틸
|
||||
├── db/
|
||||
│ ├── database.ts # SQLite 연결 + 마이그레이션
|
||||
│ ├── items.ts # items CRUD + FTS5 검색
|
||||
│ └── kg.ts # Knowledge Graph
|
||||
├── sources/
|
||||
│ ├── adapter.ts # SourceAdapter 인터페이스
|
||||
│ ├── mempalace.ts # MemPalace 연동 (선택)
|
||||
│ └── manual.ts # 직접 추가
|
||||
├── vault/
|
||||
│ ├── init.ts # 볼트 디렉토리 + SCHEMA.md 생성
|
||||
│ ├── render.ts # 아이템 → 마크다운 렌더링
|
||||
│ ├── index.ts # index.md 갱신
|
||||
│ └── log.ts # log.md 갱신
|
||||
├── wiki/
|
||||
│ ├── generator.ts # 위키 생성 오케스트레이터
|
||||
│ ├── llm.ts # z.ai API 클라이언트 (OpenAI 호환)
|
||||
│ └── prompts.ts # 시스템 프롬프트
|
||||
├── sync/
|
||||
│ ├── git.ts # git pull/push (simple-git)
|
||||
│ └── pipeline.ts # 전체 sync 파이프라인
|
||||
└── cli/
|
||||
└── commands.ts # CLI 명령어 정의
|
||||
```
|
||||
|
||||
## 핵심 규칙
|
||||
|
||||
### 코딩 규칙
|
||||
- TypeScript strict mode, ES2022 target, Node16 module resolution
|
||||
- 모든 함수와 클래스에 JSDoc 주석
|
||||
- 에러는 커스텀 에러 클래스로 래핑 (src/errors.ts)
|
||||
- async 함수는 항상 try/catch로 감싸기
|
||||
- 파일 경로는 항상 path.join() / path.resolve() 사용
|
||||
|
||||
### 볼트 규칙
|
||||
- raw/ 아래 파일은 **절대 수정하지 않음** (불변)
|
||||
- 모든 마크다운은 YAML frontmatter 포함
|
||||
- Obsidian 호환: 백링크 `[[]]`, frontmatter `---`
|
||||
- 한국어 본문, 코드/경로는 영어 원문 유지
|
||||
|
||||
### MemPalace 연동 (선택)
|
||||
- `mempalace.enabled: true` + 경로 존재 시에만 활성화
|
||||
- 없으면 자체 SQLite DB로 동작
|
||||
- ChromaDB 직접 접근하지 않음 (Python 스크립트나 MCP 경유)
|
||||
|
||||
### Git
|
||||
- 커밋 메시지: `type: description` (conventional commits)
|
||||
- 볼트 변경 후 항상 index.md 갱신
|
||||
|
||||
## 의존성
|
||||
|
||||
| 패키지 | 용도 |
|
||||
|--------|------|
|
||||
| commander | CLI 프레임워크 |
|
||||
| gray-matter | YAML frontmatter 파싱 |
|
||||
| better-sqlite3 | 자체 DB + MemPalace DB 읽기 |
|
||||
| simple-git | Git 조작 |
|
||||
| chalk | 터미널 색상 |
|
||||
| ora | 스피너 |
|
||||
| @modelcontextprotocol/sdk | MCP 서버 |
|
||||
| zod | MCP 툴 스키마 |
|
||||
|
||||
## MCP 툴
|
||||
|
||||
| 툴 | 설명 |
|
||||
|----|------|
|
||||
| `hw_status` | 볼트 + DB 통계 |
|
||||
| `hw_ingest` | 소스 → raw/ 수집 (MemPalace + 기타) |
|
||||
| `hw_add` | 아이템 직접 추가 |
|
||||
| `hw_search` | FTS5 키워드 검색 |
|
||||
| `hw_generate` | raw/ → wiki/ LLM 생성 |
|
||||
| `hw_sync` | 전체 파이프라인 |
|
||||
| `hw_read` | vault 내 파일 읽기 |
|
||||
| `hw_lint` | 정합성 검사 |
|
||||
| `hw_graph` | KG 쿼리 |
|
||||
|
||||
## 설정
|
||||
|
||||
기본 경로: `~/.habraid/data/config.json`
|
||||
레거시 fallback: `~/.config/habraid/config.json`
|
||||
|
||||
## 상세 문서
|
||||
|
||||
- `docs/architecture.md` — 전체 아키텍처 설계
|
||||
- `docs/vault-schema.md` — 볼트 디렉토리/파일 스키마
|
||||
- `docs/wiki-generation.md` — 위키 생성 프롬프트 전략
|
||||
- `docs/config-reference.md` — 설정 파일 레퍼런스
|
||||
- `PLAN.md` — v1 기획서
|
||||
- `PLAN-v2.md` — v2 독립 아키텍처 기획서
|
||||
673
README.md
Normal file
673
README.md
Normal file
@@ -0,0 +1,673 @@
|
||||
<div align="center">
|
||||
|
||||
# HaBraid
|
||||
|
||||
**Host-following memory visualization engine for Obsidian**
|
||||
|
||||
Turn long-term memory backends into a curated, graph-friendly wiki for humans and AI.
|
||||
|
||||
<p>
|
||||
<img alt="status" src="https://img.shields.io/badge/status-operational%20alpha-7c3aed">
|
||||
<img alt="runtime" src="https://img.shields.io/badge/runtime-node.js-339933">
|
||||
<img alt="transport" src="https://img.shields.io/badge/integration-MCP-1f6feb">
|
||||
<img alt="vault" src="https://img.shields.io/badge/output-Obsidian%20Wiki-8b5cf6">
|
||||
<img alt="llm" src="https://img.shields.io/badge/llm-host--following-0f766e">
|
||||
<img alt="loc" src="https://img.shields.io/badge/loc-12%2C000%2B-blueviolet">
|
||||
<img alt="files" src="https://img.shields.io/badge/source-57%20TS%20files-orange">
|
||||
</p>
|
||||
|
||||
[Architecture](docs/architecture.md) ·
|
||||
[Config Reference](docs/config-reference.md) ·
|
||||
[Wiki Generation](docs/wiki-generation.md) ·
|
||||
[Plans](docs/plans/README.md)
|
||||
|
||||
</div>
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [What is HaBraid?](#what-is-habraid)
|
||||
- [Why HaBraid exists](#why-habraid-exists)
|
||||
- [Core ideas](#core-ideas)
|
||||
- [At a glance](#at-a-glance)
|
||||
- [Demo workflow](#demo-workflow)
|
||||
- [Features](#features)
|
||||
- [Architecture diagram](#architecture-diagram)
|
||||
- [How it works](#how-it-works)
|
||||
- [Directory layout](#directory-layout)
|
||||
- [Quick start](#quick-start)
|
||||
- [MCP tools](#mcp-tools)
|
||||
- [Example MCP flows](#example-mcp-flows)
|
||||
- [Current status](#current-status)
|
||||
- [Roadmap](#roadmap)
|
||||
- [Documentation](#documentation)
|
||||
|
||||
---
|
||||
|
||||
## What is HaBraid?
|
||||
|
||||
HaBraid is a **memory visualization engine**.
|
||||
|
||||
It does **not** try to replace your memory backend. Instead, it sits between:
|
||||
|
||||
1. a long-term memory store such as **MemPalace**
|
||||
2. a curated **Obsidian wiki layer**
|
||||
3. a host agent such as **Hermes / Codex / OpenClaw**
|
||||
|
||||
Its job is to turn raw memory fragments—drawers, notes, logs, decisions, infrastructure facts, session traces—into a cleaner knowledge surface that is:
|
||||
|
||||
- readable by humans
|
||||
- re-usable by agents
|
||||
- navigable in Obsidian
|
||||
- suitable for graph view without raw-noise pollution
|
||||
|
||||
In one line:
|
||||
|
||||
> **HaBraid turns memory backends into a curated, host-following Obsidian wiki.**
|
||||
|
||||
---
|
||||
|
||||
## Why HaBraid exists
|
||||
|
||||
Raw memory systems are great at **retaining information**, but not always great at **presenting relationships**.
|
||||
|
||||
Typical problems:
|
||||
|
||||
- too many small raw records
|
||||
- duplicated context across sessions
|
||||
- hidden connections between decisions, people, projects, and infra
|
||||
- noisy graph view when raw files live in the same vault
|
||||
- backend data that agents can search, but humans can't comfortably read
|
||||
|
||||
HaBraid exists to solve that gap.
|
||||
|
||||
It takes a backend that is optimized for retention and query, and adds a layer optimized for:
|
||||
|
||||
- structure
|
||||
- linking
|
||||
- summarization
|
||||
- graph clarity
|
||||
- handoff between people and AI systems
|
||||
|
||||
---
|
||||
|
||||
## Core ideas
|
||||
|
||||
### 1. Source of truth stays in the memory backend
|
||||
|
||||
MemPalace, mem0, or any future memory backend remains the canonical store.
|
||||
|
||||
HaBraid is intentionally **not** the source of truth.
|
||||
|
||||
### 2. The wiki is the expression layer
|
||||
|
||||
The wiki is where information becomes understandable.
|
||||
|
||||
That means HaBraid focuses on:
|
||||
|
||||
- grouping scattered facts into topics
|
||||
- linking related pages with wikilinks
|
||||
- preserving important provenance in metadata
|
||||
- exposing concepts, decisions, people, infra, and projects as readable pages
|
||||
|
||||
### 3. Raw should be hidden, wiki should be visible
|
||||
|
||||
To keep Obsidian graph view useful:
|
||||
|
||||
- raw/source/debug material belongs in hidden runtime storage
|
||||
- curated wiki pages belong in the user vault
|
||||
|
||||
That means the long-term target structure is:
|
||||
|
||||
- `~/.habraid/data/raw` → internal source material
|
||||
- `~/wiki` → curated user-facing vault
|
||||
|
||||
### 4. Inference should follow the host when possible
|
||||
|
||||
HaBraid prefers **host-following generation**.
|
||||
|
||||
Instead of forcing its own internal model, it can delegate generation to the current host environment. In practice this means:
|
||||
|
||||
- HaBraid orchestrates prompts, batching, parsing, and persistence
|
||||
- the host agent chooses the actual model/policy
|
||||
- standalone fallback remains available when needed
|
||||
|
||||
---
|
||||
|
||||
## At a glance
|
||||
|
||||
| Area | What HaBraid does |
|
||||
|------|--------------------|
|
||||
| Source layer | Reads memory backends through adapters |
|
||||
| Storage layer | Maintains a local DB for item tracking and generation state |
|
||||
| Search layer | Hybrid BM25 + HNSW vector search with context enrichment |
|
||||
| Graph layer | Knowledge graph with entities, relations, and community detection |
|
||||
| Generation layer | Uses host-following or fallback LLM routes to create curated pages |
|
||||
| Lint layer | 3-tier vault validation (static, HNSW duplicates, contradiction detection) |
|
||||
| Vault layer | Writes graph-friendly wiki pages for Obsidian |
|
||||
| Integration layer | Exposes 16 MCP tools for agent-driven workflows |
|
||||
|
||||
### Good fit for
|
||||
|
||||
- long-term memory systems that are strong at storage but weak at presentation
|
||||
- Obsidian users who want cleaner graph structure
|
||||
- agent systems that need a readable shared knowledge layer
|
||||
- infra / project / decision archives that have too many raw records
|
||||
|
||||
### Not trying to be
|
||||
|
||||
- a replacement for MemPalace or other memory backends
|
||||
- a generic note-taking app
|
||||
- a fully standalone model-serving platform
|
||||
- a raw log browser inside the visible vault
|
||||
|
||||
---
|
||||
|
||||
## Demo workflow
|
||||
|
||||
A typical HaBraid loop looks like this:
|
||||
|
||||
1. new memory items arrive in the backend
|
||||
2. HaBraid ingests them into the local item DB
|
||||
3. `hw_generate` batches ungenerated items
|
||||
4. the host agent generates or updates curated wiki pages
|
||||
5. Obsidian opens `~/wiki` and shows the cleaned graph
|
||||
|
||||
### Example result
|
||||
|
||||
```text
|
||||
raw memory fragments
|
||||
├─ deploy validation logs
|
||||
├─ infra decisions
|
||||
├─ session summaries
|
||||
└─ people / project facts
|
||||
|
||||
↓ HaBraid
|
||||
|
||||
curated wiki pages
|
||||
├─ wiki/decisions/deploy-gate-signals-for-mvp-rollouts.md
|
||||
├─ wiki/guides/session-bootstrap-and-deploy-manager-prompts.md
|
||||
├─ wiki/projects/project-alpha.md
|
||||
├─ wiki/projects/mijung-ai-foodtech-platform.md
|
||||
└─ wiki/decisions/nestjs-circular-dependency-response.md
|
||||
```
|
||||
|
||||
The important change is not just format conversion.
|
||||
It is **semantic compression + linkage**.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
### Core engine
|
||||
|
||||
- **Host-following LLM route**
|
||||
- host-first generation path
|
||||
- fallback provider support (OpenAI, Ollama, ZAI)
|
||||
- route/provider/model observability in runtime state
|
||||
|
||||
- **Incremental wiki generation**
|
||||
- only ungenerated items are processed
|
||||
- DB-backed generation tracking via `wiki_generated_at`
|
||||
- restart-friendly flow
|
||||
|
||||
- **Checkpointed long-running generation**
|
||||
- subgroup-level persistence during generation
|
||||
- partial progress survives failures
|
||||
- large rebuilds no longer wait until the very end to write files
|
||||
|
||||
### Search & Retrieval
|
||||
|
||||
- **Hybrid search (BM25 + HNSW)**
|
||||
- BM25 keyword search via SQLite FTS5
|
||||
- Semantic vector search with ONNX local embeddings
|
||||
- HNSW ANN index for fast approximate nearest neighbor
|
||||
- Context enrichment mode with KG relations and wiki link re-ranking
|
||||
|
||||
### Knowledge Graph
|
||||
|
||||
- **Entity and relation management**
|
||||
- Typed entities: concept, project, person, tool, event, decision
|
||||
- Predicate-based relations with confidence scores
|
||||
- Source tracking (extracted, inferred, ambiguous)
|
||||
|
||||
- **Community detection**
|
||||
- Label Propagation Algorithm (LPA) clustering
|
||||
- Automatic community assignment for graph visualization
|
||||
- Mermaid diagram export for KG subgraphs
|
||||
|
||||
### Quality & Validation
|
||||
|
||||
- **3-tier lint system**
|
||||
- Tier 1: Static checks (orphans, broken links, frontmatter, ungenerated items)
|
||||
- Tier 2: HNSW duplicate detection (semantic near-duplicate identification)
|
||||
- Tier 3: LLM-powered contradiction detection across items
|
||||
|
||||
- **Contradiction tracking**
|
||||
- Automatic conflict detection between knowledge items
|
||||
- Resolution workflow (resolve / false positive)
|
||||
- Open/resolved status management
|
||||
|
||||
### Content
|
||||
|
||||
- **Q&A Synthesis**
|
||||
- Karpathy LLM Wiki pattern: question-answer pairs become wiki pages
|
||||
- Compound knowledge accumulation from source items
|
||||
|
||||
- **Daily work log**
|
||||
- Automatic generation activity logging
|
||||
- Date-range summaries for retrospectives
|
||||
|
||||
### Integration
|
||||
|
||||
- **MCP server**
|
||||
- 16 tools covering the full operational loop
|
||||
- Hermes config.yaml registration
|
||||
- tsx runtime for TypeScript MCP server
|
||||
|
||||
- **SQLite-backed item store**
|
||||
- local item database (500+ items in production)
|
||||
- schema integrity auto-repair on startup
|
||||
- generation metadata and sync state
|
||||
|
||||
- **Obsidian-friendly output**
|
||||
- YAML frontmatter
|
||||
- wikilinks
|
||||
- category-based page layout
|
||||
- graph-oriented curated pages
|
||||
|
||||
---
|
||||
|
||||
## Architecture diagram
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Memory backend\nMemPalace / future sources] --> B[Ingest adapters]
|
||||
B --> C[Local item DB\nSQLite + FTS5 + HNSW vectors]
|
||||
C --> D[Prompt builder + batching]
|
||||
D --> E[LLM gateway]
|
||||
E -->|preferred| F[Host-following generation]
|
||||
E -->|fallback| G[Standalone backend]
|
||||
F --> H[Curated wiki output]
|
||||
G --> H
|
||||
H --> I[~/wiki in Obsidian]
|
||||
C --> J[MCP tools\nhw_status / hw_search / hw_generate / ...]
|
||||
C --> K[Knowledge Graph\nentities + relations + communities]
|
||||
K --> J
|
||||
C --> L[3-tier Lint\nstatic + HNSW dup + contradictions]
|
||||
L --> J
|
||||
```
|
||||
|
||||
### Separation of concerns
|
||||
|
||||
- **backend** keeps durable memory
|
||||
- **HaBraid** performs orchestration, transformation, and graph extraction
|
||||
- **host** owns model selection when available
|
||||
- **Obsidian** becomes the readable interface layer
|
||||
|
||||
---
|
||||
|
||||
## How it works
|
||||
|
||||
```text
|
||||
memory backend (MemPalace / future sources)
|
||||
│
|
||||
▼
|
||||
ingest / adapters
|
||||
│
|
||||
▼
|
||||
local item DB (SQLite)
|
||||
├── FTS5 keyword index
|
||||
├── HNSW vector index
|
||||
└── KG entity/relation tables
|
||||
│
|
||||
├──→ hybrid search (BM25 + semantic)
|
||||
├──→ knowledge graph (entities + communities)
|
||||
├──→ 3-tier lint (static + dup + contradiction)
|
||||
│
|
||||
▼
|
||||
prompt builder + batching
|
||||
│
|
||||
▼
|
||||
host-following or fallback generation
|
||||
│
|
||||
▼
|
||||
curated Obsidian wiki
|
||||
```
|
||||
|
||||
The important design split is:
|
||||
|
||||
- **backend keeps memory**
|
||||
- **HaBraid structures memory**
|
||||
- **Obsidian presents memory**
|
||||
|
||||
---
|
||||
|
||||
## Directory layout
|
||||
|
||||
Current runtime-oriented structure:
|
||||
|
||||
```text
|
||||
~/.habraid/
|
||||
├── app/ # source repo (57 TS files, 12K+ LOC)
|
||||
│ ├── src/
|
||||
│ │ ├── mcp-server.ts # MCP server entry point
|
||||
│ │ ├── db/ # SQLite DB, migrations, schema
|
||||
│ │ ├── search/ # Hybrid BM25 + HNSW search
|
||||
│ │ ├── kg/ # Knowledge graph + community detection
|
||||
│ │ ├── lint/ # 3-tier lint system
|
||||
│ │ ├── wiki/ # Wiki generation engine
|
||||
│ │ └── tools/ # MCP tool handlers
|
||||
│ └── dist/ # compiled output
|
||||
└── data/ # runtime state
|
||||
├── config.json
|
||||
├── habraid.db # SQLite (items, vectors, KG, FTS5)
|
||||
├── logs/
|
||||
├── models/ # ONNX embedding model cache
|
||||
├── backups/
|
||||
└── raw/ # internal raw cache / source material
|
||||
|
||||
~/wiki/
|
||||
├── index.md
|
||||
├── overview.md
|
||||
├── log.md
|
||||
└── wiki/ # curated pages visible to the user
|
||||
├── projects/
|
||||
├── topics/
|
||||
├── decisions/
|
||||
├── guides/
|
||||
└── infrastructure/
|
||||
```
|
||||
|
||||
### Intended separation
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `~/.habraid/app` | application source code |
|
||||
| `~/.habraid/data` | runtime DB, config, logs, models, raw cache |
|
||||
| `~/wiki` | curated user-facing Obsidian vault |
|
||||
|
||||
This split is intentional:
|
||||
|
||||
- runtime state stays hidden and operational
|
||||
- user-facing graph remains cleaner
|
||||
- repo, DB, and generated output do not all fight for the same surface
|
||||
|
||||
---
|
||||
|
||||
## Quick start
|
||||
|
||||
### Requirements
|
||||
|
||||
- Node.js 18+
|
||||
- npm
|
||||
- Obsidian (optional but recommended for browsing the vault)
|
||||
- Hermes / host environment if using host-following mode
|
||||
- MemPalace if you want live backend ingestion
|
||||
|
||||
### Install
|
||||
|
||||
```bash
|
||||
cd ~/.habraid/app
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
### Config
|
||||
|
||||
Primary config path:
|
||||
|
||||
```text
|
||||
~/.habraid/data/config.json
|
||||
```
|
||||
|
||||
Related references:
|
||||
|
||||
- `docs/config-reference.md`
|
||||
- legacy fallbacks still exist for compatibility, but the runtime target is `~/.habraid/data/config.json`
|
||||
|
||||
### Run locally
|
||||
|
||||
Development:
|
||||
|
||||
```bash
|
||||
cd ~/.habraid/app
|
||||
npm run dev
|
||||
```
|
||||
|
||||
Built server:
|
||||
|
||||
```bash
|
||||
cd ~/.habraid/app
|
||||
node dist/mcp-server.js
|
||||
```
|
||||
|
||||
### Register with Hermes
|
||||
|
||||
Add to `~/.hermes/config.yaml`:
|
||||
|
||||
```yaml
|
||||
mcp_servers:
|
||||
habraid:
|
||||
command: npx
|
||||
args: ["tsx", "/path/to/.habraid/app/src/mcp-server.ts"]
|
||||
```
|
||||
|
||||
### Open the curated vault
|
||||
|
||||
```text
|
||||
~/wiki
|
||||
```
|
||||
|
||||
That is the path intended for Obsidian.
|
||||
|
||||
---
|
||||
|
||||
## MCP tools
|
||||
|
||||
HaBraid exposes 16 MCP tools under the `hw_*` namespace.
|
||||
|
||||
### Tool summary
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `hw_status` | Show vault, DB, and generation route status |
|
||||
| `hw_ingest` | Ingest source material into raw cache and local DB |
|
||||
| `hw_index` | Build vector embeddings and HNSW ANN index |
|
||||
| `hw_search` | Search items using keyword / semantic / hybrid modes |
|
||||
| `hw_add` | Add a knowledge item directly to the DB |
|
||||
| `hw_read` | Read files inside the vault |
|
||||
| `hw_generate` | Generate or update wiki pages incrementally |
|
||||
| `hw_sync` | Run the full pipeline: ingest → index → generate |
|
||||
| `hw_lint` | 3-tier vault validation (static + HNSW dup + contradictions) |
|
||||
| `hw_graph` | Knowledge graph query, entity subgraph, community detection, Mermaid export |
|
||||
| `hw_entity_add` | Add typed entities to the knowledge graph |
|
||||
| `hw_entity_search` | Search entities by name, ID, or type |
|
||||
| `hw_relation_add` | Add relations between entities |
|
||||
| `hw_contradictions` | Detect and manage contradictions across knowledge items |
|
||||
| `hw_daily_log` | Generate or read daily work log summaries |
|
||||
| `hw_synthesize` | Save Q&A as synthesized wiki pages (Karpathy LLM Wiki pattern) |
|
||||
|
||||
### Operational loop
|
||||
|
||||
```text
|
||||
ingest → index → generate → lint → search → graph → synthesize
|
||||
```
|
||||
|
||||
These cover the main workflows:
|
||||
|
||||
- **inspect**: status, daily log
|
||||
- **ingest & index**: bring data in, build embeddings
|
||||
- **generate & sync**: create curated pages
|
||||
- **search**: retrieve items with BM25 + semantic hybrid
|
||||
- **graph**: explore entities, relations, communities
|
||||
- **lint**: validate quality (orphans, broken links, duplicates, contradictions)
|
||||
- **synthesize**: accumulate compound knowledge
|
||||
|
||||
---
|
||||
|
||||
## Example MCP flows
|
||||
|
||||
### Inspect current runtime state
|
||||
|
||||
```text
|
||||
User: "HaBraid 상태 봐줘"
|
||||
→ hw_status
|
||||
→ raw files / wiki pages / DB items / last LLM route 확인
|
||||
```
|
||||
|
||||
### Ingest then generate
|
||||
|
||||
```text
|
||||
User: "새 memory 반영하고 위키 생성해줘"
|
||||
→ hw_ingest
|
||||
→ hw_index
|
||||
→ hw_generate
|
||||
```
|
||||
|
||||
### Search before opening notes
|
||||
|
||||
```text
|
||||
User: "project-alpha 관련 의사결정 찾아줘"
|
||||
→ hw_search(query="project-alpha", mode="hybrid", context=true)
|
||||
→ KG relations, wiki links, re-ranked results
|
||||
```
|
||||
|
||||
### Full pipeline
|
||||
|
||||
```text
|
||||
User: "전체 sync 돌려줘"
|
||||
→ hw_sync
|
||||
```
|
||||
|
||||
### Knowledge graph exploration
|
||||
|
||||
```text
|
||||
User: "nestjs 순환참조 관련 엔티티와 관계 보여줘"
|
||||
→ hw_entity_search(query="nestjs")
|
||||
→ hw_graph(entity="nestjs-circular-dep", depth=2, format="mermaid")
|
||||
```
|
||||
|
||||
### Lint and fix issues
|
||||
|
||||
```text
|
||||
User: "위키 품질 검사해줘"
|
||||
→ hw_lint(checks=["orphans", "broken_links", "frontmatter"])
|
||||
→ hw_lint(checks=["duplicates"]) # Tier 2: HNSW-based
|
||||
→ hw_lint(checks=["contradictions"], withLlm=true) # Tier 3: LLM-based
|
||||
```
|
||||
|
||||
### Synthesize compound knowledge
|
||||
|
||||
```text
|
||||
User: "이 질문에 대한 답변을 위키로 남겨줘"
|
||||
→ hw_synthesize(question="...", answer="...", tags=["..."])
|
||||
→ wiki/topics/ 생성
|
||||
```
|
||||
|
||||
These flows are the intended operator experience: HaBraid should feel like a structured memory engine behind a clean MCP interface.
|
||||
|
||||
---
|
||||
|
||||
## Current status
|
||||
|
||||
HaBraid is already usable in production with real data.
|
||||
|
||||
### Production metrics
|
||||
|
||||
- **500+** knowledge items in DB
|
||||
- **500+** vector embeddings indexed
|
||||
- **69** curated wiki pages generated
|
||||
- **57** TypeScript source files, **12K+** LOC
|
||||
- **16** MCP tools
|
||||
|
||||
### Stable enough to use
|
||||
|
||||
- hidden runtime layout (`~/.habraid/`)
|
||||
- MCP-driven workflow with 16 tools
|
||||
- host-following generation path
|
||||
- checkpointed regeneration
|
||||
- hybrid BM25 + HNSW search
|
||||
- knowledge graph with entities, relations, and community detection
|
||||
- 3-tier lint system (static + HNSW dup + contradictions)
|
||||
- Q&A synthesis (Karpathy LLM Wiki pattern)
|
||||
- curated wiki output in `~/wiki`
|
||||
- schema integrity auto-repair
|
||||
|
||||
### Not final yet
|
||||
|
||||
- hidden raw isolation is still being tightened
|
||||
- generation quality and category/slug consistency still need iteration
|
||||
- direct MemPalace ingest remains a separate bugfix track
|
||||
- contradiction detection LLM integration is early stage
|
||||
|
||||
Think of the current state as:
|
||||
|
||||
> **operational alpha with real output, real infra value, and active architectural cleanup**
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
### Near term
|
||||
|
||||
- move raw fully out of the user vault
|
||||
- improve graph-friendly page relationships
|
||||
- finish runtime path cleanup around hidden raw storage
|
||||
- harden continuation behavior for partial generation failures
|
||||
- expand lint auto-fix capabilities
|
||||
|
||||
### Mid term
|
||||
|
||||
- improve page deduplication and merge quality
|
||||
- tighten category placement and slug stability
|
||||
- better quality control for generated frontmatter and links
|
||||
- improve wiki index / overview / navigation surfaces
|
||||
- richer contradiction detection with cross-source verification
|
||||
|
||||
### Longer term
|
||||
|
||||
- richer hybrid retrieval with re-ranking
|
||||
- backend abstraction beyond MemPalace
|
||||
- stronger entity / concept / decision graph extraction
|
||||
- more productized onboarding and one-click setup
|
||||
- real-time sync with live memory backends
|
||||
|
||||
---
|
||||
|
||||
## Documentation
|
||||
|
||||
### Product / architecture docs
|
||||
|
||||
- `docs/architecture.md`
|
||||
- `docs/config-reference.md`
|
||||
- `docs/wiki-generation.md`
|
||||
- `docs/specs/host-following-llm.md`
|
||||
|
||||
### Planning docs
|
||||
|
||||
- `docs/plans/README.md`
|
||||
- `docs/plans/PLAN.md`
|
||||
- `docs/plans/PLAN-v2.md`
|
||||
- `docs/plans/PLAN-MCP.md`
|
||||
- `docs/plans/PLAN-VECTOR.md`
|
||||
- `docs/plans/PLAN-WIKI-LINT.md` — 3-tier lint system design
|
||||
|
||||
### Session-scoped plans
|
||||
|
||||
- `.hermes/plans/`
|
||||
|
||||
---
|
||||
|
||||
## Design stance
|
||||
|
||||
HaBraid is built around a specific product stance:
|
||||
|
||||
- memory systems should remain good at memory
|
||||
- host agents should remain in charge of inference
|
||||
- the wiki should become the shared interpretation layer
|
||||
- graph quality matters more than raw completeness inside the visible vault
|
||||
- knowledge should compound over time through synthesis
|
||||
|
||||
If the backend is the archive, HaBraid is the map.
|
||||
89
docs/architecture.md
Normal file
89
docs/architecture.md
Normal file
@@ -0,0 +1,89 @@
|
||||
# 아키텍처 설계
|
||||
|
||||
## 핵심 철학
|
||||
|
||||
HaBraid는 memory backend 자체가 아니라 **기억 시각화 계층**이다.
|
||||
|
||||
- source of truth: MemPalace / mem0 / future memory backends
|
||||
- expression layer: HaBraid wiki
|
||||
- consumer: 사람 + AI
|
||||
|
||||
즉 HaBraid는 장기 기억을 사람이 읽기 좋고 AI가 다시 활용하기 좋은 위키 표현층으로 재구성한다.
|
||||
|
||||
## 데이터 흐름
|
||||
|
||||
```text
|
||||
memory backend (MemPalace / mem0 / future backends)
|
||||
│
|
||||
▼ read-only / adapters
|
||||
┌──────────┐
|
||||
│ ingest │ source → raw/ + DB items
|
||||
└────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌──────────┐
|
||||
│ wiki │ items/raw → wiki/ (host-following LLM)
|
||||
└────┬─────┘
|
||||
│
|
||||
▼
|
||||
┌──────────┐
|
||||
│ sync │ index/log/state + optional git sync
|
||||
└──────────┘
|
||||
```
|
||||
|
||||
## LLM 실행 계층
|
||||
|
||||
```text
|
||||
prompt builder
|
||||
↓
|
||||
LLM gateway
|
||||
├─ host mode (default)
|
||||
│ └─ host가 실제 모델/정책 선택
|
||||
└─ standalone / fallback
|
||||
└─ openai | zai | ollama 직접 호출
|
||||
```
|
||||
|
||||
원칙:
|
||||
- HaBraid는 모델 선택보다 **prompt orchestration**에 집중한다.
|
||||
- 가능하면 host가 실제 inference를 담당한다.
|
||||
- host unavailable이면 configured fallback을 사용한다.
|
||||
|
||||
## 모듈 의존성
|
||||
|
||||
```text
|
||||
cli/commands.ts
|
||||
├── sync/pipeline.ts
|
||||
│ ├── sync/git.ts
|
||||
│ ├── mempalace/ingest.ts
|
||||
│ │ └── mempalace/reader.ts
|
||||
│ └── wiki/generator.ts
|
||||
│ ├── wiki/llm.ts
|
||||
│ └── wiki/prompts.ts
|
||||
├── vault/init.ts
|
||||
├── vault/render.ts
|
||||
├── vault/index.ts
|
||||
└── config.ts → types.ts
|
||||
```
|
||||
|
||||
## 에러 처리
|
||||
|
||||
모든 모듈은 `WikiEngineError` 기반 커스텀 에러 사용.
|
||||
|
||||
대표 코드:
|
||||
- `VAULT_INIT_FAILED`
|
||||
- `DB_READ_FAILED`
|
||||
- `LLM_CALL_FAILED`
|
||||
- `GIT_SYNC_FAILED`
|
||||
- `LINT_FAILED`
|
||||
- `CONFIG_LOAD_FAILED`
|
||||
|
||||
## 동시성
|
||||
|
||||
- ingest: 순차 처리
|
||||
- wiki generation: 순차 처리 (배치 단위)
|
||||
- git sync: 순차 처리
|
||||
|
||||
## 현재 알려진 분리 이슈
|
||||
|
||||
- direct MemPalace ingest는 아직 Chroma schema mismatch 문제가 남아 있다.
|
||||
- host-following LLM 전환은 이 ingest 버그와 별도 change set으로 유지한다.
|
||||
148
docs/config-reference.md
Normal file
148
docs/config-reference.md
Normal file
@@ -0,0 +1,148 @@
|
||||
# 설정 파일 레퍼런스
|
||||
|
||||
## 위치
|
||||
|
||||
기본 경로:
|
||||
- `~/.habraid/data/config.json`
|
||||
|
||||
하위 호환 / 명시 실행 경로:
|
||||
- `HABRAID_CONFIG=/path/to/config.json`
|
||||
- `WIKI_ENGINE_CONFIG=/path/to/config.json`
|
||||
- 레거시 fallback: `~/.habraid/config.json`, `~/.config/habraid/config.json`
|
||||
|
||||
## 스키마
|
||||
|
||||
```typescript
|
||||
interface WikiEngineConfig {
|
||||
vault: {
|
||||
path: string;
|
||||
git_remote?: string;
|
||||
branch: string;
|
||||
};
|
||||
db: {
|
||||
path: string;
|
||||
};
|
||||
mempalace: {
|
||||
enabled: boolean;
|
||||
path: string;
|
||||
};
|
||||
llm: {
|
||||
mode: "host" | "standalone";
|
||||
preferences?: {
|
||||
priority?: "fast" | "balanced" | "smart";
|
||||
};
|
||||
fallback?: {
|
||||
provider: "openai" | "zai" | "glm" | "ollama";
|
||||
model: string;
|
||||
api_url: string;
|
||||
api_key_env: string;
|
||||
max_tokens: number;
|
||||
};
|
||||
|
||||
// legacy compatibility fields
|
||||
provider: string;
|
||||
model: string;
|
||||
api_url: string;
|
||||
api_key_env: string;
|
||||
max_tokens: number;
|
||||
};
|
||||
sync: {
|
||||
timezone: string;
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
## 권장 기본값
|
||||
|
||||
```json
|
||||
{
|
||||
"vault": {
|
||||
"path": "~/wiki",
|
||||
"branch": "main"
|
||||
},
|
||||
"db": {
|
||||
"path": "~/.habraid/data/habraid.db"
|
||||
},
|
||||
"mempalace": {
|
||||
"enabled": true,
|
||||
"path": ""
|
||||
},
|
||||
"llm": {
|
||||
"mode": "host",
|
||||
"preferences": {
|
||||
"priority": "balanced"
|
||||
},
|
||||
"fallback": {
|
||||
"provider": "zai",
|
||||
"model": "glm-5.1",
|
||||
"api_url": "https://api.example.com/v1",
|
||||
"api_key_env": "GLM_API_KEY",
|
||||
"max_tokens": 4096
|
||||
},
|
||||
"provider": "zai",
|
||||
"model": "glm-5.1",
|
||||
"api_url": "https://api.example.com/v1",
|
||||
"api_key_env": "GLM_API_KEY",
|
||||
"max_tokens": 4096
|
||||
},
|
||||
"sync": {
|
||||
"timezone": "Asia/Seoul"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 동작 원칙
|
||||
|
||||
### `llm.mode = "host"`
|
||||
- host(Hermes/OpenClaw/Codex CLI)가 가능하면 추론을 담당
|
||||
- host bridge가 unavailable이면 `llm.fallback` 사용
|
||||
- `hw_status` / sync state에 마지막 route/provider/model이 기록될 수 있음
|
||||
|
||||
### `llm.mode = "standalone"`
|
||||
- habraid가 직접 backend 호출
|
||||
- legacy config(`provider/model/api_url/api_key_env`)는 fallback 정보를 제공한다.
|
||||
- `mode`를 명시하지 않은 기존 config도 기본적으로 host-first로 해석되고, legacy llm 필드는 fallback/runtime metadata로 유지된다.
|
||||
|
||||
## 레거시 설정 예시
|
||||
|
||||
다음 형태도 계속 읽을 수 있다.
|
||||
|
||||
```json
|
||||
{
|
||||
"llm": {
|
||||
"provider": "zai",
|
||||
"model": "glm-5.1",
|
||||
"api_url": "https://api.example.com/v1",
|
||||
"api_key_env": "GLM_API_KEY",
|
||||
"max_tokens": 4096
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
이 경우 내부 normalize 시 standalone-compatible runtime config로 변환된다.
|
||||
|
||||
## 설정 파일 로드 순서
|
||||
|
||||
1. explicit path (`loadConfig(path)` 또는 CLI `--config`)
|
||||
2. `WIKI_ENGINE_CONFIG`
|
||||
3. `HABRAID_CONFIG`
|
||||
4. `~/.habraid/data/config.json`
|
||||
5. legacy fallback (`~/.habraid/config.json`, `~/.config/habraid/config.json`)
|
||||
6. built-in defaults
|
||||
|
||||
## API 키
|
||||
|
||||
`.env` 검색 경로:
|
||||
- `~/.hermes/.env`
|
||||
- `~/.openclaw/.env`
|
||||
- `./.env`
|
||||
|
||||
예시:
|
||||
|
||||
```bash
|
||||
GLM_API_KEY=***
|
||||
OPENAI_API_KEY=***
|
||||
OPENROUTER_API_KEY=***
|
||||
```
|
||||
|
||||
실제 사용하는 키는 선택된 standalone backend 또는 fallback의 `api_key_env`를 따른다.
|
||||
86
docs/mempalace-integration.md
Normal file
86
docs/mempalace-integration.md
Normal file
@@ -0,0 +1,86 @@
|
||||
# MemPalace 연동 명세
|
||||
|
||||
## MemPalace 데이터 구조
|
||||
|
||||
MemPalace는 SQLite + ChromaDB로 구성:
|
||||
- **SQLite** (`palace.db`): 서랍 메타데이터 + Knowledge Graph
|
||||
- **ChromaDB** (`chroma_db/`): 벡터 임베딩 (직접 접근 안 함)
|
||||
|
||||
## DB 스키마 (SQLite)
|
||||
|
||||
### drawers 테이블
|
||||
|
||||
```sql
|
||||
CREATE TABLE drawers (
|
||||
id TEXT PRIMARY KEY,
|
||||
wing TEXT NOT NULL,
|
||||
room TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
added_by TEXT DEFAULT 'mcp',
|
||||
source_file TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### knowledge_graph 테이블
|
||||
|
||||
```sql
|
||||
CREATE TABLE kg_facts (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subject TEXT NOT NULL,
|
||||
predicate TEXT NOT NULL,
|
||||
object TEXT NOT NULL,
|
||||
valid_from DATE,
|
||||
valid_to DATE,
|
||||
source_closet TEXT,
|
||||
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
## reader.ts 인터페이스
|
||||
|
||||
```typescript
|
||||
interface MemPalaceReader {
|
||||
// DB 열기 (read-only, WAL 모드)
|
||||
open(dbPath: string): Promise<void>;
|
||||
|
||||
// 전체 서랍 조회
|
||||
getAllDrawers(): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
// 특정 시간 이후 변경된 서랍
|
||||
getDrawersSince(since: Date): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
// 특정 wing/room의 서랍
|
||||
getDrawersByWingRoom(wing: string, room?: string): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
// 전체 KG 팩트
|
||||
getAllFacts(): Promise<KGFact[]>;
|
||||
|
||||
// DB 닫기
|
||||
close(): void;
|
||||
}
|
||||
```
|
||||
|
||||
## ingest.ts 로직
|
||||
|
||||
1. `last_sync` 타임스탬프 읽기 (없으면 전체)
|
||||
2. `getDrawersSince(last_sync)`로 신규 서랍 조회
|
||||
3. 각 서랍을 frontmatter 포함 마크다운으로 렌더링
|
||||
4. `raw/mempalace/{wing}/{room}/{id}.md`에 저장
|
||||
5. `last_sync` 업데이트
|
||||
6. index.md, log.md 갱신
|
||||
|
||||
## 동기화 상태 저장
|
||||
|
||||
`{vault}/.sync-state.json`:
|
||||
|
||||
```json
|
||||
{
|
||||
"last_ingest": "2026-04-15T13:00:00+09:00",
|
||||
"last_wiki_update": "2026-04-15T13:05:00+09:00",
|
||||
"last_git_push": "2026-04-15T13:05:30+09:00",
|
||||
"ingested_drawers": ["abc123", "def456"],
|
||||
"wiki_pages": ["project-alpha", "project-beta"]
|
||||
}
|
||||
```
|
||||
73
docs/plans/PLAN-MCP.md
Normal file
73
docs/plans/PLAN-MCP.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# PLAN: habraid MCP 서버 추가
|
||||
|
||||
## 목표
|
||||
habraid에 MCP (Model Context Protocol) 서버를 추가해서,
|
||||
Hermes가 habraid의 기능을 도구로 직접 호출할 수 있게 한다.
|
||||
|
||||
## 배경
|
||||
현재 habraid은 CLI로만 동작함. Hermes가 위키 상태를 확인하거나
|
||||
수동으로 sync를 돌리려면 terminal에서 명령어를 쳐야 함.
|
||||
MCP 서버를 붙이면 Hermes가 자연어로 "위키 상태 확인해줘" →
|
||||
habraid MCP 도구 호출로 바로 실행 가능.
|
||||
|
||||
## 설계
|
||||
|
||||
### MCP 서버 방식
|
||||
- **stdio transport** (Hermes MCP 클라이언트와 동일 방식)
|
||||
- 진입점: `src/mcp-server.ts`
|
||||
- 실행: `npx tsx src/mcp-server.ts`
|
||||
|
||||
### 제공 도구 (Tools)
|
||||
|
||||
| 도구명 | 설명 | 파라미터 |
|
||||
|--------|------|----------|
|
||||
| wiki_status | Vault 통계 조회 | 없음 |
|
||||
| wiki_lint | Vault 무결성 검사 | 없음 |
|
||||
| wiki_ingest | MemPalace → raw/ 수집 | full (boolean, optional) |
|
||||
| wiki_generate | 위키 페이지 생성/갱신 | 없음 |
|
||||
| wiki_sync | 전체 파이프라인 실행 | noWiki (boolean, optional) |
|
||||
| wiki_read | 위키 페이지 내용 읽기 | path (string) |
|
||||
| wiki_search | 위키 페이지 검색 | query (string) |
|
||||
|
||||
### MCP 서버 구현 위치
|
||||
```
|
||||
src/
|
||||
├── mcp-server.ts # MCP 서버 진입점 (stdio transport)
|
||||
└── mcp/
|
||||
├── tools.ts # 도구 스키마 정의
|
||||
└── handlers.ts # 도구 핸들러 (기존 모듈 재사용)
|
||||
```
|
||||
|
||||
### 의존성 추가
|
||||
- `@modelcontextprotocol/sdk` — MCP 서버 프레임워크
|
||||
|
||||
### Hermes 설정
|
||||
```yaml
|
||||
mcp_servers:
|
||||
mempalace:
|
||||
command: ~/.mempalace/venv/bin/python
|
||||
args:
|
||||
- -m
|
||||
- mempalace.mcp_server
|
||||
- --palace
|
||||
- ~/.mempalace/palace
|
||||
habraid:
|
||||
command: ~/.habraid/app/node_modules/.bin/tsx
|
||||
args:
|
||||
- ~/.habraid/app/src/mcp-server.ts
|
||||
```
|
||||
|
||||
## 구현 순서
|
||||
|
||||
1. `@modelcontextprotocol/sdk` 설치
|
||||
2. `src/mcp/tools.ts` — 도구 스키마 정의
|
||||
3. `src/mcp/handlers.ts` — 핸들러 구현 (기존 모듈 래핑)
|
||||
4. `src/mcp-server.ts` — 서버 진입점
|
||||
5. 빌드 + 동작 테스트
|
||||
6. Hermes config.yaml에 MCP 서버 등록
|
||||
7. Hermes 재시작 후 도구 인식 확인
|
||||
|
||||
## 주의사항
|
||||
- 기존 CLI 기능은 그대로 유지 (MCP는 추가 기능)
|
||||
- MCP 핸들러는 CLI commands의 로직을 직접 재사용
|
||||
- config.json 경로는 환경변수나 기본 경로 사용
|
||||
126
docs/plans/PLAN-VECTOR.md
Normal file
126
docs/plans/PLAN-VECTOR.md
Normal file
@@ -0,0 +1,126 @@
|
||||
# PLAN: HaBraid Vector Search 추가
|
||||
|
||||
> 2026-04-16, Contributor
|
||||
|
||||
## 목표
|
||||
|
||||
HaBraid에 fastembed(ONNX) 기반 vector 임베딩을 추가하고,
|
||||
BM25(FTS5) + Vector 유사도를 RRF로 결합하는 **하이브리드 검색** 구현.
|
||||
|
||||
## 환경 제약
|
||||
|
||||
- CPU-only (i5-9600K, AVX2), GPU 없음
|
||||
- RAM 8GB 중 ~1.2GB 사용 가능
|
||||
- Python 3.10, Node.js
|
||||
- 디스크 16GB 여유
|
||||
|
||||
## 아키텍처
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
│ hw_search │
|
||||
│ "한국어 쿼리" │
|
||||
└─────────┬──────────────┬────────────┘
|
||||
│ │
|
||||
┌─────▼─────┐ ┌─────▼──────┐
|
||||
│ BM25 │ │ Vector │
|
||||
│ FTS5 │ │ fastembed │
|
||||
│ SQLite │ │ ONNX │
|
||||
└─────┬──────┘ └─────┬──────┘
|
||||
│ │
|
||||
│ RRF(k=60) │
|
||||
└───────┬───────┘
|
||||
▼
|
||||
하이브리드 결과
|
||||
```
|
||||
|
||||
## 모델 선택
|
||||
|
||||
**`BAAI/bge-small-en-v1.5`** (기본) 또는 **`intfloat/multilingual-e5-small`** (한국어)
|
||||
|
||||
| 모델 | 차원 | 크기 | 한국어 | 속도 |
|
||||
|---|---|---|---|---|
|
||||
| bge-small-en-v1.5 | 384 | ~130MB | ❌ | 빠름 |
|
||||
| multilingual-e5-small | 384 | ~470MB | ✅ | 보통 |
|
||||
|
||||
→ 기본: `multilingual-e5-small` (한국어 지원이 필수)
|
||||
|
||||
## 구현 계획
|
||||
|
||||
### 1. Python 임베딩 서비스 (`scripts/embed.py`)
|
||||
|
||||
```python
|
||||
# stdin으로 JSON → stdout으로 JSON
|
||||
# 모드: embed (텍스트 → 벡터), index (DB 아이템 전체 임베딩)
|
||||
# 모델 첫 로드 후 캐시 (재실행 필요 없음)
|
||||
```
|
||||
|
||||
Node.js에서 child_process로 실행. stdout/stdin JSON 통신.
|
||||
|
||||
### 2. SQLite 스키마 변경
|
||||
|
||||
```sql
|
||||
-- 임베딩 캐시 (아이템당 1행)
|
||||
CREATE TABLE IF NOT EXISTS item_vectors (
|
||||
item_id TEXT PRIMARY KEY REFERENCES items(id),
|
||||
vector BLOB NOT NULL, -- Float32 array
|
||||
model TEXT NOT NULL DEFAULT '', -- 모델명
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
-- 인덱스
|
||||
CREATE INDEX IF NOT EXISTS idx_item_vectors_model ON item_vectors(model);
|
||||
```
|
||||
|
||||
BLOB으로 384차원 float32 = 1536바이트/아이템. 503개면 ~750KB.
|
||||
|
||||
### 3. TypeScript 모듈
|
||||
|
||||
```
|
||||
src/search/
|
||||
├── bridge.ts # Python 임베딩 프로세스 관리
|
||||
├── vector.ts # 벡터 저장/검색
|
||||
└── hybrid.ts # BM25 + Vector RRF 결합
|
||||
```
|
||||
|
||||
### 4. 검색 플로우
|
||||
|
||||
```
|
||||
1. 사용자 쿼리 → BM25 검색 (top N*3)
|
||||
2. 사용자 쿼리 → 임베딩 → 코사인 유사도 검색 (top N*3)
|
||||
3. RRF 결합: score = Σ 1/(k + rank), k=60
|
||||
4. 정규화 + 상위 N개 반환
|
||||
```
|
||||
|
||||
### 5. MCP 툴 변경
|
||||
|
||||
`hw_search`에 `mode` 파라미터 추가:
|
||||
- `keyword` (기본): BM25만
|
||||
- `semantic`: Vector만
|
||||
- `hybrid`: 둘 다 (기본값 변경 고려)
|
||||
|
||||
`hw_index`: 신규 — 전체 아이템 임베딩 인덱싱
|
||||
|
||||
## 작업 순서
|
||||
|
||||
1. `pip install fastembed` 설치
|
||||
2. `scripts/embed.py` 작성
|
||||
3. `src/search/bridge.ts` — Python 프로세스 통신
|
||||
4. DB 스키마 업데이트 (item_vectors 테이블)
|
||||
5. `src/search/vector.ts` — 벡터 CRUD
|
||||
6. `src/search/hybrid.ts` — RRF 결합
|
||||
7. MCP 핸들러 업데이트
|
||||
8. 빌드 + 테스트
|
||||
|
||||
## 의존성
|
||||
|
||||
- Python: `fastembed` (ONNX Runtime 번들, PyTorch 불필요)
|
||||
- Node.js: 기존 의존성만 사용 (child_process, better-sqlite3)
|
||||
|
||||
## 완료 기준
|
||||
|
||||
- [ ] `pip install fastembed` 정상 설치
|
||||
- [ ] embed.py로 임베딩 생성 가능
|
||||
- [ ] 503개 아이템 임베딩 인덱싱 완료
|
||||
- [ ] hw_search로 하이브리드 검색 동작
|
||||
- [ ] BM25-only 폴백 (fastembed 없을 때)
|
||||
259
docs/plans/PLAN-WIKI-LINT.md
Normal file
259
docs/plans/PLAN-WIKI-LINT.md
Normal file
@@ -0,0 +1,259 @@
|
||||
# PLAN-WIKI-LINT.md — 카파시 스타일 위키 린트 시스템
|
||||
|
||||
> **상태**: ✅ 완료 (Phase 1~3 전체 구현)
|
||||
> **작성**: 2026-04-18
|
||||
> **완료**: 2026-04-18
|
||||
> **목표**: hw_lint를 frontmatter 검사에서 → 카파시 LLM-Wiki의 린트 개념에 맞는 종합 위키 정합성 검사로 확장
|
||||
|
||||
---
|
||||
|
||||
## 1. 배경
|
||||
|
||||
Andrej Karpathy의 LLM-Wiki에서 린트(Lint)는 세 가지 핵심 작업 중 하나:
|
||||
- **인제스트**: 소스 추가 → 위키 업데이트 (이미 구현됨)
|
||||
- **쿼리**: 위키 질문/검색 (이미 구현됨)
|
||||
- **린트**: 페이지 간 모순, 고립 페이지, 정합성 점검 ← **이거 지금 부실함**
|
||||
|
||||
현재 `hw_lint`는 frontmatter 누락/타입 오류만 잡는다. 위키가 500페이지 규모로 커지면 의미 없는 검사.
|
||||
|
||||
## 2. 현재 코드베이스 현황
|
||||
|
||||
### 이미 있는 것 (재사용)
|
||||
| 컴포넌트 | 파일 | 상태 |
|
||||
|----------|------|------|
|
||||
| 기본 린트 | `src/vault/lint.ts` | frontmatter만 검사. 확장 필요 |
|
||||
| 모순 감지 (규칙) | `src/wiki/contradiction.ts` | 상태/날짜/사실 충돌 감지 + contradictions 테이블 |
|
||||
| 모순 핸들러 | `handleContradictions()` | list/get/resolve/scan/stats 구현됨 |
|
||||
| KG (엔티티/관계) | `src/db/kg.ts` | 링크 기반 고립 탐지 가능 |
|
||||
| HNSW 벡터 | `src/search/hnsw.ts` + `vector.ts` | 의미적 중복 탐지 가능 |
|
||||
| FTS5 | `src/db/items.ts` | 키워드 중복 탐지 가능 |
|
||||
| content_hash | `src/db/hashing.ts` | 구버전 감지 가능 (`getChangedItems`) |
|
||||
| 위키 파일 wikilink 파싱 | `src/wiki/generator.ts` | `[[]]` 추출 로직 있음 |
|
||||
|
||||
### 새로 만들어야 할 것
|
||||
| 컴포넌트 | 설명 |
|
||||
|----------|------|
|
||||
| `src/wiki/linter.ts` | 종합 린트 오케스트레이터 (새 모듈) |
|
||||
| 확장된 `LintResult` 타입 | 세부 이슈 카테고리 포함 |
|
||||
| `handleLint()` 확장 | 기존 frontmatter + 새 린트 항목 통합 |
|
||||
| `hw_lint` 스키마 확장 | `checks` 파라미터로 선택적 실행 |
|
||||
|
||||
## 3. 린트 검사 항목
|
||||
|
||||
### Tier 1: LLM 없이 가능 (빠름, 무료)
|
||||
|
||||
#### 3.1 고립 페이지 (Orphan Pages)
|
||||
- **정의**: incoming wikilink가 0인 위키 페이지
|
||||
- **방법**:
|
||||
1. `~/wiki/wiki/` 하위 모든 .md 파일 수집
|
||||
2. 각 파일에서 `[[]]` wikilink 추출
|
||||
3. 링크 대상 역인덱스 구축 (target → [source files])
|
||||
4. incoming link가 0인 파일 = 고립
|
||||
- **예외**: `index.md`, `overview.md`, `log.md`는 글로벌 네비게이션이므로 제외
|
||||
- **출력**: `{ file, title }[]`
|
||||
|
||||
#### 3.2 파손 링크 (Broken Wikilinks)
|
||||
- **정의**: `[[]]`로 링크했는데 대상 파일이 없음
|
||||
- **방법**:
|
||||
1. 모든 위키 파일에서 `[[]]` 추출
|
||||
2. 각 링크 대상이 `~/wiki/wiki/<target>.md` 또는 `~/wiki/<target>.md`에 존재하는지 확인
|
||||
- **출력**: `{ source, target, line_number }[]`
|
||||
|
||||
#### 3.3 구버전 페이지 (Stale Pages)
|
||||
- **정의**: 원본 아이템 content가 변경됐는데 위키가 재생성 안 됨
|
||||
- **방법**: `getChangedItems(db)` 활용 — 이미 content_hash 비교 구현됨
|
||||
- **출력**: `{ item_id, title, wiki_slug, hash_changed_at }[]`
|
||||
|
||||
#### 3.4 미생성 아이템 (Ungenerated Items)
|
||||
- **정의**: DB에 아이템은 있는데 위키 페이지가 아직 없음
|
||||
- **방법**: `getWikiGenerationCounts(db)` 활용
|
||||
- **출력**: `{ total, ungenerated }[]`
|
||||
|
||||
#### 3.5 Frontmatter 검사 (기존)
|
||||
- 기존 `lintVault()` 그대로 유지
|
||||
- frontmatter 누락, type 불일치, title 누락 체크
|
||||
|
||||
### Tier 2: HNSW 활용 (빠름, 로컬 연산)
|
||||
|
||||
#### 3.6 의미적 중복 (Semantic Duplicates)
|
||||
- **정의**: 내용이 거의 같은 위키 페이지가 여러 개
|
||||
- **방법**:
|
||||
1. 모든 위키 페이지 임베딩 (이미 벡터 DB에 있으면 재사용)
|
||||
2. HNSW에서 각 페이지의 최근접 이웃 조회
|
||||
3. cosine distance < 0.15 (임계값 튜닝 필요)인 쌍을 중복 후보로 표시
|
||||
- **출력**: `{ page_a, page_b, similarity_score }[]`
|
||||
- **주의**: 같은 카테고리 내에서만 비교하면 잡음 감소
|
||||
|
||||
### Tier 3: LLM 필요 (느림, 토큰 소모)
|
||||
|
||||
#### 3.7 의미적 모순 (Semantic Contradictions)
|
||||
- **정의**: 같은 주제를 다루는데 서로 다르게 서술
|
||||
- **방법**:
|
||||
1. Tier 2의 중복 후보 중 similarity가 높은 쌍 (0.15~0.4 구간)을 모순 후보로
|
||||
2. 후보 쌍만 LLM에게 비교 요청
|
||||
3. 프롬프트: "두 페이지가 모순되는 내용을 포함하는지 검사. 모순 있으면 요약, 없으면 '모순 없음'"
|
||||
4. 기존 `contradictions` 테이블에 결과 저장
|
||||
- **출력**: 기존 Contradiction 인터페이스 재사용
|
||||
- **선택 옵션**: `hw_lint --with-llm` 또는 `checks: ["contradictions"]` 시에만 실행
|
||||
|
||||
## 4. 구현 설계
|
||||
|
||||
### 4.1 새 모듈: `src/wiki/linter.ts`
|
||||
|
||||
```typescript
|
||||
// 타입
|
||||
interface LintOptions {
|
||||
checks?: LintCheckType[]; // 비어있으면 전체
|
||||
withLlm?: boolean; // Tier 3 포함 여부
|
||||
}
|
||||
|
||||
type LintCheckType =
|
||||
| "orphans" // 3.1
|
||||
| "broken_links" // 3.2
|
||||
| "stale" // 3.3
|
||||
| "ungenerated" // 3.4
|
||||
| "frontmatter" // 3.5
|
||||
| "duplicates" // 3.6
|
||||
| "contradictions" // 3.7 (LLM)
|
||||
|
||||
interface LintReport {
|
||||
timestamp: string;
|
||||
total_checks: number;
|
||||
duration_ms: number;
|
||||
results: {
|
||||
orphans: OrphanResult;
|
||||
broken_links: BrokenLinkResult[];
|
||||
stale: StaleItem[];
|
||||
ungenerated: { total: number; count: number };
|
||||
frontmatter: string[]; // 기존 이슈
|
||||
duplicates: DuplicatePair[];
|
||||
contradictions: Contradiction[]; // LLM 결과
|
||||
};
|
||||
summary: {
|
||||
critical: number; // 파손 링크, 구버전
|
||||
warnings: number; // 고립, 중복
|
||||
info: number; // 미생성
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
### 4.2 메인 함수
|
||||
|
||||
```typescript
|
||||
export async function lintWiki(
|
||||
config: WikiEngineConfig,
|
||||
options?: LintOptions
|
||||
): Promise<LintReport>
|
||||
```
|
||||
|
||||
내부 흐름:
|
||||
1. DB 연결, vault 경로 확보
|
||||
2. 모든 위키 파일 스캔 → 파일 목록 + wikilink 역인덱스 구축
|
||||
3. options.checks에 따라 각 검사 함수 호출
|
||||
4. 결과 취합 → LintReport 반환
|
||||
|
||||
### 4.3 핸들러 확장
|
||||
|
||||
기존 `handleLint()`를 확장:
|
||||
|
||||
```typescript
|
||||
export async function handleLint(
|
||||
args: {
|
||||
checks?: string[];
|
||||
withLlm?: boolean;
|
||||
},
|
||||
config: WikiEngineConfig
|
||||
): Promise<string>
|
||||
```
|
||||
|
||||
### 4.4 MCP 툴 스키마 확장
|
||||
|
||||
`hw_lint`의 inputSchema에 파라미터 추가:
|
||||
```json
|
||||
{
|
||||
"checks": {
|
||||
"type": "array",
|
||||
"items": { "type": "string" },
|
||||
"description": "특정 검사만 실행: orphans, broken_links, stale, ungenerated, frontmatter, duplicates, contradictions"
|
||||
},
|
||||
"withLlm": {
|
||||
"type": "boolean",
|
||||
"description": "LLM 기반 모순 감지 포함 (default: false)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 5. 파일 변경 목록
|
||||
|
||||
| 파일 | 변경 유형 | 설명 |
|
||||
|------|-----------|------|
|
||||
| `src/wiki/linter.ts` | **신규** | 종합 린트 오케스트레이터 |
|
||||
| `src/types.ts` | 수정 | LintReport, LintOptions 등 타입 추가 |
|
||||
| `src/mcp/handlers.ts` | 수정 | `handleLint()` 시그니처 변경, 새 린트 호출 |
|
||||
| `src/mcp/tools.ts` | 수정 | `hw_lint` 스키마에 checks/withLlm 추가 |
|
||||
| `src/mcp-server.ts` | 수정 | handleLint 호출부 args 전달 |
|
||||
|
||||
**건드리지 않는 파일** (안정성):
|
||||
- `src/vault/lint.ts` → 기존 frontmatter 검사 함수 그대로, 새 linter.ts에서 호출만 함
|
||||
- `src/wiki/contradiction.ts` → 기존 모순 시스템 그대로, 새 linter.ts에서 결과만 참조
|
||||
- `src/wiki/generator.ts` → wikilink 추출 유틸만 import
|
||||
|
||||
## 6. 구현 순서
|
||||
|
||||
### Phase 1: 인프라 (Tier 1)
|
||||
1. `src/types.ts`에 LintReport, LintOptions 타입 추가
|
||||
2. `src/wiki/linter.ts` 신규 생성
|
||||
3. `lintOrphans()` 구현 — wikilink 역인덱스 + 고립 탐지
|
||||
4. `lintBrokenLinks()` 구현 — 파손 링크 탐지
|
||||
5. `lintStale()` 구현 — getChangedItems 래핑
|
||||
6. `lintUngenerated()` 구현 — getWikiGenerationCounts 래핑
|
||||
7. 기존 `lintVault()`를 frontmatter 서브체크로 통합
|
||||
8. `handleLint()` 확장 + 스키마 업데이트
|
||||
9. 빌드 + 스모크 테스트
|
||||
|
||||
### Phase 2: 의미 검사 (Tier 2)
|
||||
10. `lintDuplicates()` 구현 — HNSW 유사도 기반
|
||||
11. 임계값 튜닝 (0.15 시작)
|
||||
|
||||
### Phase 3: LLM 모순 (Tier 3)
|
||||
12. `lintContradictions()` 구현 — LLM 프롬프트
|
||||
13. 기존 contradictions 테이블과 통합
|
||||
14. `--withLlm` 플래그 연동
|
||||
|
||||
## 7. 검증 기준
|
||||
|
||||
- [x] `hw_lint` 실행 시 7가지 검사(orphan, broken, stale, ungenerated, frontmatter, duplicates, contradictions) 결과 반환
|
||||
- [x] 고립 페이지가 실제로 incoming link 0인 파일만 포함
|
||||
- [x] 파손 링크가 실제로 존재하지 않는 대상만 포함
|
||||
- [x] 기존 `hw_contradictions` 기능이 영향 없이 동작
|
||||
- [x] `npm run build` 에러 없음
|
||||
- [x] Phase 1은 LLM 호출 없이 5초 이내 완료 (500페이지 기준) → **266ms**
|
||||
- [x] Phase 2 (HNSW duplicates) 벡터 유사도 ≥ 0.95 → 228쌍 검출, **1.7s**
|
||||
- [x] Phase 3 (contradictions) 룰 기반 검출 동작, `--with-llm` 옵션 연동
|
||||
|
||||
## 8. 구현 결과
|
||||
|
||||
### 커밋
|
||||
- `6c711a7` — Phase 1 + DB fix + Codex 모듈 (44파일, +10,479줄)
|
||||
- `53b9a32` — Phase 2 + Phase 3 (2파일, +214줄)
|
||||
|
||||
### 스모크 테스트 결과 (2026-04-18)
|
||||
```
|
||||
Duration: 1727 ms | Checks: 6
|
||||
Orphans: 14 | Broken links: 53 | Stale: 0
|
||||
Ungenerated: 0/504 | Duplicates: 228 | Contradictions: 0
|
||||
Summary: { critical: 53, warnings: 242, info: 0 }
|
||||
```
|
||||
|
||||
### 향후 개선
|
||||
- LLM 기반 모순 검출: 현재 룰 기반만 구현, `--with-llm` 시 LLM 프롬프트로 의미적 모순 검출 추가 가능
|
||||
- 중복 임계값 튜닝: 데이터셋 성격에 따라 0.95~0.98 범위에서 조정
|
||||
|
||||
## 9. 위험 요소
|
||||
|
||||
| 위험 | 대응 |
|
||||
|------|------|
|
||||
| wikilink 파싱 불완전 (이스케이프, 중첩) | generator.ts 기존 파싱 로직 재사용 |
|
||||
| HNSW 임계값 튜닝 어려움 | 0.15에서 시작, 실패 시 조정 |
|
||||
| 대규모 위키에서 린트 속도 | 파일 스캔은 I/O-bound, 병렬화 불필요 |
|
||||
| LLM 모순 감지 토큰 비용 | Tier 3은 명시적 opt-in만 |
|
||||
221
docs/plans/PLAN-v2.md
Normal file
221
docs/plans/PLAN-v2.md
Normal file
@@ -0,0 +1,221 @@
|
||||
# PLAN: habraid 독립 아키텍처
|
||||
|
||||
> User 승인 후 진행
|
||||
|
||||
## 목표
|
||||
|
||||
habraid를 MemPalace 없이도 완전 동작하는 독립 위키 엔진으로 만든다.
|
||||
MemPalace가 있으면 추가 데이터 소스로 활용하고, 없으면 자체 기능으로 동작한다.
|
||||
|
||||
## 현재 상태
|
||||
|
||||
- MemPalace ChromaDB에서만 데이터 읽음 (의존적)
|
||||
- 검색 기능 없음 (MemPalace에 위임)
|
||||
- KG 없음
|
||||
- 세션 수집 없음
|
||||
- LLM 위키 생성만 가능
|
||||
|
||||
## 변경 아키텍처
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────┐
|
||||
│ habraid │
|
||||
│ │
|
||||
│ ┌──────────┐ ┌──────────┐ ┌───────────────┐ │
|
||||
│ │ Sources │ │ Core DB │ │ Wiki Gen │ │
|
||||
│ │ │ │ (SQLite) │ │ (LLM) │ │
|
||||
│ │ • CLI │ │ │ │ │ │
|
||||
│ │ • MCP │ │ • items │ │ • Incremental │ │
|
||||
│ │ • File │ │ • KG │ │ • Batch │ │
|
||||
│ │ • MemPal │ │ • Search │ │ • Fallback │ │
|
||||
│ └──────────┘ └──────────┘ └───────────────┘ │
|
||||
│ │
|
||||
│ ┌──────────────────────────────────────────────┐│
|
||||
│ │ Vault (Obsidian) ││
|
||||
│ │ raw/ → wiki/ → index.md + overview.md ││
|
||||
│ └──────────────────────────────────────────────┘│
|
||||
└─────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Phase 1: 자체 DB + 아이템 관리 (핵심)
|
||||
|
||||
### 1.1 자체 SQLite DB 생성
|
||||
|
||||
경로: `~/.local/share/habraid/habraid.db`
|
||||
|
||||
테이블:
|
||||
```sql
|
||||
-- 모든 지식 아이템 (drawer의 일반화)
|
||||
CREATE TABLE items (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'manual', -- manual, mempalace, cli, file, session
|
||||
category TEXT, -- projects, topics, decisions, people, infrastructure, guides
|
||||
tags TEXT, -- JSON array
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
metadata TEXT -- JSON blob for source-specific data
|
||||
);
|
||||
|
||||
-- FTS5 전문 검색
|
||||
CREATE VIRTUAL TABLE items_fts USING fts5(title, content, tags, content=items, content_rowid=rowid);
|
||||
|
||||
-- 자동 동기화 트리거
|
||||
CREATE TRIGGER items_ai AFTER INSERT ON items BEGIN
|
||||
INSERT INTO items_fts(rowid, title, content, tags) VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
CREATE TRIGGER items_ad AFTER DELETE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags) VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
END;
|
||||
CREATE TRIGGER items_au AFTER UPDATE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags) VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
INSERT INTO items_fts(rowid, title, content, tags) VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
```
|
||||
|
||||
### 1.2 MemPalace 어댑터 (선택)
|
||||
|
||||
```typescript
|
||||
// src/sources/mempalace.ts
|
||||
interface SourceAdapter {
|
||||
name: string;
|
||||
isAvailable(): boolean; // DB 파일 존재 여부
|
||||
fetchItems(since?: Date): Promise<Item[]>;
|
||||
}
|
||||
|
||||
class MemPalaceSource implements SourceAdapter {
|
||||
isAvailable(): boolean {
|
||||
return existsSync(this.config.mempalace?.path + '/chroma.sqlite3');
|
||||
}
|
||||
|
||||
async fetchItems(since?: Date): Promise<Item[]> {
|
||||
if (!this.isAvailable()) return [];
|
||||
// ChromaDB에서 drawer 읽기 → Item으로 변환
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### 1.3 MCP 툴 업데이트
|
||||
|
||||
기존 7개 + 새 툴:
|
||||
|
||||
| 툴 | 변경 |
|
||||
|-----|------|
|
||||
| `hw_status` | 이름 변경 + 자체 DB 통계 |
|
||||
| `hw_ingest` | MemPalace + 다른 소스 통합 |
|
||||
| `hw_add` | **신규** — 아이템 직접 추가 |
|
||||
| `hw_search` | **신규** — FTS5 검색 (MemPalace 없이도 동작) |
|
||||
| `hw_generate` | 기존 |
|
||||
| `hw_sync` | 기존 |
|
||||
| `hw_read` | 기존 |
|
||||
| `hw_lint` | 기존 |
|
||||
| `hw_graph` | **신규** — KG 쿼리 |
|
||||
|
||||
> `hw_` prefix로 다른 MCP 툴이랑 충돌 방지
|
||||
|
||||
## Phase 2: Knowledge Graph (선택)
|
||||
|
||||
자체 SQLite KG. MemPalace KG 있으면 참고, 없으면 자체.
|
||||
|
||||
```sql
|
||||
CREATE TABLE kg_entities (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT UNIQUE NOT NULL,
|
||||
type TEXT NOT NULL -- person, project, technology, concept, server
|
||||
);
|
||||
|
||||
CREATE TABLE kg_relations (
|
||||
id INTEGER PRIMARY KEY,
|
||||
subject_id INTEGER REFERENCES kg_entities(id),
|
||||
predicate TEXT NOT NULL,
|
||||
object_id INTEGER REFERENCES kg_entities(id),
|
||||
valid_from TEXT,
|
||||
valid_to TEXT,
|
||||
created_at TEXT NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
## Phase 3: 세션 수집 (선택, seCall 기능)
|
||||
|
||||
나중에 Claude Code, Codex CLI 세션 로그를 직접 수집하는 기능.
|
||||
지금은 Phase 1만 진행.
|
||||
|
||||
## 파일 구조 변경
|
||||
|
||||
```
|
||||
src/
|
||||
├── index.ts # CLI
|
||||
├── mcp-server.ts # MCP 서버
|
||||
├── config.ts # 설정
|
||||
├── types.ts # 타입
|
||||
├── errors.ts # 에러
|
||||
├── utils.ts # 유틸
|
||||
├── db/
|
||||
│ ├── database.ts # SQLite 연결 + 마이그레이션
|
||||
│ ├── items.ts # items CRUD + FTS5
|
||||
│ └── kg.ts # Knowledge Graph (Phase 2)
|
||||
├── sources/
|
||||
│ ├── adapter.ts # SourceAdapter 인터페이스
|
||||
│ ├── mempalace.ts # MemPalace 연동 (선택)
|
||||
│ ├── manual.ts # 직접 추가
|
||||
│ └── file.ts # 파일 임포트 (향후)
|
||||
├── vault/
|
||||
│ ├── init.ts
|
||||
│ ├── render.ts
|
||||
│ ├── index.ts
|
||||
│ └── log.ts
|
||||
├── wiki/
|
||||
│ ├── generator.ts
|
||||
│ ├── llm.ts
|
||||
│ └── prompts.ts
|
||||
├── sync/
|
||||
│ ├── git.ts
|
||||
│ └── pipeline.ts
|
||||
└── cli/
|
||||
└── commands.ts
|
||||
```
|
||||
|
||||
## 설정 파일
|
||||
|
||||
`~/.config/habraid/config.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"vault": { "path": "~/wiki", "branch": "main" },
|
||||
"db": { "path": "~/.local/share/habraid/habraid.db" },
|
||||
"mempalace": {
|
||||
"enabled": true,
|
||||
"path": "~/.mempalace/palace"
|
||||
},
|
||||
"llm": {
|
||||
"provider": "zai",
|
||||
"model": "glm-5.1",
|
||||
"api_url": "https://api.example.com/v1",
|
||||
"api_key_env": "GLM_API_KEY",
|
||||
"max_tokens": 4096
|
||||
},
|
||||
"sync": { "timezone": "Asia/Seoul" }
|
||||
}
|
||||
```
|
||||
|
||||
- `mempalace.enabled: false` → MemPalace 완전 무시
|
||||
- `mempalace.enabled: true` + 경로 없음 → 자동으로 비활성화
|
||||
- `mempalace.enabled: true` + 경로 있음 → 연동
|
||||
|
||||
## 작업 순서
|
||||
|
||||
1. **db/ 모듈 작성** — SQLite 스키마, items CRUD, FTS5
|
||||
2. **sources/ 어댑터 작성** — SourceAdapter 인터페이스 + MemPalace 어댑터 + Manual 어댑터
|
||||
3. **MCP 핸들러 업데이트** — 기존 핸들러를 DB 기반으로 전환 + 새 툴 추가
|
||||
4. **ingest 재작성** — DB에서 raw/ 생성 (MemPalace 없이도 동작)
|
||||
5. **검색 구현** — hw_search (FTS5)
|
||||
6. **테스트** — MemPalace 있을 때 / 없을 때 모두
|
||||
7. **CLI 업데이트** — 새 명령어 반영
|
||||
|
||||
## 완료 기준
|
||||
|
||||
- [ ] MemPalace 없이 `hw_add` + `hw_search` + `hw_generate` 동작
|
||||
- [ ] MemPalace 있을 때 자동 감지 + 연동
|
||||
- [ ] 기존 raw/ 503개 데이터 마이그레이션
|
||||
- [ ] MCP 9개 툴 모두 정상
|
||||
411
docs/plans/PLAN.md
Normal file
411
docs/plans/PLAN.md
Normal file
@@ -0,0 +1,411 @@
|
||||
# Generator Batch Processing Implementation Plan
|
||||
|
||||
> **For Codex:** Read this plan carefully. Implement all tasks sequentially. Commit after each task.
|
||||
|
||||
**Goal:** Fix wiki generator to process 500+ drawers in room-based batches with proper LLM calls, and fix fallback to produce one merged page per room instead of per-drawer (which overwrites itself).
|
||||
|
||||
**Architecture:** Group drawers by room, send each room as one LLM batch (max 25 drawers). LLM generates consolidated wiki pages per room. If LLM fails, merge all drawers in that room into one fallback page.
|
||||
|
||||
**Tech Stack:** TypeScript, Node.js, z.ai OpenAI-compatible API, gray-matter
|
||||
|
||||
---
|
||||
|
||||
## Problem Analysis
|
||||
|
||||
### Current Issues
|
||||
|
||||
1. **`updateWiki()` in `generator.ts`** sends ALL 503 drawers in a single LLM call → token overflow → LLM fails → falls to fallback
|
||||
2. **`buildFallbackFiles()`** creates one file per drawer but uses `room-summary` as slug → same room drawers overwrite each other → only last drawer's content survives
|
||||
3. **`readRawDrawers()`** reads all 503 drawers into memory at once (fine for 503, but prompt construction is the bottleneck)
|
||||
|
||||
### Solution
|
||||
|
||||
1. Group drawers by `room` after reading
|
||||
2. Process each room group as a separate LLM call (batch within room if >25 drawers)
|
||||
3. Fix fallback to merge all drawers in a room into ONE page
|
||||
4. LLM timeout set to 120s per batch, 2s delay between batches
|
||||
|
||||
---
|
||||
|
||||
## Files to Modify
|
||||
|
||||
- `src/wiki/generator.ts` — Main changes: batch processing, fixed fallback
|
||||
- `src/wiki/prompts.ts` — Adjust prompt for per-room batch context
|
||||
- `src/wiki/llm.ts` — Add timeout parameter to generate()
|
||||
- `src/vault/render.ts` — Fix `renderFallbackWikiMarkdown` to accept multiple drawers
|
||||
|
||||
---
|
||||
|
||||
## Task 1: Add timeout to ZaiLlmClient
|
||||
|
||||
**Objective:** Allow callers to set custom timeout per request.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/wiki/llm.ts`
|
||||
|
||||
**Step 1: Update generate() method signature and implementation**
|
||||
|
||||
Add an optional `timeoutMs` parameter (default 60000ms) to the `generate` method. Pass it to `fetch` via `AbortController`.
|
||||
|
||||
```typescript
|
||||
async generate(systemPrompt: string, userPrompt: string, timeoutMs: number = 60000): Promise<string> {
|
||||
try {
|
||||
const apiKey = process.env[this.config.llm.api_key_env];
|
||||
if (!apiKey) {
|
||||
throw new LlmCallError(
|
||||
`Missing API key environment variable ${this.config.llm.api_key_env} for wiki generation.`,
|
||||
);
|
||||
}
|
||||
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
const response = await fetch(`${this.config.llm.api_url}/chat/completions`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${apiKey}`,
|
||||
},
|
||||
signal: controller.signal,
|
||||
body: JSON.stringify({
|
||||
model: this.config.llm.model,
|
||||
max_tokens: this.config.llm.max_tokens,
|
||||
messages: [
|
||||
{ role: "system", content: systemPrompt },
|
||||
{ role: "user", content: userPrompt },
|
||||
],
|
||||
}),
|
||||
});
|
||||
|
||||
clearTimeout(timeout);
|
||||
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new LlmCallError(`LLM request failed with ${response.status}: ${body}`);
|
||||
}
|
||||
|
||||
const data = (await response.json()) as {
|
||||
choices?: Array<{ message?: { content?: string } }>;
|
||||
};
|
||||
const content = data.choices?.[0]?.message?.content;
|
||||
if (!content) {
|
||||
throw new LlmCallError("LLM response did not contain any message content.");
|
||||
}
|
||||
|
||||
return content;
|
||||
} catch (error) {
|
||||
if (error instanceof LlmCallError) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
throw new LlmCallError("Failed to call the z.ai LLM backend.", error as Error);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Update LLMClient interface**
|
||||
|
||||
```typescript
|
||||
generate(systemPrompt: string, userPrompt: string, timeoutMs?: number): Promise<string>;
|
||||
```
|
||||
|
||||
**Step 3: Commit**
|
||||
```bash
|
||||
git add src/wiki/llm.ts
|
||||
git commit -m "feat: add timeout parameter to LLM client"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 2: Fix fallback to merge drawers per room
|
||||
|
||||
**Objective:** Instead of one fallback file per drawer (which overwrites), create one merged page per room.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/vault/render.ts`
|
||||
|
||||
**Step 1: Add `renderMergedFallbackWikiMarkdown` function**
|
||||
|
||||
Add a new function that takes an array of drawers from the same room and produces ONE wiki page:
|
||||
|
||||
```typescript
|
||||
export function renderMergedFallbackWikiMarkdown(drawers: MemPalaceDrawer[]): string {
|
||||
if (drawers.length === 0) {
|
||||
return "";
|
||||
}
|
||||
|
||||
const first = drawers[0];
|
||||
const title = `${humanizeSegment(first.room)} 정리`;
|
||||
const category = inferWikiCategory(first.wing, first.room);
|
||||
const allIds = drawers.map((d) => d.id);
|
||||
const allTags = [...new Set(drawers.flatMap((d) => d.tags))];
|
||||
|
||||
const frontmatter: WikiFrontmatter = {
|
||||
type: "wiki",
|
||||
category,
|
||||
title,
|
||||
created: toDateString(first.createdAt),
|
||||
updated: toDateString(new Date()),
|
||||
sources: allIds,
|
||||
tags: allTags,
|
||||
status: "draft",
|
||||
agent: first.addedBy,
|
||||
};
|
||||
|
||||
const sections = drawers.map((drawer) => {
|
||||
const header = drawer.content.match(/^#\s+(.+)$/m);
|
||||
const sectionTitle = header ? header[1] : drawer.id;
|
||||
return [
|
||||
`### ${sectionTitle}`,
|
||||
"",
|
||||
drawer.content,
|
||||
"",
|
||||
].join("\n");
|
||||
});
|
||||
|
||||
const body = [
|
||||
`# ${title}`,
|
||||
"",
|
||||
"## 개요",
|
||||
`${first.wing}/${first.room} 서랍의 원본 내용을 정리한 초안입니다. 총 ${drawers.length}개 서랍.`,
|
||||
"",
|
||||
"## 핵심 내용",
|
||||
"",
|
||||
...sections,
|
||||
"## 원본 서랍",
|
||||
...drawers.map((d) => `- [[raw/mempalace/${d.wing}/${d.room}/${d.id}|${d.id}]]`),
|
||||
].join("\n");
|
||||
|
||||
return stringifyFrontmatter(body, frontmatter);
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Commit**
|
||||
```bash
|
||||
git add src/vault/render.ts
|
||||
git commit -m "feat: add merged fallback renderer for room-based grouping"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 3: Rewrite generator with batch processing
|
||||
|
||||
**Objective:** Group drawers by room, call LLM per room batch, use merged fallback on failure.
|
||||
|
||||
**Files:**
|
||||
- Modify: `src/wiki/generator.ts`
|
||||
|
||||
**Step 1: Replace `updateWiki` function**
|
||||
|
||||
The new `updateWiki` should:
|
||||
1. Read raw drawers
|
||||
2. Group by room
|
||||
3. For each room group (split into batches of 25 if needed):
|
||||
a. Call LLM with per-room prompt
|
||||
b. Parse generated files
|
||||
c. On LLM failure, use `renderMergedFallbackWikiMarkdown` for the whole room
|
||||
4. Write all files
|
||||
5. Update sync state and rebuild overview
|
||||
|
||||
```typescript
|
||||
const ROOM_BATCH_SIZE = 25;
|
||||
const LLM_TIMEOUT_MS = 120_000;
|
||||
const BATCH_DELAY_MS = 2000;
|
||||
|
||||
export async function updateWiki(config: WikiEngineConfig): Promise<WikiUpdateResult> {
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
const state = await readSyncState(vaultPath);
|
||||
const existingWikiFiles = await listFilesRecursive(path.join(vaultPath, "wiki"), ".md");
|
||||
|
||||
const sourceDrawers = readRawDrawers(vaultPath);
|
||||
|
||||
if (sourceDrawers.length === 0) {
|
||||
await rebuildOverview(vaultPath);
|
||||
return { filesWritten: 0, pageSlugs: [], filePaths: [] };
|
||||
}
|
||||
|
||||
// Group by room
|
||||
const roomGroups = groupDrawersByRoom(sourceDrawers);
|
||||
const llmClient = new ZaiLlmClient(config);
|
||||
|
||||
const allFiles: GeneratedWikiFile[] = [];
|
||||
|
||||
for (const [roomKey, roomDrawers] of roomGroups) {
|
||||
const batches = splitIntoBatches(roomDrawers, ROOM_BATCH_SIZE);
|
||||
|
||||
for (let i = 0; i < batches.length; i++) {
|
||||
const batch = batches[i];
|
||||
|
||||
try {
|
||||
const prompt = await buildIncrementalWikiPrompt(batch, existingWikiFiles, vaultPath);
|
||||
const response = await llmClient.generate(WIKI_SYSTEM_PROMPT, prompt, LLM_TIMEOUT_MS);
|
||||
const files = parseGeneratedWikiFiles(response);
|
||||
|
||||
if (files.length > 0) {
|
||||
allFiles.push(...files);
|
||||
} else {
|
||||
// LLM returned empty — use merged fallback for this batch
|
||||
allFiles.push(buildMergedFallbackFile(batch));
|
||||
}
|
||||
} catch (error) {
|
||||
// LLM failed — use merged fallback
|
||||
allFiles.push(buildMergedFallbackFile(batch));
|
||||
}
|
||||
|
||||
// Rate limit between batches
|
||||
if (i < batches.length - 1 || roomKey !== roomGroups[roomGroups.length - 1]?.[0]) {
|
||||
await sleep(BATCH_DELAY_MS);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Write all files
|
||||
const writtenPaths: string[] = [];
|
||||
const pageSlugs: string[] = [];
|
||||
|
||||
for (const file of allFiles) {
|
||||
const absolutePath = path.join(vaultPath, file.path);
|
||||
await writeTextFile(absolutePath, file.content);
|
||||
writtenPaths.push(absolutePath);
|
||||
pageSlugs.push(path.basename(absolutePath, ".md"));
|
||||
}
|
||||
|
||||
// Update state
|
||||
const nextState: SyncState = {
|
||||
...state,
|
||||
last_wiki_update: toIsoTimestamp(),
|
||||
wiki_pages: [...new Set([...state.wiki_pages, ...pageSlugs])],
|
||||
};
|
||||
|
||||
await writeSyncState(vaultPath, nextState);
|
||||
await rebuildOverview(vaultPath);
|
||||
await updateVaultIndex(vaultPath, nextState);
|
||||
|
||||
const logEntry: SyncLogEntry = {
|
||||
time: new Date().toLocaleString("sv-SE", { timeZone: config.sync.timezone }).replace("T", " "),
|
||||
action: "wiki",
|
||||
target: "raw -> wiki",
|
||||
result: `+${writtenPaths.length} pages`,
|
||||
};
|
||||
await appendSyncLog(vaultPath, [logEntry]);
|
||||
|
||||
return { filesWritten: writtenPaths.length, pageSlugs, filePaths: writtenPaths };
|
||||
} catch (error) {
|
||||
throw new LlmCallError("Wiki generation failed.", error as Error);
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Step 2: Add helper functions**
|
||||
|
||||
```typescript
|
||||
/** Groups drawers by their room field. Returns entries sorted by drawer count (largest first). */
|
||||
function groupDrawersByRoom(drawers: MemPalaceDrawer[]): [string, MemPalaceDrawer[]][] {
|
||||
const groups = new Map<string, MemPalaceDrawer[]>();
|
||||
for (const drawer of drawers) {
|
||||
const key = `${drawer.wing}/${drawer.room}`;
|
||||
if (!groups.has(key)) {
|
||||
groups.set(key, []);
|
||||
}
|
||||
groups.get(key)!.push(drawer);
|
||||
}
|
||||
return [...groups.entries()].sort((a, b) => b[1].length - a[1].length);
|
||||
}
|
||||
|
||||
/** Splits an array into batches of given size. */
|
||||
function splitIntoBatches<T>(items: T[], batchSize: number): T[][] {
|
||||
const batches: T[][] = [];
|
||||
for (let i = 0; i < items.length; i += batchSize) {
|
||||
batches.push(items.slice(i, i + batchSize));
|
||||
}
|
||||
return batches;
|
||||
}
|
||||
|
||||
/** Builds a single merged fallback file for a batch of drawers. */
|
||||
function buildMergedFallbackFile(batch: MemPalaceDrawer[]): GeneratedWikiFile {
|
||||
const first = batch[0];
|
||||
const content = renderMergedFallbackWikiMarkdown(batch);
|
||||
const slug = toKebabCase(`${first.room}-summary`);
|
||||
const category = inferWikiCategory(first.wing, first.room);
|
||||
return {
|
||||
path: path.join("wiki", category, `${slug}.md`),
|
||||
content,
|
||||
};
|
||||
}
|
||||
|
||||
/** Simple promise-based sleep. */
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => setTimeout(resolve, ms));
|
||||
}
|
||||
```
|
||||
|
||||
**Step 3: Remove old `buildFallbackFiles` function**
|
||||
|
||||
Delete the old function that creates one file per drawer.
|
||||
|
||||
**Step 4: Add required imports**
|
||||
|
||||
Add at the top of generator.ts:
|
||||
```typescript
|
||||
import { renderMergedFallbackWikiMarkdown, inferWikiCategory } from "../vault/render.js";
|
||||
```
|
||||
|
||||
Remove the old import of `renderFallbackWikiMarkdown` if it exists.
|
||||
|
||||
**Step 5: Commit**
|
||||
```bash
|
||||
git add src/wiki/generator.ts
|
||||
git commit -m "feat: room-based batch processing for wiki generation"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 4: Clean up temporary files
|
||||
|
||||
**Objective:** Remove the temporary cli-generate.ts script.
|
||||
|
||||
**Files:**
|
||||
- Delete: `src/cli-generate.ts`
|
||||
|
||||
**Step 1: Delete the file**
|
||||
```bash
|
||||
rm src/cli-generate.ts
|
||||
git add -A
|
||||
git commit -m "chore: remove temporary cli-generate script"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Task 5: Verify TypeScript compilation
|
||||
|
||||
**Step 1: Run type check**
|
||||
```bash
|
||||
npx tsc --noEmit
|
||||
```
|
||||
|
||||
Expected: No errors. If there are type errors, fix them.
|
||||
|
||||
**Step 2: Commit any fixes**
|
||||
```bash
|
||||
git add -A
|
||||
git commit -m "fix: resolve type errors"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
After all tasks, run:
|
||||
```bash
|
||||
cd ~/.habraid/app
|
||||
source ~/.hermes/.env
|
||||
npx tsx src/index.ts generate
|
||||
```
|
||||
|
||||
Expected behavior:
|
||||
- Drawers grouped by room (7 rooms: memory, general, agents, workflows, state, security, protocol, diary)
|
||||
- Each room processed as separate LLM batch
|
||||
- If LLM succeeds: 2-5 wiki pages per room
|
||||
- If LLM fails: 1 merged fallback page per room (not per drawer)
|
||||
- Total: ~10-30 wiki pages instead of 503 identical overwrites
|
||||
12
docs/plans/README.md
Normal file
12
docs/plans/README.md
Normal file
@@ -0,0 +1,12 @@
|
||||
# Plans
|
||||
|
||||
이 폴더는 HaBraid의 중장기 구현 계획 문서를 보관해.
|
||||
|
||||
## 포함 문서
|
||||
|
||||
- `PLAN.md` — 초기 batch processing / generator 개선 계획
|
||||
- `PLAN-v2.md` — habraid 독립 아키텍처 계획
|
||||
- `PLAN-MCP.md` — MCP 서버 도입 계획
|
||||
- `PLAN-VECTOR.md` — vector / hybrid search 계획
|
||||
|
||||
세션 단위 작업 계획은 루트의 `.hermes/plans/` 아래에 따로 쌓여.
|
||||
144
docs/specs/host-following-llm.md
Normal file
144
docs/specs/host-following-llm.md
Normal file
@@ -0,0 +1,144 @@
|
||||
# Host-Following LLM Spec
|
||||
|
||||
**Status:** proposed
|
||||
**Owner:** habraid
|
||||
**Scope:** wiki generation routing, config schema, host integration boundary
|
||||
|
||||
---
|
||||
|
||||
## Goal
|
||||
|
||||
HaBraid의 wiki generation이 특정 provider/model에 고정되지 않고, 기본적으로 현재 host(Hermes/OpenClaw/Codex CLI)의 추론 정책을 따르도록 만든다.
|
||||
|
||||
## Context
|
||||
|
||||
현재 habraid는 `src/wiki/llm.ts`에서 직접 외부 LLM backend를 호출한다. 이 구조는 다음 문제를 만든다.
|
||||
|
||||
- host와 habraid의 모델이 분리된다.
|
||||
- quota / auth / provider routing이 이중화된다.
|
||||
- 사용자는 "지금 네가 쓰는 모델로 해"를 기대하지만 habraid는 자체 config를 따른다.
|
||||
|
||||
HaBraid의 본질은 모델 선택이 아니라 **memory backend를 wiki 표현층으로 변환하는 orchestration** 이다.
|
||||
|
||||
## Product Principles
|
||||
|
||||
1. HaBraid는 저장소가 아니라 기억 시각화 계층이다.
|
||||
2. source of truth는 MemPalace/mem0 같은 memory backend에 있다.
|
||||
3. 위키는 사람과 AI의 공용 인터페이스다.
|
||||
4. 모델 선택은 기본적으로 host가 한다.
|
||||
5. raw memory는 의미 단위(topic/entity/decision/timeline)로 재구성된다.
|
||||
|
||||
## Constraints
|
||||
|
||||
- 기존 standalone 실행 경로를 완전히 깨면 안 된다.
|
||||
- 레거시 config(`provider/model/api_url/api_key_env`)는 하위 호환해야 한다.
|
||||
- exact host model string은 있으면 좋지만 필수 의존성으로 삼지 않는다.
|
||||
- host bridge가 unavailable일 수 있으므로 fallback 경로가 필요하다.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
- 이번 변경에서 MemPalace direct ingest 문제를 해결하지 않는다.
|
||||
- 새로운 memory backend(mem0 등)를 실제로 붙이지 않는다.
|
||||
- host와 완전한 양방향 protocol을 이번 단계에서 확정하지 않는다.
|
||||
|
||||
## Desired Architecture
|
||||
|
||||
```text
|
||||
items/raw
|
||||
↓
|
||||
grouping + prompt building
|
||||
↓
|
||||
LLM gateway
|
||||
├─ host mode (default)
|
||||
│ └─ host가 실제 모델/정책 선택
|
||||
└─ standalone mode / fallback
|
||||
└─ openai | zai | ollama 직접 호출
|
||||
↓
|
||||
response normalization
|
||||
↓
|
||||
wiki files + sync state + logs
|
||||
```
|
||||
|
||||
## Routing Rules
|
||||
|
||||
### Default behavior
|
||||
- `llm.mode = "host"` 이면 host route를 먼저 시도한다.
|
||||
- host route가 사용 가능하면 실제 생성은 host가 수행한다.
|
||||
- host가 어떤 provider/model을 썼는지는 optional metadata로만 기록한다.
|
||||
|
||||
### Fallback behavior
|
||||
- host route unavailable이면 configured fallback을 사용한다.
|
||||
- fallback은 explicit config일 때만 활성화한다.
|
||||
- fallback도 없으면 명확한 에러를 반환한다.
|
||||
|
||||
### Standalone behavior
|
||||
- `llm.mode = "standalone"` 이면 기존처럼 직접 backend를 호출한다.
|
||||
- 레거시 config는 내부적으로 standalone mode로 normalize한다.
|
||||
|
||||
## Config Shape
|
||||
|
||||
```json
|
||||
{
|
||||
"llm": {
|
||||
"mode": "host",
|
||||
"preferences": {
|
||||
"priority": "balanced"
|
||||
},
|
||||
"fallback": {
|
||||
"provider": "ollama",
|
||||
"model": "qwen2.5:14b",
|
||||
"api_url": "http://localhost:11434"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Legacy compatibility
|
||||
|
||||
```json
|
||||
{
|
||||
"llm": {
|
||||
"provider": "zai",
|
||||
"model": "glm-5.1",
|
||||
"api_url": "https://api.example.com/v1",
|
||||
"api_key_env": "GLM_API_KEY",
|
||||
"max_tokens": 4096
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
위 형태는 내부적으로 fallback/runtime config로 흡수된다. `mode`를 명시하지 않은 레거시 config는 기본적으로 `host` 모드를 따르고, 위 값들은 fallback/backend metadata로 유지된다.
|
||||
|
||||
## Host Bridge Contract
|
||||
|
||||
```ts
|
||||
interface HostInferenceBridge {
|
||||
isAvailable(): Promise<boolean>;
|
||||
generate(request: {
|
||||
systemPrompt: string;
|
||||
userPrompt: string;
|
||||
maxTokens?: number;
|
||||
preference?: "fast" | "balanced" | "smart";
|
||||
}): Promise<{
|
||||
text: string;
|
||||
model?: string;
|
||||
provider?: string;
|
||||
}>;
|
||||
}
|
||||
```
|
||||
|
||||
## Observability
|
||||
|
||||
최소한 다음 정보는 기록 가능해야 한다.
|
||||
|
||||
- 마지막 generation route (`host` / `fallback` / `standalone`)
|
||||
- 마지막 provider/model (알 수 있을 때만)
|
||||
- fallback 사용 여부
|
||||
|
||||
## Done when
|
||||
|
||||
- `llm.mode = host | standalone` 구조가 코드와 문서에 반영된다.
|
||||
- host unavailable 시 fallback 규칙이 명확히 동작한다.
|
||||
- generator가 특정 provider/model을 직접 전제하지 않는다.
|
||||
- status/log에서 마지막 LLM route를 볼 수 있다.
|
||||
- 기존 standalone config 사용자는 깨지지 않는다.
|
||||
125
docs/vault-schema.md
Normal file
125
docs/vault-schema.md
Normal file
@@ -0,0 +1,125 @@
|
||||
# 볼트 스키마
|
||||
|
||||
## 디렉토리 구조
|
||||
|
||||
```
|
||||
{vault}/
|
||||
├── SCHEMA.md # 이 파일 (컨벤션 문서)
|
||||
├── index.md # 세션 인덱스
|
||||
├── log.md # 작업 로그
|
||||
├── overview.md # 전체 요약 (AI 생성)
|
||||
│
|
||||
├── raw/ # ← 불변 원본
|
||||
│ └── mempalace/
|
||||
│ ├── infrastructure/
|
||||
│ │ ├── server-config/
|
||||
│ │ │ └── abc123.md
|
||||
│ │ └── network-topology/
|
||||
│ │ └── def456.md
|
||||
│ ├── projects/
|
||||
│ │ └── project-beta/
|
||||
│ │ └── ghi789.md
|
||||
│ └── ...
|
||||
│
|
||||
├── wiki/ # ← AI 생성 정리본
|
||||
│ ├── projects/
|
||||
│ │ └── project-beta.md
|
||||
│ ├── topics/
|
||||
│ │ └── fsrs-algorithm.md
|
||||
│ ├── decisions/
|
||||
│ │ └── 2026-04-15-nestjs-circular-deps.md
|
||||
│ ├── people/
|
||||
│ │ └── sisters.md
|
||||
│ ├── infrastructure/
|
||||
│ │ └── server-cluster.md
|
||||
│ └── guides/
|
||||
│ └── deployment-workflow.md
|
||||
│
|
||||
└── graph/
|
||||
└── graph.json # KG 내보내기
|
||||
```
|
||||
|
||||
## frontmatter 스키마
|
||||
|
||||
### raw/ 마크다운 (MemPalace 서랍)
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: drawer
|
||||
source: mempalace
|
||||
wing: {wing_name}
|
||||
room: {room_name}
|
||||
drawer_id: {unique_id}
|
||||
created: "YYYY-MM-DD"
|
||||
agent: {sister_name}
|
||||
tags: [tag1, tag2]
|
||||
---
|
||||
|
||||
# [{wing} / {room}]
|
||||
|
||||
{서랍 원본 내용 그대로}
|
||||
```
|
||||
|
||||
### wiki/ 마크다운 (AI 생성)
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: wiki
|
||||
category: projects|topics|decisions|people|infrastructure|guides
|
||||
title: 페이지 제목
|
||||
created: "YYYY-MM-DD"
|
||||
updated: "YYYY-MM-DD"
|
||||
sources: [drawer_id_1, drawer_id_2] # 참조한 원본
|
||||
tags: [tag1, tag2]
|
||||
status: draft|stable|archived
|
||||
agent: {생성한 agents}
|
||||
---
|
||||
|
||||
# {제목}
|
||||
|
||||
{AI가 정리한 내용}
|
||||
```
|
||||
|
||||
### index.md
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: index
|
||||
vault_path: /path/to/vault
|
||||
last_updated: "YYYY-MM-DDTHH:mm:ss"
|
||||
---
|
||||
|
||||
# Wiki Index
|
||||
|
||||
## Recent
|
||||
|
||||
- [[raw/mempalace/infrastructure/server-config/abc123|Server 설정]] — contributor, 2026-04-15
|
||||
- [[wiki/infrastructure/server-cluster|Server 클러스터 구성]] — 2026-04-15
|
||||
```
|
||||
|
||||
### log.md
|
||||
|
||||
```yaml
|
||||
---
|
||||
type: log
|
||||
---
|
||||
|
||||
# Sync Log
|
||||
|
||||
| 시간 | 작업 | 대상 | 결과 |
|
||||
|------|------|------|------|
|
||||
| 2026-04-15 13:00 | ingest | mempalace → raw | +5 drawers |
|
||||
| 2026-04-15 13:01 | wiki | raw → wiki | +2 pages |
|
||||
```
|
||||
|
||||
## 파일 명명 규칙
|
||||
|
||||
- raw/: `{drawer_id}.md` (ID 기반, 변경 불가)
|
||||
- wiki/: `{kebab-case-title}.md` (주제 기반, 사람이 읽을 수 있게)
|
||||
- 디렉토리: 영어 소문자 + 하이픈
|
||||
|
||||
## 백링크 규칙
|
||||
|
||||
- wiki/ 페이지 간: `[[페이지 제목]]` 또는 `[[경로/페이지|표시 텍스트]]`
|
||||
- wiki → raw: `[[raw/mempalace/{wing}/{room}/{id}|표시 텍스트]]`
|
||||
- overview.md에서 모든 wiki/ 페이지 링크
|
||||
105
docs/wiki-generation.md
Normal file
105
docs/wiki-generation.md
Normal file
@@ -0,0 +1,105 @@
|
||||
# 위키 생성 프롬프트 전략
|
||||
|
||||
## 역할
|
||||
|
||||
HaBraid는 기억 저장소를 사람이 읽고 AI가 다시 활용할 수 있는 위키 표현층으로 바꾸는 엔진이다.
|
||||
|
||||
- 목표는 단순 요약이 아니라 **정리 + 재구성**
|
||||
- raw memory를 topic / entity / decision / timeline 같은 의미 단위로 재편성
|
||||
- 결과물은 human-readable 이면서 agent-readable 해야 함
|
||||
|
||||
## LLM 실행 방식
|
||||
|
||||
### 기본: host-following
|
||||
- `llm.mode = host`
|
||||
- 가능하면 host(Hermes/OpenClaw/Codex CLI)가 실제 생성 수행
|
||||
- HaBraid는 prompt 생성, batching, 파싱, 저장을 담당
|
||||
|
||||
### fallback / standalone
|
||||
- host route unavailable이면 `llm.fallback` 사용
|
||||
- standalone mode에서는 habraid가 직접 backend 호출
|
||||
- 지원 대상: `zai`, `openai`, `ollama`
|
||||
|
||||
```typescript
|
||||
interface LLMClient {
|
||||
generate(systemPrompt: string, userPrompt: string, timeoutMs?: number): Promise<{
|
||||
text: string;
|
||||
metadata: {
|
||||
route: "host" | "fallback" | "standalone";
|
||||
provider?: string;
|
||||
model?: string;
|
||||
fallbackUsed: boolean;
|
||||
};
|
||||
}>;
|
||||
}
|
||||
```
|
||||
|
||||
## 프롬프트 구조
|
||||
|
||||
### 시스템 프롬프트 (항상 포함)
|
||||
|
||||
```text
|
||||
당신은 HaBraid wiki 관리 에이전트입니다.
|
||||
|
||||
## 역할
|
||||
memory backend에 저장된 인프라, 프로젝트, 의사결정 지식을 Obsidian 위키 페이지로 정리합니다.
|
||||
|
||||
## 철학
|
||||
- "요약"이 아니라 "정리"가 목표입니다.
|
||||
- 원본을 다시 열지 않아도 될 정도로 충분히 구조화하세요.
|
||||
- 기술 결정: 왜 A를 선택했는지, 트레이드오프를 구체적으로
|
||||
- 에러/해결: 에러 메시지, 원인, 해결 방법을 그대로
|
||||
- 코드/설정: 스니펫, 명령어, 경로를 원문 그대로 포함
|
||||
- 한국어 본문, 코드/경로/명령어는 영어 원문 유지
|
||||
|
||||
## 출력 형식
|
||||
- YAML frontmatter 포함 마크다운
|
||||
- Obsidian 백링크 [[]] 사용
|
||||
- 마크다운 헤딩으로 구조화
|
||||
```
|
||||
|
||||
### 증분 업데이트 프롬프트
|
||||
|
||||
```text
|
||||
## 새로 수집된 데이터
|
||||
- item 수: N개
|
||||
- Wings: {wing 목록}
|
||||
- Rooms: {room 목록}
|
||||
|
||||
## 새 item 내용
|
||||
{각 item의 wing/room/content}
|
||||
|
||||
## 기존 위키 페이지 목록
|
||||
{wiki/ 디렉토리의 파일 목록}
|
||||
|
||||
## 작업
|
||||
1. 새 내용을 분석하세요
|
||||
2. 기존 wiki/ 페이지와 관련 있으면 해당 페이지에 내용 추가
|
||||
3. 새로운 주제면 적절한 카테고리에 새 페이지 생성
|
||||
4. SCHEMA/frontmatter 규칙을 따르세요
|
||||
5. sources에 source drawer/item ID를 반드시 포함하세요
|
||||
```
|
||||
|
||||
## 증분 업데이트 로직
|
||||
|
||||
1. DB에서 `wiki_generated_at IS NULL` 아이템 조회
|
||||
2. raw/ 또는 DB content를 prompt 컨텍스트로 구성
|
||||
3. 기존 wiki/ 페이지 목록 포함
|
||||
4. LLM 응답 파싱 → 파일로 저장
|
||||
5. 생성된 아이템은 DB에 generated 상태 기록
|
||||
6. `.sync-state.json`에 마지막 LLM route/provider/model 기록
|
||||
7. overview/index/log 갱신
|
||||
|
||||
## 배치 전략
|
||||
|
||||
- room 단위 그룹핑
|
||||
- room 내부는 subgroup/batch 분할
|
||||
- 큰 초기 rebuild에서는 첫 room이 매우 커질 수 있으므로 `maxRooms=1`이 항상 가벼운 smoke test는 아님
|
||||
- 진행 상황은 `/tmp/habraid-wiki-gen.log`에 기록
|
||||
|
||||
## 토큰/시간 관리
|
||||
|
||||
- 긴 content는 prompt용으로 잘라서 사용
|
||||
- 초기 full generation은 시간이 오래 걸릴 수 있음
|
||||
- host-following 구조에서도 batching은 유지한다
|
||||
- route metadata를 남겨 실제 어떤 경로(host/fallback/standalone)로 생성됐는지 추적한다
|
||||
3849
package-lock.json
generated
Normal file
3849
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
45
package.json
Normal file
45
package.json
Normal file
@@ -0,0 +1,45 @@
|
||||
{
|
||||
"name": "habraid",
|
||||
"version": "0.1.0",
|
||||
"description": "Obsidian wiki engine for HaBraid Team — knowledge vault with MemPalace integration",
|
||||
"type": "module",
|
||||
"main": "dist/index.js",
|
||||
"bin": {
|
||||
"habraid": "dist/mcp-server.js"
|
||||
},
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"dev": "tsx src/index.ts",
|
||||
"start": "node dist/index.js",
|
||||
"lint": "tsc --noEmit",
|
||||
"test": "vitest run",
|
||||
"test:watch": "vitest"
|
||||
},
|
||||
"keywords": [
|
||||
"obsidian",
|
||||
"wiki",
|
||||
"mempalace",
|
||||
"knowledge-base"
|
||||
],
|
||||
"author": "contributor",
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"@modelcontextprotocol/sdk": "^1.29.0",
|
||||
"@xenova/transformers": "^2.17.2",
|
||||
"better-sqlite3": "^11.0.0",
|
||||
"chalk": "^5.3.0",
|
||||
"commander": "^12.0.0",
|
||||
"dotenv": "^17.4.2",
|
||||
"gray-matter": "^4.0.3",
|
||||
"hnswlib-node": "^3.0.0",
|
||||
"ora": "^8.0.0",
|
||||
"simple-git": "^3.0.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/better-sqlite3": "^7.6.0",
|
||||
"@types/node": "^22.0.0",
|
||||
"tsx": "^4.0.0",
|
||||
"typescript": "^5.9.3",
|
||||
"vitest": "^4.1.4"
|
||||
}
|
||||
}
|
||||
193
src/__tests__/community.test.ts
Normal file
193
src/__tests__/community.test.ts
Normal file
@@ -0,0 +1,193 @@
|
||||
/**
|
||||
* Community detection tests for HaBraid.
|
||||
*
|
||||
* Tests LPA community detection, isolated nodes, and persistence.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import { addEntity, addRelation } from "../db/kg.js";
|
||||
import { detectCommunities } from "../graph/community.js";
|
||||
import {
|
||||
saveCommunities,
|
||||
loadCommunities,
|
||||
getEntityCommunityId,
|
||||
} from "../db/communities.js";
|
||||
import { createTestDb } from "./setup.js";
|
||||
|
||||
describe("Community Detection", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
describe("detectCommunities (LPA)", () => {
|
||||
it("should return empty for empty graph", () => {
|
||||
const communities = detectCommunities(db);
|
||||
expect(communities).toEqual([]);
|
||||
});
|
||||
|
||||
it("should detect a single community in a connected graph", () => {
|
||||
// Create a triangle: A - B - C - A
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
addEntity(db, { id: "b", name: "B", type: "concept" });
|
||||
addEntity(db, { id: "c", name: "C", type: "concept" });
|
||||
|
||||
addRelation(db, { subjectId: "a", predicate: "relates", objectId: "b" });
|
||||
addRelation(db, { subjectId: "b", predicate: "relates", objectId: "c" });
|
||||
addRelation(db, { subjectId: "c", predicate: "relates", objectId: "a" });
|
||||
|
||||
const communities = detectCommunities(db);
|
||||
|
||||
expect(communities.length).toBe(1);
|
||||
expect(communities[0].size).toBe(3);
|
||||
expect(communities[0].entities.sort()).toEqual(["a", "b", "c"]);
|
||||
});
|
||||
|
||||
it("should detect separate communities for disconnected subgraphs", () => {
|
||||
// Subgraph 1: A - B
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
addEntity(db, { id: "b", name: "B", type: "concept" });
|
||||
addRelation(db, { subjectId: "a", predicate: "relates", objectId: "b" });
|
||||
|
||||
// Subgraph 2: C - D
|
||||
addEntity(db, { id: "c", name: "C", type: "concept" });
|
||||
addEntity(db, { id: "d", name: "D", type: "concept" });
|
||||
addRelation(db, { subjectId: "c", predicate: "relates", objectId: "d" });
|
||||
|
||||
const communities = detectCommunities(db);
|
||||
|
||||
expect(communities.length).toBe(2);
|
||||
});
|
||||
|
||||
it("should assign isolated nodes to their own communities", () => {
|
||||
// Isolated nodes with no relations
|
||||
addEntity(db, { id: "iso1", name: "Isolated1", type: "concept" });
|
||||
addEntity(db, { id: "iso2", name: "Isolated2", type: "person" });
|
||||
addEntity(db, { id: "iso3", name: "Isolated3", type: "tool" });
|
||||
|
||||
const communities = detectCommunities(db);
|
||||
|
||||
// Each isolated node forms its own community
|
||||
expect(communities.length).toBe(3);
|
||||
|
||||
// Each community should have exactly one entity
|
||||
for (const community of communities) {
|
||||
expect(community.size).toBe(1);
|
||||
expect(community.entities.length).toBe(1);
|
||||
}
|
||||
});
|
||||
|
||||
it("should include dominantType in communities", () => {
|
||||
addEntity(db, { id: "a", name: "A", type: "tool" });
|
||||
addEntity(db, { id: "b", name: "B", type: "tool" });
|
||||
addRelation(db, { subjectId: "a", predicate: "relates", objectId: "b" });
|
||||
|
||||
const communities = detectCommunities(db);
|
||||
|
||||
expect(communities[0].dominantType).toBe("tool");
|
||||
});
|
||||
|
||||
it("should produce communities with valid IDs and labels", () => {
|
||||
addEntity(db, { id: "a", name: "Alpha", type: "concept" });
|
||||
|
||||
const communities = detectCommunities(db);
|
||||
|
||||
expect(communities[0].id).toMatch(/^community-\d+$/);
|
||||
expect(communities[0].label).toBeTruthy();
|
||||
});
|
||||
});
|
||||
|
||||
describe("Community Persistence (save/load)", () => {
|
||||
it("should save and load communities correctly", () => {
|
||||
// First add entities so the FK constraints are satisfied
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
addEntity(db, { id: "b", name: "B", type: "concept" });
|
||||
addEntity(db, { id: "c", name: "C", type: "concept" });
|
||||
|
||||
const communities = [
|
||||
{
|
||||
id: "community-0",
|
||||
label: "Test Community",
|
||||
entities: ["a", "b"],
|
||||
size: 2,
|
||||
dominantType: "concept",
|
||||
},
|
||||
{
|
||||
id: "community-1",
|
||||
label: "Isolated",
|
||||
entities: ["c"],
|
||||
size: 1,
|
||||
dominantType: "concept",
|
||||
},
|
||||
];
|
||||
|
||||
saveCommunities(db, communities);
|
||||
const loaded = loadCommunities(db);
|
||||
|
||||
expect(loaded.length).toBe(2);
|
||||
expect(loaded[0].id).toBe("community-0");
|
||||
expect(loaded[0].entities.sort()).toEqual(["a", "b"]);
|
||||
expect(loaded[1].id).toBe("community-1");
|
||||
expect(loaded[1].entities).toEqual(["c"]);
|
||||
});
|
||||
|
||||
it("should replace existing communities on save", () => {
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
addEntity(db, { id: "b", name: "B", type: "concept" });
|
||||
|
||||
saveCommunities(db, [
|
||||
{
|
||||
id: "community-0",
|
||||
label: "Old",
|
||||
entities: ["a"],
|
||||
size: 1,
|
||||
dominantType: "concept",
|
||||
},
|
||||
]);
|
||||
|
||||
saveCommunities(db, [
|
||||
{
|
||||
id: "community-0",
|
||||
label: "New",
|
||||
entities: ["a", "b"],
|
||||
size: 2,
|
||||
dominantType: "concept",
|
||||
},
|
||||
]);
|
||||
|
||||
const loaded = loadCommunities(db);
|
||||
expect(loaded.length).toBe(1);
|
||||
expect(loaded[0].entities.sort()).toEqual(["a", "b"]);
|
||||
});
|
||||
|
||||
it("should look up entity community ID", () => {
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
|
||||
saveCommunities(db, [
|
||||
{
|
||||
id: "community-0",
|
||||
label: "Test",
|
||||
entities: ["a"],
|
||||
size: 1,
|
||||
dominantType: "concept",
|
||||
},
|
||||
]);
|
||||
|
||||
const communityId = getEntityCommunityId(db, "a");
|
||||
expect(communityId).toBe("community-0");
|
||||
});
|
||||
|
||||
it("should return null for entity not in any community", () => {
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
|
||||
const communityId = getEntityCommunityId(db, "a");
|
||||
expect(communityId).toBeNull();
|
||||
});
|
||||
});
|
||||
});
|
||||
418
src/__tests__/contradiction.test.ts
Normal file
418
src/__tests__/contradiction.test.ts
Normal file
@@ -0,0 +1,418 @@
|
||||
/**
|
||||
* Contradiction detection tests for HaBraid.
|
||||
*
|
||||
* Tests status conflicts, date conflicts, tag conflicts,
|
||||
* markContradiction, getContradictions, and resolveContradiction.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import { createItem } from "../db/items.js";
|
||||
import {
|
||||
detectContradictions,
|
||||
markContradiction,
|
||||
getContradictions,
|
||||
resolveContradiction,
|
||||
getContradictionById,
|
||||
scanAllContradictions,
|
||||
getContradictionStats,
|
||||
detectWikiContradictions,
|
||||
} from "../wiki/contradiction.js";
|
||||
import { createTestDb, sampleCreateParams } from "./setup.js";
|
||||
|
||||
describe("Contradiction Detection", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
describe("detectContradictions - status conflicts", () => {
|
||||
it("should detect status conflicts for same project", () => {
|
||||
// Create an existing item about project X with status "active"
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project X status",
|
||||
content: "Project X is active",
|
||||
metadata: { project: "ProjectX", status: "active" },
|
||||
}));
|
||||
|
||||
// Create a new item about same project with different status
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Project X update",
|
||||
content: "Project X is now deprecated",
|
||||
metadata: { project: "ProjectX", status: "deprecated" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
|
||||
expect(contradictions.length).toBeGreaterThanOrEqual(1);
|
||||
const statusConflict = contradictions.find((c) => c.field === "status");
|
||||
expect(statusConflict).toBeDefined();
|
||||
expect(statusConflict!.value_a).toBe("active");
|
||||
expect(statusConflict!.value_b).toBe("deprecated");
|
||||
expect(statusConflict!.severity).toBe("high");
|
||||
});
|
||||
|
||||
it("should not detect conflict when status is the same", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project X status",
|
||||
content: "Project X is active",
|
||||
metadata: { project: "ProjectX", status: "active" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Project X note",
|
||||
content: "Project X remains active",
|
||||
metadata: { project: "ProjectX", status: "active" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
const statusConflict = contradictions.find((c) => c.field === "status");
|
||||
expect(statusConflict).toBeUndefined();
|
||||
});
|
||||
|
||||
it("should not detect conflict for different projects", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project A status",
|
||||
content: "Project A is active",
|
||||
metadata: { project: "ProjectA", status: "active" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Project B status",
|
||||
content: "Project B is deprecated",
|
||||
metadata: { project: "ProjectB", status: "deprecated" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
expect(contradictions.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectContradictions - date conflicts", () => {
|
||||
it("should detect date conflicts for same event", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Event note 1",
|
||||
content: "The conference was on Jan 15",
|
||||
metadata: { event: "TechConf 2025", date: "2025-01-15" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Event note 2",
|
||||
content: "The conference was on Feb 20",
|
||||
metadata: { event: "TechConf 2025", event_date: "2025-02-20" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
const dateConflict = contradictions.find((c) => c.field === "date");
|
||||
|
||||
expect(dateConflict).toBeDefined();
|
||||
expect(dateConflict!.severity).toBe("medium");
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectContradictions - tag conflicts", () => {
|
||||
it("should detect active/deprecated tag conflicts", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Entity A",
|
||||
content: "About entity A",
|
||||
tags: ["active"],
|
||||
metadata: { entity: "MyEntity" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Entity A update",
|
||||
content: "About entity A updated",
|
||||
tags: ["deprecated"],
|
||||
metadata: { entity: "MyEntity" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
const tagConflict = contradictions.find((c) => c.field === "tags");
|
||||
|
||||
expect(tagConflict).toBeDefined();
|
||||
});
|
||||
|
||||
it("should detect stable/experimental tag conflicts", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Entity B",
|
||||
content: "About entity B",
|
||||
tags: ["stable"],
|
||||
metadata: { entity: "EntityB" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Entity B update",
|
||||
content: "About entity B updated",
|
||||
tags: ["experimental"],
|
||||
metadata: { entity: "EntityB" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
const tagConflict = contradictions.find((c) => c.field === "tags");
|
||||
|
||||
expect(tagConflict).toBeDefined();
|
||||
});
|
||||
|
||||
it("should not detect conflict when tags are compatible", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Entity C",
|
||||
content: "About entity C",
|
||||
tags: ["active", "production"],
|
||||
metadata: { entity: "EntityC" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Entity C update",
|
||||
content: "About entity C updated",
|
||||
tags: ["active", "stable"],
|
||||
metadata: { entity: "EntityC" },
|
||||
}));
|
||||
|
||||
const contradictions = detectContradictions(db, newItem);
|
||||
const tagConflict = contradictions.find((c) => c.field === "tags");
|
||||
expect(tagConflict).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("markContradiction / getContradictions", () => {
|
||||
it("should mark and retrieve a contradiction", () => {
|
||||
const input = {
|
||||
item_a_id: "item-a",
|
||||
item_b_id: "item-b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high" as const,
|
||||
};
|
||||
|
||||
const marked = markContradiction(db, input);
|
||||
|
||||
expect(marked.id).toBeDefined();
|
||||
expect(marked.item_a_id).toBe("item-a");
|
||||
expect(marked.item_b_id).toBe("item-b");
|
||||
expect(marked.field).toBe("status");
|
||||
expect(marked.status).toBe("open");
|
||||
});
|
||||
|
||||
it("should retrieve contradictions filtered by status", () => {
|
||||
markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high",
|
||||
});
|
||||
|
||||
const open = getContradictions(db, "open");
|
||||
const resolved = getContradictions(db, "resolved");
|
||||
|
||||
expect(open.length).toBe(1);
|
||||
expect(resolved.length).toBe(0);
|
||||
});
|
||||
|
||||
it("should retrieve all contradictions without filter", () => {
|
||||
markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high",
|
||||
});
|
||||
markContradiction(db, {
|
||||
item_a_id: "c",
|
||||
item_b_id: "d",
|
||||
field: "date",
|
||||
value_a: "2025-01-01",
|
||||
value_b: "2025-02-01",
|
||||
severity: "medium",
|
||||
});
|
||||
|
||||
const all = getContradictions(db);
|
||||
expect(all.length).toBe(2);
|
||||
});
|
||||
|
||||
it("should avoid duplicate open contradictions for same pair+field", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project X",
|
||||
content: "Active",
|
||||
metadata: { project: "ProjectX", status: "active" },
|
||||
}));
|
||||
|
||||
const newItem = createItem(db, sampleCreateParams({
|
||||
title: "Project X update",
|
||||
content: "Deprecated",
|
||||
metadata: { project: "ProjectX", status: "deprecated" },
|
||||
}));
|
||||
|
||||
// First detection
|
||||
const first = detectContradictions(db, newItem);
|
||||
expect(first.length).toBeGreaterThanOrEqual(1);
|
||||
|
||||
// Second detection should not create duplicates
|
||||
const second = detectContradictions(db, newItem);
|
||||
expect(second.length).toBe(0);
|
||||
|
||||
const all = getContradictions(db);
|
||||
const statusConflicts = all.filter((c) => c.field === "status");
|
||||
expect(statusConflicts.length).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("resolveContradiction", () => {
|
||||
it("should mark a contradiction as resolved", () => {
|
||||
const marked = markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high",
|
||||
});
|
||||
|
||||
const resolved = resolveContradiction(db, marked.id!, {
|
||||
resolution: "Accepted deprecated status as correct",
|
||||
status: "resolved",
|
||||
});
|
||||
|
||||
expect(resolved).toBeDefined();
|
||||
expect(resolved!.status).toBe("resolved");
|
||||
expect(resolved!.resolution).toBe("Accepted deprecated status as correct");
|
||||
});
|
||||
|
||||
it("should mark a contradiction as false_positive", () => {
|
||||
const marked = markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "date",
|
||||
value_a: "2025-01-01",
|
||||
value_b: "2025-01-02",
|
||||
severity: "medium",
|
||||
});
|
||||
|
||||
const resolved = resolveContradiction(db, marked.id!, {
|
||||
resolution: "Different time zones, not a conflict",
|
||||
status: "false_positive",
|
||||
});
|
||||
|
||||
expect(resolved!.status).toBe("false_positive");
|
||||
});
|
||||
|
||||
it("should return undefined for non-existent ID", () => {
|
||||
const result = resolveContradiction(db, 99999, {
|
||||
resolution: "N/A",
|
||||
status: "resolved",
|
||||
});
|
||||
|
||||
expect(result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("getContradictionById", () => {
|
||||
it("should retrieve a contradiction by ID", () => {
|
||||
const marked = markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high",
|
||||
});
|
||||
|
||||
const retrieved = getContradictionById(db, marked.id!);
|
||||
expect(retrieved).toBeDefined();
|
||||
expect(retrieved!.id).toBe(marked.id);
|
||||
});
|
||||
|
||||
it("should return undefined for non-existent ID", () => {
|
||||
const result = getContradictionById(db, 99999);
|
||||
expect(result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("scanAllContradictions", () => {
|
||||
it("should scan all items and find contradictions", () => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project X status",
|
||||
content: "Active",
|
||||
metadata: { project: "ProjectX", status: "active" },
|
||||
}));
|
||||
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Project X update",
|
||||
content: "Deprecated",
|
||||
metadata: { project: "ProjectX", status: "deprecated" },
|
||||
}));
|
||||
|
||||
const count = scanAllContradictions(db);
|
||||
expect(count).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getContradictionStats", () => {
|
||||
it("should return correct stats", () => {
|
||||
markContradiction(db, {
|
||||
item_a_id: "a",
|
||||
item_b_id: "b",
|
||||
field: "status",
|
||||
value_a: "active",
|
||||
value_b: "deprecated",
|
||||
severity: "high",
|
||||
});
|
||||
|
||||
markContradiction(db, {
|
||||
item_a_id: "c",
|
||||
item_b_id: "d",
|
||||
field: "date",
|
||||
value_a: "2025-01-01",
|
||||
value_b: "2025-02-01",
|
||||
severity: "medium",
|
||||
});
|
||||
|
||||
const stats = getContradictionStats(db);
|
||||
|
||||
expect(stats.total).toBe(2);
|
||||
expect(stats.open).toBe(2);
|
||||
expect(stats.resolved).toBe(0);
|
||||
expect(stats.falsePositive).toBe(0);
|
||||
expect(stats.bySeverity).toEqual({ high: 1, medium: 1 });
|
||||
});
|
||||
});
|
||||
|
||||
describe("detectWikiContradictions", () => {
|
||||
it("should detect internal contradictions in wiki content", () => {
|
||||
const wikiContent = `
|
||||
# Project Status
|
||||
|
||||
- **status**: active
|
||||
- **status**: deprecated
|
||||
`;
|
||||
|
||||
const contradictions = detectWikiContradictions(db, wikiContent, "test-slug");
|
||||
|
||||
expect(contradictions.length).toBeGreaterThanOrEqual(1);
|
||||
const statusConflict = contradictions.find((c) =>
|
||||
c.field === "wiki.status",
|
||||
);
|
||||
expect(statusConflict).toBeDefined();
|
||||
});
|
||||
|
||||
it("should not detect contradictions in consistent wiki content", () => {
|
||||
const wikiContent = `
|
||||
# Project Status
|
||||
|
||||
- **status**: active
|
||||
- **date**: 2025-01-01
|
||||
`;
|
||||
|
||||
const contradictions = detectWikiContradictions(db, wikiContent, "test-slug");
|
||||
expect(contradictions.length).toBe(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
123
src/__tests__/database.test.ts
Normal file
123
src/__tests__/database.test.ts
Normal file
@@ -0,0 +1,123 @@
|
||||
/**
|
||||
* Database schema tests for HaBraid.
|
||||
*
|
||||
* Tests that the full schema migration runs correctly and produces
|
||||
* all expected tables, indexes, and triggers.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import { getSchemaVersion } from "../db/database.js";
|
||||
import { createTestDb } from "./setup.js";
|
||||
|
||||
describe("Database Schema", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("should have schema version 7 after migration", () => {
|
||||
const version = getSchemaVersion(db);
|
||||
expect(version).toBe(7);
|
||||
});
|
||||
|
||||
it("should have all expected tables", () => {
|
||||
const tables = db
|
||||
.prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' ORDER BY name",
|
||||
)
|
||||
.all() as Array<{ name: string }>;
|
||||
|
||||
const tableNames = tables.map((t) => t.name);
|
||||
|
||||
expect(tableNames).toContain("items");
|
||||
expect(tableNames).toContain("schema_version");
|
||||
expect(tableNames).toContain("item_vectors");
|
||||
expect(tableNames).toContain("kg_entities");
|
||||
expect(tableNames).toContain("kg_relations");
|
||||
expect(tableNames).toContain("contradictions");
|
||||
expect(tableNames).toContain("kg_communities");
|
||||
expect(tableNames).toContain("kg_community_members");
|
||||
});
|
||||
|
||||
it("should have FTS5 virtual table", () => {
|
||||
const tables = db
|
||||
.prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='table' AND name='items_fts'",
|
||||
)
|
||||
.all() as Array<{ name: string }>;
|
||||
|
||||
expect(tables.length).toBe(1);
|
||||
expect(tables[0].name).toBe("items_fts");
|
||||
});
|
||||
|
||||
it("should have WAL mode or memory mode", () => {
|
||||
// In-memory databases report "memory" instead of "wal"
|
||||
const result = db.pragma("journal_mode") as Array<{ journal_mode: string }>;
|
||||
expect(["wal", "memory"]).toContain(result[0].journal_mode);
|
||||
});
|
||||
|
||||
it("should have foreign keys enabled", () => {
|
||||
const result = db.pragma("foreign_keys") as Array<{ foreign_keys: number }>;
|
||||
expect(result[0].foreign_keys).toBe(1);
|
||||
});
|
||||
|
||||
it("should have all expected triggers", () => {
|
||||
const triggers = db
|
||||
.prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='trigger' ORDER BY name",
|
||||
)
|
||||
.all() as Array<{ name: string }>;
|
||||
|
||||
const triggerNames = triggers.map((t) => t.name);
|
||||
|
||||
expect(triggerNames).toContain("items_ai");
|
||||
expect(triggerNames).toContain("items_ad");
|
||||
expect(triggerNames).toContain("items_au");
|
||||
});
|
||||
|
||||
it("should have expected indexes", () => {
|
||||
const indexes = db
|
||||
.prepare(
|
||||
"SELECT name FROM sqlite_master WHERE type='index' AND name NOT LIKE 'sqlite_%' ORDER BY name",
|
||||
)
|
||||
.all() as Array<{ name: string }>;
|
||||
|
||||
const indexNames = indexes.map((i) => i.name);
|
||||
|
||||
expect(indexNames).toContain("idx_item_vectors_model");
|
||||
expect(indexNames).toContain("idx_kg_relations_subject");
|
||||
expect(indexNames).toContain("idx_kg_relations_object");
|
||||
expect(indexNames).toContain("idx_kg_entities_type");
|
||||
expect(indexNames).toContain("idx_contradictions_status");
|
||||
expect(indexNames).toContain("idx_contradictions_item_a");
|
||||
expect(indexNames).toContain("idx_contradictions_item_b");
|
||||
expect(indexNames).toContain("idx_contradictions_field");
|
||||
expect(indexNames).toContain("idx_kg_community_members_entity");
|
||||
});
|
||||
|
||||
it("items table should have content_hash column", () => {
|
||||
const cols = db.pragma("table_info(items)") as Array<{
|
||||
name: string;
|
||||
type: string;
|
||||
}>;
|
||||
const colNames = cols.map((c) => c.name);
|
||||
|
||||
expect(colNames).toContain("content_hash");
|
||||
});
|
||||
|
||||
it("contradictions table should have slug columns", () => {
|
||||
const cols = db.pragma("table_info(contradictions)") as Array<{
|
||||
name: string;
|
||||
}>;
|
||||
const colNames = cols.map((c) => c.name);
|
||||
|
||||
expect(colNames).toContain("item_a_slug");
|
||||
expect(colNames).toContain("item_b_slug");
|
||||
});
|
||||
});
|
||||
220
src/__tests__/hashing.test.ts
Normal file
220
src/__tests__/hashing.test.ts
Normal file
@@ -0,0 +1,220 @@
|
||||
/**
|
||||
* Content hashing tests for HaBraid.
|
||||
*
|
||||
* Tests computeItemHash determinism, getChangedItems,
|
||||
* getWikiGenerationCounts, and hash mismatch detection.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import {
|
||||
computeItemHash,
|
||||
getChangedItems,
|
||||
getWikiGenerationCounts,
|
||||
markItemHash,
|
||||
getGeneratedSlugs,
|
||||
} from "../db/hashing.js";
|
||||
import { createItem, updateItem } from "../db/items.js";
|
||||
import { createTestDb, sampleCreateParams, sampleItem } from "./setup.js";
|
||||
|
||||
describe("Content Hashing", () => {
|
||||
describe("computeItemHash", () => {
|
||||
it("should be deterministic", () => {
|
||||
const item = sampleItem();
|
||||
const hash1 = computeItemHash(item);
|
||||
const hash2 = computeItemHash(item);
|
||||
|
||||
expect(hash1).toBe(hash2);
|
||||
});
|
||||
|
||||
it("should produce different hashes for different content", () => {
|
||||
const item1 = sampleItem({ content: "Content A" });
|
||||
const item2 = sampleItem({ content: "Content B" });
|
||||
|
||||
expect(computeItemHash(item1)).not.toBe(computeItemHash(item2));
|
||||
});
|
||||
|
||||
it("should produce different hashes for different titles", () => {
|
||||
const item1 = sampleItem({ title: "Title A" });
|
||||
const item2 = sampleItem({ title: "Title B" });
|
||||
|
||||
expect(computeItemHash(item1)).not.toBe(computeItemHash(item2));
|
||||
});
|
||||
|
||||
it("should produce different hashes for different tags", () => {
|
||||
const item1 = sampleItem({ tags: ["a", "b"] });
|
||||
const item2 = sampleItem({ tags: ["c", "d"] });
|
||||
|
||||
expect(computeItemHash(item1)).not.toBe(computeItemHash(item2));
|
||||
});
|
||||
|
||||
it("should produce the same hash regardless of tag order", () => {
|
||||
const item1 = sampleItem({ tags: ["alpha", "beta", "gamma"] });
|
||||
const item2 = sampleItem({ tags: ["gamma", "alpha", "beta"] });
|
||||
|
||||
expect(computeItemHash(item1)).toBe(computeItemHash(item2));
|
||||
});
|
||||
|
||||
it("should produce different hashes for different categories", () => {
|
||||
const item1 = sampleItem({ category: "projects" });
|
||||
const item2 = sampleItem({ category: "topics" });
|
||||
|
||||
expect(computeItemHash(item1)).not.toBe(computeItemHash(item2));
|
||||
});
|
||||
|
||||
it("should return a 64-char hex string (SHA-256)", () => {
|
||||
const item = sampleItem();
|
||||
const hash = computeItemHash(item);
|
||||
|
||||
expect(hash).toHaveLength(64);
|
||||
expect(hash).toMatch(/^[0-9a-f]+$/);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getChangedItems", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("should detect no changes for freshly created items without wiki generation", () => {
|
||||
createItem(db, sampleCreateParams());
|
||||
|
||||
const changed = getChangedItems(db);
|
||||
expect(changed.length).toBe(0);
|
||||
});
|
||||
|
||||
it("should detect items with hash mismatch after update", () => {
|
||||
const item = createItem(db, sampleCreateParams());
|
||||
|
||||
// Mark it as wiki-generated (simulates first generation)
|
||||
const now = new Date().toISOString();
|
||||
db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
).run(now, "test-slug", item.id);
|
||||
|
||||
// Now update the item content (which changes the hash)
|
||||
updateItem(db, item.id, { content: "Updated content that differs" });
|
||||
|
||||
// Directly tamper with the stored hash to simulate a stale hash
|
||||
db.prepare(
|
||||
"UPDATE items SET content_hash = '0000000000000000000000000000000000000000000000000000000000000000' WHERE id = ?",
|
||||
).run(item.id);
|
||||
|
||||
const changed = getChangedItems(db);
|
||||
expect(changed.length).toBe(1);
|
||||
expect(changed[0].id).toBe(item.id);
|
||||
});
|
||||
});
|
||||
|
||||
describe("markItemHash", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("should update the stored hash for an item", () => {
|
||||
const item = createItem(db, sampleCreateParams());
|
||||
|
||||
markItemHash(db, item.id, "abcdef1234567890".repeat(4));
|
||||
|
||||
const row = db
|
||||
.prepare("SELECT content_hash FROM items WHERE id = ?")
|
||||
.get(item.id) as { content_hash: string };
|
||||
|
||||
expect(row.content_hash).toBe("abcdef1234567890".repeat(4));
|
||||
});
|
||||
});
|
||||
|
||||
describe("getWikiGenerationCounts", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("should report correct counts for empty database", () => {
|
||||
const counts = getWikiGenerationCounts(db);
|
||||
|
||||
expect(counts.total).toBe(0);
|
||||
expect(counts.generated).toBe(0);
|
||||
expect(counts.stale).toBe(0);
|
||||
expect(counts.ungenerated).toBe(0);
|
||||
});
|
||||
|
||||
it("should report ungenerated items correctly", () => {
|
||||
createItem(db, sampleCreateParams());
|
||||
createItem(db, sampleCreateParams());
|
||||
|
||||
const counts = getWikiGenerationCounts(db);
|
||||
|
||||
expect(counts.total).toBe(2);
|
||||
expect(counts.ungenerated).toBe(2);
|
||||
expect(counts.generated).toBe(0);
|
||||
});
|
||||
|
||||
it("should report generated items correctly", () => {
|
||||
const item = createItem(db, sampleCreateParams());
|
||||
|
||||
db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
).run(new Date().toISOString(), "test-slug", item.id);
|
||||
|
||||
const counts = getWikiGenerationCounts(db);
|
||||
|
||||
expect(counts.total).toBe(1);
|
||||
expect(counts.generated).toBe(1);
|
||||
expect(counts.ungenerated).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getGeneratedSlugs", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
it("should return empty array when no items generated", () => {
|
||||
createItem(db, sampleCreateParams());
|
||||
const slugs = getGeneratedSlugs(db);
|
||||
|
||||
expect(slugs).toEqual([]);
|
||||
});
|
||||
|
||||
it("should return distinct slugs of generated items", () => {
|
||||
const item1 = createItem(db, sampleCreateParams());
|
||||
const item2 = createItem(db, sampleCreateParams());
|
||||
|
||||
const now = new Date().toISOString();
|
||||
db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
).run(now, "slug-a", item1.id);
|
||||
db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
).run(now, "slug-b", item2.id);
|
||||
|
||||
const slugs = getGeneratedSlugs(db);
|
||||
|
||||
expect(slugs.sort()).toEqual(["slug-a", "slug-b"]);
|
||||
});
|
||||
});
|
||||
});
|
||||
263
src/__tests__/items.test.ts
Normal file
263
src/__tests__/items.test.ts
Normal file
@@ -0,0 +1,263 @@
|
||||
/**
|
||||
* Items CRUD tests for HaBraid.
|
||||
*
|
||||
* Tests createItem, getItem, updateItem, deleteItem,
|
||||
* FTS5 search, content_hash computation, and upsertItem.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import {
|
||||
createItem,
|
||||
getItem,
|
||||
updateItem,
|
||||
deleteItem,
|
||||
searchItems,
|
||||
upsertItem,
|
||||
getItemCount,
|
||||
listItems,
|
||||
} from "../db/items.js";
|
||||
import { computeItemHash } from "../db/hashing.js";
|
||||
import { createTestDb, sampleCreateParams, sampleItem } from "./setup.js";
|
||||
|
||||
describe("Items CRUD", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
describe("createItem", () => {
|
||||
it("should create an item and return it", () => {
|
||||
const params = sampleCreateParams();
|
||||
const item = createItem(db, params);
|
||||
|
||||
expect(item.id).toBeTruthy();
|
||||
expect(item.id).toMatch(/^manual-/);
|
||||
expect(item.title).toBe("Test Item");
|
||||
expect(item.content).toBe("This is test content for a knowledge item.");
|
||||
expect(item.source).toBe("manual");
|
||||
expect(item.category).toBe("topics");
|
||||
expect(item.tags).toEqual(["test", "sample"]);
|
||||
expect(item.createdAt).toBeTruthy();
|
||||
expect(item.updatedAt).toBeTruthy();
|
||||
});
|
||||
|
||||
it("should compute content_hash on create", () => {
|
||||
const params = sampleCreateParams();
|
||||
const item = createItem(db, params);
|
||||
|
||||
// Read the raw row to check content_hash
|
||||
const row = db
|
||||
.prepare("SELECT content_hash FROM items WHERE id = ?")
|
||||
.get(item.id) as { content_hash: string };
|
||||
|
||||
expect(row.content_hash).toBeTruthy();
|
||||
expect(row.content_hash).toHaveLength(64); // SHA-256 hex digest
|
||||
});
|
||||
|
||||
it("should use 'manual' as default source", () => {
|
||||
const params = sampleCreateParams({ source: undefined });
|
||||
const item = createItem(db, params);
|
||||
|
||||
expect(item.source).toBe("manual");
|
||||
});
|
||||
|
||||
it("should default tags and metadata to empty", () => {
|
||||
const params = sampleCreateParams({ tags: undefined, metadata: undefined });
|
||||
const item = createItem(db, params);
|
||||
|
||||
expect(item.tags).toEqual([]);
|
||||
expect(item.metadata).toEqual({});
|
||||
});
|
||||
});
|
||||
|
||||
describe("getItem", () => {
|
||||
it("should retrieve a created item by ID", () => {
|
||||
const params = sampleCreateParams();
|
||||
const created = createItem(db, params);
|
||||
const retrieved = getItem(db, created.id);
|
||||
|
||||
expect(retrieved).toBeDefined();
|
||||
expect(retrieved!.id).toBe(created.id);
|
||||
expect(retrieved!.title).toBe(created.title);
|
||||
expect(retrieved!.content).toBe(created.content);
|
||||
});
|
||||
|
||||
it("should return undefined for non-existent item", () => {
|
||||
const result = getItem(db, "nonexistent-id");
|
||||
expect(result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("updateItem", () => {
|
||||
it("should update title and content", () => {
|
||||
const created = createItem(db, sampleCreateParams());
|
||||
const updated = updateItem(db, created.id, {
|
||||
title: "Updated Title",
|
||||
content: "Updated content",
|
||||
});
|
||||
|
||||
expect(updated).toBeDefined();
|
||||
expect(updated!.title).toBe("Updated Title");
|
||||
expect(updated!.content).toBe("Updated content");
|
||||
});
|
||||
|
||||
it("should update tags", () => {
|
||||
const created = createItem(db, sampleCreateParams());
|
||||
const updated = updateItem(db, created.id, {
|
||||
tags: ["new-tag-1", "new-tag-2"],
|
||||
});
|
||||
|
||||
expect(updated!.tags).toEqual(["new-tag-1", "new-tag-2"]);
|
||||
});
|
||||
|
||||
it("should recompute content_hash on update", () => {
|
||||
const created = createItem(db, sampleCreateParams());
|
||||
|
||||
const hashBefore = db
|
||||
.prepare("SELECT content_hash FROM items WHERE id = ?")
|
||||
.get(created.id) as { content_hash: string };
|
||||
|
||||
updateItem(db, created.id, { content: "Completely different content" });
|
||||
|
||||
const hashAfter = db
|
||||
.prepare("SELECT content_hash FROM items WHERE id = ?")
|
||||
.get(created.id) as { content_hash: string };
|
||||
|
||||
expect(hashAfter.content_hash).not.toBe(hashBefore.content_hash);
|
||||
});
|
||||
|
||||
it("should return undefined for non-existent item", () => {
|
||||
const result = updateItem(db, "nonexistent", { title: "X" });
|
||||
expect(result).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("deleteItem", () => {
|
||||
it("should delete an existing item", () => {
|
||||
const created = createItem(db, sampleCreateParams());
|
||||
const deleted = deleteItem(db, created.id);
|
||||
|
||||
expect(deleted).toBe(true);
|
||||
expect(getItem(db, created.id)).toBeUndefined();
|
||||
});
|
||||
|
||||
it("should return false for non-existent item", () => {
|
||||
const deleted = deleteItem(db, "nonexistent");
|
||||
expect(deleted).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("FTS5 search", () => {
|
||||
beforeEach(() => {
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Rust programming language",
|
||||
content: "Rust is a systems programming language focused on safety and performance.",
|
||||
tags: ["rust", "programming"],
|
||||
}));
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "TypeScript guide",
|
||||
content: "TypeScript adds type safety to JavaScript development.",
|
||||
tags: ["typescript", "programming"],
|
||||
}));
|
||||
createItem(db, sampleCreateParams({
|
||||
title: "Cooking recipes",
|
||||
content: "How to make pasta carbonara with eggs and cheese.",
|
||||
tags: ["cooking", "food"],
|
||||
}));
|
||||
});
|
||||
|
||||
it("should find items by title keyword", () => {
|
||||
const results = searchItems(db, { query: "programming" });
|
||||
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
const titles = results.map((r) => r.item.title);
|
||||
expect(titles.some((t) => t.includes("programming") || t.includes("Rust") || t.includes("TypeScript"))).toBe(true);
|
||||
});
|
||||
|
||||
it("should find items by content keyword", () => {
|
||||
const results = searchItems(db, { query: "pasta" });
|
||||
|
||||
expect(results.length).toBeGreaterThanOrEqual(1);
|
||||
expect(results[0].item.title).toContain("Cooking");
|
||||
});
|
||||
|
||||
it("should return empty results for non-matching query", () => {
|
||||
const results = searchItems(db, { query: "quantum_xyz_nonexistent" });
|
||||
expect(results.length).toBe(0);
|
||||
});
|
||||
|
||||
it("should include rank and snippet in results", () => {
|
||||
const results = searchItems(db, { query: "Rust" });
|
||||
|
||||
if (results.length > 0) {
|
||||
expect(results[0].rank).toBeDefined();
|
||||
expect(typeof results[0].rank).toBe("number");
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe("upsertItem", () => {
|
||||
it("should insert a new item", () => {
|
||||
const item = sampleItem({ id: "manual-upsert-new-001" });
|
||||
const result = upsertItem(db, item);
|
||||
|
||||
expect(result.id).toBe("manual-upsert-new-001");
|
||||
expect(result.title).toBe(item.title);
|
||||
});
|
||||
|
||||
it("should update an existing item on conflict", () => {
|
||||
const item = sampleItem({ id: "manual-upsert-conflict-001", title: "Original" });
|
||||
upsertItem(db, item);
|
||||
|
||||
const updated = sampleItem({
|
||||
id: "manual-upsert-conflict-001",
|
||||
title: "Updated Title",
|
||||
updatedAt: new Date().toISOString(),
|
||||
});
|
||||
const result = upsertItem(db, updated);
|
||||
|
||||
expect(result.title).toBe("Updated Title");
|
||||
expect(getItemCount(db)).toBe(1);
|
||||
});
|
||||
|
||||
it("should compute content_hash on upsert", () => {
|
||||
const item = sampleItem({ id: "manual-upsert-hash-001" });
|
||||
upsertItem(db, item);
|
||||
|
||||
const row = db
|
||||
.prepare("SELECT content_hash FROM items WHERE id = ?")
|
||||
.get(item.id) as { content_hash: string };
|
||||
|
||||
expect(row.content_hash).toBeTruthy();
|
||||
expect(row.content_hash).toHaveLength(64);
|
||||
});
|
||||
});
|
||||
|
||||
describe("listItems and getItemCount", () => {
|
||||
it("should list items ordered by created_at DESC", () => {
|
||||
createItem(db, sampleCreateParams({ title: "First" }));
|
||||
createItem(db, sampleCreateParams({ title: "Second" }));
|
||||
|
||||
const items = listItems(db);
|
||||
expect(items.length).toBe(2);
|
||||
// Both items exist; most recent should come first if timestamps differ
|
||||
const titles = items.map((i) => i.title);
|
||||
expect(titles).toContain("First");
|
||||
expect(titles).toContain("Second");
|
||||
});
|
||||
|
||||
it("should count items correctly", () => {
|
||||
expect(getItemCount(db)).toBe(0);
|
||||
createItem(db, sampleCreateParams());
|
||||
expect(getItemCount(db)).toBe(1);
|
||||
createItem(db, sampleCreateParams());
|
||||
expect(getItemCount(db)).toBe(2);
|
||||
});
|
||||
});
|
||||
});
|
||||
301
src/__tests__/kg.test.ts
Normal file
301
src/__tests__/kg.test.ts
Normal file
@@ -0,0 +1,301 @@
|
||||
/**
|
||||
* Knowledge Graph tests for HaBraid.
|
||||
*
|
||||
* Tests entity CRUD, relations, search, BFS traversal,
|
||||
* deletion cascading, and graph statistics.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
|
||||
import {
|
||||
addEntity,
|
||||
getEntity,
|
||||
searchEntities,
|
||||
deleteEntity,
|
||||
addRelation,
|
||||
getRelations,
|
||||
getEntityNeighbors,
|
||||
getGraphStats,
|
||||
} from "../db/kg.js";
|
||||
import { createTestDb } from "./setup.js";
|
||||
|
||||
describe("Knowledge Graph", () => {
|
||||
let db: Database.Database;
|
||||
|
||||
beforeEach(() => {
|
||||
db = createTestDb();
|
||||
});
|
||||
|
||||
afterEach(() => {
|
||||
db.close();
|
||||
});
|
||||
|
||||
describe("addEntity / getEntity", () => {
|
||||
it("should add an entity and retrieve it", () => {
|
||||
addEntity(db, {
|
||||
id: "rust-lang",
|
||||
name: "Rust",
|
||||
type: "tool",
|
||||
metadata: { version: "1.75" },
|
||||
});
|
||||
|
||||
const entity = getEntity(db, "rust-lang");
|
||||
|
||||
expect(entity).toBeDefined();
|
||||
expect(entity!.id).toBe("rust-lang");
|
||||
expect(entity!.name).toBe("Rust");
|
||||
expect(entity!.type).toBe("tool");
|
||||
expect(entity!.metadata).toEqual({ version: "1.75" });
|
||||
expect(entity!.outgoingRelations).toBe(0);
|
||||
expect(entity!.incomingRelations).toBe(0);
|
||||
});
|
||||
|
||||
it("should default type to 'concept'", () => {
|
||||
addEntity(db, { id: "test-entity", name: "Test" });
|
||||
const entity = getEntity(db, "test-entity");
|
||||
|
||||
expect(entity!.type).toBe("concept");
|
||||
});
|
||||
|
||||
it("should upsert on duplicate id", () => {
|
||||
addEntity(db, { id: "rust-lang", name: "Rust", type: "tool" });
|
||||
addEntity(db, { id: "rust-lang", name: "Rust Language", type: "concept" });
|
||||
|
||||
const entity = getEntity(db, "rust-lang");
|
||||
expect(entity!.name).toBe("Rust Language");
|
||||
expect(entity!.type).toBe("concept");
|
||||
});
|
||||
|
||||
it("should return undefined for non-existent entity", () => {
|
||||
const entity = getEntity(db, "nonexistent");
|
||||
expect(entity).toBeUndefined();
|
||||
});
|
||||
});
|
||||
|
||||
describe("addRelation / getRelations", () => {
|
||||
beforeEach(() => {
|
||||
addEntity(db, { id: "rust", name: "Rust", type: "tool" });
|
||||
addEntity(db, { id: "memory-safety", name: "Memory Safety", type: "concept" });
|
||||
addEntity(db, { id: "performance", name: "Performance", type: "concept" });
|
||||
});
|
||||
|
||||
it("should add a relation and retrieve it", () => {
|
||||
const relId = addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
confidence: 0.95,
|
||||
evidence: "Rust ownership model",
|
||||
});
|
||||
|
||||
expect(relId).toBeGreaterThan(0);
|
||||
|
||||
const relations = getRelations(db, "rust", "outgoing");
|
||||
expect(relations.length).toBe(1);
|
||||
expect(relations[0].subjectId).toBe("rust");
|
||||
expect(relations[0].predicate).toBe("provides");
|
||||
expect(relations[0].objectId).toBe("memory-safety");
|
||||
expect(relations[0].confidence).toBe(0.95);
|
||||
});
|
||||
|
||||
it("should retrieve incoming relations", () => {
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
});
|
||||
|
||||
const incoming = getRelations(db, "memory-safety", "incoming");
|
||||
expect(incoming.length).toBe(1);
|
||||
expect(incoming[0].subjectId).toBe("rust");
|
||||
});
|
||||
|
||||
it("should retrieve both incoming and outgoing with 'both'", () => {
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
});
|
||||
addRelation(db, {
|
||||
subjectId: "performance",
|
||||
predicate: "requires",
|
||||
objectId: "rust",
|
||||
});
|
||||
|
||||
const both = getRelations(db, "rust", "both");
|
||||
expect(both.length).toBe(2);
|
||||
});
|
||||
|
||||
it("should update outgoing/incoming counts in getEntity", () => {
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
});
|
||||
|
||||
const rust = getEntity(db, "rust");
|
||||
expect(rust!.outgoingRelations).toBe(1);
|
||||
expect(rust!.incomingRelations).toBe(0);
|
||||
|
||||
const safety = getEntity(db, "memory-safety");
|
||||
expect(safety!.outgoingRelations).toBe(0);
|
||||
expect(safety!.incomingRelations).toBe(1);
|
||||
});
|
||||
|
||||
it("should upsert relation on duplicate (subject, predicate, object, source)", () => {
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
confidence: 0.8,
|
||||
});
|
||||
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "memory-safety",
|
||||
confidence: 0.95,
|
||||
});
|
||||
|
||||
const relations = getRelations(db, "rust", "outgoing");
|
||||
// Should be upserted, not duplicated
|
||||
expect(relations.length).toBe(1);
|
||||
expect(relations[0].confidence).toBe(0.95);
|
||||
});
|
||||
});
|
||||
|
||||
describe("searchEntities", () => {
|
||||
beforeEach(() => {
|
||||
addEntity(db, { id: "rust-lang", name: "Rust Programming", type: "tool" });
|
||||
addEntity(db, { id: "rust-crate", name: "Serde Crate", type: "tool" });
|
||||
addEntity(db, { id: "python", name: "Python", type: "tool" });
|
||||
});
|
||||
|
||||
it("should find entities by name", () => {
|
||||
const results = searchEntities(db, "Rust");
|
||||
expect(results.length).toBe(2);
|
||||
});
|
||||
|
||||
it("should find entities by id", () => {
|
||||
const results = searchEntities(db, "python");
|
||||
expect(results.length).toBe(1);
|
||||
expect(results[0].id).toBe("python");
|
||||
});
|
||||
|
||||
it("should find entities by type", () => {
|
||||
const results = searchEntities(db, "tool");
|
||||
expect(results.length).toBe(3);
|
||||
});
|
||||
|
||||
it("should return empty for non-matching query", () => {
|
||||
const results = searchEntities(db, "nonexistent");
|
||||
expect(results.length).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe("deleteEntity", () => {
|
||||
it("should delete an entity", () => {
|
||||
addEntity(db, { id: "rust", name: "Rust", type: "tool" });
|
||||
const deleted = deleteEntity(db, "rust");
|
||||
|
||||
expect(deleted).toBe(true);
|
||||
expect(getEntity(db, "rust")).toBeUndefined();
|
||||
});
|
||||
|
||||
it("should cascade delete relations (manually due to FK constraint)", () => {
|
||||
addEntity(db, { id: "rust", name: "Rust", type: "tool" });
|
||||
addEntity(db, { id: "safety", name: "Safety", type: "concept" });
|
||||
addRelation(db, {
|
||||
subjectId: "rust",
|
||||
predicate: "provides",
|
||||
objectId: "safety",
|
||||
});
|
||||
|
||||
// Delete the subject entity's relations first, then the entity
|
||||
// (The schema uses REFERENCES but without ON DELETE CASCADE)
|
||||
db.prepare("DELETE FROM kg_relations WHERE subject_id = ?").run("rust");
|
||||
deleteEntity(db, "rust");
|
||||
|
||||
// The relation referencing rust should be gone
|
||||
const safetyRelations = getRelations(db, "safety", "incoming");
|
||||
expect(safetyRelations.length).toBe(0);
|
||||
});
|
||||
|
||||
it("should return false for non-existent entity", () => {
|
||||
const deleted = deleteEntity(db, "nonexistent");
|
||||
expect(deleted).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getEntityNeighbors (BFS)", () => {
|
||||
beforeEach(() => {
|
||||
// Create a small graph: A → B → C → D, A → E
|
||||
addEntity(db, { id: "a", name: "A", type: "concept" });
|
||||
addEntity(db, { id: "b", name: "B", type: "concept" });
|
||||
addEntity(db, { id: "c", name: "C", type: "concept" });
|
||||
addEntity(db, { id: "d", name: "D", type: "concept" });
|
||||
addEntity(db, { id: "e", name: "E", type: "concept" });
|
||||
|
||||
addRelation(db, { subjectId: "a", predicate: "relates", objectId: "b" });
|
||||
addRelation(db, { subjectId: "b", predicate: "relates", objectId: "c" });
|
||||
addRelation(db, { subjectId: "c", predicate: "relates", objectId: "d" });
|
||||
addRelation(db, { subjectId: "a", predicate: "relates", objectId: "e" });
|
||||
});
|
||||
|
||||
it("should find immediate neighbors at depth 1", () => {
|
||||
const result = getEntityNeighbors(db, "a", 1);
|
||||
|
||||
// At depth 1, BFS processes: d=0 (node a) and d=1 (nodes b, e)
|
||||
// Entities visited: a, b, e (and c is added to next level but not processed)
|
||||
// Note: c is discovered as a neighbor of b but only processed at d=2
|
||||
// However, c gets added to visitedEntities when found as a neighbor at d=1
|
||||
// actually no - c is added to nextLevel only, not visitedEntities
|
||||
expect(result.entities.length).toBe(3); // a, b, e
|
||||
|
||||
const entityIds = result.entities.map((e) => e.id).sort();
|
||||
expect(entityIds).toContain("a");
|
||||
expect(entityIds).toContain("b");
|
||||
expect(entityIds).toContain("e");
|
||||
});
|
||||
|
||||
it("should traverse deeper at depth 2", () => {
|
||||
const result = getEntityNeighbors(db, "a", 2);
|
||||
|
||||
expect(result.entities.length).toBe(4); // a, b, e, c
|
||||
const entityIds = result.entities.map((e) => e.id).sort();
|
||||
expect(entityIds).toContain("c");
|
||||
});
|
||||
|
||||
it("should traverse the full chain at depth 3+", () => {
|
||||
const result = getEntityNeighbors(db, "a", 4);
|
||||
|
||||
const entityIds = result.entities.map((e) => e.id).sort();
|
||||
expect(entityIds).toEqual(["a", "b", "c", "d", "e"]);
|
||||
});
|
||||
});
|
||||
|
||||
describe("getGraphStats", () => {
|
||||
it("should return zero stats for empty graph", () => {
|
||||
const stats = getGraphStats(db);
|
||||
|
||||
expect(stats.totalEntities).toBe(0);
|
||||
expect(stats.totalRelations).toBe(0);
|
||||
expect(Object.keys(stats.entityTypes).length).toBe(0);
|
||||
});
|
||||
|
||||
it("should count entities and relations correctly", () => {
|
||||
addEntity(db, { id: "a", name: "A", type: "tool" });
|
||||
addEntity(db, { id: "b", name: "B", type: "tool" });
|
||||
addEntity(db, { id: "c", name: "C", type: "concept" });
|
||||
addRelation(db, { subjectId: "a", predicate: "uses", objectId: "b" });
|
||||
addRelation(db, { subjectId: "a", predicate: "requires", objectId: "c" });
|
||||
|
||||
const stats = getGraphStats(db);
|
||||
|
||||
expect(stats.totalEntities).toBe(3);
|
||||
expect(stats.totalRelations).toBe(2);
|
||||
expect(stats.entityTypes).toEqual({ tool: 2, concept: 1 });
|
||||
expect(stats.topPredicates.length).toBeGreaterThan(0);
|
||||
});
|
||||
});
|
||||
});
|
||||
248
src/__tests__/setup.ts
Normal file
248
src/__tests__/setup.ts
Normal file
@@ -0,0 +1,248 @@
|
||||
/**
|
||||
* Test utilities for HaBraid tests.
|
||||
*
|
||||
* Provides in-memory SQLite database helpers, sample data generators,
|
||||
* and common test setup/teardown functions.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
import type { Database as DatabaseType } from "better-sqlite3";
|
||||
|
||||
import { openDatabase } from "../db/database.js";
|
||||
import type { Item, CreateItemParams } from "../types.js";
|
||||
|
||||
/**
|
||||
* Creates a fresh in-memory SQLite database with full schema migrations applied.
|
||||
*
|
||||
* @returns Connected better-sqlite3 Database instance (in-memory).
|
||||
*/
|
||||
export function createTestDb(): DatabaseType {
|
||||
// openDatabase calls mkdirSync which fails for ":memory:" —
|
||||
// so we replicate the essential setup inline.
|
||||
const db = new Database(":memory:");
|
||||
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("foreign_keys = ON");
|
||||
|
||||
// Re-use openDatabase but with a workaround: it calls mkdirSync on dirname(":memory:")
|
||||
// which is "." and should exist. Let's just call it properly.
|
||||
db.close();
|
||||
|
||||
// Use the real openDatabase with a temp-like path won't work for :memory:
|
||||
// because it tries to create dirs. So we manually run schema + migrations.
|
||||
const freshDb = new Database(":memory:");
|
||||
freshDb.pragma("journal_mode = WAL");
|
||||
freshDb.pragma("foreign_keys = ON");
|
||||
|
||||
// Import schema SQL by calling openDatabase logic indirectly —
|
||||
// actually, let's just use the openDatabase function but patch mkdirSync.
|
||||
// Since we can't easily patch that, let's just create the schema manually here.
|
||||
// We'll import the schema from database.ts by executing the same statements.
|
||||
|
||||
freshDb.close();
|
||||
|
||||
// Simplest approach: use a temp file, open it, then return it.
|
||||
// But we want :memory: for speed and no file I/O.
|
||||
// Let's just re-run the schema SQL manually here by importing the source.
|
||||
|
||||
// Actually the cleanest way: use openDatabase with a real temp path,
|
||||
// but that creates file I/O. Instead, let's inline the schema setup.
|
||||
return createInMemoryDb();
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an in-memory database with the full HaBraid schema.
|
||||
* This avoids the mkdirSync issue in openDatabase.
|
||||
*
|
||||
* @returns Connected in-memory database.
|
||||
*/
|
||||
function createInMemoryDb(): DatabaseType {
|
||||
const db = new Database(":memory:");
|
||||
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("foreign_keys = ON");
|
||||
|
||||
// Schema v1 — initial tables
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS items (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'manual',
|
||||
category TEXT,
|
||||
tags TEXT DEFAULT '[]',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
metadata TEXT DEFAULT '{}',
|
||||
wiki_generated_at TEXT,
|
||||
wiki_slug TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_version (
|
||||
version INTEGER PRIMARY KEY
|
||||
);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS items_fts USING fts5(
|
||||
title, content, tags,
|
||||
content=items,
|
||||
content_rowid=rowid
|
||||
);
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_ai AFTER INSERT ON items BEGIN
|
||||
INSERT INTO items_fts(rowid, title, content, tags)
|
||||
VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_ad AFTER DELETE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags)
|
||||
VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_au AFTER UPDATE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags)
|
||||
VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
INSERT INTO items_fts(rowid, title, content, tags)
|
||||
VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
|
||||
INSERT OR IGNORE INTO schema_version (version) VALUES (1);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS item_vectors (
|
||||
item_id TEXT PRIMARY KEY REFERENCES items(id),
|
||||
vector BLOB NOT NULL,
|
||||
model TEXT NOT NULL DEFAULT 'paraphrase-multilingual-MiniLM-L12-v2',
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_item_vectors_model ON item_vectors(model);
|
||||
`);
|
||||
|
||||
// Migration v2 — wiki fields already in v1 schema above
|
||||
db.exec(`UPDATE schema_version SET version = 2`);
|
||||
|
||||
// Migration v3 — KG tables
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS kg_entities (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL DEFAULT 'concept',
|
||||
wiki_slug TEXT,
|
||||
first_seen TEXT DEFAULT (datetime('now')),
|
||||
last_seen TEXT DEFAULT (datetime('now')),
|
||||
metadata TEXT DEFAULT '{}')
|
||||
;
|
||||
CREATE TABLE IF NOT EXISTS kg_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subject_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
predicate TEXT NOT NULL,
|
||||
object_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
confidence REAL DEFAULT 1.0,
|
||||
source TEXT NOT NULL DEFAULT 'extracted',
|
||||
evidence TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
UNIQUE(subject_id, predicate, object_id, source))
|
||||
;
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_relations_subject ON kg_relations(subject_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_relations_object ON kg_relations(object_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_entities_type ON kg_entities(type);
|
||||
`);
|
||||
db.exec(`UPDATE schema_version SET version = 3`);
|
||||
|
||||
// Migration v4 — content_hash
|
||||
db.exec(`ALTER TABLE items ADD COLUMN content_hash TEXT`);
|
||||
db.exec(`UPDATE schema_version SET version = 4`);
|
||||
|
||||
// Migration v5 — contradictions
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS contradictions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
item_a_id TEXT NOT NULL,
|
||||
item_b_id TEXT NOT NULL,
|
||||
field TEXT NOT NULL,
|
||||
value_a TEXT NOT NULL,
|
||||
value_b TEXT NOT NULL,
|
||||
severity TEXT DEFAULT 'medium',
|
||||
status TEXT DEFAULT 'open',
|
||||
resolution TEXT,
|
||||
detected_at TEXT DEFAULT (datetime('now')),
|
||||
resolved_at TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_contradictions_status ON contradictions(status);
|
||||
CREATE INDEX IF NOT EXISTS idx_contradictions_item_a ON contradictions(item_a_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_contradictions_item_b ON contradictions(item_b_id);
|
||||
`);
|
||||
db.exec(`UPDATE schema_version SET version = 5`);
|
||||
|
||||
// Migration v6 — communities
|
||||
db.exec(`
|
||||
CREATE TABLE IF NOT EXISTS kg_communities (
|
||||
id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL,
|
||||
dominant_type TEXT,
|
||||
entity_count INTEGER DEFAULT 0,
|
||||
detected_at TEXT DEFAULT (datetime('now'))
|
||||
);
|
||||
CREATE TABLE IF NOT EXISTS kg_community_members (
|
||||
community_id TEXT NOT NULL REFERENCES kg_communities(id),
|
||||
entity_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
PRIMARY KEY (community_id, entity_id)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_kg_community_members_entity ON kg_community_members(entity_id);
|
||||
`);
|
||||
db.exec(`UPDATE schema_version SET version = 6`);
|
||||
|
||||
// Migration v7 — contradiction slugs
|
||||
db.exec(`
|
||||
ALTER TABLE contradictions ADD COLUMN item_a_slug TEXT;
|
||||
ALTER TABLE contradictions ADD COLUMN item_b_slug TEXT;
|
||||
CREATE INDEX IF NOT EXISTS idx_contradictions_field ON contradictions(field);
|
||||
`);
|
||||
db.exec(`UPDATE schema_version SET version = 7`);
|
||||
|
||||
return db;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a sample CreateItemParams object for testing.
|
||||
*
|
||||
* @param overrides Partial overrides for the default params.
|
||||
* @returns CreateItemParams.
|
||||
*/
|
||||
export function sampleCreateParams(
|
||||
overrides: Partial<CreateItemParams> = {},
|
||||
): CreateItemParams {
|
||||
return {
|
||||
title: "Test Item",
|
||||
content: "This is test content for a knowledge item.",
|
||||
source: "manual",
|
||||
category: "topics",
|
||||
tags: ["test", "sample"],
|
||||
metadata: {},
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a sample Item object for testing.
|
||||
*
|
||||
* @param overrides Partial overrides for the default item.
|
||||
* @returns Item.
|
||||
*/
|
||||
export function sampleItem(
|
||||
overrides: Partial<Item> = {},
|
||||
): Item {
|
||||
return {
|
||||
id: "manual-test1234-abcd",
|
||||
title: "Sample Item",
|
||||
content: "Sample content for testing.",
|
||||
source: "manual",
|
||||
category: "topics",
|
||||
tags: ["test"],
|
||||
createdAt: new Date("2025-01-01T00:00:00Z").toISOString(),
|
||||
updatedAt: new Date("2025-01-01T00:00:00Z").toISOString(),
|
||||
metadata: {},
|
||||
wikiGeneratedAt: null,
|
||||
wikiSlug: null,
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
317
src/cli/commands.ts
Normal file
317
src/cli/commands.ts
Normal file
@@ -0,0 +1,317 @@
|
||||
import path from "node:path";
|
||||
import { Command } from "commander";
|
||||
import chalk from "chalk";
|
||||
import ora from "ora";
|
||||
|
||||
import { buildInitConfig, loadConfig, resolveConfigPath, saveConfig } from "../config.js";
|
||||
import { LintError, WikiEngineError } from "../errors.js";
|
||||
import { exportGraphToFile, exportCommunityGraph, exportMermaidCommunities } from "../graph/export.js";
|
||||
import { ingestMemPalace } from "../mempalace/ingest.js";
|
||||
import { runSyncPipeline } from "../sync/pipeline.js";
|
||||
import type { VaultStatus, WikiEngineConfig } from "../types.js";
|
||||
import { readSyncState, listFilesRecursive, toDateString } from "../utils.js";
|
||||
import { initializeVault } from "../vault/init.js";
|
||||
import { lintVault } from "../vault/lint.js";
|
||||
import { updateWiki, getWikiStatus } from "../wiki/generator.js";
|
||||
import { ensureGitRepo } from "../sync/git.js";
|
||||
import { openDatabase } from "../db/database.js";
|
||||
import { generateDailyLog, generateDailyLogRange, writeDailyLog } from "../vault/daily-log.js";
|
||||
|
||||
/**
|
||||
* Registers all CLI commands.
|
||||
*
|
||||
* @param program Commander program instance.
|
||||
*/
|
||||
export function registerCommands(program: Command): void {
|
||||
program
|
||||
.command("init")
|
||||
.description("Initialize a new wiki vault")
|
||||
.option("--vault <path>", "Vault path")
|
||||
.option("--git <url>", "Git remote URL")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { vault?: string; git?: string; config?: string }) => {
|
||||
await withSpinner("Initializing vault", async () => {
|
||||
const { config, configPath } = await loadConfig(options.config);
|
||||
const nextConfig = buildInitConfig(config, {
|
||||
vaultPath: options.vault,
|
||||
gitRemote: options.git,
|
||||
});
|
||||
await initializeVault(nextConfig);
|
||||
await ensureGitRepo(nextConfig.vault.path, nextConfig.vault.branch, nextConfig.vault.git_remote);
|
||||
await saveConfig(resolveConfigPath(options.config ?? configPath), nextConfig);
|
||||
return `Vault initialized at ${nextConfig.vault.path}`;
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("ingest")
|
||||
.description("Ingest MemPalace drawers into raw/")
|
||||
.option("--full", "Run a full ingest")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { full?: boolean; config?: string }) => {
|
||||
await withSpinner("Ingesting MemPalace drawers", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const result = await ingestMemPalace(config, { full: options.full });
|
||||
return `Processed ${result.drawersProcessed} drawers, wrote ${result.drawersWritten}`;
|
||||
});
|
||||
});
|
||||
|
||||
const wikiCommand = program.command("wiki").description("Wiki page operations");
|
||||
|
||||
wikiCommand
|
||||
.command("update")
|
||||
.description("Generate or update wiki pages")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { config?: string }) => {
|
||||
await withSpinner("Updating wiki pages", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const result = await updateWiki(config);
|
||||
return `Wrote ${result.filesWritten} wiki pages`;
|
||||
});
|
||||
});
|
||||
|
||||
wikiCommand
|
||||
.command("status")
|
||||
.description("Show wiki page status")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { config?: string }) => {
|
||||
await withSpinner("Reading wiki status", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const status = await getWikiStatus(config.vault.path);
|
||||
return formatWikiStatus(status);
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("sync")
|
||||
.description("Run the full sync pipeline")
|
||||
.option("--no-wiki", "Skip wiki generation")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { noWiki?: boolean; config?: string }) => {
|
||||
await withSpinner("Running sync pipeline", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
await runSyncPipeline(config, { noWiki: options.noWiki });
|
||||
return "Sync completed";
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("status")
|
||||
.description("Show vault statistics")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { config?: string }) => {
|
||||
await withSpinner("Collecting vault status", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const status = await getVaultStatus(config);
|
||||
return [
|
||||
`raw: ${status.rawCount}`,
|
||||
`wiki: ${status.wikiCount}`,
|
||||
`last_ingest: ${status.lastIngest ?? "-"}`,
|
||||
`last_wiki_update: ${status.lastWikiUpdate ?? "-"}`,
|
||||
`last_git_push: ${status.lastGitPush ?? "-"}`,
|
||||
].join("\n");
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("lint")
|
||||
.description("Validate vault integrity")
|
||||
.option("--contradictions", "Also scan for contradictions in the knowledge base")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { contradictions?: boolean; config?: string }) => {
|
||||
await withSpinner("Linting vault", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const result = await lintVault(config.vault.path);
|
||||
const messages: string[] = [];
|
||||
|
||||
if (!result.ok) {
|
||||
messages.push(...result.issues);
|
||||
}
|
||||
|
||||
if (options.contradictions) {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const { scanAllContradictions, getContradictionStats } = await import("../wiki/contradiction.js");
|
||||
const newCount = scanAllContradictions(db);
|
||||
const stats = getContradictionStats(db);
|
||||
if (stats.open > 0) {
|
||||
messages.push(`Contradictions: ${stats.open} open (${stats.total} total, ${newCount} new)`);
|
||||
messages.push(` By severity: high=${stats.bySeverity["high"] ?? 0}, medium=${stats.bySeverity["medium"] ?? 0}, low=${stats.bySeverity["low"] ?? 0}`);
|
||||
} else {
|
||||
messages.push(`Contradictions: clean (0 open, ${stats.total} total)`);
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
}
|
||||
|
||||
if (messages.some((m) => m.includes("Contradictions:") && m.includes("open"))) {
|
||||
throw new LintError(`Vault lint found issues:\n${messages.join("\n")}`);
|
||||
}
|
||||
|
||||
if (!result.ok) {
|
||||
throw new LintError(`Vault lint failed:\n${messages.join("\n")}`);
|
||||
}
|
||||
|
||||
return messages.length > 0 ? messages.join("\n") : "Vault lint passed";
|
||||
});
|
||||
});
|
||||
|
||||
program
|
||||
.command("log [date]")
|
||||
.description("Generate or show daily work log (YYYY-MM-DD). Defaults to today.")
|
||||
.option("--start <date>", "Range start date (YYYY-MM-DD)")
|
||||
.option("--end <date>", "Range end date (YYYY-MM-DD)")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (date: string | undefined, options: { start?: string; end?: string; config?: string }) => {
|
||||
await withSpinner("Generating daily log", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
let markdown: string;
|
||||
let filePath: string;
|
||||
|
||||
if (options.start && options.end) {
|
||||
markdown = generateDailyLogRange(db, options.start, options.end);
|
||||
// For range, write with start-end filename
|
||||
filePath = await writeDailyLog(db, config.vault.path, `${options.start}~${options.end}`.replace(/~/g, ""));
|
||||
} else {
|
||||
const targetDate = date ?? toDateString(new Date());
|
||||
markdown = generateDailyLog(db, targetDate);
|
||||
filePath = await writeDailyLog(db, config.vault.path, targetDate);
|
||||
}
|
||||
|
||||
return `Daily log written to ${filePath}`;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
const graphCommand = program.command("graph").description("Knowledge graph operations");
|
||||
|
||||
graphCommand
|
||||
.command("export")
|
||||
.description("Export KG facts into graph/graph.json")
|
||||
.option("--config <path>", "Config path")
|
||||
.action(async (options: { config?: string }) => {
|
||||
await withSpinner("Exporting graph", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const outputPath = await exportGraphToFile(config);
|
||||
return `Exported graph to ${outputPath}`;
|
||||
});
|
||||
});
|
||||
|
||||
graphCommand
|
||||
.command("communities")
|
||||
.description("Detect and export KG communities using Label Propagation Algorithm")
|
||||
.option("--config <path>", "Config path")
|
||||
.option("--mermaid", "Output as Mermaid diagram instead of JSON")
|
||||
.action(async (options: { config?: string; mermaid?: boolean }) => {
|
||||
await withSpinner("Detecting communities", async () => {
|
||||
const { config } = await loadConfig(options.config);
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
if (options.mermaid) {
|
||||
const mermaid = exportMermaidCommunities(db);
|
||||
process.stdout.write(`${mermaid}\n`);
|
||||
return "Mermaid diagram output above";
|
||||
}
|
||||
|
||||
const communityGraph = exportCommunityGraph(db);
|
||||
const summary = [
|
||||
`Communities: ${communityGraph.communities.length}`,
|
||||
`Inter-community edges: ${communityGraph.edges.length}`,
|
||||
`Total entities clustered: ${communityGraph.communities.reduce((sum, c) => sum + c.size, 0)}`,
|
||||
];
|
||||
for (const c of communityGraph.communities.slice(0, 10)) {
|
||||
summary.push(` ${c.id}: ${c.label} (${c.size} entities, type: ${c.dominantType})`);
|
||||
}
|
||||
if (communityGraph.communities.length > 10) {
|
||||
summary.push(` ... and ${communityGraph.communities.length - 10} more`);
|
||||
}
|
||||
return summary.join("\n");
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Wraps a CLI action with a spinner and project-aware error reporting.
|
||||
*
|
||||
* @param text Spinner label.
|
||||
* @param fn Async action.
|
||||
*/
|
||||
async function withSpinner(text: string, fn: () => Promise<string>): Promise<void> {
|
||||
const spinner = ora(text).start();
|
||||
|
||||
try {
|
||||
const message = await fn();
|
||||
spinner.succeed(message);
|
||||
} catch (error) {
|
||||
const formatted = formatError(error);
|
||||
spinner.fail(formatted);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads high-level vault stats.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @returns Status object.
|
||||
*/
|
||||
async function getVaultStatus(config: WikiEngineConfig): Promise<VaultStatus> {
|
||||
try {
|
||||
const state = await readSyncState(config.vault.path);
|
||||
const [rawFiles, wikiFiles] = await Promise.all([
|
||||
listFilesRecursive(path.join(config.vault.path, "raw"), ".md"),
|
||||
listFilesRecursive(path.join(config.vault.path, "wiki"), ".md"),
|
||||
]);
|
||||
|
||||
return {
|
||||
rawCount: rawFiles.length,
|
||||
wikiCount: wikiFiles.length,
|
||||
dbItemCount: 0,
|
||||
lastIngest: state.last_ingest,
|
||||
lastWikiUpdate: state.last_wiki_update,
|
||||
lastGitPush: state.last_git_push,
|
||||
};
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Formats a wiki status summary for terminal output.
|
||||
*
|
||||
* @param status Status object.
|
||||
* @returns Summary string.
|
||||
*/
|
||||
function formatWikiStatus(status: { pageCount: number; categories: Record<string, number> }): string {
|
||||
const lines = [`pages: ${status.pageCount}`];
|
||||
for (const [category, count] of Object.entries(status.categories).sort(([left], [right]) => left.localeCompare(right))) {
|
||||
lines.push(`${category}: ${count}`);
|
||||
}
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts known project errors into concise terminal text.
|
||||
*
|
||||
* @param error Unknown thrown value.
|
||||
* @returns Readable message.
|
||||
*/
|
||||
function formatError(error: unknown): string {
|
||||
if (error instanceof WikiEngineError) {
|
||||
return `${chalk.red(error.code)} ${error.message}`;
|
||||
}
|
||||
|
||||
if (error instanceof Error) {
|
||||
return chalk.red(error.message);
|
||||
}
|
||||
|
||||
return chalk.red(String(error));
|
||||
}
|
||||
263
src/config.ts
Normal file
263
src/config.ts
Normal file
@@ -0,0 +1,263 @@
|
||||
import path from "node:path";
|
||||
import { promises as fs } from "node:fs";
|
||||
|
||||
import { ConfigError } from "./errors.js";
|
||||
import type { StandaloneLlmConfig, WikiEngineConfig } from "./types.js";
|
||||
import { DEFAULT_CONFIG, expandHomeDir, resolvePath, writeTextFile } from "./utils.js";
|
||||
|
||||
/**
|
||||
* Legacy llm config shape kept for backwards compatibility.
|
||||
*/
|
||||
interface LegacyLlmConfig {
|
||||
provider?: string;
|
||||
model?: string;
|
||||
api_url?: string;
|
||||
api_key_env?: string;
|
||||
max_tokens?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Partial config shape used when loading JSON from disk.
|
||||
*/
|
||||
interface PartialWikiEngineConfig {
|
||||
vault?: Partial<WikiEngineConfig["vault"]>;
|
||||
db?: Partial<WikiEngineConfig["db"]>;
|
||||
mempalace?: Partial<WikiEngineConfig["mempalace"]>;
|
||||
llm?: Partial<WikiEngineConfig["llm"]> & LegacyLlmConfig;
|
||||
sync?: Partial<WikiEngineConfig["sync"]>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the default config file location.
|
||||
*
|
||||
* @returns Absolute config path.
|
||||
*/
|
||||
export function getDefaultConfigPath(): string {
|
||||
return resolvePath("~/.habraid/data/config.json");
|
||||
}
|
||||
|
||||
export function getLegacyConfigPath(): string {
|
||||
return resolvePath("~/.habraid/config.json");
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves the effective config path according to the documented precedence.
|
||||
*
|
||||
* @param explicitPath Optional CLI-provided path.
|
||||
* @returns Absolute config path.
|
||||
*/
|
||||
export function resolveConfigPath(explicitPath?: string): string {
|
||||
if (explicitPath) {
|
||||
return resolvePath(explicitPath);
|
||||
}
|
||||
|
||||
const envPath = process.env.WIKI_ENGINE_CONFIG ?? process.env.HABRAID_CONFIG;
|
||||
if (envPath) {
|
||||
return resolvePath(envPath);
|
||||
}
|
||||
|
||||
const defaultPath = getDefaultConfigPath();
|
||||
return defaultPath;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a standalone/fallback config from legacy flat llm fields.
|
||||
*/
|
||||
function buildStandaloneLlmConfig(source: LegacyLlmConfig, fallback: StandaloneLlmConfig): StandaloneLlmConfig {
|
||||
return {
|
||||
provider: (source.provider ?? fallback.provider) as StandaloneLlmConfig["provider"],
|
||||
model: source.model ?? fallback.model,
|
||||
api_url: source.api_url ?? fallback.api_url,
|
||||
api_key_env: source.api_key_env ?? fallback.api_key_env,
|
||||
max_tokens: source.max_tokens ?? fallback.max_tokens,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Deep-merges two configuration objects with simple nested objects.
|
||||
*
|
||||
* @param base Base config object.
|
||||
* @param override Override object.
|
||||
* @returns Merged config.
|
||||
*/
|
||||
function mergeConfig(base: WikiEngineConfig, override: PartialWikiEngineConfig): WikiEngineConfig {
|
||||
const baseStandalone: StandaloneLlmConfig = {
|
||||
provider: (base.llm.fallback?.provider ?? base.llm.provider) as StandaloneLlmConfig["provider"],
|
||||
model: base.llm.fallback?.model ?? base.llm.model,
|
||||
api_url: base.llm.fallback?.api_url ?? base.llm.api_url,
|
||||
api_key_env: base.llm.fallback?.api_key_env ?? base.llm.api_key_env,
|
||||
max_tokens: base.llm.fallback?.max_tokens ?? base.llm.max_tokens,
|
||||
};
|
||||
const fallback = buildStandaloneLlmConfig(override.llm ?? {}, baseStandalone);
|
||||
const mode = override.llm?.mode ?? base.llm.mode;
|
||||
|
||||
return {
|
||||
vault: {
|
||||
...base.vault,
|
||||
...override.vault,
|
||||
},
|
||||
db: {
|
||||
...base.db,
|
||||
...override.db,
|
||||
},
|
||||
mempalace: {
|
||||
...base.mempalace,
|
||||
...override.mempalace,
|
||||
},
|
||||
llm: {
|
||||
...base.llm,
|
||||
...override.llm,
|
||||
mode,
|
||||
preferences: {
|
||||
...base.llm.preferences,
|
||||
...override.llm?.preferences,
|
||||
},
|
||||
fallback: override.llm?.fallback
|
||||
? {
|
||||
...fallback,
|
||||
...override.llm.fallback,
|
||||
}
|
||||
: (mode === "standalone" ? fallback : base.llm.fallback),
|
||||
// Preserve new-style backend config blocks
|
||||
host: override.llm?.host ?? base.llm.host,
|
||||
openai: override.llm?.openai ?? base.llm.openai,
|
||||
ollama: override.llm?.ollama ?? base.llm.ollama,
|
||||
zai: override.llm?.zai ?? base.llm.zai,
|
||||
provider: fallback.provider,
|
||||
model: fallback.model,
|
||||
api_url: fallback.api_url,
|
||||
api_key_env: fallback.api_key_env,
|
||||
max_tokens: fallback.max_tokens,
|
||||
},
|
||||
sync: {
|
||||
...base.sync,
|
||||
...override.sync,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes path-bearing config values.
|
||||
*
|
||||
* @param config Config to normalize.
|
||||
* @returns Normalized config.
|
||||
*/
|
||||
function normalizeConfig(config: WikiEngineConfig): WikiEngineConfig {
|
||||
return {
|
||||
...config,
|
||||
vault: {
|
||||
...config.vault,
|
||||
path: resolvePath(config.vault.path),
|
||||
git_remote: config.vault.git_remote,
|
||||
},
|
||||
db: {
|
||||
path: resolvePath(config.db.path),
|
||||
},
|
||||
mempalace: {
|
||||
...config.mempalace,
|
||||
path: config.mempalace.path ? resolvePath(config.mempalace.path) : "",
|
||||
},
|
||||
llm: {
|
||||
...config.llm,
|
||||
fallback: config.llm.fallback
|
||||
? {
|
||||
...config.llm.fallback,
|
||||
api_url: config.llm.fallback.api_url ? resolvePathIfFileLike(config.llm.fallback.api_url) : config.llm.fallback.api_url,
|
||||
}
|
||||
: undefined,
|
||||
api_url: resolvePathIfFileLike(config.llm.api_url),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Keeps HTTP URLs untouched while resolving file-like paths.
|
||||
*/
|
||||
function resolvePathIfFileLike(input: string): string {
|
||||
if (/^https?:\/\//.test(input) || /^http:\/\//.test(input)) {
|
||||
return input;
|
||||
}
|
||||
if (input.startsWith("~/") || input.startsWith("./") || input.startsWith("../") || input.startsWith("/")) {
|
||||
return resolvePath(input);
|
||||
}
|
||||
return input;
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads configuration from disk or returns defaults.
|
||||
*
|
||||
* @param explicitPath Optional CLI-provided config path.
|
||||
* @returns Effective config and source path.
|
||||
*/
|
||||
export async function loadConfig(
|
||||
explicitPath?: string,
|
||||
): Promise<{ config: WikiEngineConfig; configPath: string }> {
|
||||
try {
|
||||
const configPath = resolveConfigPath(explicitPath);
|
||||
let loaded: PartialWikiEngineConfig = {};
|
||||
|
||||
try {
|
||||
const raw = await fs.readFile(configPath, "utf8");
|
||||
loaded = JSON.parse(raw) as PartialWikiEngineConfig;
|
||||
} catch (error) {
|
||||
const errno = (error as NodeJS.ErrnoException).code;
|
||||
if (errno === "ENOENT" && !explicitPath && configPath === getDefaultConfigPath()) {
|
||||
try {
|
||||
const raw = await fs.readFile(getLegacyConfigPath(), "utf8");
|
||||
loaded = JSON.parse(raw) as PartialWikiEngineConfig;
|
||||
} catch (legacyError) {
|
||||
if ((legacyError as NodeJS.ErrnoException).code !== "ENOENT") {
|
||||
throw legacyError;
|
||||
}
|
||||
}
|
||||
} else if (errno !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
const merged = mergeConfig(DEFAULT_CONFIG, loaded);
|
||||
return {
|
||||
config: normalizeConfig(merged),
|
||||
configPath,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new ConfigError("Failed to load habraid configuration.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes a config file to disk, creating parent directories as needed.
|
||||
*
|
||||
* @param configPath Destination config path.
|
||||
* @param config Config object to write.
|
||||
*/
|
||||
export async function saveConfig(configPath: string, config: WikiEngineConfig): Promise<void> {
|
||||
try {
|
||||
const normalizedPath = resolvePath(configPath);
|
||||
await fs.mkdir(path.dirname(normalizedPath), { recursive: true });
|
||||
await writeTextFile(normalizedPath, `${JSON.stringify(config, null, 2)}\n`);
|
||||
} catch (error) {
|
||||
throw new ConfigError("Failed to save habraid configuration.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Produces an initialized config object with optional vault overrides.
|
||||
*
|
||||
* @param currentConfig Base config.
|
||||
* @param options Init overrides.
|
||||
* @returns Updated config.
|
||||
*/
|
||||
export function buildInitConfig(
|
||||
currentConfig: WikiEngineConfig,
|
||||
options: { vaultPath?: string; gitRemote?: string },
|
||||
): WikiEngineConfig {
|
||||
return normalizeConfig({
|
||||
...currentConfig,
|
||||
vault: {
|
||||
...currentConfig.vault,
|
||||
path: options.vaultPath ? expandHomeDir(options.vaultPath) : currentConfig.vault.path,
|
||||
git_remote: options.gitRemote ?? currentConfig.vault.git_remote,
|
||||
},
|
||||
});
|
||||
}
|
||||
93
src/db/communities.ts
Normal file
93
src/db/communities.ts
Normal file
@@ -0,0 +1,93 @@
|
||||
/**
|
||||
* Community DB 연산 모듈.
|
||||
*
|
||||
* 감지된 커뮤니티를 kg_communities / kg_community_members 테이블에
|
||||
* 저장하고 조회하는 기능을 제공한다.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Community } from "../graph/community.js";
|
||||
|
||||
// ─── 공개 함수 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 커뮤니티를 DB에 저장한다.
|
||||
* 기존 커뮤니티 데이터를 모두 삭제하고 새로 저장한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param communities 저장할 커뮤니티 목록.
|
||||
*/
|
||||
export function saveCommunities(db: Database.Database, communities: Community[]): void {
|
||||
const saveAll = db.transaction(() => {
|
||||
// 기존 데이터 삭제
|
||||
db.prepare("DELETE FROM kg_community_members").run();
|
||||
db.prepare("DELETE FROM kg_communities").run();
|
||||
|
||||
// 커뮤니티 저장
|
||||
const insertCommunity = db.prepare(`
|
||||
INSERT INTO kg_communities (id, label, dominant_type, entity_count)
|
||||
VALUES (?, ?, ?, ?)
|
||||
`);
|
||||
|
||||
const insertMember = db.prepare(`
|
||||
INSERT INTO kg_community_members (community_id, entity_id)
|
||||
VALUES (?, ?)
|
||||
`);
|
||||
|
||||
for (const community of communities) {
|
||||
insertCommunity.run(
|
||||
community.id,
|
||||
community.label,
|
||||
community.dominantType,
|
||||
community.size,
|
||||
);
|
||||
|
||||
for (const entityId of community.entities) {
|
||||
insertMember.run(community.id, entityId);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
saveAll();
|
||||
}
|
||||
|
||||
/**
|
||||
* DB에서 모든 커뮤니티를 로드한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 커뮤니티 목록.
|
||||
*/
|
||||
export function loadCommunities(db: Database.Database): Community[] {
|
||||
const communityRows = db.prepare(
|
||||
"SELECT id, label, dominant_type, entity_count FROM kg_communities ORDER BY entity_count DESC",
|
||||
).all() as Array<{ id: string; label: string; dominant_type: string; entity_count: number }>;
|
||||
|
||||
return communityRows.map((row) => {
|
||||
const members = db.prepare(
|
||||
"SELECT entity_id FROM kg_community_members WHERE community_id = ?",
|
||||
).all(row.id) as Array<{ entity_id: string }>;
|
||||
|
||||
return {
|
||||
id: row.id,
|
||||
label: row.label,
|
||||
entities: members.map((m) => m.entity_id),
|
||||
size: row.entity_count,
|
||||
dominantType: row.dominant_type,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티가 속한 커뮤니티 ID를 반환한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 엔티티 ID.
|
||||
* @returns 커뮤니티 ID 또는 null.
|
||||
*/
|
||||
export function getEntityCommunityId(db: Database.Database, entityId: string): string | null {
|
||||
const row = db.prepare(
|
||||
"SELECT community_id FROM kg_community_members WHERE entity_id = ?",
|
||||
).get(entityId) as { community_id: string } | undefined;
|
||||
return row?.community_id ?? null;
|
||||
}
|
||||
349
src/db/database.ts
Normal file
349
src/db/database.ts
Normal file
@@ -0,0 +1,349 @@
|
||||
/**
|
||||
* SQLite database connection and schema migration for habraid.
|
||||
*
|
||||
* Manages the local items database used for knowledge storage and FTS5 search.
|
||||
* The database is created automatically on first connection.
|
||||
*/
|
||||
|
||||
import Database from "better-sqlite3";
|
||||
import path from "node:path";
|
||||
import { mkdirSync } from "node:fs";
|
||||
|
||||
import { DbReadError } from "../errors.js";
|
||||
|
||||
/** Current schema version for migration tracking. */
|
||||
const SCHEMA_VERSION = 7;
|
||||
|
||||
/**
|
||||
* SQL statements for initial schema creation.
|
||||
*/
|
||||
const SCHEMA_SQL = `
|
||||
CREATE TABLE IF NOT EXISTS items (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'manual',
|
||||
category TEXT,
|
||||
tags TEXT DEFAULT '[]',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
metadata TEXT DEFAULT '{}',
|
||||
wiki_generated_at TEXT,
|
||||
wiki_slug TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS schema_version (
|
||||
version INTEGER PRIMARY KEY
|
||||
);
|
||||
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS items_fts USING fts5(
|
||||
title, content, tags,
|
||||
content=items,
|
||||
content_rowid=rowid
|
||||
);
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_ai AFTER INSERT ON items BEGIN
|
||||
INSERT INTO items_fts(rowid, title, content, tags)
|
||||
VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_ad AFTER DELETE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags)
|
||||
VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
END;
|
||||
|
||||
CREATE TRIGGER IF NOT EXISTS items_au AFTER UPDATE ON items BEGIN
|
||||
INSERT INTO items_fts(items_fts, rowid, title, content, tags)
|
||||
VALUES('delete', old.rowid, old.title, old.content, old.tags);
|
||||
INSERT INTO items_fts(rowid, title, content, tags)
|
||||
VALUES (new.rowid, new.title, new.content, new.tags);
|
||||
END;
|
||||
|
||||
INSERT OR IGNORE INTO schema_version (version) SELECT 1 WHERE NOT EXISTS (SELECT 1 FROM schema_version);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS item_vectors (
|
||||
item_id TEXT PRIMARY KEY REFERENCES items(id),
|
||||
vector BLOB NOT NULL,
|
||||
model TEXT NOT NULL DEFAULT 'paraphrase-multilingual-MiniLM-L12-v2',
|
||||
updated_at TEXT NOT NULL
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_item_vectors_model ON item_vectors(model);
|
||||
`;
|
||||
|
||||
/**
|
||||
* Migration statements to run for each version bump.
|
||||
* Index = target version (1-indexed). Runs only if current version < target.
|
||||
*/
|
||||
const MIGRATIONS: Record<number, string[]> = {
|
||||
2: [
|
||||
`ALTER TABLE items ADD COLUMN wiki_generated_at TEXT`,
|
||||
`ALTER TABLE items ADD COLUMN wiki_slug TEXT`,
|
||||
],
|
||||
3: [
|
||||
`CREATE TABLE IF NOT EXISTS kg_entities (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL DEFAULT 'concept',
|
||||
wiki_slug TEXT,
|
||||
first_seen TEXT DEFAULT (datetime('now')),
|
||||
last_seen TEXT DEFAULT (datetime('now')),
|
||||
metadata TEXT DEFAULT '{}')
|
||||
`,
|
||||
`CREATE TABLE IF NOT EXISTS kg_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subject_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
predicate TEXT NOT NULL,
|
||||
object_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
confidence REAL DEFAULT 1.0,
|
||||
source TEXT NOT NULL DEFAULT 'extracted',
|
||||
evidence TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
UNIQUE(subject_id, predicate, object_id, source))
|
||||
`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_kg_relations_subject ON kg_relations(subject_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_kg_relations_object ON kg_relations(object_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_kg_entities_type ON kg_entities(type)`,
|
||||
],
|
||||
4: [
|
||||
`ALTER TABLE items ADD COLUMN content_hash TEXT`,
|
||||
],
|
||||
5: [
|
||||
`CREATE TABLE IF NOT EXISTS contradictions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
item_a_id TEXT NOT NULL,
|
||||
item_b_id TEXT NOT NULL,
|
||||
field TEXT NOT NULL,
|
||||
value_a TEXT NOT NULL,
|
||||
value_b TEXT NOT NULL,
|
||||
severity TEXT DEFAULT 'medium',
|
||||
status TEXT DEFAULT 'open',
|
||||
resolution TEXT,
|
||||
detected_at TEXT DEFAULT (datetime('now')),
|
||||
resolved_at TEXT
|
||||
)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_contradictions_status ON contradictions(status)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_contradictions_item_a ON contradictions(item_a_id)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_contradictions_item_b ON contradictions(item_b_id)`,
|
||||
],
|
||||
6: [
|
||||
`CREATE TABLE IF NOT EXISTS kg_communities (
|
||||
id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL,
|
||||
dominant_type TEXT,
|
||||
entity_count INTEGER DEFAULT 0,
|
||||
detected_at TEXT DEFAULT (datetime('now'))
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS kg_community_members (
|
||||
community_id TEXT NOT NULL REFERENCES kg_communities(id),
|
||||
entity_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
PRIMARY KEY (community_id, entity_id)
|
||||
)`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_kg_community_members_entity ON kg_community_members(entity_id)`,
|
||||
],
|
||||
7: [
|
||||
`ALTER TABLE contradictions ADD COLUMN item_a_slug TEXT`,
|
||||
`ALTER TABLE contradictions ADD COLUMN item_b_slug TEXT`,
|
||||
`CREATE INDEX IF NOT EXISTS idx_contradictions_field ON contradictions(field)`,
|
||||
],
|
||||
};
|
||||
|
||||
/**
|
||||
* Opens (or creates) the habraid database and runs migrations.
|
||||
*
|
||||
* @param dbPath Absolute path to the SQLite database file.
|
||||
* @returns Connected better-sqlite3 Database instance.
|
||||
*/
|
||||
export function openDatabase(dbPath: string): Database.Database {
|
||||
try {
|
||||
// Ensure parent directory exists
|
||||
mkdirSync(path.dirname(dbPath), { recursive: true });
|
||||
|
||||
const db = new Database(dbPath);
|
||||
|
||||
// Enable WAL mode for better concurrent read performance
|
||||
db.pragma("journal_mode = WAL");
|
||||
db.pragma("foreign_keys = ON");
|
||||
|
||||
// Run schema creation
|
||||
db.exec(SCHEMA_SQL);
|
||||
|
||||
// Run incremental migrations
|
||||
runMigrations(db);
|
||||
|
||||
return db;
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Failed to open database at ${dbPath}: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs pending schema migrations sequentially.
|
||||
* Each version's statements run in a transaction.
|
||||
* Statements that fail with "duplicate column" are silently skipped.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
*/
|
||||
function runMigrations(db: Database.Database): void {
|
||||
const currentVersion = getSchemaVersion(db);
|
||||
|
||||
for (let target = currentVersion + 1; target <= SCHEMA_VERSION; target += 1) {
|
||||
const statements = MIGRATIONS[target];
|
||||
if (!statements) continue;
|
||||
|
||||
const migrate = db.transaction(() => {
|
||||
for (const sql of statements) {
|
||||
try {
|
||||
db.exec(sql);
|
||||
} catch (err: unknown) {
|
||||
// Ignore duplicate column errors — column already exists
|
||||
if (err instanceof Error && err.message.includes("duplicate column name")) {
|
||||
continue;
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
db.prepare("DELETE FROM schema_version WHERE version != ?").run(target);
|
||||
db.prepare("INSERT OR REPLACE INTO schema_version (version) VALUES (?)").run(target);
|
||||
});
|
||||
|
||||
migrate();
|
||||
}
|
||||
|
||||
// After all migrations, verify that no tables are missing.
|
||||
// This guards against corrupted DBs where schema_version was advanced
|
||||
// but the actual DDL never ran (e.g. DB file copied mid-migration).
|
||||
ensureSchemaIntegrity(db);
|
||||
}
|
||||
|
||||
/**
|
||||
* Collects every table name expected by migrations and the initial schema,
|
||||
* then re-runs `CREATE TABLE IF NOT EXISTS` for any that are missing.
|
||||
*
|
||||
* Only covers CREATE TABLE statements (not ALTERs or indexes) since those
|
||||
* are idempotent with IF NOT EXISTS and are the critical structural elements.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
*/
|
||||
function ensureSchemaIntegrity(db: Database.Database): void {
|
||||
// All CREATE TABLE statements that should exist in a fully-migrated DB.
|
||||
const tableDDLs: string[] = [
|
||||
// From initial SCHEMA_SQL
|
||||
`CREATE TABLE IF NOT EXISTS items (
|
||||
id TEXT PRIMARY KEY,
|
||||
title TEXT NOT NULL,
|
||||
content TEXT NOT NULL,
|
||||
source TEXT NOT NULL DEFAULT 'manual',
|
||||
category TEXT,
|
||||
tags TEXT DEFAULT '[]',
|
||||
created_at TEXT NOT NULL,
|
||||
updated_at TEXT NOT NULL,
|
||||
metadata TEXT DEFAULT '{}',
|
||||
wiki_generated_at TEXT,
|
||||
wiki_slug TEXT,
|
||||
content_hash TEXT
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS schema_version (
|
||||
version INTEGER PRIMARY KEY
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS item_vectors (
|
||||
item_id TEXT PRIMARY KEY REFERENCES items(id),
|
||||
vector BLOB NOT NULL,
|
||||
model TEXT NOT NULL DEFAULT 'paraphrase-multilingual-MiniLM-L12-v2',
|
||||
updated_at TEXT NOT NULL
|
||||
)`,
|
||||
// Migration 3 — KG tables (the ones that were missing)
|
||||
`CREATE TABLE IF NOT EXISTS kg_entities (
|
||||
id TEXT PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
type TEXT NOT NULL DEFAULT 'concept',
|
||||
wiki_slug TEXT,
|
||||
first_seen TEXT DEFAULT (datetime('now')),
|
||||
last_seen TEXT DEFAULT (datetime('now')),
|
||||
metadata TEXT DEFAULT '{}')
|
||||
`,
|
||||
`CREATE TABLE IF NOT EXISTS kg_relations (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
subject_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
predicate TEXT NOT NULL,
|
||||
object_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
confidence REAL DEFAULT 1.0,
|
||||
source TEXT NOT NULL DEFAULT 'extracted',
|
||||
evidence TEXT,
|
||||
created_at TEXT DEFAULT (datetime('now')),
|
||||
UNIQUE(subject_id, predicate, object_id, source))
|
||||
`,
|
||||
// Migration 5 — contradictions
|
||||
`CREATE TABLE IF NOT EXISTS contradictions (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
item_a_id TEXT NOT NULL,
|
||||
item_b_id TEXT NOT NULL,
|
||||
field TEXT NOT NULL,
|
||||
value_a TEXT NOT NULL,
|
||||
value_b TEXT NOT NULL,
|
||||
severity TEXT DEFAULT 'medium',
|
||||
status TEXT DEFAULT 'open',
|
||||
resolution TEXT,
|
||||
detected_at TEXT DEFAULT (datetime('now')),
|
||||
resolved_at TEXT,
|
||||
item_a_slug TEXT,
|
||||
item_b_slug TEXT
|
||||
)`,
|
||||
// Migration 6 — communities
|
||||
`CREATE TABLE IF NOT EXISTS kg_communities (
|
||||
id TEXT PRIMARY KEY,
|
||||
label TEXT NOT NULL,
|
||||
dominant_type TEXT,
|
||||
entity_count INTEGER DEFAULT 0,
|
||||
detected_at TEXT DEFAULT (datetime('now'))
|
||||
)`,
|
||||
`CREATE TABLE IF NOT EXISTS kg_community_members (
|
||||
community_id TEXT NOT NULL REFERENCES kg_communities(id),
|
||||
entity_id TEXT NOT NULL REFERENCES kg_entities(id),
|
||||
PRIMARY KEY (community_id, entity_id)
|
||||
)`,
|
||||
];
|
||||
|
||||
// Get the set of existing tables
|
||||
const existingTables = new Set(
|
||||
(db.prepare("SELECT name FROM sqlite_master WHERE type='table' AND name NOT LIKE 'sqlite_%'").all() as { name: string }[])
|
||||
.map((r) => r.name),
|
||||
);
|
||||
|
||||
// Extract table name from each DDL and create if missing
|
||||
for (const ddl of tableDDLs) {
|
||||
const match = ddl.match(/CREATE\s+TABLE\s+IF\s+NOT\s+EXISTS\s+(\w+)/i);
|
||||
if (!match) continue;
|
||||
const tableName = match[1];
|
||||
if (!existingTables.has(tableName)) {
|
||||
db.exec(ddl);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Closes the database connection safely.
|
||||
*
|
||||
* @param db Database instance to close.
|
||||
*/
|
||||
export function closeDatabase(db: Database.Database): void {
|
||||
try {
|
||||
db.close();
|
||||
} catch {
|
||||
// Ignore close errors
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the current schema version.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Schema version number.
|
||||
*/
|
||||
export function getSchemaVersion(db: Database.Database): number {
|
||||
const row = db.prepare("SELECT MAX(version) as version FROM schema_version").get() as { version: number | null };
|
||||
return row.version ?? 0;
|
||||
}
|
||||
195
src/db/hashing.ts
Normal file
195
src/db/hashing.ts
Normal file
@@ -0,0 +1,195 @@
|
||||
/**
|
||||
* Content hashing utilities for incremental wiki updates.
|
||||
*
|
||||
* SHA-256 hashing of item content fields (title + content + tags + category)
|
||||
* to detect changes and trigger wiki page regeneration.
|
||||
*/
|
||||
|
||||
import crypto from "node:crypto";
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Item } from "../types.js";
|
||||
|
||||
/**
|
||||
* Computes a SHA-256 hash of an item's content-significant fields.
|
||||
*
|
||||
* The hash covers title, content, tags (sorted), and category so that
|
||||
* any meaningful change is detected.
|
||||
*
|
||||
* @param item Item to hash.
|
||||
* @returns Hex-encoded SHA-256 digest.
|
||||
*/
|
||||
export function computeItemHash(item: Item): string {
|
||||
const parts = [
|
||||
item.title,
|
||||
item.content,
|
||||
JSON.stringify([...item.tags].sort()),
|
||||
item.category ?? "",
|
||||
];
|
||||
const raw = parts.join("\n");
|
||||
return crypto.createHash("sha256").update(raw, "utf8").digest("hex");
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items whose current content hash differs from the stored hash,
|
||||
* or whose stored hash is NULL. Only items that already have a wiki page
|
||||
* (wiki_generated_at IS NOT NULL) are considered "changed".
|
||||
*
|
||||
* Gracefully handles databases that do not yet have the content_hash column
|
||||
* by returning an empty array.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Items that need wiki page regeneration.
|
||||
*/
|
||||
export function getChangedItems(db: Database.Database): Item[] {
|
||||
try {
|
||||
const rows = db.prepare(`
|
||||
SELECT i.* FROM items i
|
||||
WHERE i.wiki_generated_at IS NOT NULL
|
||||
AND i.content_hash IS NULL
|
||||
`).all() as ItemRowForHash[];
|
||||
|
||||
const changedItems: Item[] = [];
|
||||
|
||||
// Check items with NULL hash (migrated from older schema)
|
||||
for (const row of rows) {
|
||||
const item = rowToItemForHash(row);
|
||||
const currentHash = computeItemHash(item);
|
||||
// NULL hash means migrated — compute and store it
|
||||
markItemHash(db, item.id, currentHash);
|
||||
// These aren't truly "changed", just uninitialized — skip them
|
||||
}
|
||||
|
||||
// Check items where stored hash differs from current hash
|
||||
const allGeneratedRows = db.prepare(`
|
||||
SELECT i.* FROM items i
|
||||
WHERE i.wiki_generated_at IS NOT NULL
|
||||
AND i.content_hash IS NOT NULL
|
||||
`).all() as ItemRowForHash[];
|
||||
|
||||
for (const row of allGeneratedRows) {
|
||||
const item = rowToItemForHash(row);
|
||||
const currentHash = computeItemHash(item);
|
||||
if (currentHash !== row.content_hash) {
|
||||
changedItems.push(item);
|
||||
}
|
||||
}
|
||||
|
||||
return changedItems;
|
||||
} catch (error) {
|
||||
// Gracefully handle missing content_hash column (old DB)
|
||||
if (error instanceof Error && error.message.includes("no such column")) {
|
||||
return [];
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Updates the stored content_hash for an item.
|
||||
*
|
||||
* Gracefully handles databases that do not yet have the content_hash column.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param itemId Item ID.
|
||||
* @param hash New hash value.
|
||||
*/
|
||||
export function markItemHash(db: Database.Database, itemId: string, hash: string): void {
|
||||
try {
|
||||
db.prepare("UPDATE items SET content_hash = ? WHERE id = ?").run(hash, itemId);
|
||||
} catch (error) {
|
||||
// Gracefully handle missing content_hash column (old DB)
|
||||
if (error instanceof Error && error.message.includes("no such column")) {
|
||||
return;
|
||||
}
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns all wiki_slug values for items that have been generated.
|
||||
*
|
||||
* Gracefully handles databases that do not yet have the wiki_slug column.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Array of wiki slugs.
|
||||
*/
|
||||
export function getGeneratedSlugs(db: Database.Database): string[] {
|
||||
try {
|
||||
const rows = db.prepare(
|
||||
"SELECT DISTINCT wiki_slug FROM items WHERE wiki_slug IS NOT NULL",
|
||||
).all() as { wiki_slug: string }[];
|
||||
return rows.map((r) => r.wiki_slug);
|
||||
} catch {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns counts for wiki generation status.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Object with total, generated, stale, and ungenerated counts.
|
||||
*/
|
||||
export function getWikiGenerationCounts(db: Database.Database): {
|
||||
total: number;
|
||||
generated: number;
|
||||
stale: number;
|
||||
ungenerated: number;
|
||||
} {
|
||||
try {
|
||||
const totalRow = db.prepare("SELECT COUNT(*) as count FROM items").get() as { count: number };
|
||||
const ungenRow = db.prepare(
|
||||
"SELECT COUNT(*) as count FROM items WHERE wiki_generated_at IS NULL",
|
||||
).get() as { count: number };
|
||||
const total = totalRow.count;
|
||||
const ungenerated = ungenRow.count;
|
||||
const generated = total - ungenerated;
|
||||
|
||||
// Stale = generated items with hash mismatch
|
||||
const stale = getChangedItems(db).length;
|
||||
|
||||
return { total, generated, stale, ungenerated };
|
||||
} catch {
|
||||
return { total: 0, generated: 0, stale: 0, ungenerated: 0 };
|
||||
}
|
||||
}
|
||||
|
||||
// ── Internal types ────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Row shape from items table including content_hash.
|
||||
*/
|
||||
interface ItemRowForHash {
|
||||
id: string;
|
||||
title: string;
|
||||
content: string;
|
||||
source: string;
|
||||
category: string | null;
|
||||
tags: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
metadata: string;
|
||||
wiki_generated_at: string | null;
|
||||
wiki_slug: string | null;
|
||||
content_hash: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a raw database row into an Item object.
|
||||
*/
|
||||
function rowToItemForHash(row: ItemRowForHash): Item {
|
||||
return {
|
||||
id: row.id,
|
||||
title: row.title,
|
||||
content: row.content,
|
||||
source: row.source as Item["source"],
|
||||
category: row.category as Item["category"],
|
||||
tags: JSON.parse(row.tags || "[]"),
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
metadata: JSON.parse(row.metadata || "{}"),
|
||||
wikiGeneratedAt: row.wiki_generated_at,
|
||||
wikiSlug: row.wiki_slug,
|
||||
};
|
||||
}
|
||||
376
src/db/items.ts
Normal file
376
src/db/items.ts
Normal file
@@ -0,0 +1,376 @@
|
||||
/**
|
||||
* Item CRUD operations and FTS5 full-text search.
|
||||
*
|
||||
* All operations work on the items table managed by {@link module:db/database}.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Item, CreateItemParams, SearchParams, SearchResult } from "../types.js";
|
||||
import { generateItemId, toIsoTimestamp } from "../utils.js";
|
||||
import { computeItemHash } from "./hashing.js";
|
||||
|
||||
/**
|
||||
* Row shape from the items table.
|
||||
*/
|
||||
interface ItemRow {
|
||||
id: string;
|
||||
title: string;
|
||||
content: string;
|
||||
source: string;
|
||||
category: string | null;
|
||||
tags: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
metadata: string;
|
||||
wiki_generated_at: string | null;
|
||||
wiki_slug: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a raw database row into an Item object.
|
||||
*
|
||||
* @param row Database row.
|
||||
* @returns Parsed Item.
|
||||
*/
|
||||
function rowToItem(row: ItemRow): Item {
|
||||
return {
|
||||
id: row.id,
|
||||
title: row.title,
|
||||
content: row.content,
|
||||
source: row.source as Item["source"],
|
||||
category: row.category as Item["category"],
|
||||
tags: JSON.parse(row.tags || "[]"),
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
metadata: JSON.parse(row.metadata || "{}"),
|
||||
wikiGeneratedAt: row.wiki_generated_at,
|
||||
wikiSlug: row.wiki_slug,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Inserts a new item into the database.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param params Item creation parameters.
|
||||
* @returns The created Item.
|
||||
*/
|
||||
export function createItem(db: Database.Database, params: CreateItemParams): Item {
|
||||
const now = toIsoTimestamp();
|
||||
const id = generateItemId(params.source ?? "manual");
|
||||
const tags = JSON.stringify(params.tags ?? []);
|
||||
const metadata = JSON.stringify(params.metadata ?? {});
|
||||
|
||||
// Compute content_hash from params (build a temporary Item-like object)
|
||||
const tempItem: Item = {
|
||||
id,
|
||||
title: params.title,
|
||||
content: params.content,
|
||||
source: (params.source ?? "manual") as Item["source"],
|
||||
category: params.category ?? null,
|
||||
tags: params.tags ?? [],
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
metadata: params.metadata ?? {},
|
||||
};
|
||||
const contentHash = computeItemHash(tempItem);
|
||||
|
||||
db.prepare(`
|
||||
INSERT INTO items (id, title, content, source, category, tags, created_at, updated_at, metadata, content_hash)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
`).run(
|
||||
id,
|
||||
params.title,
|
||||
params.content,
|
||||
params.source ?? "manual",
|
||||
params.category ?? null,
|
||||
tags,
|
||||
now,
|
||||
now,
|
||||
metadata,
|
||||
contentHash,
|
||||
);
|
||||
|
||||
return getItem(db, id)!;
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves a single item by ID.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param id Item ID.
|
||||
* @returns The Item or undefined if not found.
|
||||
*/
|
||||
export function getItem(db: Database.Database, id: string): Item | undefined {
|
||||
const row = db.prepare("SELECT * FROM items WHERE id = ?").get(id) as ItemRow | undefined;
|
||||
return row ? rowToItem(row) : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns all items, ordered by creation date descending.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param limit Maximum items to return (default 100).
|
||||
* @param offset Number of items to skip.
|
||||
* @returns Array of Items.
|
||||
*/
|
||||
export function listItems(db: Database.Database, limit = 100, offset = 0): Item[] {
|
||||
const rows = db.prepare("SELECT * FROM items ORDER BY created_at DESC LIMIT ? OFFSET ?").all(limit, offset) as ItemRow[];
|
||||
return rows.map(rowToItem);
|
||||
}
|
||||
|
||||
/**
|
||||
* Updates an existing item's content and/or metadata.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param id Item ID.
|
||||
* @param updates Fields to update.
|
||||
* @returns Updated Item or undefined if not found.
|
||||
*/
|
||||
export function updateItem(
|
||||
db: Database.Database,
|
||||
id: string,
|
||||
updates: Partial<Pick<CreateItemParams, "title" | "content" | "category" | "tags" | "metadata">>,
|
||||
): Item | undefined {
|
||||
const existing = getItem(db, id);
|
||||
if (!existing) return undefined;
|
||||
|
||||
const now = toIsoTimestamp();
|
||||
const title = updates.title ?? existing.title;
|
||||
const content = updates.content ?? existing.content;
|
||||
const category = updates.category !== undefined ? updates.category : existing.category;
|
||||
const tags = JSON.stringify(updates.tags ?? existing.tags);
|
||||
const metadata = JSON.stringify(updates.metadata ?? existing.metadata);
|
||||
|
||||
// Compute new content_hash from merged values
|
||||
const mergedItem: Item = {
|
||||
...existing,
|
||||
title,
|
||||
content,
|
||||
category,
|
||||
tags: updates.tags ?? existing.tags,
|
||||
};
|
||||
const contentHash = computeItemHash(mergedItem);
|
||||
|
||||
db.prepare(`
|
||||
UPDATE items
|
||||
SET title = ?, content = ?, category = ?, tags = ?, updated_at = ?, metadata = ?, content_hash = ?
|
||||
WHERE id = ?
|
||||
`).run(title, content, category, tags, now, metadata, contentHash, id);
|
||||
|
||||
return getItem(db, id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes an item by ID.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param id Item ID.
|
||||
* @returns Whether the item was deleted.
|
||||
*/
|
||||
export function deleteItem(db: Database.Database, id: string): boolean {
|
||||
const result = db.prepare("DELETE FROM items WHERE id = ?").run(id);
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Searches items using FTS5 full-text search.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param params Search parameters.
|
||||
* @returns Search results with rank and snippets.
|
||||
*/
|
||||
export function searchItems(db: Database.Database, params: SearchParams): SearchResult[] {
|
||||
const limit = params.limit ?? 20;
|
||||
const offset = params.offset ?? 0;
|
||||
|
||||
// Use FTS5 MATCH with BM25 ranking
|
||||
const query = params.query.replace(/"/g, '""'); // Escape double quotes
|
||||
|
||||
// Build WHERE clauses for source/category filtering
|
||||
const conditions: string[] = [];
|
||||
const sqlParams: unknown[] = [`"${query}"`];
|
||||
|
||||
if (params.sourceFilter && params.sourceFilter.length > 0) {
|
||||
const placeholders = params.sourceFilter.map(() => "?").join(", ");
|
||||
conditions.push(`i.source IN (${placeholders})`);
|
||||
sqlParams.push(...params.sourceFilter);
|
||||
}
|
||||
|
||||
if (params.categoryFilter && params.categoryFilter.length > 0) {
|
||||
const placeholders = params.categoryFilter.map(() => "?").join(", ");
|
||||
conditions.push(`i.category IN (${placeholders})`);
|
||||
sqlParams.push(...params.categoryFilter);
|
||||
}
|
||||
|
||||
const whereClause = conditions.length > 0 ? `AND ${conditions.join(" AND ")}` : "";
|
||||
|
||||
const rows = db.prepare(`
|
||||
SELECT
|
||||
i.*,
|
||||
bm25(items_fts) as rank,
|
||||
snippet(items_fts, -1, '>>>', '<<<', '...', 32) as snippet
|
||||
FROM items_fts f
|
||||
JOIN items i ON i.rowid = f.rowid
|
||||
WHERE items_fts MATCH ? ${whereClause}
|
||||
ORDER BY rank
|
||||
LIMIT ? OFFSET ?
|
||||
`).all(...sqlParams, limit, offset) as (ItemRow & { rank: number; snippet: string })[];
|
||||
|
||||
return rows.map((row) => ({
|
||||
item: rowToItem(row),
|
||||
rank: row.rank,
|
||||
snippet: row.snippet,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the total number of items in the database.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Item count.
|
||||
*/
|
||||
export function getItemCount(db: Database.Database): number {
|
||||
const row = db.prepare("SELECT COUNT(*) as count FROM items").get() as { count: number };
|
||||
return row.count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items filtered by source type.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param source Source type to filter by.
|
||||
* @param limit Maximum items to return.
|
||||
* @returns Filtered Items.
|
||||
*/
|
||||
export function getItemsBySource(db: Database.Database, source: string, limit = 100): Item[] {
|
||||
const rows = db.prepare("SELECT * FROM items WHERE source = ? ORDER BY created_at DESC LIMIT ?").all(source, limit) as ItemRow[];
|
||||
return rows.map(rowToItem);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items created after a given date.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param since Cutoff date.
|
||||
* @returns Items created after the date.
|
||||
*/
|
||||
export function getItemsSince(db: Database.Database, since: Date): Item[] {
|
||||
const rows = db.prepare("SELECT * FROM items WHERE created_at > ? ORDER BY created_at DESC").all(since.toISOString()) as ItemRow[];
|
||||
return rows.map(rowToItem);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns distinct source types present in the database.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Array of source type strings.
|
||||
*/
|
||||
export function getDistinctSources(db: Database.Database): string[] {
|
||||
const rows = db.prepare("SELECT DISTINCT source FROM items ORDER BY source").all() as { source: string }[];
|
||||
return rows.map((r) => r.source);
|
||||
}
|
||||
|
||||
/**
|
||||
* Upserts an item — inserts if not exists, updates if it does.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param item Item to upsert.
|
||||
* @returns The upserted Item.
|
||||
*/
|
||||
export function upsertItem(db: Database.Database, item: Item): Item {
|
||||
const tags = JSON.stringify(item.tags);
|
||||
const metadata = JSON.stringify(item.metadata);
|
||||
const contentHash = computeItemHash(item);
|
||||
|
||||
db.prepare(`
|
||||
INSERT INTO items (id, title, content, source, category, tags, created_at, updated_at, metadata, content_hash)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
title = excluded.title,
|
||||
content = excluded.content,
|
||||
category = excluded.category,
|
||||
tags = excluded.tags,
|
||||
updated_at = excluded.updated_at,
|
||||
metadata = excluded.metadata,
|
||||
content_hash = excluded.content_hash
|
||||
`).run(
|
||||
item.id,
|
||||
item.title,
|
||||
item.content,
|
||||
item.source,
|
||||
item.category,
|
||||
tags,
|
||||
item.createdAt,
|
||||
item.updatedAt,
|
||||
metadata,
|
||||
contentHash,
|
||||
);
|
||||
|
||||
return getItem(db, item.id)!;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items that have not yet been processed into wiki pages.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Items without wiki generation timestamp.
|
||||
*/
|
||||
export function getUngeneratedItems(db: Database.Database): Item[] {
|
||||
const rows = db.prepare(
|
||||
"SELECT * FROM items WHERE wiki_generated_at IS NULL ORDER BY created_at ASC",
|
||||
).all() as ItemRow[];
|
||||
return rows.map(rowToItem);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the count of items that have not been processed into wiki pages.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Count of ungenerated items.
|
||||
*/
|
||||
export function getUngeneratedCount(db: Database.Database): number {
|
||||
const row = db.prepare(
|
||||
"SELECT COUNT(*) as count FROM items WHERE wiki_generated_at IS NULL",
|
||||
).get() as { count: number };
|
||||
return row.count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks items as having been processed into wiki pages.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param itemIds IDs of items to mark as generated.
|
||||
* @param slug Wiki page slug the items were generated into.
|
||||
*/
|
||||
export function markItemsGenerated(db: Database.Database, itemIds: string[], slug: string): void {
|
||||
const now = toIsoTimestamp();
|
||||
const stmt = db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
);
|
||||
const markMany = db.transaction((ids: string[]) => {
|
||||
for (const id of ids) {
|
||||
stmt.run(now, slug, id);
|
||||
}
|
||||
});
|
||||
markMany(itemIds);
|
||||
}
|
||||
|
||||
/**
|
||||
* Resets wiki generation tracking for specified items, allowing re-generation.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param itemIds IDs of items to reset.
|
||||
*/
|
||||
export function resetWikiGeneration(db: Database.Database, itemIds: string[]): void {
|
||||
const stmt = db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = NULL, wiki_slug = NULL WHERE id = ?",
|
||||
);
|
||||
const resetMany = db.transaction((ids: string[]) => {
|
||||
for (const id of ids) {
|
||||
stmt.run(id);
|
||||
}
|
||||
});
|
||||
resetMany(itemIds);
|
||||
}
|
||||
379
src/db/kg.ts
Normal file
379
src/db/kg.ts
Normal file
@@ -0,0 +1,379 @@
|
||||
/**
|
||||
* Knowledge Graph 데이터베이스 연산.
|
||||
*
|
||||
* 엔티티(kg_entities)와 관계(kg_relations) 테이블에 대한 CRUD + 검색 + 그래프 순회 기능을 제공한다.
|
||||
* 모든 연산은 {@link module:db/database}에서 관리하는 SQLite 연결을 사용한다.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
// ─── 타입 정의 ────────────────────────────────────────────────────────
|
||||
|
||||
/** 엔티티 유형 */
|
||||
export type EntityType = "concept" | "project" | "person" | "tool" | "event" | "decision";
|
||||
|
||||
/** 관계 출처 */
|
||||
export type RelationSource = "extracted" | "inferred" | "ambiguous";
|
||||
|
||||
/** 관계 방향 */
|
||||
export type RelationDirection = "incoming" | "outgoing" | "both";
|
||||
|
||||
/** 엔티티 추가 파라미터 */
|
||||
export interface AddEntityParams {
|
||||
id: string;
|
||||
name: string;
|
||||
type?: EntityType;
|
||||
wikiSlug?: string;
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** 관계 추가 파라미터 */
|
||||
export interface AddRelationParams {
|
||||
subjectId: string;
|
||||
predicate: string;
|
||||
objectId: string;
|
||||
confidence?: number;
|
||||
source?: RelationSource;
|
||||
evidence?: string;
|
||||
}
|
||||
|
||||
/** 엔티티 행 (DB 로우) */
|
||||
interface EntityRow {
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
wiki_slug: string | null;
|
||||
first_seen: string;
|
||||
last_seen: string;
|
||||
metadata: string;
|
||||
}
|
||||
|
||||
/** 관계 행 (DB 로우) */
|
||||
interface RelationRow {
|
||||
id: number;
|
||||
subject_id: string;
|
||||
predicate: string;
|
||||
object_id: string;
|
||||
confidence: number;
|
||||
source: string;
|
||||
evidence: string | null;
|
||||
created_at: string;
|
||||
}
|
||||
|
||||
/** 엔티티 + 관계 수 정보 */
|
||||
export interface EntityWithStats {
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
wikiSlug: string | null;
|
||||
firstSeen: string;
|
||||
lastSeen: string;
|
||||
metadata: Record<string, unknown>;
|
||||
outgoingRelations: number;
|
||||
incomingRelations: number;
|
||||
}
|
||||
|
||||
/** 관계 정보 */
|
||||
export interface RelationInfo {
|
||||
id: number;
|
||||
subjectId: string;
|
||||
predicate: string;
|
||||
objectId: string;
|
||||
confidence: number;
|
||||
source: string;
|
||||
evidence: string | null;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
/** 그래프 통계 */
|
||||
export interface GraphStats {
|
||||
totalEntities: number;
|
||||
totalRelations: number;
|
||||
entityTypes: Record<string, number>;
|
||||
topPredicates: Array<{ predicate: string; count: number }>;
|
||||
}
|
||||
|
||||
/** 이웃 노드 (BFS 순회 결과) */
|
||||
export interface NeighborResult {
|
||||
entities: EntityWithStats[];
|
||||
relations: RelationInfo[];
|
||||
}
|
||||
|
||||
// ─── 내부 유틸 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* DB 로우를 EntityWithStats로 변환한다.
|
||||
*
|
||||
* @param row DB 엔티티 로우.
|
||||
* @param outgoingCount 나가는 관계 수.
|
||||
* @param incomingCount 들어오는 관계 수.
|
||||
* @returns 변환된 엔티티 객체.
|
||||
*/
|
||||
function rowToEntityWithStats(
|
||||
row: EntityRow,
|
||||
outgoingCount: number,
|
||||
incomingCount: number,
|
||||
): EntityWithStats {
|
||||
return {
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
type: row.type,
|
||||
wikiSlug: row.wiki_slug,
|
||||
firstSeen: row.first_seen,
|
||||
lastSeen: row.last_seen,
|
||||
metadata: JSON.parse(row.metadata || "{}"),
|
||||
outgoingRelations: outgoingCount,
|
||||
incomingRelations: incomingCount,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* DB 로우를 RelationInfo로 변환한다.
|
||||
*
|
||||
* @param row DB 관계 로우.
|
||||
* @returns 변환된 관계 객체.
|
||||
*/
|
||||
function rowToRelationInfo(row: RelationRow): RelationInfo {
|
||||
return {
|
||||
id: row.id,
|
||||
subjectId: row.subject_id,
|
||||
predicate: row.predicate,
|
||||
objectId: row.object_id,
|
||||
confidence: row.confidence,
|
||||
source: row.source,
|
||||
evidence: row.evidence,
|
||||
createdAt: row.created_at,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── 엔티티 연산 ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 엔티티를 추가하거나 갱신한다 (upsert).
|
||||
* 이미 존재하면 name, type, wiki_slug, last_seen, metadata를 갱신한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param params 엔티티 추가 파라미터.
|
||||
* @returns 추가/갱신된 엔티티.
|
||||
*/
|
||||
export function addEntity(db: Database.Database, params: AddEntityParams): EntityWithStats {
|
||||
const metadata = JSON.stringify(params.metadata ?? {});
|
||||
const type = params.type ?? "concept";
|
||||
|
||||
db.prepare(`
|
||||
INSERT INTO kg_entities (id, name, type, wiki_slug, metadata)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(id) DO UPDATE SET
|
||||
name = excluded.name,
|
||||
type = excluded.type,
|
||||
wiki_slug = COALESCE(excluded.wiki_slug, kg_entities.wiki_slug),
|
||||
last_seen = datetime('now'),
|
||||
metadata = excluded.metadata
|
||||
`).run(params.id, params.name, type, params.wikiSlug ?? null, metadata);
|
||||
|
||||
return getEntity(db, params.id)!;
|
||||
}
|
||||
|
||||
/**
|
||||
* ID로 엔티티를 조회한다. 관계 수 정보를 포함한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param id 엔티티 ID (slug 형식).
|
||||
* @returns 엔티티 정보 또는 undefined.
|
||||
*/
|
||||
export function getEntity(db: Database.Database, id: string): EntityWithStats | undefined {
|
||||
const row = db.prepare("SELECT * FROM kg_entities WHERE id = ?").get(id) as EntityRow | undefined;
|
||||
if (!row) return undefined;
|
||||
|
||||
const outgoing = db.prepare("SELECT COUNT(*) as count FROM kg_relations WHERE subject_id = ?").get(id) as { count: number };
|
||||
const incoming = db.prepare("SELECT COUNT(*) as count FROM kg_relations WHERE object_id = ?").get(id) as { count: number };
|
||||
|
||||
return rowToEntityWithStats(row, outgoing.count, incoming.count);
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티를 텍스트 쿼리로 검색한다.
|
||||
* FTS5가 없으므로 LIKE 기반 검색을 사용한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param query 검색어.
|
||||
* @param limit 최대 결과 수 (기본값: 20).
|
||||
* @returns 매칭된 엔티티 목록.
|
||||
*/
|
||||
export function searchEntities(db: Database.Database, query: string, limit = 20): EntityWithStats[] {
|
||||
const pattern = `%${query}%`;
|
||||
const rows = db.prepare(`
|
||||
SELECT * FROM kg_entities
|
||||
WHERE id LIKE ? OR name LIKE ? OR type LIKE ?
|
||||
ORDER BY last_seen DESC
|
||||
LIMIT ?
|
||||
`).all(pattern, pattern, pattern, limit) as EntityRow[];
|
||||
|
||||
return rows.map((row) => {
|
||||
const outgoing = db.prepare("SELECT COUNT(*) as count FROM kg_relations WHERE subject_id = ?").get(row.id) as { count: number };
|
||||
const incoming = db.prepare("SELECT COUNT(*) as count FROM kg_relations WHERE object_id = ?").get(row.id) as { count: number };
|
||||
return rowToEntityWithStats(row, outgoing.count, incoming.count);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티를 삭제한다. 연관된 관계도 모두 삭제된다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param id 엔티티 ID.
|
||||
* @returns 삭제 여부.
|
||||
*/
|
||||
export function deleteEntity(db: Database.Database, id: string): boolean {
|
||||
// CASCADE로 관계도 삭제됨 (foreign_keys = ON)
|
||||
const result = db.prepare("DELETE FROM kg_entities WHERE id = ?").run(id);
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
// ─── 관계 연산 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 관계를 추가하거나 갱신한다 (upsert).
|
||||
* 동일한 (subject_id, predicate, object_id, source) 조합이면 갱신한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param params 관계 추가 파라미터.
|
||||
* @returns 추가/갱신된 관계 ID.
|
||||
*/
|
||||
export function addRelation(db: Database.Database, params: AddRelationParams): number {
|
||||
const confidence = params.confidence ?? 1.0;
|
||||
const source = params.source ?? "extracted";
|
||||
|
||||
const result = db.prepare(`
|
||||
INSERT INTO kg_relations (subject_id, predicate, object_id, confidence, source, evidence)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON CONFLICT(subject_id, predicate, object_id, source) DO UPDATE SET
|
||||
confidence = excluded.confidence,
|
||||
evidence = COALESCE(excluded.evidence, kg_relations.evidence)
|
||||
`).run(params.subjectId, params.predicate, params.objectId, confidence, source, params.evidence ?? null);
|
||||
|
||||
// AUTOINCREMENT이므로 lastInsertRowid 사용
|
||||
return Number(result.lastInsertRowid);
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티의 관계를 방향별로 조회한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 엔티티 ID.
|
||||
* @param direction 관계 방향 (기본값: 'both').
|
||||
* @returns 관계 목록.
|
||||
*/
|
||||
export function getRelations(db: Database.Database, entityId: string, direction: RelationDirection = "both"): RelationInfo[] {
|
||||
let sql: string;
|
||||
|
||||
if (direction === "outgoing") {
|
||||
sql = "SELECT * FROM kg_relations WHERE subject_id = ? ORDER BY created_at DESC";
|
||||
} else if (direction === "incoming") {
|
||||
sql = "SELECT * FROM kg_relations WHERE object_id = ? ORDER BY created_at DESC";
|
||||
} else {
|
||||
sql = "SELECT * FROM kg_relations WHERE subject_id = ? OR object_id = ? ORDER BY created_at DESC";
|
||||
}
|
||||
|
||||
const rows = direction === "both"
|
||||
? db.prepare(sql).all(entityId, entityId) as RelationRow[]
|
||||
: db.prepare(sql).all(entityId) as RelationRow[];
|
||||
|
||||
return rows.map(rowToRelationInfo);
|
||||
}
|
||||
|
||||
/**
|
||||
* 단일 관계를 ID로 삭제한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param id 관계 ID.
|
||||
* @returns 삭제 여부.
|
||||
*/
|
||||
export function deleteRelation(db: Database.Database, id: number): boolean {
|
||||
const result = db.prepare("DELETE FROM kg_relations WHERE id = ?").run(id);
|
||||
return result.changes > 0;
|
||||
}
|
||||
|
||||
// ─── 그래프 통계 ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 지식 그래프 전체 통계를 반환한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 그래프 통계.
|
||||
*/
|
||||
export function getGraphStats(db: Database.Database): GraphStats {
|
||||
const entityCount = db.prepare("SELECT COUNT(*) as count FROM kg_entities").get() as { count: number };
|
||||
const relationCount = db.prepare("SELECT COUNT(*) as count FROM kg_relations").get() as { count: number };
|
||||
|
||||
const typeRows = db.prepare(
|
||||
"SELECT type, COUNT(*) as count FROM kg_entities GROUP BY type ORDER BY count DESC",
|
||||
).all() as Array<{ type: string; count: number }>;
|
||||
|
||||
const predicateRows = db.prepare(
|
||||
"SELECT predicate, COUNT(*) as count FROM kg_relations GROUP BY predicate ORDER BY count DESC LIMIT 10",
|
||||
).all() as Array<{ predicate: string; count: number }>;
|
||||
|
||||
const entityTypes: Record<string, number> = {};
|
||||
for (const row of typeRows) {
|
||||
entityTypes[row.type] = row.count;
|
||||
}
|
||||
|
||||
return {
|
||||
totalEntities: entityCount.count,
|
||||
totalRelations: relationCount.count,
|
||||
entityTypes,
|
||||
topPredicates: predicateRows,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── 그래프 순회 ──────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 엔티티의 이웃을 BFS로 순회하여 서브그래프를 반환한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 시작 엔티티 ID.
|
||||
* @param depth 순회 깊이 (기본값: 1).
|
||||
* @returns 발견된 엔티티와 관계.
|
||||
*/
|
||||
export function getEntityNeighbors(db: Database.Database, entityId: string, depth = 1): NeighborResult {
|
||||
const visitedEntities = new Set<string>();
|
||||
const relations: RelationInfo[] = [];
|
||||
let currentLevel = new Set<string>([entityId]);
|
||||
|
||||
for (let d = 0; d <= depth; d += 1) {
|
||||
const nextLevel = new Set<string>();
|
||||
|
||||
for (const eid of currentLevel) {
|
||||
if (visitedEntities.has(eid)) continue;
|
||||
visitedEntities.add(eid);
|
||||
|
||||
// 나가는 관계
|
||||
const outRows = db.prepare(
|
||||
"SELECT * FROM kg_relations WHERE subject_id = ?",
|
||||
).all(eid) as RelationRow[];
|
||||
|
||||
// 들어오는 관계
|
||||
const inRows = db.prepare(
|
||||
"SELECT * FROM kg_relations WHERE object_id = ?",
|
||||
).all(eid) as RelationRow[];
|
||||
|
||||
for (const row of [...outRows, ...inRows]) {
|
||||
relations.push(rowToRelationInfo(row));
|
||||
if (!visitedEntities.has(row.subject_id)) nextLevel.add(row.subject_id);
|
||||
if (!visitedEntities.has(row.object_id)) nextLevel.add(row.object_id);
|
||||
}
|
||||
}
|
||||
|
||||
currentLevel = nextLevel;
|
||||
}
|
||||
|
||||
// 모든 발견된 엔티티 정보 수집
|
||||
const entities: EntityWithStats[] = [];
|
||||
for (const eid of visitedEntities) {
|
||||
const entity = getEntity(db, eid);
|
||||
if (entity) entities.push(entity);
|
||||
}
|
||||
|
||||
return { entities, relations };
|
||||
}
|
||||
117
src/errors.ts
Normal file
117
src/errors.ts
Normal file
@@ -0,0 +1,117 @@
|
||||
/**
|
||||
* Base project error with a stable code.
|
||||
*/
|
||||
export class WikiEngineError extends Error {
|
||||
public readonly code: string;
|
||||
public override readonly cause?: Error;
|
||||
|
||||
/**
|
||||
* Creates a typed project error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param code Stable machine-readable error code.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, code: string, cause?: Error) {
|
||||
super(message);
|
||||
this.name = "WikiEngineError";
|
||||
this.code = code;
|
||||
this.cause = cause;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when vault initialization fails.
|
||||
*/
|
||||
export class VaultInitError extends WikiEngineError {
|
||||
/**
|
||||
* Creates a vault initialization error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "VAULT_INIT_FAILED", cause);
|
||||
this.name = "VaultInitError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when reading the MemPalace database fails.
|
||||
*/
|
||||
export class DbReadError extends WikiEngineError {
|
||||
/**
|
||||
* Creates a database read error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "DB_READ_FAILED", cause);
|
||||
this.name = "DbReadError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when the LLM backend call fails.
|
||||
*/
|
||||
export class LlmCallError extends WikiEngineError {
|
||||
/**
|
||||
* Creates an LLM call error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "LLM_CALL_FAILED", cause);
|
||||
this.name = "LlmCallError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when git synchronization fails.
|
||||
*/
|
||||
export class GitSyncError extends WikiEngineError {
|
||||
/**
|
||||
* Creates a git sync error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "GIT_SYNC_FAILED", cause);
|
||||
this.name = "GitSyncError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when vault linting fails.
|
||||
*/
|
||||
export class LintError extends WikiEngineError {
|
||||
/**
|
||||
* Creates a lint error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "LINT_FAILED", cause);
|
||||
this.name = "LintError";
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Error thrown when configuration loading fails.
|
||||
*/
|
||||
export class ConfigError extends WikiEngineError {
|
||||
/**
|
||||
* Creates a configuration error.
|
||||
*
|
||||
* @param message Human-readable error message.
|
||||
* @param cause Optional underlying cause.
|
||||
*/
|
||||
constructor(message: string, cause?: Error) {
|
||||
super(message, "CONFIG_LOAD_FAILED", cause);
|
||||
this.name = "ConfigError";
|
||||
}
|
||||
}
|
||||
485
src/graph/community.ts
Normal file
485
src/graph/community.ts
Normal file
@@ -0,0 +1,485 @@
|
||||
/**
|
||||
* Community detection module for the Knowledge Graph.
|
||||
*
|
||||
* Implements the Label Propagation Algorithm (LPA) to automatically cluster
|
||||
* KG entities into communities/topics based on the kg_relations graph.
|
||||
*
|
||||
* Communities are recalculated on demand (not real-time) and stored in the DB
|
||||
* for fast lookups.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
// ─── 타입 정의 ────────────────────────────────────────────────────────
|
||||
|
||||
/** 커뮤니티 정보 */
|
||||
export interface Community {
|
||||
/** 커뮤니티 ID (community-0, community-1, ...) */
|
||||
id: string;
|
||||
/** 자동 생성된 라벨 (우세 엔티티 타입/태그 기반) */
|
||||
label: string;
|
||||
/** 커뮤니티에 속한 엔티티 ID 목록 */
|
||||
entities: string[];
|
||||
/** 엔티티 수 */
|
||||
size: number;
|
||||
/** 커뮤니티 내 가장 흔한 엔티티 타입 */
|
||||
dominantType: string;
|
||||
}
|
||||
|
||||
/** 커뮤니티 상세 정보 */
|
||||
export interface CommunityDetails extends Community {
|
||||
/** 다른 커뮤니티와의 관계 */
|
||||
relations: { predicate: string; targetCommunity: string; count: number }[];
|
||||
/** 관계 수 기준 상위 엔티티 */
|
||||
topEntities: { id: string; name: string; type: string; relationCount: number }[];
|
||||
}
|
||||
|
||||
/** 커뮤니티 레벨 그래프 */
|
||||
export interface CommunityGraph {
|
||||
/** 커뮤니티 노드 목록 */
|
||||
communities: Community[];
|
||||
/** 커뮤니티 간 엣지 */
|
||||
edges: { source: string; target: string; weight: number; predicates: string[] }[];
|
||||
}
|
||||
|
||||
// ─── 내부 타입 ────────────────────────────────────────────────────────
|
||||
|
||||
/** DB 엔티티 행 */
|
||||
interface EntityRow {
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
}
|
||||
|
||||
/** DB 관계 행 */
|
||||
interface RelationRow {
|
||||
subject_id: string;
|
||||
predicate: string;
|
||||
object_id: string;
|
||||
}
|
||||
|
||||
// ─── LPA 구현 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Label Propagation Algorithm(LPA)으로 커뮤니티를 감지한다.
|
||||
*
|
||||
* 1. kg_relations로부터 인접 리스트를 구축한다.
|
||||
* 2. 각 엔티티의 초기 라벨을 엔티티 타입으로 시드한다 (안정성).
|
||||
* 3. 반복적으로 각 노드가 이웃 중 가장 빈번한 라벨을 채택한다.
|
||||
* 4. 수렴하거나 최대 반복에 도달하면 종료한다.
|
||||
* 5. 동일 라벨끼리 그룹화하여 커뮤니티를 생성한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 감지된 커뮤니티 목록.
|
||||
*/
|
||||
export function detectCommunities(db: Database.Database): Community[] {
|
||||
// 1. 모든 엔티티 로드
|
||||
const entities = db.prepare("SELECT id, name, type FROM kg_entities").all() as EntityRow[];
|
||||
|
||||
if (entities.length === 0) return [];
|
||||
|
||||
const entityMap = new Map<string, EntityRow>();
|
||||
for (const e of entities) {
|
||||
entityMap.set(e.id, e);
|
||||
}
|
||||
|
||||
// 2. 모든 관계 로드
|
||||
const relations = db.prepare(
|
||||
"SELECT subject_id, predicate, object_id FROM kg_relations",
|
||||
).all() as RelationRow[];
|
||||
|
||||
// 3. 인접 리스트 구축 (무방향)
|
||||
const adjacency = new Map<string, Set<string>>();
|
||||
for (const eid of entityMap.keys()) {
|
||||
adjacency.set(eid, new Set());
|
||||
}
|
||||
|
||||
for (const rel of relations) {
|
||||
if (entityMap.has(rel.subject_id) && entityMap.has(rel.object_id)) {
|
||||
adjacency.get(rel.subject_id)!.add(rel.object_id);
|
||||
adjacency.get(rel.object_id)!.add(rel.subject_id);
|
||||
}
|
||||
}
|
||||
|
||||
// 4. 초기 라벨: 엔티티 타입을 시드로 사용 (안정성 향상)
|
||||
const labels = new Map<string, string>();
|
||||
for (const e of entities) {
|
||||
labels.set(e.id, `${e.type}:${e.id}`);
|
||||
}
|
||||
|
||||
// 5. LPA 반복
|
||||
const MAX_ITERATIONS = 100;
|
||||
const entityIds = entities.map((e) => e.id);
|
||||
|
||||
for (let iter = 0; iter < MAX_ITERATIONS; iter++) {
|
||||
let changed = 0;
|
||||
|
||||
// 랜덤 순서로 순회 (편향 방지)
|
||||
const order = [...entityIds];
|
||||
shuffleArray(order);
|
||||
|
||||
for (const eid of order) {
|
||||
const neighbors = adjacency.get(eid);
|
||||
if (!neighbors || neighbors.size === 0) continue;
|
||||
|
||||
// 이웃 라벨 빈도 집계
|
||||
const labelCounts = new Map<string, number>();
|
||||
for (const nid of neighbors) {
|
||||
const nl = labels.get(nid);
|
||||
if (nl !== undefined) {
|
||||
labelCounts.set(nl, (labelCounts.get(nl) ?? 0) + 1);
|
||||
}
|
||||
}
|
||||
|
||||
// 가장 빈번한 라벨 찾기 (동점 시 기존 라벨 우선 → 안정성)
|
||||
let maxCount = 0;
|
||||
let bestLabel = labels.get(eid)!;
|
||||
|
||||
for (const [label, count] of labelCounts) {
|
||||
if (count > maxCount || (count === maxCount && label < bestLabel)) {
|
||||
maxCount = count;
|
||||
bestLabel = label;
|
||||
}
|
||||
}
|
||||
|
||||
if (bestLabel !== labels.get(eid)) {
|
||||
labels.set(eid, bestLabel);
|
||||
changed++;
|
||||
}
|
||||
}
|
||||
|
||||
// 수렴 체크
|
||||
if (changed === 0) break;
|
||||
}
|
||||
|
||||
// 6. 라벨별로 엔티티 그룹화
|
||||
const groups = new Map<string, string[]>();
|
||||
for (const [eid, label] of labels) {
|
||||
if (!groups.has(label)) {
|
||||
groups.set(label, []);
|
||||
}
|
||||
groups.get(label)!.push(eid);
|
||||
}
|
||||
|
||||
// 7. 커뮤니티 객체 생성
|
||||
const communities: Community[] = [];
|
||||
let communityIndex = 0;
|
||||
|
||||
// 크기 내림차순 정렬
|
||||
const sortedGroups = [...groups.entries()].sort((a, b) => b[1].length - a[1].length);
|
||||
|
||||
for (const [, entityIds] of sortedGroups) {
|
||||
// 우세 타입 계산
|
||||
const typeCounts = new Map<string, number>();
|
||||
for (const eid of entityIds) {
|
||||
const entity = entityMap.get(eid);
|
||||
if (entity) {
|
||||
typeCounts.set(entity.type, (typeCounts.get(entity.type) ?? 0) + 1);
|
||||
}
|
||||
}
|
||||
|
||||
let dominantType = "concept";
|
||||
let maxTypeCount = 0;
|
||||
for (const [type, count] of typeCounts) {
|
||||
if (count > maxTypeCount) {
|
||||
maxTypeCount = count;
|
||||
dominantType = type;
|
||||
}
|
||||
}
|
||||
|
||||
// 라벨 생성: 우세 타입 + 대표 엔티티 이름
|
||||
const representativeEntity = entityMap.get(entityIds[0]);
|
||||
const representativeName = representativeEntity?.name ?? "unknown";
|
||||
const label = `${dominantType}: ${representativeName}`;
|
||||
|
||||
communities.push({
|
||||
id: `community-${communityIndex}`,
|
||||
label,
|
||||
entities: entityIds,
|
||||
size: entityIds.length,
|
||||
dominantType,
|
||||
});
|
||||
|
||||
communityIndex++;
|
||||
}
|
||||
|
||||
return communities;
|
||||
}
|
||||
|
||||
/**
|
||||
* 특정 엔티티가 속한 커뮤니티를 반환한다.
|
||||
* DB에 저장된 커뮤니티 정보를 사용한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 엔티티 ID.
|
||||
* @returns 커뮤니티 정보 또는 null.
|
||||
*/
|
||||
export function getEntityCommunity(db: Database.Database, entityId: string): Community | null {
|
||||
// kg_community_members에서 커뮤니티 ID 조회
|
||||
const memberRow = db.prepare(
|
||||
"SELECT community_id FROM kg_community_members WHERE entity_id = ?",
|
||||
).get(entityId) as { community_id: string } | undefined;
|
||||
|
||||
if (!memberRow) return null;
|
||||
|
||||
// 커뮤니티 정보 로드
|
||||
const communityRow = db.prepare(
|
||||
"SELECT id, label, dominant_type, entity_count FROM kg_communities WHERE id = ?",
|
||||
).get(memberRow.community_id) as {
|
||||
id: string; label: string; dominant_type: string; entity_count: number;
|
||||
} | undefined;
|
||||
|
||||
if (!communityRow) return null;
|
||||
|
||||
// 멤버 목록 로드
|
||||
const members = db.prepare(
|
||||
"SELECT entity_id FROM kg_community_members WHERE community_id = ?",
|
||||
).all(communityRow.id) as Array<{ entity_id: string }>;
|
||||
|
||||
return {
|
||||
id: communityRow.id,
|
||||
label: communityRow.label,
|
||||
entities: members.map((m) => m.entity_id),
|
||||
size: communityRow.entity_count,
|
||||
dominantType: communityRow.dominant_type,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 커뮤니티 상세 정보를 반환한다.
|
||||
* 커뮤니티 간 관계와 상위 엔티티를 포함한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param communityId 커뮤니티 ID.
|
||||
* @returns 커뮤니티 상세 정보.
|
||||
*/
|
||||
export function getCommunityDetails(db: Database.Database, communityId: string): CommunityDetails | null {
|
||||
const community = getEntityCommunityById(db, communityId);
|
||||
if (!community) return null;
|
||||
|
||||
const communityEntitySet = new Set(community.entities);
|
||||
|
||||
// 상위 엔티티 (관계 수 기준)
|
||||
const topEntities: CommunityDetails["topEntities"] = [];
|
||||
for (const eid of community.entities.slice(0, 20)) {
|
||||
const entityRow = db.prepare("SELECT id, name, type FROM kg_entities WHERE id = ?").get(eid) as EntityRow | undefined;
|
||||
if (!entityRow) continue;
|
||||
|
||||
const outCount = db.prepare("SELECT COUNT(*) as c FROM kg_relations WHERE subject_id = ?").get(eid) as { c: number };
|
||||
const inCount = db.prepare("SELECT COUNT(*) as c FROM kg_relations WHERE object_id = ?").get(eid) as { c: number };
|
||||
|
||||
topEntities.push({
|
||||
id: entityRow.id,
|
||||
name: entityRow.name,
|
||||
type: entityRow.type,
|
||||
relationCount: outCount.c + inCount.c,
|
||||
});
|
||||
}
|
||||
|
||||
topEntities.sort((a, b) => b.relationCount - a.relationCount);
|
||||
|
||||
// 다른 커뮤니티와의 관계
|
||||
const interCommunityRelations = new Map<string, { predicates: Set<string>; count: number }>();
|
||||
|
||||
for (const eid of community.entities) {
|
||||
const outgoing = db.prepare(
|
||||
"SELECT predicate, object_id FROM kg_relations WHERE subject_id = ?",
|
||||
).all(eid) as Array<{ predicate: string; object_id: string }>;
|
||||
|
||||
const incoming = db.prepare(
|
||||
"SELECT predicate, subject_id FROM kg_relations WHERE object_id = ?",
|
||||
).all(eid) as Array<{ predicate: string; subject_id: string }>;
|
||||
|
||||
for (const rel of outgoing) {
|
||||
if (!communityEntitySet.has(rel.object_id)) {
|
||||
const targetCommunity = getEntityCommunityId(db, rel.object_id);
|
||||
if (targetCommunity) {
|
||||
const key = targetCommunity;
|
||||
if (!interCommunityRelations.has(key)) {
|
||||
interCommunityRelations.set(key, { predicates: new Set(), count: 0 });
|
||||
}
|
||||
interCommunityRelations.get(key)!.predicates.add(rel.predicate);
|
||||
interCommunityRelations.get(key)!.count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const rel of incoming) {
|
||||
if (!communityEntitySet.has(rel.subject_id)) {
|
||||
const targetCommunity = getEntityCommunityId(db, rel.subject_id);
|
||||
if (targetCommunity) {
|
||||
const key = targetCommunity;
|
||||
if (!interCommunityRelations.has(key)) {
|
||||
interCommunityRelations.set(key, { predicates: new Set(), count: 0 });
|
||||
}
|
||||
interCommunityRelations.get(key)!.predicates.add(rel.predicate);
|
||||
interCommunityRelations.get(key)!.count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const relations: CommunityDetails["relations"] = [];
|
||||
for (const [targetCommunity, data] of interCommunityRelations) {
|
||||
// 대표 predicate 선택 (가장 빈번한 것)
|
||||
const predicates = [...data.predicates];
|
||||
relations.push({
|
||||
predicate: predicates[0] ?? "related_to",
|
||||
targetCommunity,
|
||||
count: data.count,
|
||||
});
|
||||
}
|
||||
|
||||
relations.sort((a, b) => b.count - a.count);
|
||||
|
||||
return {
|
||||
...community,
|
||||
relations,
|
||||
topEntities: topEntities.slice(0, 10),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 커뮤니티 레벨 그래프를 반환한다.
|
||||
* 노드 = 커뮤니티, 엣지 = 커뮤니티 간 관계.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 커뮤니티 그래프.
|
||||
*/
|
||||
export function getCommunityGraph(db: Database.Database): CommunityGraph {
|
||||
const communities = loadCommunitiesFromDb(db);
|
||||
|
||||
// 각 엔티티 → 커뮤니티 매핑
|
||||
const entityToCommunity = new Map<string, string>();
|
||||
for (const community of communities) {
|
||||
for (const eid of community.entities) {
|
||||
entityToCommunity.set(eid, community.id);
|
||||
}
|
||||
}
|
||||
|
||||
// 커뮤니티 간 엣지 수집
|
||||
const edgeMap = new Map<string, { weight: number; predicates: Set<string> }>();
|
||||
|
||||
const allRelations = db.prepare(
|
||||
"SELECT subject_id, predicate, object_id FROM kg_relations",
|
||||
).all() as RelationRow[];
|
||||
|
||||
for (const rel of allRelations) {
|
||||
const sourceCommunity = entityToCommunity.get(rel.subject_id);
|
||||
const targetCommunity = entityToCommunity.get(rel.object_id);
|
||||
|
||||
if (!sourceCommunity || !targetCommunity || sourceCommunity === targetCommunity) continue;
|
||||
|
||||
const edgeKey = sourceCommunity < targetCommunity
|
||||
? `${sourceCommunity}|${targetCommunity}`
|
||||
: `${targetCommunity}|${sourceCommunity}`;
|
||||
|
||||
if (!edgeMap.has(edgeKey)) {
|
||||
edgeMap.set(edgeKey, { weight: 0, predicates: new Set() });
|
||||
}
|
||||
|
||||
const edge = edgeMap.get(edgeKey)!;
|
||||
edge.weight++;
|
||||
edge.predicates.add(rel.predicate);
|
||||
}
|
||||
|
||||
const edges: CommunityGraph["edges"] = [];
|
||||
for (const [key, data] of edgeMap) {
|
||||
const [source, target] = key.split("|");
|
||||
edges.push({
|
||||
source,
|
||||
target,
|
||||
weight: data.weight,
|
||||
predicates: [...data.predicates],
|
||||
});
|
||||
}
|
||||
|
||||
edges.sort((a, b) => b.weight - a.weight);
|
||||
|
||||
return { communities, edges };
|
||||
}
|
||||
|
||||
// ─── DB 로드 헬퍼 ────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* DB에서 모든 커뮤니티를 로드한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 커뮤니티 목록.
|
||||
*/
|
||||
function loadCommunitiesFromDb(db: Database.Database): Community[] {
|
||||
const communityRows = db.prepare(
|
||||
"SELECT id, label, dominant_type, entity_count FROM kg_communities ORDER BY entity_count DESC",
|
||||
).all() as Array<{ id: string; label: string; dominant_type: string; entity_count: number }>;
|
||||
|
||||
return communityRows.map((row) => {
|
||||
const members = db.prepare(
|
||||
"SELECT entity_id FROM kg_community_members WHERE community_id = ?",
|
||||
).all(row.id) as Array<{ entity_id: string }>;
|
||||
|
||||
return {
|
||||
id: row.id,
|
||||
label: row.label,
|
||||
entities: members.map((m) => m.entity_id),
|
||||
size: row.entity_count,
|
||||
dominantType: row.dominant_type,
|
||||
};
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* 커뮤니티 ID로 커뮤니티를 조회한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param communityId 커뮤니티 ID.
|
||||
* @returns 커뮤니티 정보 또는 null.
|
||||
*/
|
||||
function getEntityCommunityById(db: Database.Database, communityId: string): Community | null {
|
||||
const communityRow = db.prepare(
|
||||
"SELECT id, label, dominant_type, entity_count FROM kg_communities WHERE id = ?",
|
||||
).get(communityId) as {
|
||||
id: string; label: string; dominant_type: string; entity_count: number;
|
||||
} | undefined;
|
||||
|
||||
if (!communityRow) return null;
|
||||
|
||||
const members = db.prepare(
|
||||
"SELECT entity_id FROM kg_community_members WHERE community_id = ?",
|
||||
).all(communityRow.id) as Array<{ entity_id: string }>;
|
||||
|
||||
return {
|
||||
id: communityRow.id,
|
||||
label: communityRow.label,
|
||||
entities: members.map((m) => m.entity_id),
|
||||
size: communityRow.entity_count,
|
||||
dominantType: communityRow.dominant_type,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티가 속한 커뮤니티 ID를 반환한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 엔티티 ID.
|
||||
* @returns 커뮤니티 ID 또는 null.
|
||||
*/
|
||||
function getEntityCommunityId(db: Database.Database, entityId: string): string | null {
|
||||
const row = db.prepare(
|
||||
"SELECT community_id FROM kg_community_members WHERE entity_id = ?",
|
||||
).get(entityId) as { community_id: string } | undefined;
|
||||
return row?.community_id ?? null;
|
||||
}
|
||||
|
||||
// ─── 유틸 ────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 배열을 Fisher-Yates 알고리즘으로 섞는다.
|
||||
*
|
||||
* @param arr 섞을 배열 (in-place).
|
||||
*/
|
||||
function shuffleArray(arr: string[]): void {
|
||||
for (let i = arr.length - 1; i > 0; i--) {
|
||||
const j = Math.floor(Math.random() * (i + 1));
|
||||
[arr[i], arr[j]] = [arr[j], arr[i]];
|
||||
}
|
||||
}
|
||||
450
src/graph/export.ts
Normal file
450
src/graph/export.ts
Normal file
@@ -0,0 +1,450 @@
|
||||
/**
|
||||
* 지식 그래프 내보내기 모듈.
|
||||
*
|
||||
* KG 엔티티와 관계를 JSON 형식으로 내보낸다.
|
||||
* 전체 그래프, 서브그래프, 통계 내보내기를 지원한다.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import { DbReadError } from "../errors.js";
|
||||
import type { GraphExport, WikiEngineConfig } from "../types.js";
|
||||
import {
|
||||
getEntityNeighbors,
|
||||
getGraphStats as getKgGraphStats,
|
||||
searchEntities,
|
||||
type EntityWithStats,
|
||||
type RelationInfo,
|
||||
} from "../db/kg.js";
|
||||
import { toIsoTimestamp } from "../utils.js";
|
||||
import { openDatabase } from "../db/database.js";
|
||||
import {
|
||||
detectCommunities,
|
||||
getCommunityGraph as getCommunityGraphFromCommunity,
|
||||
} from "./community.js";
|
||||
import { saveCommunities, loadCommunities } from "../db/communities.js";
|
||||
|
||||
// ─── 내보내기 타입 ─────────────────────────────────────────────────────
|
||||
|
||||
/** 내보내기용 노드 */
|
||||
export interface ExportNode {
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
wikiSlug: string | null;
|
||||
relationCount: number;
|
||||
}
|
||||
|
||||
/** 내보내기용 엣지 */
|
||||
export interface ExportEdge {
|
||||
source: string;
|
||||
target: string;
|
||||
predicate: string;
|
||||
confidence: number;
|
||||
sourceType: string;
|
||||
}
|
||||
|
||||
/** 내보내기 결과 */
|
||||
export interface ExportResult {
|
||||
nodes: ExportNode[];
|
||||
edges: ExportEdge[];
|
||||
}
|
||||
|
||||
/** 내보내기 옵션 */
|
||||
export interface ExportOptions {
|
||||
/** 엔티티 유형 필터 */
|
||||
type?: string;
|
||||
/** 최대 노드 수 */
|
||||
limit?: number;
|
||||
}
|
||||
|
||||
/** 통계 내보내기 결과 */
|
||||
export interface StatsExport {
|
||||
totalEntities: number;
|
||||
totalRelations: number;
|
||||
entityTypes: Record<string, number>;
|
||||
topPredicates: Array<{ predicate: string; count: number }>;
|
||||
}
|
||||
|
||||
// ─── 내부 유틸 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* EntityWithStats를 ExportNode로 변환한다.
|
||||
*
|
||||
* @param entity 엔티티 정보.
|
||||
* @returns 내보내기용 노드.
|
||||
*/
|
||||
function toExportNode(entity: EntityWithStats): ExportNode {
|
||||
return {
|
||||
id: entity.id,
|
||||
name: entity.name,
|
||||
type: entity.type,
|
||||
wikiSlug: entity.wikiSlug,
|
||||
relationCount: entity.outgoingRelations + entity.incomingRelations,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* RelationInfo를 ExportEdge로 변환한다.
|
||||
*
|
||||
* @param relation 관계 정보.
|
||||
* @returns 내보내기용 엣지.
|
||||
*/
|
||||
function toExportEdge(relation: RelationInfo): ExportEdge {
|
||||
return {
|
||||
source: relation.subjectId,
|
||||
target: relation.objectId,
|
||||
predicate: relation.predicate,
|
||||
confidence: relation.confidence,
|
||||
sourceType: relation.source,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── 공개 함수 ─────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 전체 지식 그래프를 JSON으로 내보낸다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param options 내보내기 옵션.
|
||||
* @returns 노드와 엣지 목록.
|
||||
*/
|
||||
export function exportGraph(db: Database.Database, options?: ExportOptions): ExportResult {
|
||||
try {
|
||||
const limit = options?.limit ?? 1000;
|
||||
const typeFilter = options?.type;
|
||||
|
||||
// 엔티티 조회
|
||||
let entityRows;
|
||||
if (typeFilter) {
|
||||
entityRows = db.prepare(
|
||||
"SELECT * FROM kg_entities WHERE type = ? ORDER BY last_seen DESC LIMIT ?",
|
||||
).all(typeFilter, limit) as Array<{
|
||||
id: string; name: string; type: string; wiki_slug: string | null;
|
||||
first_seen: string; last_seen: string; metadata: string;
|
||||
}>;
|
||||
} else {
|
||||
entityRows = db.prepare(
|
||||
"SELECT * FROM kg_entities ORDER BY last_seen DESC LIMIT ?",
|
||||
).all(limit) as Array<{
|
||||
id: string; name: string; type: string; wiki_slug: string | null;
|
||||
first_seen: string; last_seen: string; metadata: string;
|
||||
}>;
|
||||
}
|
||||
|
||||
const entityIds = new Set(entityRows.map((r) => r.id));
|
||||
|
||||
// 관계 조회 (엔티티 범위 내)
|
||||
const relationRows = db.prepare(
|
||||
"SELECT * FROM kg_relations WHERE subject_id IN (SELECT id FROM kg_entities ORDER BY last_seen DESC LIMIT ?) OR object_id IN (SELECT id FROM kg_entities ORDER BY last_seen DESC LIMIT ?)",
|
||||
).all(limit, limit) as Array<{
|
||||
id: number; subject_id: string; predicate: string; object_id: string;
|
||||
confidence: number; source: string; evidence: string | null; created_at: string;
|
||||
}>;
|
||||
|
||||
// 필터링: 양쪽 엔티티 모두 세트에 있는 관계만
|
||||
const filteredRelations = relationRows.filter(
|
||||
(r) => entityIds.has(r.subject_id) && entityIds.has(r.object_id),
|
||||
);
|
||||
|
||||
// 노드에 관계 수 계산
|
||||
const outgoingCounts: Record<string, number> = {};
|
||||
const incomingCounts: Record<string, number> = {};
|
||||
for (const r of filteredRelations) {
|
||||
outgoingCounts[r.subject_id] = (outgoingCounts[r.subject_id] ?? 0) + 1;
|
||||
incomingCounts[r.object_id] = (incomingCounts[r.object_id] ?? 0) + 1;
|
||||
}
|
||||
|
||||
const nodes: ExportNode[] = entityRows.map((row) => ({
|
||||
id: row.id,
|
||||
name: row.name,
|
||||
type: row.type,
|
||||
wikiSlug: row.wiki_slug,
|
||||
relationCount: (outgoingCounts[row.id] ?? 0) + (incomingCounts[row.id] ?? 0),
|
||||
}));
|
||||
|
||||
const edges: ExportEdge[] = filteredRelations.map((row) => ({
|
||||
source: row.subject_id,
|
||||
target: row.object_id,
|
||||
predicate: row.predicate,
|
||||
confidence: row.confidence,
|
||||
sourceType: row.source,
|
||||
}));
|
||||
|
||||
return { nodes, edges };
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Graph export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 특정 엔티티 주변의 서브그래프를 내보낸다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 중심 엔티티 ID.
|
||||
* @param depth 순회 깊이 (기본값: 1).
|
||||
* @returns 서브그래프 노드와 엣지.
|
||||
*/
|
||||
export function exportSubgraph(db: Database.Database, entityId: string, depth = 1): ExportResult {
|
||||
try {
|
||||
const { entities, relations } = getEntityNeighbors(db, entityId, depth);
|
||||
|
||||
return {
|
||||
nodes: entities.map(toExportNode),
|
||||
edges: relations.map(toExportEdge),
|
||||
};
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Subgraph export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 그래프 통계를 내보낸다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 그래프 통계.
|
||||
*/
|
||||
export function exportGraphStats(db: Database.Database): StatsExport {
|
||||
try {
|
||||
return getKgGraphStats(db);
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Graph stats export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 설정 기반 그래프 내보내기 (CLI 호환).
|
||||
*
|
||||
* DB를 열고 KG 그래프를 JSON 파일로 저장한 후 파일 경로를 반환한다.
|
||||
* 기존 CLI `habraid graph` 명령어와의 호환성을 위해 유지한다.
|
||||
*
|
||||
* @param config 위키 엔진 설정.
|
||||
* @returns 출력 파일 경로.
|
||||
*/
|
||||
export async function exportGraphToFile(config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const result = exportGraph(db);
|
||||
const document: GraphExport = {
|
||||
generatedAt: toIsoTimestamp(),
|
||||
nodes: result.nodes.map((n) => ({ id: n.id, type: "entity" as const })),
|
||||
edges: result.edges.map((e) => ({
|
||||
source: e.source,
|
||||
target: e.target,
|
||||
predicate: e.predicate,
|
||||
})),
|
||||
};
|
||||
|
||||
const { writeTextFile: writeFile } = await import("../utils.js");
|
||||
const path = await import("node:path");
|
||||
const outputPath = path.join(config.vault.path, "graph", "graph.json");
|
||||
await writeFile(outputPath, `${JSON.stringify(document, null, 2)}\n`);
|
||||
return outputPath;
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Graph file export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
// ─── 커뮤니티 내보내기 ─────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 커뮤니티를 감지하고 DB에 저장한 뒤, 커뮤니티 그래프를 JSON으로 내보낸다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 커뮤니티 그래프 JSON 객체.
|
||||
*/
|
||||
export function exportCommunityGraph(db: Database.Database): {
|
||||
communities: Array<{
|
||||
id: string;
|
||||
label: string;
|
||||
dominantType: string;
|
||||
size: number;
|
||||
entities: string[];
|
||||
}>;
|
||||
edges: Array<{
|
||||
source: string;
|
||||
target: string;
|
||||
weight: number;
|
||||
predicates: string[];
|
||||
}>;
|
||||
} {
|
||||
try {
|
||||
// 커뮤니티 감지 및 저장
|
||||
const communities = detectCommunities(db);
|
||||
saveCommunities(db, communities);
|
||||
|
||||
// 커뮤니티 그래프 생성
|
||||
const graph = getCommunityGraphFromCommunity(db);
|
||||
|
||||
return {
|
||||
communities: graph.communities.map((c) => ({
|
||||
id: c.id,
|
||||
label: c.label,
|
||||
dominantType: c.dominantType,
|
||||
size: c.size,
|
||||
entities: c.entities,
|
||||
})),
|
||||
edges: graph.edges,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Community graph export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 지식 그래프를 Mermaid 다이어그램으로 내보낸다.
|
||||
*
|
||||
* 전체 그래프 또는 특정 엔티티 중심의 서브그래프를
|
||||
* Mermaid flowchart 형식으로 생성한다.
|
||||
* 엔티티 유형별로 색상을 구분한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 중심 엔티티 ID (없으면 전체 그래프).
|
||||
* @param options 내보내기 옵션 (maxNodes, depth).
|
||||
* @returns Mermaid 다이어그램 문자열.
|
||||
*/
|
||||
export function exportMermaidGraph(
|
||||
db: Database.Database,
|
||||
entityId?: string,
|
||||
options?: { maxNodes?: number; depth?: number },
|
||||
): string {
|
||||
try {
|
||||
const maxNodes = options?.maxNodes ?? 20;
|
||||
const depth = options?.depth ?? 2;
|
||||
|
||||
// 엔티티 유형별 색상
|
||||
const TYPE_COLORS: Record<string, string> = {
|
||||
project: "#4A90D9",
|
||||
person: "#27AE60",
|
||||
concept: "#E67E22",
|
||||
tool: "#8E44AD",
|
||||
event: "#E74C3C",
|
||||
decision: "#F39C12",
|
||||
};
|
||||
const DEFAULT_COLOR = "#95A5A6";
|
||||
|
||||
function safeId(id: string): string {
|
||||
return id.replace(/[^a-zA-Z0-9가-힣_-]/g, "_").replace(/^-+|-+$/g, "") || "node_unknown";
|
||||
}
|
||||
function safeLbl(text: string): string {
|
||||
return text.replace(/"/g, "'").replace(/\n/g, " ").slice(0, 40);
|
||||
}
|
||||
function color(type: string): string {
|
||||
return TYPE_COLORS[type] ?? DEFAULT_COLOR;
|
||||
}
|
||||
|
||||
let result: ExportResult;
|
||||
|
||||
if (entityId) {
|
||||
result = exportSubgraph(db, entityId, depth);
|
||||
} else {
|
||||
result = exportGraph(db, { limit: maxNodes });
|
||||
}
|
||||
|
||||
if (result.nodes.length === 0) return "graph LR\n empty[\"(엔티티 없음)\"]";
|
||||
|
||||
const limitedNodes = result.nodes.slice(0, maxNodes);
|
||||
const nodeIds = new Set(limitedNodes.map((n) => n.id));
|
||||
|
||||
const limitedEdges = result.edges.filter(
|
||||
(e) => nodeIds.has(e.source) && nodeIds.has(e.target),
|
||||
);
|
||||
|
||||
const lines: string[] = ["graph LR"];
|
||||
|
||||
// 노드 정의
|
||||
for (const node of limitedNodes) {
|
||||
const nid = safeId(node.id);
|
||||
const lbl = safeLbl(node.name);
|
||||
lines.push(` ${nid}["${lbl}"]`);
|
||||
}
|
||||
|
||||
lines.push("");
|
||||
|
||||
// 엣지 정의
|
||||
for (const edge of limitedEdges) {
|
||||
const sid = safeId(edge.source);
|
||||
const tid = safeId(edge.target);
|
||||
const pred = edge.predicate.length > 15 ? edge.predicate.slice(0, 12) + "..." : edge.predicate;
|
||||
lines.push(` ${sid} -->|"${pred}"| ${tid}`);
|
||||
}
|
||||
|
||||
// 스타일
|
||||
lines.push("");
|
||||
for (const node of limitedNodes) {
|
||||
const nid = safeId(node.id);
|
||||
const c = color(node.type);
|
||||
lines.push(` style ${nid} fill:${c},color:#fff,stroke:#333`);
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Mermaid graph export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* 커뮤니티 그래프를 Mermaid 다이어그램으로 내보낸다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns Mermaid 다이어그램 문자열.
|
||||
*/
|
||||
export function exportMermaidCommunities(db: Database.Database): string {
|
||||
try {
|
||||
// 저장된 커뮤니티가 없으면 감지
|
||||
let communities = loadCommunities(db);
|
||||
if (communities.length === 0) {
|
||||
communities = detectCommunities(db);
|
||||
saveCommunities(db, communities);
|
||||
}
|
||||
|
||||
const graph = getCommunityGraphFromCommunity(db);
|
||||
const lines: string[] = ["graph TD"];
|
||||
|
||||
// 커뮤니티 노드
|
||||
for (const community of graph.communities) {
|
||||
const safeId = community.id.replace(/-/g, "_");
|
||||
const shortLabel = community.label.length > 30
|
||||
? `${community.label.slice(0, 27)}...`
|
||||
: community.label;
|
||||
lines.push(` ${safeId}["${shortLabel}<br/>(${community.size} entities)"]`);
|
||||
}
|
||||
|
||||
// 엣지
|
||||
for (const edge of graph.edges.slice(0, 50)) {
|
||||
const sourceId = edge.source.replace(/-/g, "_");
|
||||
const targetId = edge.target.replace(/-/g, "_");
|
||||
const label = edge.predicates.slice(0, 2).join(", ");
|
||||
lines.push(` ${sourceId} -->|"${label} (${edge.weight})"| ${targetId}`);
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
} catch (error) {
|
||||
throw new DbReadError(
|
||||
`Mermaid community export failed: ${error instanceof Error ? error.message : String(error)}`,
|
||||
error instanceof Error ? error : undefined,
|
||||
);
|
||||
}
|
||||
}
|
||||
27
src/index.ts
Normal file
27
src/index.ts
Normal file
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
import { Command } from "commander";
|
||||
|
||||
import { registerCommands } from "./cli/commands.js";
|
||||
|
||||
/**
|
||||
* Builds and runs the CLI program.
|
||||
*/
|
||||
async function main(): Promise<void> {
|
||||
try {
|
||||
const program = new Command();
|
||||
program
|
||||
.name("habraid")
|
||||
.description("Obsidian wiki engine for HaBraid Team")
|
||||
.version("0.1.0");
|
||||
|
||||
registerCommands(program);
|
||||
await program.parseAsync(process.argv);
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.stack ?? error.message : String(error);
|
||||
process.stderr.write(`${message}\n`);
|
||||
process.exitCode = 1;
|
||||
}
|
||||
}
|
||||
|
||||
void main();
|
||||
295
src/mcp-server.ts
Normal file
295
src/mcp-server.ts
Normal file
@@ -0,0 +1,295 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* habraid MCP Server entry point.
|
||||
*
|
||||
* Runs as a stdio-based MCP server that Hermes can spawn as a child process.
|
||||
* Exposes habraid operations as MCP tools with `hw_` prefix.
|
||||
*
|
||||
* Zero-config: on first run, auto-detects MemPalace, creates ~/.habraid/{app,data},
|
||||
* initializes DB, and starts watching for changes.
|
||||
*
|
||||
* Usage:
|
||||
* npx habraid
|
||||
* node dist/mcp-server.js
|
||||
*
|
||||
* Environment:
|
||||
* HABRAID_CONFIG — path to config.json (optional, defaults to ~/.habraid/data/config.json)
|
||||
*/
|
||||
|
||||
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
||||
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
||||
import { z } from "zod";
|
||||
|
||||
import {
|
||||
handleStatus,
|
||||
handleLint,
|
||||
handleIngest,
|
||||
handleGenerate,
|
||||
handleSync,
|
||||
handleRead,
|
||||
handleSearch,
|
||||
handleAdd,
|
||||
handleGraph,
|
||||
handleIndex,
|
||||
handleEntityAdd,
|
||||
handleEntitySearch,
|
||||
handleRelationAdd,
|
||||
handleDailyLog,
|
||||
handleSynthesize,
|
||||
handleContradictions,
|
||||
} from "./mcp/handlers.js";
|
||||
import { loadConfig } from "./config.js";
|
||||
import { runSetup, type SetupResult } from "./setup.js";
|
||||
import { startWatcher, stopWatcher } from "./sync/watcher.js";
|
||||
import { ensureSync, getPendingWikiCount } from "./sync/auto-sync.js";
|
||||
import type { WikiEngineConfig } from "./types.js";
|
||||
|
||||
function log(msg: string): void {
|
||||
process.stderr.write(`[HaBraid] ${msg}\n`);
|
||||
}
|
||||
|
||||
/** Cached config across all handlers. */
|
||||
let cachedConfig: WikiEngineConfig | null = null;
|
||||
|
||||
/**
|
||||
* Returns the cached config, loading it if necessary.
|
||||
*/
|
||||
async function getConfig(): Promise<WikiEngineConfig> {
|
||||
if (!cachedConfig) {
|
||||
const { config } = await loadConfig(process.env.HABRAID_CONFIG);
|
||||
cachedConfig = config;
|
||||
}
|
||||
return cachedConfig;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lazy sync wrapper — runs before handler logic.
|
||||
* Returns the config for the handler to use.
|
||||
*/
|
||||
async function syncedConfig(): Promise<WikiEngineConfig> {
|
||||
const config = await getConfig();
|
||||
await ensureSync(config);
|
||||
return config;
|
||||
}
|
||||
|
||||
/**
|
||||
* Watcher callback — runs auto-sync when files change.
|
||||
*/
|
||||
async function onFileChange(config: WikiEngineConfig): Promise<void> {
|
||||
const result = await ensureSync(config);
|
||||
if (result.hadNewItems) {
|
||||
log(`Auto-sync: ingested ${result.ingested} new items`);
|
||||
const pending = getPendingWikiCount(config);
|
||||
if (pending > 0) {
|
||||
log(`Auto-sync: ${pending} items pending wiki generation (run hw_generate)`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
// ── Zero-config setup ─────────────────────────────────────────
|
||||
const { config } = await loadConfig(process.env.HABRAID_CONFIG);
|
||||
cachedConfig = config;
|
||||
|
||||
const setupResult: SetupResult = await runSetup(config);
|
||||
|
||||
if (setupResult.firstRun) {
|
||||
log("First run detected. Initializing...");
|
||||
}
|
||||
|
||||
for (const msg of setupResult.messages) {
|
||||
log(msg);
|
||||
}
|
||||
|
||||
// ── Start file watcher ────────────────────────────────────────
|
||||
startWatcher(config, onFileChange);
|
||||
|
||||
// ── MCP Server ────────────────────────────────────────────────
|
||||
const server = new McpServer({
|
||||
name: "habraid",
|
||||
version: "0.2.0",
|
||||
});
|
||||
|
||||
// hw_status — vault + DB statistics
|
||||
server.tool("hw_status", "Show vault and database statistics — raw files, wiki pages, DB items, last sync timestamps.", {}, async () => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleStatus(config) }] };
|
||||
});
|
||||
|
||||
// hw_lint — comprehensive wiki lint
|
||||
server.tool("hw_lint", "Run comprehensive wiki lint checks including orphans, broken links, stale pages, ungenerated items, and frontmatter validation.", {
|
||||
checks: z.array(z.string()).optional().describe("Specific checks to run: orphans, broken_links, stale, ungenerated, frontmatter, duplicates, contradictions"),
|
||||
withLlm: z.boolean().optional().describe("Include LLM-based contradiction checks when available (default: false)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleLint(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_ingest — source → raw/ + DB
|
||||
server.tool("hw_ingest", "Ingest data from all configured sources (MemPalace, etc.) into raw/ and the local DB. Set full=true to re-ingest everything.", {
|
||||
full: z.boolean().optional().describe("Re-ingest all items, not just new ones (default: false)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleIngest(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_add — direct item creation
|
||||
server.tool("hw_add", "Add a knowledge item directly to the local DB. Provide title and content at minimum.", {
|
||||
title: z.string().describe("Item title"),
|
||||
content: z.string().describe("Item content (markdown)"),
|
||||
category: z.string().optional().describe("Category: projects, topics, decisions, people, infrastructure, guides"),
|
||||
tags: z.string().optional().describe("Comma-separated tags"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleAdd(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_search — hybrid search (BM25 + vector + RRF)
|
||||
server.tool("hw_search", "Search knowledge items using hybrid search (BM25 keyword + vector semantic). Modes: hybrid (default), keyword, semantic. Set context=true to enrich results with KG relations, wiki links, and re-rank by context coherence.", {
|
||||
query: z.string().describe("Search query — keywords or natural language"),
|
||||
mode: z.string().optional().describe("Search mode: hybrid (default), keyword, or semantic"),
|
||||
limit: z.number().optional().describe("Max results (default: 20)"),
|
||||
context: z.boolean().optional().describe("Enable context enrichment with KG relations, wiki links, category hierarchy, and context-aware re-ranking (default: false)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleSearch(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_index — build vector embeddings
|
||||
server.tool("hw_index", "Build vector embeddings for all items in the DB. Required before semantic/hybrid search. Uses ONNX local embedding. Automatically builds HNSW ANN index for fast search.", {
|
||||
force: z.boolean().optional().describe("Re-index all items (default: false)"),
|
||||
rebuildHnsw: z.boolean().optional().describe("Rebuild only the HNSW ANN index without re-embedding (default: false)"),
|
||||
rebuildCommunities: z.boolean().optional().describe("Rebuild community detection (LPA) for KG entities (default: false)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleIndex(args ?? {}, config) }] };
|
||||
});
|
||||
|
||||
// hw_generate — LLM wiki generation
|
||||
server.tool("hw_generate", "Generate or update wiki pages from items using LLM. Detects changed items via content hashing. Use force=true to regenerate all pages.", {
|
||||
limit: z.number().optional().describe("Max rooms to process per call (default: all). Use 5-10 for batched generation."),
|
||||
force: z.boolean().optional().describe("Regenerate all wiki pages, not just new/changed ones (default: false)"),
|
||||
backend: z.string().optional().describe("Override the LLM backend for this request: host, openai, ollama, or zai"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleGenerate({ maxRooms: args.limit, force: args.force, backend: args.backend }, config) }] };
|
||||
});
|
||||
|
||||
// hw_sync — full pipeline
|
||||
server.tool("hw_sync", "Run the full sync pipeline: ingest → index → wiki generate.", {
|
||||
noWiki: z.boolean().optional().describe("Skip wiki generation (default: false)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleSync(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_read — read vault file
|
||||
server.tool("hw_read", "Read a wiki or raw page by relative path within the vault.", {
|
||||
path: z.string().describe('Relative file path, e.g. "wiki/projects/project-beta.md"'),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleRead(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_graph — knowledge graph query
|
||||
server.tool("hw_graph", "Query the knowledge graph. Supports multiple modes: no args for stats, entity for subgraph, export for full JSON, search for entity search, mode=communities for community detection. Use format=mermaid for Mermaid diagram output.", {
|
||||
entity: z.string().optional().describe("Entity name or ID to query (returns subgraph)"),
|
||||
depth: z.number().optional().describe("Max traversal depth for subgraph (default: 1)"),
|
||||
export: z.boolean().optional().describe("Export full graph as JSON (default: false)"),
|
||||
search: z.string().optional().describe("Search entities by name or ID"),
|
||||
mode: z.string().optional().describe("Query mode: 'communities' for community detection graph with LPA clustering"),
|
||||
format: z.string().optional().describe("Output format: 'json' (default) or 'mermaid' for Mermaid diagram string"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleGraph(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_entity_add — add KG entity
|
||||
server.tool("hw_entity_add", "Add an entity to the knowledge graph. Entities are nodes identified by a slug ID.", {
|
||||
id: z.string().describe("Entity ID in slug format (e.g. 'nestjs-circular-dep')"),
|
||||
name: z.string().describe("Display name (e.g. 'NestJS 순환참조')"),
|
||||
type: z.string().optional().describe("Entity type: concept, project, person, tool, event, decision (default: concept)"),
|
||||
wikiSlug: z.string().optional().describe("Link to an existing wiki page slug"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleEntityAdd(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_entity_search — search KG entities
|
||||
server.tool("hw_entity_search", "Search entities in the knowledge graph by name, ID, or type.", {
|
||||
query: z.string().describe("Search query — matches against entity name, ID, or type"),
|
||||
limit: z.number().optional().describe("Max results (default: 20)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleEntitySearch(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_relation_add — add KG relation
|
||||
server.tool("hw_relation_add", "Add a relation between two entities in the knowledge graph.", {
|
||||
subjectId: z.string().describe("Source entity ID (slug)"),
|
||||
predicate: z.string().describe("Relationship predicate (e.g. 'fixes', 'depends_on', 'related_to')"),
|
||||
objectId: z.string().describe("Target entity ID (slug)"),
|
||||
confidence: z.number().optional().describe("Confidence score 0.0-1.0 (default: 1.0)"),
|
||||
source: z.string().optional().describe("Source type: extracted, inferred, or ambiguous (default: extracted)"),
|
||||
evidence: z.string().optional().describe("Brief description of why this relation exists"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleRelationAdd(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_daily_log — daily work log
|
||||
server.tool("hw_daily_log", "Generate or read a daily work log summarizing wiki generation activity. Without parameters, returns today's log.", {
|
||||
date: z.string().optional().describe("Specific date (YYYY-MM-DD). Defaults to today."),
|
||||
start: z.string().optional().describe("Range start date (YYYY-MM-DD) for multi-day summary."),
|
||||
end: z.string().optional().describe("Range end date (YYYY-MM-DD) for multi-day summary."),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleDailyLog(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_synthesize — Q&A → wiki synthesis page
|
||||
server.tool("hw_synthesize", "질문-답변을 위키 합성 페이지로 저장. 지식이 복리로 쌓임. Karpathy LLM Wiki 패턴.", {
|
||||
question: z.string().describe("원래 질문"),
|
||||
answer: z.string().describe("답변 내용"),
|
||||
tags: z.array(z.string()).optional().describe("태그 목록"),
|
||||
source_item_ids: z.array(z.string()).optional().describe("소스 아이템 ID 목록"),
|
||||
category: z.string().optional().describe("위키 카테고리 (기본값: topics)"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleSynthesize(args, config) }] };
|
||||
});
|
||||
|
||||
// hw_contradictions — contradiction detection and management
|
||||
server.tool("hw_contradictions", "Detect and manage contradictions/conflicts in the knowledge base. Actions: list, get, resolve, scan, stats.", {
|
||||
action: z.string().optional().describe("Action: list (default), get, resolve, scan, or stats"),
|
||||
id: z.number().optional().describe("Contradiction ID (for get/resolve)"),
|
||||
status: z.string().optional().describe("Filter by status: open, resolved, false_positive (for list)"),
|
||||
resolution: z.string().optional().describe("Resolution description (for resolve)"),
|
||||
resolve_as: z.string().optional().describe("Resolve as: resolved (default) or false_positive"),
|
||||
}, async (args) => {
|
||||
const config = await syncedConfig();
|
||||
return { content: [{ type: "text" as const, text: await handleContradictions(args, config) }] };
|
||||
});
|
||||
|
||||
// ── Connect ───────────────────────────────────────────────────
|
||||
const transport = new StdioServerTransport();
|
||||
await server.connect(transport);
|
||||
|
||||
log("Ready. 16 MCP tools available (hw_*).");
|
||||
|
||||
// Cleanup on exit
|
||||
process.on("SIGINT", () => {
|
||||
stopWatcher();
|
||||
process.exit(0);
|
||||
});
|
||||
process.on("SIGTERM", () => {
|
||||
stopWatcher();
|
||||
process.exit(0);
|
||||
});
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
process.stderr.write(`Fatal: ${error}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
978
src/mcp/handlers.ts
Normal file
978
src/mcp/handlers.ts
Normal file
@@ -0,0 +1,978 @@
|
||||
/**
|
||||
* MCP tool handlers for habraid v2.
|
||||
*
|
||||
* All handlers receive config from the MCP server (which handles
|
||||
* setup and auto-sync externally). No config loading inside handlers.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { openDatabase, closeDatabase } from "../db/database.js";
|
||||
import {
|
||||
createItem,
|
||||
getItem,
|
||||
searchItems,
|
||||
getItemCount,
|
||||
getItemsBySource,
|
||||
listItems,
|
||||
upsertItem,
|
||||
} from "../db/items.js";
|
||||
import {
|
||||
addEntity,
|
||||
addRelation,
|
||||
getEntity,
|
||||
getRelations,
|
||||
searchEntities,
|
||||
getGraphStats,
|
||||
getEntityNeighbors,
|
||||
deleteEntity,
|
||||
deleteRelation,
|
||||
} from "../db/kg.js";
|
||||
import { exportGraph, exportSubgraph, exportGraphStats, exportCommunityGraph, exportMermaidCommunities, exportMermaidGraph } from "../graph/export.js";
|
||||
import { ingestMemPalace } from "../mempalace/ingest.js";
|
||||
import { updateWiki } from "../wiki/generator.js";
|
||||
import { runSyncPipeline } from "../sync/pipeline.js";
|
||||
import { createAllAdapters } from "../sources/adapter.js";
|
||||
// Auto-register adapters
|
||||
import "../sources/mempalace.js";
|
||||
import "../sources/manual.js";
|
||||
import {
|
||||
readTextFileIfExists,
|
||||
readSyncState,
|
||||
toDateString,
|
||||
toIsoTimestamp,
|
||||
writeSyncState,
|
||||
} from "../utils.js";
|
||||
import { generateDailyLog, generateDailyLogRange, writeDailyLog } from "../vault/daily-log.js";
|
||||
import { hybridSearch, type SearchMode } from "../search/hybrid.js";
|
||||
import { embedBatch } from "../search/embedder.js";
|
||||
import { storeVector, getVectorCount, getIndexedItemIds, rebuildHnswIndex, invalidateHnswIndex } from "../search/vector.js";
|
||||
import { isHnswAvailable, HnswIndex } from "../search/hnsw.js";
|
||||
import { getWikiGenerationCounts } from "../db/hashing.js";
|
||||
import type { Item, LintCheckType, WikiCategory, WikiEngineConfig } from "../types.js";
|
||||
import { createBackend, getBackendModeName } from "../wiki/backends/index.js";
|
||||
import { lintWiki } from "../wiki/linter.js";
|
||||
import {
|
||||
createSynthesis,
|
||||
findRelatedItems,
|
||||
createSynthesisKGRelations,
|
||||
renderSynthesisWikiPage,
|
||||
generateSynthesisSlug,
|
||||
} from "../wiki/synthesis.js";
|
||||
import { writeTextFile } from "../utils.js";
|
||||
import {
|
||||
getContradictions,
|
||||
getContradictionById,
|
||||
resolveContradiction,
|
||||
scanAllContradictions,
|
||||
getContradictionStats,
|
||||
} from "../wiki/contradiction.js";
|
||||
|
||||
/**
|
||||
* Handler: hw_status
|
||||
*/
|
||||
export async function handleStatus(config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
const rawDir = path.join(vaultPath, "raw");
|
||||
const wikiDir = path.join(vaultPath, "wiki");
|
||||
|
||||
const rawCount = fs.existsSync(rawDir) ? countFilesRecursive(rawDir) : 0;
|
||||
const wikiCount = fs.existsSync(wikiDir) ? countFilesRecursive(wikiDir) : 0;
|
||||
|
||||
const state = fs.existsSync(path.join(vaultPath, ".sync-state.json"))
|
||||
? await readSyncState(vaultPath) : null;
|
||||
|
||||
let dbItemCount = 0;
|
||||
let vectorCount = 0;
|
||||
let hnswStatus = "unavailable";
|
||||
let wikiGenCounts = { total: 0, generated: 0, stale: 0, ungenerated: 0 };
|
||||
let llmBackendAvailable = false;
|
||||
const llmBackendMode = getBackendModeName(config);
|
||||
try {
|
||||
const backend = createBackend(config);
|
||||
llmBackendAvailable = await backend.isAvailable();
|
||||
} catch {
|
||||
llmBackendAvailable = false;
|
||||
}
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
dbItemCount = getItemCount(db);
|
||||
vectorCount = getVectorCount(db);
|
||||
try {
|
||||
wikiGenCounts = getWikiGenerationCounts(db);
|
||||
} catch {
|
||||
// Gracefully handle missing content_hash column (old DB)
|
||||
}
|
||||
if (isHnswAvailable()) {
|
||||
const indexPath = HnswIndex.getDefaultIndexPath(config.db.path);
|
||||
if (HnswIndex.indexExists(indexPath)) {
|
||||
hnswStatus = "indexed";
|
||||
} else if (vectorCount > 0) {
|
||||
hnswStatus = "pending_build";
|
||||
} else {
|
||||
hnswStatus = "no_vectors";
|
||||
}
|
||||
}
|
||||
db.close();
|
||||
} catch {
|
||||
// DB not yet initialized
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
vault_path: vaultPath,
|
||||
raw_files: rawCount,
|
||||
wiki_pages: wikiCount,
|
||||
db_items: dbItemCount,
|
||||
wiki_generated: wikiGenCounts.generated,
|
||||
wiki_stale: wikiGenCounts.stale,
|
||||
wiki_ungenerated: wikiGenCounts.ungenerated,
|
||||
vector_indexed: vectorCount,
|
||||
hnsw_status: hnswStatus,
|
||||
mempalace_path: config.mempalace.path || "(not detected)",
|
||||
llm_mode: config.llm.mode,
|
||||
llm_backend: llmBackendMode,
|
||||
llm_backend_available: llmBackendAvailable,
|
||||
llm_fallback_provider: config.llm.fallback?.provider ?? null,
|
||||
last_ingest: state?.last_ingest ?? null,
|
||||
last_wiki_update: state?.last_wiki_update ?? null,
|
||||
last_llm_route: state?.last_llm_route ?? null,
|
||||
last_llm_provider: state?.last_llm_provider ?? null,
|
||||
last_llm_model: state?.last_llm_model ?? null,
|
||||
last_fallback_used: state?.last_fallback_used ?? null,
|
||||
}, null, 2);
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_lint
|
||||
*/
|
||||
export async function handleLint(args: { checks?: string[]; withLlm?: boolean }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const result = await lintWiki(config, {
|
||||
checks: args.checks as LintCheckType[] | undefined,
|
||||
withLlm: args.withLlm,
|
||||
});
|
||||
return JSON.stringify(result, null, 2);
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_ingest
|
||||
*/
|
||||
export async function handleIngest(args: { full?: boolean }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const results: { source: string; count: number }[] = [];
|
||||
|
||||
// 1. MemPalace ingest
|
||||
if (config.mempalace.enabled) {
|
||||
try {
|
||||
const ingestResult = await ingestMemPalace(config, { full: args.full ?? false });
|
||||
results.push({ source: "mempalace", count: ingestResult.drawersWritten });
|
||||
} catch (error) {
|
||||
results.push({ source: "mempalace", count: 0, error: formatError(error) } as unknown as { source: string; count: number });
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Source adapters → DB
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const adapters = createAllAdapters(config);
|
||||
for (const adapter of adapters) {
|
||||
if (!adapter.isAvailable()) continue;
|
||||
const items = await adapter.fetchItems();
|
||||
let synced = 0;
|
||||
for (const item of items) {
|
||||
upsertItem(db, item);
|
||||
synced++;
|
||||
}
|
||||
if (synced > 0) {
|
||||
results.push({ source: adapter.name, count: synced });
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
|
||||
// Update sync state
|
||||
const state = await readSyncState(config.vault.path);
|
||||
await writeSyncState(config.vault.path, {
|
||||
...state,
|
||||
last_ingest: toIsoTimestamp(),
|
||||
});
|
||||
|
||||
return JSON.stringify({ results, total_sources: results.length }, null, 2);
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_add
|
||||
*/
|
||||
export async function handleAdd(args: {
|
||||
title: string;
|
||||
content: string;
|
||||
category?: string;
|
||||
tags?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const validCategories: WikiCategory[] = ["projects", "topics", "decisions", "people", "infrastructure", "guides"];
|
||||
const category = args.category && validCategories.includes(args.category as WikiCategory)
|
||||
? (args.category as WikiCategory) : null;
|
||||
|
||||
const tags = args.tags ? args.tags.split(",").map((t) => t.trim()).filter(Boolean) : [];
|
||||
|
||||
const item = createItem(db, {
|
||||
title: args.title,
|
||||
content: args.content,
|
||||
source: "manual",
|
||||
category,
|
||||
tags,
|
||||
metadata: {},
|
||||
});
|
||||
|
||||
// Non-blocking contradiction detection after item creation
|
||||
try {
|
||||
const { detectContradictions } = await import("../wiki/contradiction.js");
|
||||
const contradictions = detectContradictions(db, item);
|
||||
if (contradictions.length > 0) {
|
||||
return JSON.stringify({
|
||||
status: "created",
|
||||
id: item.id,
|
||||
title: item.title,
|
||||
category: item.category,
|
||||
tags: item.tags,
|
||||
warnings: {
|
||||
contradictions: contradictions.length,
|
||||
message: `${contradictions.length} contradiction(s) detected with existing items`,
|
||||
},
|
||||
}, null, 2);
|
||||
}
|
||||
} catch {
|
||||
// Contradiction detection failure must not block item creation
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
status: "created", id: item.id, title: item.title, category: item.category, tags: item.tags,
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_generate
|
||||
*/
|
||||
export async function handleGenerate(args: { maxRooms?: number; force?: boolean; backend?: string }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const result = await updateWiki(config, { maxRooms: args.maxRooms, force: args.force, backend: args.backend });
|
||||
return JSON.stringify({
|
||||
files_written: result.filesWritten,
|
||||
page_slugs: result.pageSlugs,
|
||||
file_paths: result.filePaths,
|
||||
new_items: result.newItems ?? 0,
|
||||
changed_items: result.changedItems ?? 0,
|
||||
}, null, 2);
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_sync
|
||||
*/
|
||||
export async function handleSync(args: { noWiki?: boolean }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
await runSyncPipeline(config, { noWiki: args.noWiki ?? false });
|
||||
return JSON.stringify({ status: "completed" }, null, 2);
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_read
|
||||
*/
|
||||
export async function handleRead(args: { path: string }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const filePath = path.resolve(config.vault.path, args.path);
|
||||
if (!filePath.startsWith(path.resolve(config.vault.path))) {
|
||||
return JSON.stringify({ error: "Path traversal not allowed" });
|
||||
}
|
||||
const content = await readTextFileIfExists(filePath);
|
||||
if (content === undefined) {
|
||||
return JSON.stringify({ error: `File not found: ${args.path}` });
|
||||
}
|
||||
return content;
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_search
|
||||
*/
|
||||
export async function handleSearch(args: { query: string; mode?: string; limit?: number; context?: boolean }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const mode = (args.mode ?? "hybrid") as SearchMode;
|
||||
if (!["keyword", "semantic", "hybrid"].includes(mode)) {
|
||||
return JSON.stringify({ error: `Invalid mode '${mode}'. Use: keyword, semantic, or hybrid` });
|
||||
}
|
||||
|
||||
const enrich = args.context === true;
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const { results, enrichedResults, mode: usedMode, fallback } = await hybridSearch(
|
||||
db, args.query, mode, args.limit ?? 20, config.db.path, enrich,
|
||||
);
|
||||
|
||||
const response: Record<string, unknown> = {
|
||||
query: args.query,
|
||||
mode: usedMode,
|
||||
matches: results.length,
|
||||
fallback,
|
||||
};
|
||||
|
||||
if (enrich && enrichedResults) {
|
||||
response.context_enriched = true;
|
||||
response.results = enrichedResults.map((er) => ({
|
||||
id: er.item.item.id,
|
||||
title: er.item.item.title,
|
||||
source: er.item.item.source,
|
||||
category: er.item.item.category,
|
||||
snippet: er.item.snippet,
|
||||
score: er.item.rank,
|
||||
context: {
|
||||
category: er.context.category,
|
||||
tags: er.context.tags,
|
||||
relatedEntities: er.context.relatedEntities,
|
||||
linkedPages: er.context.linkedPages,
|
||||
temporal: er.context.temporalContext,
|
||||
source: er.context.sourceContext,
|
||||
},
|
||||
contextScore: er.contextScore,
|
||||
rerankPosition: er.rerankPosition,
|
||||
}));
|
||||
} else {
|
||||
response.results = results.map((r) => ({
|
||||
id: r.item.id, title: r.item.title, source: r.item.source,
|
||||
category: r.item.category, snippet: r.snippet, score: r.score, matched_by: r.sources,
|
||||
}));
|
||||
}
|
||||
|
||||
return JSON.stringify(response, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_index
|
||||
*
|
||||
* Builds vector embeddings and HNSW ANN index for semantic search.
|
||||
*/
|
||||
export async function handleIndex(args: { force?: boolean; rebuildHnsw?: boolean; rebuildCommunities?: boolean }, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
// 커뮤니티 재구축 모드
|
||||
if (args.rebuildCommunities) {
|
||||
const communityGraph = exportCommunityGraph(db);
|
||||
return JSON.stringify({
|
||||
status: "communities_rebuilt",
|
||||
communities: communityGraph.communities.length,
|
||||
edges: communityGraph.edges.length,
|
||||
total_entities: communityGraph.communities.reduce((sum, c) => sum + c.size, 0),
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
const totalItems = getItemCount(db);
|
||||
const alreadyIndexed = getVectorCount(db);
|
||||
|
||||
if (totalItems === 0) {
|
||||
return JSON.stringify({ error: "No items in DB. Run hw_ingest first." });
|
||||
}
|
||||
|
||||
// HNSW-only rebuild mode
|
||||
if (args.rebuildHnsw) {
|
||||
if (alreadyIndexed === 0) {
|
||||
return JSON.stringify({ error: "No vectors indexed. Run hw_index first to generate embeddings." });
|
||||
}
|
||||
invalidateHnswIndex();
|
||||
let hnswCount = 0;
|
||||
try {
|
||||
hnswCount = rebuildHnswIndex(db, config.db.path);
|
||||
} catch {
|
||||
return JSON.stringify({ error: "HNSW rebuild failed. hnswlib-node may not be available.", hnsw_available: isHnswAvailable() });
|
||||
}
|
||||
return JSON.stringify({
|
||||
status: "hnsw_rebuilt", vectors_indexed: hnswCount, hnsw_available: isHnswAvailable(),
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
let itemsToIndex: Item[];
|
||||
if (args.force) {
|
||||
itemsToIndex = listItems(db, totalItems);
|
||||
} else {
|
||||
const indexedIds = new Set(getIndexedItemIds(db));
|
||||
const allItems = listItems(db, totalItems);
|
||||
itemsToIndex = allItems.filter((item) => !indexedIds.has(item.id));
|
||||
}
|
||||
|
||||
if (itemsToIndex.length === 0) {
|
||||
return JSON.stringify({
|
||||
status: "already_indexed", total_items: totalItems, indexed: alreadyIndexed, new: 0,
|
||||
});
|
||||
}
|
||||
|
||||
const BATCH_SIZE = 32;
|
||||
let indexed = 0;
|
||||
for (let i = 0; i < itemsToIndex.length; i += BATCH_SIZE) {
|
||||
const batch = itemsToIndex.slice(i, i + BATCH_SIZE);
|
||||
const texts = batch.map((item) => `${item.title}\n${item.content}`);
|
||||
const vectors = await embedBatch(texts);
|
||||
for (let j = 0; j < batch.length; j++) {
|
||||
storeVector(db, batch[j].id, vectors[j]);
|
||||
indexed++;
|
||||
}
|
||||
}
|
||||
|
||||
// Invalidate and rebuild HNSW index after new embeddings
|
||||
invalidateHnswIndex();
|
||||
let hnswCount = 0;
|
||||
let hnswBuilt = false;
|
||||
try {
|
||||
hnswCount = rebuildHnswIndex(db, config.db.path);
|
||||
hnswBuilt = hnswCount > 0;
|
||||
} catch {
|
||||
// HNSW build failed — embeddings still stored in SQLite
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
status: "indexed", total_items: totalItems,
|
||||
previously_indexed: alreadyIndexed, newly_indexed: indexed,
|
||||
total_indexed: alreadyIndexed + indexed,
|
||||
hnsw_built: hnswBuilt, hnsw_indexed: hnswCount,
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_graph
|
||||
*
|
||||
* 다양한 모드로 지식 그래프를 질의한다:
|
||||
* - 인자 없음 → 통계 + 개요
|
||||
* - entity → 엔티티 상세 + 이웃
|
||||
* - export → 전체 그래프 JSON
|
||||
* - search → 엔티티 검색
|
||||
*/
|
||||
export async function handleGraph(args: {
|
||||
entity?: string;
|
||||
depth?: number;
|
||||
export?: boolean;
|
||||
search?: string;
|
||||
mode?: string;
|
||||
format?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
// communities 모드
|
||||
if (args.mode === "communities") {
|
||||
const communityGraph = exportCommunityGraph(db);
|
||||
const result: Record<string, unknown> = {
|
||||
mode: "communities",
|
||||
communities: communityGraph.communities.length,
|
||||
edges: communityGraph.edges.length,
|
||||
total_entities: communityGraph.communities.reduce((sum, c) => sum + c.size, 0),
|
||||
graph: communityGraph,
|
||||
};
|
||||
if (args.format === "mermaid") {
|
||||
result.mermaid = exportMermaidCommunities(db);
|
||||
}
|
||||
return JSON.stringify(result, null, 2);
|
||||
}
|
||||
|
||||
// mermaid format 모드 (entity 또는 전체)
|
||||
if (args.format === "mermaid") {
|
||||
const depth = args.depth ?? 2;
|
||||
const mermaid = exportMermaidGraph(db, args.entity, { maxNodes: 20, depth });
|
||||
return JSON.stringify({ format: "mermaid", entity: args.entity ?? null, mermaid }, null, 2);
|
||||
}
|
||||
|
||||
// export 모드
|
||||
if (args.export) {
|
||||
const result = exportGraph(db);
|
||||
return JSON.stringify({
|
||||
exported: true,
|
||||
nodes: result.nodes.length,
|
||||
edges: result.edges.length,
|
||||
graph: result,
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
// search 모드
|
||||
if (args.search) {
|
||||
const results = searchEntities(db, args.search);
|
||||
return JSON.stringify({
|
||||
query: args.search,
|
||||
matches: results.length,
|
||||
entities: results.map((e) => ({
|
||||
id: e.id,
|
||||
name: e.name,
|
||||
type: e.type,
|
||||
wikiSlug: e.wikiSlug,
|
||||
relationCount: e.outgoingRelations + e.incomingRelations,
|
||||
})),
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
// entity 모드
|
||||
if (args.entity) {
|
||||
const depth = args.depth ?? 1;
|
||||
const entity = getEntity(db, args.entity);
|
||||
if (!entity) {
|
||||
// slug로 정확히 매칭 안 되면 검색 시도
|
||||
const found = searchEntities(db, args.entity, 5);
|
||||
if (found.length === 0) {
|
||||
return JSON.stringify({ error: `Entity not found: ${args.entity}` });
|
||||
}
|
||||
const matched = found[0];
|
||||
const subgraph = getEntityNeighbors(db, matched.id, depth);
|
||||
return JSON.stringify({
|
||||
entity: matched,
|
||||
depth,
|
||||
neighbors: subgraph.entities.length,
|
||||
relations: subgraph.relations.length,
|
||||
graph: {
|
||||
nodes: subgraph.entities.map((e) => ({
|
||||
id: e.id, name: e.name, type: e.type,
|
||||
relationCount: e.outgoingRelations + e.incomingRelations,
|
||||
})),
|
||||
edges: subgraph.relations.map((r) => ({
|
||||
source: r.subjectId, target: r.objectId,
|
||||
predicate: r.predicate, confidence: r.confidence,
|
||||
})),
|
||||
},
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
const subgraph = getEntityNeighbors(db, entity.id, depth);
|
||||
return JSON.stringify({
|
||||
entity,
|
||||
depth,
|
||||
neighbors: subgraph.entities.length,
|
||||
relations: subgraph.relations.length,
|
||||
graph: {
|
||||
nodes: subgraph.entities.map((e) => ({
|
||||
id: e.id, name: e.name, type: e.type,
|
||||
relationCount: e.outgoingRelations + e.incomingRelations,
|
||||
})),
|
||||
edges: subgraph.relations.map((r) => ({
|
||||
source: r.subjectId, target: r.objectId,
|
||||
predicate: r.predicate, confidence: r.confidence,
|
||||
})),
|
||||
},
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
// 기본: 통계 + 최근 엔티티
|
||||
const stats = getGraphStats(db);
|
||||
const recentEntities = searchEntities(db, "*", 10);
|
||||
return JSON.stringify({
|
||||
stats,
|
||||
recentEntities: recentEntities.map((e) => ({
|
||||
id: e.id, name: e.name, type: e.type,
|
||||
relationCount: e.outgoingRelations + e.incomingRelations,
|
||||
})),
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_entity_add
|
||||
*
|
||||
* 지식 그래프에 엔티티를 추가한다.
|
||||
*/
|
||||
export async function handleEntityAdd(args: {
|
||||
id: string;
|
||||
name: string;
|
||||
type?: string;
|
||||
wikiSlug?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const entity = addEntity(db, {
|
||||
id: args.id,
|
||||
name: args.name,
|
||||
type: (args.type as "concept" | "project" | "person" | "tool" | "event" | "decision") ?? "concept",
|
||||
wikiSlug: args.wikiSlug,
|
||||
});
|
||||
return JSON.stringify({ status: "created", entity }, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_entity_search
|
||||
*
|
||||
* 엔티티를 검색한다.
|
||||
*/
|
||||
export async function handleEntitySearch(args: {
|
||||
query: string;
|
||||
limit?: number;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const results = searchEntities(db, args.query, args.limit ?? 20);
|
||||
return JSON.stringify({
|
||||
query: args.query,
|
||||
matches: results.length,
|
||||
entities: results.map((e) => ({
|
||||
id: e.id,
|
||||
name: e.name,
|
||||
type: e.type,
|
||||
wikiSlug: e.wikiSlug,
|
||||
relationCount: e.outgoingRelations + e.incomingRelations,
|
||||
})),
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_relation_add
|
||||
*
|
||||
* 지식 그래프에 관계를 추가한다.
|
||||
*/
|
||||
export async function handleRelationAdd(args: {
|
||||
subjectId: string;
|
||||
predicate: string;
|
||||
objectId: string;
|
||||
confidence?: number;
|
||||
source?: string;
|
||||
evidence?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const relationId = addRelation(db, {
|
||||
subjectId: args.subjectId,
|
||||
predicate: args.predicate,
|
||||
objectId: args.objectId,
|
||||
confidence: args.confidence,
|
||||
source: (args.source as "extracted" | "inferred" | "ambiguous") ?? "extracted",
|
||||
evidence: args.evidence,
|
||||
});
|
||||
return JSON.stringify({
|
||||
status: "created",
|
||||
relationId,
|
||||
subject: args.subjectId,
|
||||
predicate: args.predicate,
|
||||
object: args.objectId,
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Handler: hw_daily_log
|
||||
*
|
||||
* 일일 작업 로그를 생성하거나 반환한다.
|
||||
* - date 지정 시 해당 날짜 로그
|
||||
* - start/end 지정 시 기간 로그
|
||||
* - 인자 없으면 오늘 날짜 로그
|
||||
*/
|
||||
export async function handleDailyLog(args: {
|
||||
date?: string;
|
||||
start?: string;
|
||||
end?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
let markdown: string;
|
||||
let filePath: string | undefined;
|
||||
|
||||
if (args.start && args.end) {
|
||||
// Range mode
|
||||
markdown = generateDailyLogRange(db, args.start, args.end);
|
||||
} else {
|
||||
// Single day mode
|
||||
const targetDate = args.date ?? toDateString(new Date());
|
||||
markdown = generateDailyLog(db, targetDate);
|
||||
// Also write the file
|
||||
filePath = await writeDailyLog(db, config.vault.path, targetDate);
|
||||
}
|
||||
|
||||
return JSON.stringify({
|
||||
status: "generated",
|
||||
date: args.start && args.end ? `${args.start}~${args.end}` : (args.date ?? toDateString(new Date())),
|
||||
file_path: filePath ?? null,
|
||||
content: markdown,
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Handler: hw_synthesize
|
||||
*
|
||||
* 질문-답변을 위키 합성 페이지로 저장한다. 지식이 복리로 쌓이는 Karpathy LLM Wiki 패턴.
|
||||
*
|
||||
* 동작 순서:
|
||||
* 1. 관련 기존 아이템 검색 (hybrid search)
|
||||
* 2. DB에 synthesis 아이템 생성
|
||||
* 3. 위키 마크다운 페이지 렌더링 및 저장
|
||||
* 4. KG에 엔티티/관계 생성 (derived_from, answers)
|
||||
* 5. 결과 반환
|
||||
*/
|
||||
export async function handleSynthesize(args: {
|
||||
question: string;
|
||||
answer: string;
|
||||
tags?: string[];
|
||||
source_item_ids?: string[];
|
||||
category?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const validCategories: WikiCategory[] = ["projects", "topics", "decisions", "people", "infrastructure", "guides"];
|
||||
const category = args.category && validCategories.includes(args.category as WikiCategory)
|
||||
? (args.category as WikiCategory) : "topics" as WikiCategory;
|
||||
|
||||
// 1. 관련 아이템 검색
|
||||
const relatedItems = await findRelatedItems(db, args.question, 5, config.db.path);
|
||||
|
||||
// 2. synthesis 아이템 생성
|
||||
const itemId = createSynthesis(db, {
|
||||
question: args.question,
|
||||
answer: args.answer,
|
||||
sourceItemIds: args.source_item_ids,
|
||||
tags: args.tags,
|
||||
category,
|
||||
});
|
||||
|
||||
// Get the created item to access metadata
|
||||
const createdItem = getItem(db, itemId);
|
||||
if (!createdItem) {
|
||||
return JSON.stringify({ error: "Failed to create synthesis item" });
|
||||
}
|
||||
|
||||
const slug = generateSynthesisSlug(args.question);
|
||||
|
||||
// 3. 소스 아이템의 wikiSlug 수집
|
||||
const sourceSlugs: string[] = [];
|
||||
if (args.source_item_ids) {
|
||||
for (const sourceId of args.source_item_ids) {
|
||||
const sourceItem = getItem(db, sourceId);
|
||||
if (sourceItem?.wikiSlug) {
|
||||
sourceSlugs.push(sourceItem.wikiSlug);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 4. 위키 마크다운 페이지 렌더링
|
||||
const wikiContent = renderSynthesisWikiPage(
|
||||
args.question,
|
||||
args.answer,
|
||||
slug,
|
||||
args.tags ?? [],
|
||||
category,
|
||||
(createdItem.metadata as Record<string, unknown>).confidence as number ?? 0.8,
|
||||
sourceSlugs,
|
||||
relatedItems.map((item) => ({ title: item.title, wikiSlug: item.wikiSlug ?? null })),
|
||||
);
|
||||
|
||||
// 5. 파일 저장
|
||||
const wikiDir = path.join(config.vault.path, "wiki", category);
|
||||
const wikiFilePath = path.join(wikiDir, `${slug}.md`);
|
||||
await writeTextFile(wikiFilePath, wikiContent);
|
||||
|
||||
// 6. KG 관계 생성
|
||||
createSynthesisKGRelations(
|
||||
db,
|
||||
slug,
|
||||
createdItem.title,
|
||||
args.source_item_ids ?? [],
|
||||
relatedItems,
|
||||
);
|
||||
|
||||
return JSON.stringify({
|
||||
status: "synthesized",
|
||||
item_id: itemId,
|
||||
slug,
|
||||
wiki_path: path.relative(config.vault.path, wikiFilePath),
|
||||
related_items: relatedItems.length,
|
||||
tags: args.tags ?? [],
|
||||
category,
|
||||
}, null, 2);
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Helpers (internal) ────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Handler: hw_contradictions
|
||||
*
|
||||
* Detect and manage contradictions/conflicts in the knowledge base.
|
||||
*/
|
||||
export async function handleContradictions(args: {
|
||||
action?: string;
|
||||
id?: number;
|
||||
status?: string;
|
||||
resolution?: string;
|
||||
resolve_as?: string;
|
||||
}, config: WikiEngineConfig): Promise<string> {
|
||||
try {
|
||||
const action = args.action ?? "list";
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
switch (action) {
|
||||
case "list": {
|
||||
const contradictions = getContradictions(db, args.status);
|
||||
return JSON.stringify({
|
||||
action: "list",
|
||||
status_filter: args.status ?? "all",
|
||||
count: contradictions.length,
|
||||
contradictions: contradictions.map((c) => ({
|
||||
id: c.id,
|
||||
item_a: c.item_a_id,
|
||||
item_b: c.item_b_id,
|
||||
field: c.field,
|
||||
value_a: c.value_a,
|
||||
value_b: c.value_b,
|
||||
severity: c.severity,
|
||||
status: c.status,
|
||||
detected_at: c.detected_at,
|
||||
})),
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
case "get": {
|
||||
if (!args.id) {
|
||||
return JSON.stringify({ error: "id is required for get action" });
|
||||
}
|
||||
const contradiction = getContradictionById(db, args.id);
|
||||
if (!contradiction) {
|
||||
return JSON.stringify({ error: `Contradiction ${args.id} not found` });
|
||||
}
|
||||
return JSON.stringify({ action: "get", contradiction }, null, 2);
|
||||
}
|
||||
|
||||
case "resolve": {
|
||||
if (!args.id) {
|
||||
return JSON.stringify({ error: "id is required for resolve action" });
|
||||
}
|
||||
if (!args.resolution) {
|
||||
return JSON.stringify({ error: "resolution description is required for resolve action" });
|
||||
}
|
||||
const resolveStatus = args.resolve_as === "false_positive" ? "false_positive" : "resolved";
|
||||
const updated = resolveContradiction(db, args.id, {
|
||||
resolution: args.resolution,
|
||||
status: resolveStatus,
|
||||
});
|
||||
if (!updated) {
|
||||
return JSON.stringify({ error: `Contradiction ${args.id} not found` });
|
||||
}
|
||||
return JSON.stringify({
|
||||
action: "resolve",
|
||||
status: "resolved",
|
||||
contradiction: updated,
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
case "scan": {
|
||||
const newCount = scanAllContradictions(db);
|
||||
const stats = getContradictionStats(db);
|
||||
return JSON.stringify({
|
||||
action: "scan",
|
||||
new_contradictions: newCount,
|
||||
stats,
|
||||
}, null, 2);
|
||||
}
|
||||
|
||||
case "stats": {
|
||||
const stats = getContradictionStats(db);
|
||||
return JSON.stringify({ action: "stats", stats }, null, 2);
|
||||
}
|
||||
|
||||
default:
|
||||
return JSON.stringify({ error: `Unknown action '${action}'. Use: list, get, resolve, scan, stats` });
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
} catch (error) {
|
||||
return JSON.stringify({ error: formatError(error) });
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Original Helpers (internal) ────────────────────────────────────────────
|
||||
|
||||
function countFilesRecursive(dir: string): number {
|
||||
let count = 0;
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name.startsWith(".")) continue;
|
||||
if (entry.isDirectory()) {
|
||||
count += countFilesRecursive(path.join(dir, entry.name));
|
||||
} else if (entry.name.endsWith(".md")) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
function formatError(error: unknown): string {
|
||||
return error instanceof Error ? error.message : String(error);
|
||||
}
|
||||
422
src/mcp/tools.ts
Normal file
422
src/mcp/tools.ts
Normal file
@@ -0,0 +1,422 @@
|
||||
/**
|
||||
* MCP tool schema definitions for habraid v2.
|
||||
*
|
||||
* All tools use the `hw_` prefix to avoid conflicts with other MCP servers.
|
||||
* Maps to handlers in {@link module:mcp/handlers}.
|
||||
*/
|
||||
|
||||
/** Single MCP tool definition. */
|
||||
export interface McpToolDef {
|
||||
name: string;
|
||||
description: string;
|
||||
inputSchema: {
|
||||
type: "object";
|
||||
properties?: Record<string, unknown>;
|
||||
required?: string[];
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* All tools exposed by the habraid MCP server.
|
||||
*/
|
||||
export const WIKI_TOOLS: McpToolDef[] = [
|
||||
{
|
||||
name: "hw_status",
|
||||
description:
|
||||
"Show vault and database statistics — raw file count, wiki page count, DB items, " +
|
||||
"last sync timestamps. Use this to quickly check the health and freshness of the wiki.",
|
||||
inputSchema: { type: "object", properties: {} },
|
||||
},
|
||||
{
|
||||
name: "hw_lint",
|
||||
description:
|
||||
"Run comprehensive wiki lint checks. Supports orphan pages, broken wikilinks, stale pages, " +
|
||||
"ungenerated items, and frontmatter validation. Returns a structured lint report.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
checks: {
|
||||
type: "array",
|
||||
items: { type: "string" },
|
||||
description: "Specific checks to run: orphans, broken_links, stale, ungenerated, frontmatter, duplicates, contradictions",
|
||||
},
|
||||
withLlm: {
|
||||
type: "boolean",
|
||||
description: "Include LLM-based contradiction checks when available (default: false)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_ingest",
|
||||
description:
|
||||
"Ingest data from all configured sources into raw/ and the local DB. " +
|
||||
"MemPalace is one source; others can be added. Set full=true to re-ingest everything.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
full: {
|
||||
type: "boolean",
|
||||
description: "Re-ingest all items, not just new ones (default: false)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_add",
|
||||
description:
|
||||
"Add a knowledge item directly to the local database. " +
|
||||
"Provide at minimum a title and content. Optionally specify category and tags.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
title: {
|
||||
type: "string",
|
||||
description: "Item title",
|
||||
},
|
||||
content: {
|
||||
type: "string",
|
||||
description: "Item content (markdown)",
|
||||
},
|
||||
category: {
|
||||
type: "string",
|
||||
description: "Category: projects, topics, decisions, people, infrastructure, guides",
|
||||
},
|
||||
tags: {
|
||||
type: "string",
|
||||
description: "Comma-separated tags",
|
||||
},
|
||||
},
|
||||
required: ["title", "content"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_search",
|
||||
description:
|
||||
"Search knowledge items using hybrid search (BM25 keyword + vector semantic). " +
|
||||
"Modes: 'hybrid' (default) combines both with RRF fusion, 'keyword' for FTS5 only, " +
|
||||
"'semantic' for vector similarity only. Returns ranked results with snippets. " +
|
||||
"Set context=true to enrich results with hierarchical metadata from the knowledge graph " +
|
||||
"and wiki structure, and re-rank based on context coherence.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
query: {
|
||||
type: "string",
|
||||
description: "Search query — keywords or natural language",
|
||||
},
|
||||
mode: {
|
||||
type: "string",
|
||||
description: "Search mode: hybrid (default), keyword, or semantic",
|
||||
},
|
||||
limit: {
|
||||
type: "number",
|
||||
description: "Max results (default: 20)",
|
||||
},
|
||||
context: {
|
||||
type: "boolean",
|
||||
description: "Enable context enrichment: adds KG relations, wiki links, category hierarchy, temporal/source context, and re-ranks results by context coherence (default: false)",
|
||||
},
|
||||
},
|
||||
required: ["query"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_index",
|
||||
description:
|
||||
"Build vector embeddings for all items in the DB. " +
|
||||
"Required before semantic/hybrid search works. " +
|
||||
"Uses fastembed (ONNX) — Python 3 + pip install fastembed needed. " +
|
||||
"Set force=true to re-index everything.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
force: {
|
||||
type: "boolean",
|
||||
description: "Re-index all items, not just new ones (default: false)",
|
||||
},
|
||||
rebuildCommunities: {
|
||||
type: "boolean",
|
||||
description: "Rebuild community detection (LPA) for KG entities (default: false)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_generate",
|
||||
description:
|
||||
"Generate or update wiki pages from items using LLM. " +
|
||||
"Groups items by category, sends them to the LLM for analysis, " +
|
||||
"and creates or updates wiki pages under wiki/. " +
|
||||
"Detects changed items via content hashing and regenerates stale pages. " +
|
||||
"Use force=true to regenerate all pages. Use backend to override the LLM backend " +
|
||||
"(host, openai, ollama, zai). Requires LLM API access.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
force: {
|
||||
type: "boolean",
|
||||
description: "Regenerate all wiki pages, not just new/changed ones (default: false)",
|
||||
},
|
||||
backend: {
|
||||
type: "string",
|
||||
description: "Override the LLM backend for this request: host, openai, ollama, or zai",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_sync",
|
||||
description:
|
||||
"Run the full sync pipeline: git pull → ingest → wiki generate → git commit → git push. " +
|
||||
"This is the main command for keeping the wiki up to date. " +
|
||||
"Set noWiki=true to skip the LLM wiki generation step.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
noWiki: {
|
||||
type: "boolean",
|
||||
description: "Skip wiki generation, only ingest and sync raw files (default: false)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_read",
|
||||
description:
|
||||
"Read the content of a wiki page by its relative path under the vault. " +
|
||||
"Returns the full markdown content including frontmatter. " +
|
||||
'Example paths: "wiki/projects/project-beta.md", "raw/mempalace/infrastructure/server-config.md"',
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
path: {
|
||||
type: "string",
|
||||
description: "Relative file path within the vault",
|
||||
},
|
||||
},
|
||||
required: ["path"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_graph",
|
||||
description:
|
||||
"Query the knowledge graph. Supports multiple modes: " +
|
||||
"(1) no args → stats + overview, " +
|
||||
"(2) entity=<name> → entity details + neighbors (BFS subgraph), " +
|
||||
"(3) export=true → full graph JSON export, " +
|
||||
"(5) search=<query> → search entities by name/id/type, " +
|
||||
"(6) mode=communities → community detection graph with LPA clustering. " +
|
||||
"Use format=mermaid to get Mermaid diagram output instead of JSON.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
entity: {
|
||||
type: "string",
|
||||
description: "Entity name or ID to query (returns subgraph)",
|
||||
},
|
||||
depth: {
|
||||
type: "number",
|
||||
description: "Max traversal depth for subgraph (default: 1)",
|
||||
},
|
||||
export: {
|
||||
type: "boolean",
|
||||
description: "Export full graph as JSON (default: false)",
|
||||
},
|
||||
search: {
|
||||
type: "string",
|
||||
description: "Search entities by name or ID",
|
||||
},
|
||||
mode: {
|
||||
type: "string",
|
||||
description: "Query mode: 'communities' for community detection graph with LPA clustering",
|
||||
},
|
||||
format: {
|
||||
type: "string",
|
||||
description: "Output format: 'json' (default) or 'mermaid' for Mermaid diagram string",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_entity_add",
|
||||
description:
|
||||
"Add an entity to the knowledge graph. Entities are nodes identified by a slug ID " +
|
||||
"(e.g. 'nestjs-circular-dep'). Types: concept, project, person, tool, event, decision.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
id: {
|
||||
type: "string",
|
||||
description: "Entity ID in slug format (e.g. 'nestjs-circular-dep')",
|
||||
},
|
||||
name: {
|
||||
type: "string",
|
||||
description: "Display name (e.g. 'NestJS 순환참조')",
|
||||
},
|
||||
type: {
|
||||
type: "string",
|
||||
description: "Entity type: concept, project, person, tool, event, decision (default: concept)",
|
||||
},
|
||||
wikiSlug: {
|
||||
type: "string",
|
||||
description: "Link to an existing wiki page slug",
|
||||
},
|
||||
},
|
||||
required: ["id", "name"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_entity_search",
|
||||
description:
|
||||
"Search entities in the knowledge graph by name, ID, or type. " +
|
||||
"Returns matching entities with relation counts.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
query: {
|
||||
type: "string",
|
||||
description: "Search query — matches against entity name, ID, or type",
|
||||
},
|
||||
limit: {
|
||||
type: "number",
|
||||
description: "Max results (default: 20)",
|
||||
},
|
||||
},
|
||||
required: ["query"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_relation_add",
|
||||
description:
|
||||
"Add a relation between two entities in the knowledge graph. " +
|
||||
"Common predicates: fixes, depends_on, related_to, implements, creates, uses, blocks. " +
|
||||
"Source types: extracted (from content), inferred (by logic), ambiguous (needs review).",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
subjectId: {
|
||||
type: "string",
|
||||
description: "Source entity ID (slug)",
|
||||
},
|
||||
predicate: {
|
||||
type: "string",
|
||||
description: "Relationship predicate (e.g. 'fixes', 'depends_on', 'related_to')",
|
||||
},
|
||||
objectId: {
|
||||
type: "string",
|
||||
description: "Target entity ID (slug)",
|
||||
},
|
||||
confidence: {
|
||||
type: "number",
|
||||
description: "Confidence score 0.0-1.0 (default: 1.0)",
|
||||
},
|
||||
source: {
|
||||
type: "string",
|
||||
description: "Source type: extracted, inferred, or ambiguous (default: extracted)",
|
||||
},
|
||||
evidence: {
|
||||
type: "string",
|
||||
description: "Brief description of why this relation exists",
|
||||
},
|
||||
},
|
||||
required: ["subjectId", "predicate", "objectId"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_daily_log",
|
||||
description:
|
||||
"Generate or read a daily work log summarizing wiki generation activity. " +
|
||||
"Without parameters, returns today's log. " +
|
||||
"Provide 'date' for a specific day (YYYY-MM-DD), or 'start'/'end' for a date range. " +
|
||||
"Logs are written to vault/daily/YYYY-MM-DD.md with Obsidian-compatible backlinks.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
date: {
|
||||
type: "string",
|
||||
description: "Specific date to generate log for (YYYY-MM-DD). Defaults to today.",
|
||||
},
|
||||
start: {
|
||||
type: "string",
|
||||
description: "Range start date (YYYY-MM-DD) — use with 'end' for multi-day summary.",
|
||||
},
|
||||
end: {
|
||||
type: "string",
|
||||
description: "Range end date (YYYY-MM-DD) — use with 'start' for multi-day summary.",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_synthesize",
|
||||
description:
|
||||
"질문-답변을 위키 합성 페이지로 저장. 지식이 복리로 쌓임. " +
|
||||
"Karpathy LLM Wiki 패턴: 좋은 Q&A를 위키에 등록하여 누적 지식 베이스를 구축. " +
|
||||
"관련 기존 아이템을 자동 검색하여 링크하고, KG에 관계를 생성함.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
question: {
|
||||
type: "string",
|
||||
description: "원래 질문",
|
||||
},
|
||||
answer: {
|
||||
type: "string",
|
||||
description: "답변 내용",
|
||||
},
|
||||
tags: {
|
||||
type: "array",
|
||||
items: { type: "string" },
|
||||
description: "태그 목록",
|
||||
},
|
||||
source_item_ids: {
|
||||
type: "array",
|
||||
items: { type: "string" },
|
||||
description: "이 합성의 근거가 된 소스 아이템 ID 목록",
|
||||
},
|
||||
category: {
|
||||
type: "string",
|
||||
description: "위키 카테고리 (기본값: topics)",
|
||||
},
|
||||
},
|
||||
required: ["question", "answer"],
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "hw_contradictions",
|
||||
description:
|
||||
"Detect and manage contradictions/conflicts in the knowledge base. " +
|
||||
"Supports multiple actions: " +
|
||||
"(1) action=list — list contradictions (optionally filter by status), " +
|
||||
"(2) action=get — get a specific contradiction by ID, " +
|
||||
"(3) action=resolve — mark a contradiction as resolved or false_positive, " +
|
||||
"(4) action=scan — run a full contradiction scan across all items, " +
|
||||
"(5) action=stats — show contradiction statistics.",
|
||||
inputSchema: {
|
||||
type: "object",
|
||||
properties: {
|
||||
action: {
|
||||
type: "string",
|
||||
description: "Action: list, get, resolve, scan, or stats (default: list)",
|
||||
},
|
||||
id: {
|
||||
type: "number",
|
||||
description: "Contradiction ID (required for get and resolve actions)",
|
||||
},
|
||||
status: {
|
||||
type: "string",
|
||||
description: "Filter by status: open, resolved, false_positive (for list action)",
|
||||
},
|
||||
resolution: {
|
||||
type: "string",
|
||||
description: "Resolution description (required for resolve action)",
|
||||
},
|
||||
resolve_as: {
|
||||
type: "string",
|
||||
description: "Resolution status: resolved or false_positive (default: resolved)",
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
];
|
||||
97
src/mempalace/ingest.ts
Normal file
97
src/mempalace/ingest.ts
Normal file
@@ -0,0 +1,97 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { DbReadError } from "../errors.js";
|
||||
import type { IngestResult, SyncLogEntry, SyncState, WikiEngineConfig } from "../types.js";
|
||||
import { appendSyncLog } from "../vault/log.js";
|
||||
import { updateVaultIndex } from "../vault/index.js";
|
||||
import { getRawDrawerFileName, renderRawDrawerMarkdown } from "../vault/render.js";
|
||||
import { SqliteMemPalaceReader } from "./reader.js";
|
||||
import { pathExists, readSyncState, toIsoTimestamp, writeSyncState, writeTextFile } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Options for an ingest run.
|
||||
*/
|
||||
export interface IngestOptions {
|
||||
full?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ingests new MemPalace drawers into immutable raw markdown files.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @param options Ingest options.
|
||||
* @returns Ingest summary.
|
||||
*/
|
||||
export async function ingestMemPalace(
|
||||
config: WikiEngineConfig,
|
||||
options: IngestOptions = {},
|
||||
): Promise<IngestResult> {
|
||||
const reader = new SqliteMemPalaceReader();
|
||||
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
const dbPath = path.join(config.mempalace.path, "chroma.sqlite3");
|
||||
const state = await readSyncState(vaultPath);
|
||||
|
||||
await reader.open(dbPath);
|
||||
|
||||
const drawers =
|
||||
options.full || !state.last_ingest
|
||||
? await reader.getAllDrawers()
|
||||
: await reader.getDrawersSince(new Date(state.last_ingest));
|
||||
|
||||
const rawFilePaths: string[] = [];
|
||||
const writtenIds: string[] = [];
|
||||
const skippedExisting: string[] = [];
|
||||
|
||||
for (const drawer of drawers) {
|
||||
const outputPath = path.join(
|
||||
vaultPath,
|
||||
"raw",
|
||||
"mempalace",
|
||||
drawer.wing,
|
||||
drawer.room,
|
||||
getRawDrawerFileName(drawer.id),
|
||||
);
|
||||
|
||||
if (await pathExists(outputPath)) {
|
||||
skippedExisting.push(drawer.id);
|
||||
rawFilePaths.push(outputPath);
|
||||
continue;
|
||||
}
|
||||
|
||||
await writeTextFile(outputPath, renderRawDrawerMarkdown(drawer));
|
||||
rawFilePaths.push(outputPath);
|
||||
writtenIds.push(drawer.id);
|
||||
}
|
||||
|
||||
const nextState: SyncState = {
|
||||
...state,
|
||||
last_ingest: toIsoTimestamp(),
|
||||
ingested_drawers: [...new Set([...state.ingested_drawers, ...writtenIds])],
|
||||
};
|
||||
|
||||
await writeSyncState(vaultPath, nextState);
|
||||
await updateVaultIndex(vaultPath, nextState);
|
||||
|
||||
const logEntry: SyncLogEntry = {
|
||||
time: new Date().toLocaleString("sv-SE", { timeZone: config.sync.timezone }).replace("T", " "),
|
||||
action: "ingest",
|
||||
target: "mempalace -> raw",
|
||||
result: `+${writtenIds.length} drawers`,
|
||||
};
|
||||
await appendSyncLog(vaultPath, [logEntry]);
|
||||
|
||||
return {
|
||||
drawersProcessed: drawers.length,
|
||||
drawersWritten: writtenIds.length,
|
||||
drawerIds: writtenIds,
|
||||
rawFilePaths,
|
||||
skippedExisting,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new DbReadError("MemPalace ingest failed.", error as Error);
|
||||
} finally {
|
||||
reader.close();
|
||||
}
|
||||
}
|
||||
222
src/mempalace/reader.ts
Normal file
222
src/mempalace/reader.ts
Normal file
@@ -0,0 +1,222 @@
|
||||
import Database from "better-sqlite3";
|
||||
import path from "node:path";
|
||||
|
||||
import { DbReadError } from "../errors.js";
|
||||
import type { KGFact, MemPalaceDrawer } from "../types.js";
|
||||
import { extractTags } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Public reader contract for MemPalace storage.
|
||||
*/
|
||||
export interface MemPalaceReader {
|
||||
/**
|
||||
* Opens the SQLite database in read-only mode.
|
||||
*
|
||||
* @param dbPath SQLite file path.
|
||||
*/
|
||||
open(dbPath: string): Promise<void>;
|
||||
|
||||
/**
|
||||
* Loads every drawer row.
|
||||
*
|
||||
* @returns All drawers.
|
||||
*/
|
||||
getAllDrawers(): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
/**
|
||||
* Loads drawers updated since a point in time.
|
||||
*
|
||||
* @param since Timestamp threshold.
|
||||
* @returns Matching drawers.
|
||||
*/
|
||||
getDrawersSince(since: Date): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
/**
|
||||
* Loads drawers for a wing and optional room.
|
||||
*
|
||||
* @param wing Wing name.
|
||||
* @param room Optional room name.
|
||||
* @returns Matching drawers.
|
||||
*/
|
||||
getDrawersByWingRoom(wing: string, room?: string): Promise<MemPalaceDrawer[]>;
|
||||
|
||||
/**
|
||||
* Loads all knowledge-graph facts.
|
||||
*
|
||||
* @returns All facts.
|
||||
*/
|
||||
getAllFacts(): Promise<KGFact[]>;
|
||||
|
||||
/**
|
||||
* Closes the database handle.
|
||||
*/
|
||||
close(): void;
|
||||
}
|
||||
|
||||
/**
|
||||
* Better-sqlite3 backed MemPalace reader.
|
||||
*/
|
||||
export class SqliteMemPalaceReader implements MemPalaceReader {
|
||||
private db?: Database.Database;
|
||||
|
||||
/**
|
||||
* Opens the configured database in read-only mode.
|
||||
*
|
||||
* @param dbPath SQLite file path.
|
||||
*/
|
||||
async open(dbPath: string): Promise<void> {
|
||||
try {
|
||||
this.db = new Database(path.resolve(dbPath), {
|
||||
readonly: true,
|
||||
fileMustExist: true,
|
||||
});
|
||||
this.db.pragma("journal_mode = WAL");
|
||||
} catch (error) {
|
||||
throw new DbReadError(`Failed to open MemPalace DB at ${dbPath}.`, error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads every drawer row from the database.
|
||||
*
|
||||
* @returns All drawers.
|
||||
*/
|
||||
async getAllDrawers(): Promise<MemPalaceDrawer[]> {
|
||||
try {
|
||||
return this.queryDrawers(
|
||||
`SELECT id, wing, room, content, added_by, source_file, created_at, updated_at
|
||||
FROM drawers
|
||||
ORDER BY datetime(updated_at) ASC`,
|
||||
);
|
||||
} catch (error) {
|
||||
throw new DbReadError("Failed to query all drawers.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads drawers updated after the provided date.
|
||||
*
|
||||
* @param since Threshold time.
|
||||
* @returns Matching drawers.
|
||||
*/
|
||||
async getDrawersSince(since: Date): Promise<MemPalaceDrawer[]> {
|
||||
try {
|
||||
return this.queryDrawers(
|
||||
`SELECT id, wing, room, content, added_by, source_file, created_at, updated_at
|
||||
FROM drawers
|
||||
WHERE datetime(updated_at) >= datetime(?)
|
||||
ORDER BY datetime(updated_at) ASC`,
|
||||
since.toISOString(),
|
||||
);
|
||||
} catch (error) {
|
||||
throw new DbReadError("Failed to query drawers since timestamp.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads drawers filtered by wing and optional room.
|
||||
*
|
||||
* @param wing Wing name.
|
||||
* @param room Optional room name.
|
||||
* @returns Matching drawers.
|
||||
*/
|
||||
async getDrawersByWingRoom(wing: string, room?: string): Promise<MemPalaceDrawer[]> {
|
||||
try {
|
||||
if (room) {
|
||||
return this.queryDrawers(
|
||||
`SELECT id, wing, room, content, added_by, source_file, created_at, updated_at
|
||||
FROM drawers
|
||||
WHERE wing = ? AND room = ?
|
||||
ORDER BY datetime(updated_at) ASC`,
|
||||
wing,
|
||||
room,
|
||||
);
|
||||
}
|
||||
|
||||
return this.queryDrawers(
|
||||
`SELECT id, wing, room, content, added_by, source_file, created_at, updated_at
|
||||
FROM drawers
|
||||
WHERE wing = ?
|
||||
ORDER BY datetime(updated_at) ASC`,
|
||||
wing,
|
||||
);
|
||||
} catch (error) {
|
||||
throw new DbReadError("Failed to query drawers by wing/room.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads every knowledge-graph fact from SQLite.
|
||||
*
|
||||
* @returns All facts.
|
||||
*/
|
||||
async getAllFacts(): Promise<KGFact[]> {
|
||||
try {
|
||||
const db = this.getDb();
|
||||
const rows = db
|
||||
.prepare(
|
||||
`SELECT id, subject, predicate, object, valid_from, valid_to, source_closet, created_at
|
||||
FROM kg_facts
|
||||
ORDER BY id ASC`,
|
||||
)
|
||||
.all() as Array<Record<string, unknown>>;
|
||||
|
||||
return rows.map((row) => ({
|
||||
id: Number(row.id),
|
||||
subject: String(row.subject),
|
||||
predicate: String(row.predicate),
|
||||
object: String(row.object),
|
||||
validFrom: row.valid_from ? String(row.valid_from) : null,
|
||||
validTo: row.valid_to ? String(row.valid_to) : null,
|
||||
sourceCloset: row.source_closet ? String(row.source_closet) : null,
|
||||
createdAt: String(row.created_at),
|
||||
}));
|
||||
} catch (error) {
|
||||
throw new DbReadError("Failed to query knowledge graph facts.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Closes the database connection if it is open.
|
||||
*/
|
||||
close(): void {
|
||||
this.db?.close();
|
||||
this.db = undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a drawer query and maps rows into application types.
|
||||
*
|
||||
* @param sql SQL statement.
|
||||
* @param params Statement parameters.
|
||||
* @returns Drawer rows.
|
||||
*/
|
||||
private queryDrawers(sql: string, ...params: unknown[]): MemPalaceDrawer[] {
|
||||
const db = this.getDb();
|
||||
const rows = db.prepare(sql).all(...params) as Array<Record<string, unknown>>;
|
||||
return rows.map((row) => ({
|
||||
id: String(row.id),
|
||||
wing: String(row.wing),
|
||||
room: String(row.room),
|
||||
content: String(row.content),
|
||||
addedBy: String(row.added_by ?? "mcp"),
|
||||
sourceFile: row.source_file ? String(row.source_file) : undefined,
|
||||
createdAt: String(row.created_at),
|
||||
updatedAt: String(row.updated_at),
|
||||
tags: extractTags(String(row.content)),
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the active database or throws when not opened.
|
||||
*
|
||||
* @returns Active database handle.
|
||||
*/
|
||||
private getDb(): Database.Database {
|
||||
if (!this.db) {
|
||||
throw new DbReadError("MemPalace DB is not open.");
|
||||
}
|
||||
|
||||
return this.db;
|
||||
}
|
||||
}
|
||||
406
src/search/context.ts
Normal file
406
src/search/context.ts
Normal file
@@ -0,0 +1,406 @@
|
||||
/**
|
||||
* Context tree builder for search result enrichment.
|
||||
*
|
||||
* Inspired by QMD's "context trees" — hierarchical metadata enrichment
|
||||
* that improves search result quality by connecting items via their
|
||||
* category hierarchy, KG relations, wiki links, temporal proximity,
|
||||
* and source provenance.
|
||||
*
|
||||
* Context enrichment is **opt-in** via the `enrichContext` parameter.
|
||||
* Default search behaviour is never changed.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import { getItem } from "../db/items.js";
|
||||
import {
|
||||
getEntity,
|
||||
getRelations,
|
||||
searchEntities,
|
||||
} from "../db/kg.js";
|
||||
import type { Item, SearchResult, WikiCategory } from "../types.js";
|
||||
|
||||
// ─── Public types ─────────────────────────────────────────────────────
|
||||
|
||||
/** A single related KG entity. */
|
||||
export interface RelatedEntity {
|
||||
id: string;
|
||||
name: string;
|
||||
type: string;
|
||||
relation: string;
|
||||
}
|
||||
|
||||
/** Temporal metadata for a search result. */
|
||||
export interface TemporalContext {
|
||||
created: string;
|
||||
updated: string;
|
||||
generated?: string;
|
||||
}
|
||||
|
||||
/** Source provenance metadata. */
|
||||
export interface SourceContext {
|
||||
source: string;
|
||||
wing?: string;
|
||||
room?: string;
|
||||
}
|
||||
|
||||
/** Hierarchical context node for a single search result. */
|
||||
export interface ContextNode {
|
||||
itemId: string;
|
||||
title: string;
|
||||
category: string;
|
||||
tags: string[];
|
||||
relatedEntities: RelatedEntity[];
|
||||
linkedPages: string[];
|
||||
temporalContext: TemporalContext;
|
||||
sourceContext: SourceContext;
|
||||
}
|
||||
|
||||
/** A search result enriched with context tree data and re-ranking info. */
|
||||
export interface EnrichedSearchResult {
|
||||
item: SearchResult;
|
||||
context: ContextNode;
|
||||
/** How related this result is to other results in the set (0–1). */
|
||||
contextScore: number;
|
||||
/** New position after context-aware re-ranking. */
|
||||
rerankPosition: number;
|
||||
}
|
||||
|
||||
// ─── Internal helpers ─────────────────────────────────────────────────
|
||||
|
||||
/** Regex to find Obsidian-style wiki links `[[page-slug]]`. */
|
||||
const WIKI_LINK_RE = /\[\[([^\]#|]+)(?:[|#][^\]]*)?\]\]/g;
|
||||
|
||||
/**
|
||||
* Extract `[[...]]` linked page slugs from markdown content.
|
||||
*
|
||||
* @param content Markdown content.
|
||||
* @returns Array of linked page slugs (unique).
|
||||
*/
|
||||
function extractWikiLinks(content: string): string[] {
|
||||
const links = new Set<string>();
|
||||
let match: RegExpExecArray | null;
|
||||
const re = new RegExp(WIKI_LINK_RE.source, WIKI_LINK_RE.flags);
|
||||
while ((match = re.exec(content)) !== null) {
|
||||
links.add(match[1].trim());
|
||||
}
|
||||
return [...links];
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a flat category path from an item's category + tags.
|
||||
*
|
||||
* Tags that look like subcategory tokens are included in the hierarchy.
|
||||
* For simplicity we treat the category as the root and tags as leaves.
|
||||
*
|
||||
* @param category Item category (may be null).
|
||||
* @param tags Item tags.
|
||||
* @returns Array of category-like strings.
|
||||
*/
|
||||
function buildCategoryHierarchy(category: WikiCategory | null, tags: string[]): string[] {
|
||||
const hierarchy: string[] = [];
|
||||
if (category) {
|
||||
hierarchy.push(category);
|
||||
}
|
||||
// Tags serve as subcategory leaves
|
||||
for (const tag of tags) {
|
||||
hierarchy.push(tag);
|
||||
}
|
||||
return hierarchy;
|
||||
}
|
||||
|
||||
/**
|
||||
* Look up KG entities whose `wikiSlug` matches the item's `wikiSlug`,
|
||||
* or whose name/id matches the item title.
|
||||
* Returns entities and the predicates of their immediate relations.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param item The item to find related entities for.
|
||||
* @returns Related entities with relation labels.
|
||||
*/
|
||||
function gatherRelatedEntities(
|
||||
db: Database.Database,
|
||||
item: Item,
|
||||
): RelatedEntity[] {
|
||||
const entities: RelatedEntity[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
// 1) Match via item's wikiSlug → entity
|
||||
if (item.wikiSlug) {
|
||||
const entity = getEntity(db, item.wikiSlug);
|
||||
if (entity) {
|
||||
const relations = getRelations(db, entity.id, "both");
|
||||
for (const rel of relations) {
|
||||
// Determine the "other" side of the relation
|
||||
const isSubject = rel.subjectId === entity.id;
|
||||
const otherId = isSubject ? rel.objectId : rel.subjectId;
|
||||
const otherEntity = getEntity(db, otherId);
|
||||
if (otherEntity && !seen.has(otherId)) {
|
||||
seen.add(otherId);
|
||||
entities.push({
|
||||
id: otherEntity.id,
|
||||
name: otherEntity.name,
|
||||
type: otherEntity.type,
|
||||
relation: rel.predicate,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 2) Match via title search → entity
|
||||
if (item.title) {
|
||||
const matched = searchEntities(db, item.title, 5);
|
||||
for (const ent of matched) {
|
||||
if (seen.has(ent.id)) continue;
|
||||
seen.add(ent.id);
|
||||
const relations = getRelations(db, ent.id, "both");
|
||||
// Add the matched entity itself
|
||||
entities.push({
|
||||
id: ent.id,
|
||||
name: ent.name,
|
||||
type: ent.type,
|
||||
relation: "matched_by_title",
|
||||
});
|
||||
// Add its direct neighbors too (limited)
|
||||
for (const rel of relations.slice(0, 3)) {
|
||||
const isSubject = rel.subjectId === ent.id;
|
||||
const otherId = isSubject ? rel.objectId : rel.subjectId;
|
||||
if (seen.has(otherId)) continue;
|
||||
seen.add(otherId);
|
||||
const otherEntity = getEntity(db, otherId);
|
||||
if (otherEntity) {
|
||||
entities.push({
|
||||
id: otherEntity.id,
|
||||
name: otherEntity.name,
|
||||
type: otherEntity.type,
|
||||
relation: rel.predicate,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return entities;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build source provenance context from item metadata.
|
||||
*
|
||||
* @param item The item.
|
||||
* @returns Source context.
|
||||
*/
|
||||
function buildSourceContext(item: Item): SourceContext {
|
||||
const meta = item.metadata ?? {};
|
||||
return {
|
||||
source: item.source,
|
||||
wing: typeof meta.wing === "string" ? meta.wing : undefined,
|
||||
room: typeof meta.room === "string" ? meta.room : undefined,
|
||||
};
|
||||
}
|
||||
|
||||
// ─── Public API ───────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Build context tree nodes for a set of item IDs.
|
||||
*
|
||||
* For each item the context tree gathers:
|
||||
* - Category hierarchy (category + tags)
|
||||
* - KG relations (connected entities)
|
||||
* - Wiki page links (`[[...]]`)
|
||||
* - Temporal context (created, updated, generated)
|
||||
* - Source context (provenance)
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param itemIds IDs of items to build context for.
|
||||
* @returns Context nodes for all found items.
|
||||
*/
|
||||
export function buildContextTree(
|
||||
db: Database.Database,
|
||||
itemIds: string[],
|
||||
): ContextNode[] {
|
||||
const nodes: ContextNode[] = [];
|
||||
|
||||
for (const id of itemIds) {
|
||||
const item = getItem(db, id);
|
||||
if (!item) continue;
|
||||
|
||||
const categories = buildCategoryHierarchy(item.category, item.tags);
|
||||
const relatedEntities = gatherRelatedEntities(db, item);
|
||||
const linkedPages = extractWikiLinks(item.content);
|
||||
const sourceCtx = buildSourceContext(item);
|
||||
|
||||
nodes.push({
|
||||
itemId: item.id,
|
||||
title: item.title,
|
||||
category: categories[0] ?? "uncategorized",
|
||||
tags: item.tags,
|
||||
relatedEntities,
|
||||
linkedPages,
|
||||
temporalContext: {
|
||||
created: item.createdAt,
|
||||
updated: item.updatedAt,
|
||||
generated: item.wikiGeneratedAt ?? undefined,
|
||||
},
|
||||
sourceContext: sourceCtx,
|
||||
});
|
||||
}
|
||||
|
||||
return nodes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Compute a context-coherence score for each result against the rest of the set.
|
||||
*
|
||||
* The score is based on:
|
||||
* - Shared categories between items
|
||||
* - Shared tags between items
|
||||
* - Shared KG entity neighbors
|
||||
* - Shared wiki page links
|
||||
*
|
||||
* Scores are normalised to 0–1 range. The formula counts the number of
|
||||
* overlapping signals between a given item and every other item in the
|
||||
* result set, divided by the maximum possible overlap.
|
||||
*
|
||||
* This is a **simple, fast, deterministic** calculation — no LLM needed.
|
||||
*
|
||||
* @param nodes Context nodes for all results.
|
||||
* @returns Array of scores in the same order as `nodes`.
|
||||
*/
|
||||
function computeContextScores(nodes: ContextNode[]): number[] {
|
||||
const n = nodes.length;
|
||||
if (n <= 1) return nodes.map(() => 0);
|
||||
|
||||
// Pre-compute sets for fast intersection
|
||||
const categories = nodes.map((nd) => new Set([nd.category, ...nd.tags]));
|
||||
const entityIds = nodes.map((nd) => new Set(nd.relatedEntities.map((e) => e.id)));
|
||||
const pageLinks = nodes.map((nd) => new Set(nd.linkedPages));
|
||||
|
||||
const scores: number[] = [];
|
||||
|
||||
for (let i = 0; i < n; i++) {
|
||||
let overlap = 0;
|
||||
let maxOverlap = 0;
|
||||
|
||||
for (let j = 0; j < n; j++) {
|
||||
if (i === j) continue;
|
||||
|
||||
// Category & tag overlap
|
||||
const catInter = intersectionSize(categories[i], categories[j]);
|
||||
overlap += catInter;
|
||||
maxOverlap += Math.max(categories[i].size, 1);
|
||||
|
||||
// KG entity overlap
|
||||
const entInter = intersectionSize(entityIds[i], entityIds[j]);
|
||||
overlap += entInter;
|
||||
maxOverlap += Math.max(entityIds[i].size, 1);
|
||||
|
||||
// Wiki link overlap
|
||||
const linkInter = intersectionSize(pageLinks[i], pageLinks[j]);
|
||||
overlap += linkInter;
|
||||
maxOverlap += Math.max(pageLinks[i].size, 1);
|
||||
}
|
||||
|
||||
// Normalise: avoid division by zero
|
||||
const score = maxOverlap > 0 ? overlap / maxOverlap : 0;
|
||||
scores.push(Math.min(score, 1));
|
||||
}
|
||||
|
||||
return scores;
|
||||
}
|
||||
|
||||
/**
|
||||
* Count the number of elements in the intersection of two sets.
|
||||
*/
|
||||
function intersectionSize<T>(a: Set<T>, b: Set<T>): number {
|
||||
let count = 0;
|
||||
for (const v of a) {
|
||||
if (b.has(v)) count++;
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Enrich search results with context tree data and re-rank by context coherence.
|
||||
*
|
||||
* Items that share categories, KG neighbors, or wiki links with other results
|
||||
* get a context-score boost and are re-ranked higher.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param results Raw search results.
|
||||
* @returns Enriched results sorted by combined score.
|
||||
*/
|
||||
export function enrichSearchResults(
|
||||
db: Database.Database,
|
||||
results: SearchResult[],
|
||||
): EnrichedSearchResult[] {
|
||||
if (results.length === 0) return [];
|
||||
|
||||
// 1. Build context tree for all result item IDs
|
||||
const itemIds = results.map((r) => r.item.id);
|
||||
const nodes = buildContextTree(db, itemIds);
|
||||
|
||||
// Build a map for fast lookup
|
||||
const nodeMap = new Map<string, ContextNode>();
|
||||
for (const node of nodes) {
|
||||
nodeMap.set(node.itemId, node);
|
||||
}
|
||||
|
||||
// 2. Compute context coherence scores
|
||||
const scores = computeContextScores(nodes);
|
||||
|
||||
// 3. Combine original search rank with context score
|
||||
const enriched: EnrichedSearchResult[] = results.map((r, idx) => {
|
||||
const node = nodeMap.get(r.item.id) ?? {
|
||||
itemId: r.item.id,
|
||||
title: r.item.title,
|
||||
category: r.item.category ?? "uncategorized",
|
||||
tags: r.item.tags,
|
||||
relatedEntities: [],
|
||||
linkedPages: extractWikiLinks(r.item.content),
|
||||
temporalContext: {
|
||||
created: r.item.createdAt,
|
||||
updated: r.item.updatedAt,
|
||||
generated: r.item.wikiGeneratedAt ?? undefined,
|
||||
},
|
||||
sourceContext: buildSourceContext(r.item),
|
||||
};
|
||||
|
||||
const nodeIdx = nodes.findIndex((n) => n.itemId === r.item.id);
|
||||
const ctxScore = nodeIdx >= 0 ? scores[nodeIdx] : 0;
|
||||
|
||||
return {
|
||||
item: r,
|
||||
context: node,
|
||||
contextScore: Math.round(ctxScore * 100) / 100,
|
||||
rerankPosition: 0, // placeholder, set below
|
||||
};
|
||||
});
|
||||
|
||||
// 4. Re-rank: blend original rank position with context score.
|
||||
// Items with high context coherence move up.
|
||||
// We use a weighted blend: 70% original rank score, 30% context score.
|
||||
// Original score is derived from inverse position in the result set.
|
||||
const maxOriginalScore = enriched.length > 0 ? enriched[0].item.rank : 1;
|
||||
|
||||
const scored = enriched.map((e, idx) => {
|
||||
const originalNorm = maxOriginalScore !== 0
|
||||
? Math.abs(e.item.rank) / Math.abs(maxOriginalScore)
|
||||
: 1 / (idx + 1);
|
||||
// Normalise: better rank = higher value (ranks are negative for BM25)
|
||||
const rankScore = Math.min(Math.abs(originalNorm), 1);
|
||||
const combined = 0.7 * rankScore + 0.3 * e.contextScore;
|
||||
return { ...e, _combined: combined };
|
||||
});
|
||||
|
||||
// Sort by combined score descending
|
||||
scored.sort((a, b) => b._combined - a._combined);
|
||||
|
||||
// Assign re-rank positions
|
||||
const final = scored.map((e, idx) => {
|
||||
const { _combined, ...rest } = e;
|
||||
return { ...rest, rerankPosition: idx + 1 };
|
||||
});
|
||||
|
||||
return final;
|
||||
}
|
||||
105
src/search/embedder.ts
Normal file
105
src/search/embedder.ts
Normal file
@@ -0,0 +1,105 @@
|
||||
/**
|
||||
* Local ONNX embedder for HaBraid.
|
||||
*
|
||||
* Uses @xenova/transformers to run embedding inference in Node.js
|
||||
* without any Python dependency. Model is auto-downloaded on first use.
|
||||
*
|
||||
* Model: Xenova/paraphrase-multilingual-MiniLM-L12-v2 (384-dim, ~120MB)
|
||||
* - Multilingual with Korean support
|
||||
* - CPU-only, no GPU needed
|
||||
*/
|
||||
|
||||
import { pipeline } from "@xenova/transformers";
|
||||
import type { FeatureExtractionPipeline } from "@xenova/transformers";
|
||||
|
||||
const MODEL_ID = "Xenova/paraphrase-multilingual-MiniLM-L12-v2";
|
||||
const VECTOR_DIM = 384;
|
||||
|
||||
/** Singleton embedder — lazy-loaded, reused across calls. */
|
||||
let embedder: FeatureExtractionPipeline | null = null;
|
||||
let loading: Promise<FeatureExtractionPipeline> | null = null;
|
||||
|
||||
/**
|
||||
* Returns (and lazily loads) the embedding pipeline.
|
||||
*
|
||||
* First call downloads the model (~120MB) to HuggingFace cache.
|
||||
* Subsequent calls reuse the loaded model.
|
||||
*/
|
||||
async function getEmbedder(): Promise<FeatureExtractionPipeline> {
|
||||
if (embedder) return embedder;
|
||||
if (loading) return loading;
|
||||
|
||||
loading = pipeline("feature-extraction", MODEL_ID, {
|
||||
progress_callback: (progress: { status: string; file?: string; progress?: number }) => {
|
||||
if (progress.status === "progress" && progress.file) {
|
||||
// Log to stderr so it doesn't interfere with MCP stdio
|
||||
process.stderr.write(
|
||||
`[embed] Downloading ${progress.file}: ${Math.round(progress.progress ?? 0)}%\n`,
|
||||
);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
embedder = await loading;
|
||||
loading = null;
|
||||
return embedder;
|
||||
}
|
||||
|
||||
/**
|
||||
* Embeds a single text string.
|
||||
*
|
||||
* @param text Text to embed.
|
||||
* @returns Float array of length 384.
|
||||
*/
|
||||
export async function embed(text: string): Promise<number[]> {
|
||||
const pipe = await getEmbedder();
|
||||
const output = await pipe(text, { pooling: "mean", normalize: true });
|
||||
return Array.from(output.data as Float32Array);
|
||||
}
|
||||
|
||||
/**
|
||||
* Embeds multiple texts in a batch.
|
||||
*
|
||||
* @param texts Texts to embed.
|
||||
* @returns Array of float arrays.
|
||||
*/
|
||||
export async function embedBatch(texts: string[]): Promise<number[][]> {
|
||||
const pipe = await getEmbedder();
|
||||
const output = await pipe(texts, { pooling: "mean", normalize: true });
|
||||
|
||||
// output.dims = [batch_size, 384]
|
||||
const dim = output.dims[output.dims.length - 1];
|
||||
const data = output.data as Float32Array;
|
||||
const results: number[][] = [];
|
||||
|
||||
for (let i = 0; i < texts.length; i++) {
|
||||
const start = i * dim;
|
||||
results.push(Array.from(data.slice(start, start + dim)));
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if the embedder is ready (loads model if needed).
|
||||
*
|
||||
* @returns true if embedding is available.
|
||||
*/
|
||||
export async function isReady(): Promise<boolean> {
|
||||
try {
|
||||
await getEmbedder();
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/** Returns the embedding dimension. */
|
||||
export function getDimension(): number {
|
||||
return VECTOR_DIM;
|
||||
}
|
||||
|
||||
/** Returns the model ID. */
|
||||
export function getModelId(): string {
|
||||
return MODEL_ID;
|
||||
}
|
||||
358
src/search/hnsw.ts
Normal file
358
src/search/hnsw.ts
Normal file
@@ -0,0 +1,358 @@
|
||||
/**
|
||||
* HNSW-based ANN index for scalable vector search.
|
||||
*
|
||||
* Wraps hnswlib-node to provide persistent HNSW indexing with
|
||||
* string-based item ID mapping (hnswlib uses numeric labels).
|
||||
*
|
||||
* Falls back gracefully if hnswlib-node is unavailable.
|
||||
*/
|
||||
|
||||
import path from "node:path";
|
||||
import fs from "node:fs";
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import { VECTOR_DIM } from "./vector.js";
|
||||
|
||||
/** HNSW index configuration. */
|
||||
export interface HnswConfig {
|
||||
/** Maximum number of elements the index can hold. */
|
||||
maxElements: number;
|
||||
/** M parameter — max outgoing connections per layer (default: 16). */
|
||||
m: number;
|
||||
/** efConstruction — build-time accuracy/speed tradeoff (default: 200). */
|
||||
efConstruction: number;
|
||||
/** ef — search-time accuracy/speed tradeoff (default: 100). */
|
||||
ef: number;
|
||||
}
|
||||
|
||||
/** Default HNSW parameters tuned for ~1000-10000 vectors. */
|
||||
const DEFAULT_HNSW_CONFIG: HnswConfig = {
|
||||
maxElements: 10000,
|
||||
m: 16,
|
||||
efConstruction: 200,
|
||||
ef: 100,
|
||||
};
|
||||
|
||||
/** Search result from HNSW index. */
|
||||
export interface HnswSearchResult {
|
||||
/** Item ID string. */
|
||||
itemId: string;
|
||||
/** Cosine distance (0 = identical, 2 = opposite). */
|
||||
distance: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Manages an HNSW index with persistent storage and ID mapping.
|
||||
*
|
||||
* hnswlib uses numeric labels (0..N-1) internally. This class maintains
|
||||
* a bidirectional mapping between numeric labels and string item IDs.
|
||||
* The mapping is stored in a companion JSON file alongside the binary index.
|
||||
*/
|
||||
export class HnswIndex {
|
||||
private index: InstanceType<typeof import("hnswlib-node").HierarchicalNSW> | null = null;
|
||||
private labelToId = new Map<number, string>();
|
||||
private idToLabel = new Map<string, number>();
|
||||
private nextLabel = 0;
|
||||
private config: HnswConfig;
|
||||
private dimensions: number;
|
||||
private loaded = false;
|
||||
|
||||
/**
|
||||
* Creates a new HNSW index manager.
|
||||
*
|
||||
* @param dimensions Vector dimensions (default: 384).
|
||||
* @param config HNSW parameters.
|
||||
*/
|
||||
constructor(dimensions: number = VECTOR_DIM, config: Partial<HnswConfig> = {}) {
|
||||
this.dimensions = dimensions;
|
||||
this.config = { ...DEFAULT_HNSW_CONFIG, ...config };
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether the index is loaded and ready for search.
|
||||
*/
|
||||
get isReady(): boolean {
|
||||
return this.loaded && this.index !== null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Current number of items in the index.
|
||||
*/
|
||||
get size(): number {
|
||||
return this.labelToId.size;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds the index from scratch using all vectors in the database.
|
||||
*
|
||||
* @param db Connected database with item_vectors table.
|
||||
* @returns Number of items indexed.
|
||||
*/
|
||||
buildIndex(db: Database.Database): number {
|
||||
this.ensureHnswlib();
|
||||
|
||||
const rows = db.prepare(
|
||||
"SELECT item_id, vector FROM item_vectors",
|
||||
).all() as Array<{ item_id: string; vector: Buffer }>;
|
||||
|
||||
if (rows.length === 0) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
// Ensure max capacity
|
||||
const maxElements = Math.max(this.config.maxElements, rows.length * 2);
|
||||
this.config.maxElements = maxElements;
|
||||
|
||||
// Create fresh index
|
||||
this.index = new (this.getHnswlib()).HierarchicalNSW("cosine", this.dimensions);
|
||||
this.index.initIndex(maxElements, this.config.m, this.config.efConstruction);
|
||||
this.index.setEf(this.config.ef);
|
||||
|
||||
// Reset mappings
|
||||
this.labelToId.clear();
|
||||
this.idToLabel.clear();
|
||||
this.nextLabel = 0;
|
||||
|
||||
// Add all vectors
|
||||
for (const row of rows) {
|
||||
const vec = this.bufferToNumberArray(row.vector);
|
||||
const label = this.nextLabel++;
|
||||
this.index.addPoint(vec, label);
|
||||
this.labelToId.set(label, row.item_id);
|
||||
this.idToLabel.set(row.item_id, label);
|
||||
}
|
||||
|
||||
this.loaded = true;
|
||||
return rows.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Adds a single item to the index.
|
||||
*
|
||||
* @param itemId Item ID string.
|
||||
* @param vector Embedding vector.
|
||||
*/
|
||||
addItem(itemId: string, vector: number[]): void {
|
||||
this.ensureHnswlib();
|
||||
|
||||
// If item already exists, remove old mapping
|
||||
if (this.idToLabel.has(itemId)) {
|
||||
const oldLabel = this.idToLabel.get(itemId)!;
|
||||
this.index!.markDelete(oldLabel);
|
||||
this.labelToId.delete(oldLabel);
|
||||
this.idToLabel.delete(itemId);
|
||||
}
|
||||
|
||||
// Resize if needed
|
||||
if (this.nextLabel >= this.config.maxElements) {
|
||||
const newMax = this.config.maxElements * 2;
|
||||
this.index!.resizeIndex(newMax);
|
||||
this.config.maxElements = newMax;
|
||||
}
|
||||
|
||||
const label = this.nextLabel++;
|
||||
this.index!.addPoint(vector, label, true);
|
||||
this.labelToId.set(label, itemId);
|
||||
this.idToLabel.set(itemId, label);
|
||||
}
|
||||
|
||||
/**
|
||||
* Searches for the K nearest neighbors of a query vector.
|
||||
*
|
||||
* @param query Query embedding vector.
|
||||
* @param k Number of nearest neighbors to return.
|
||||
* @returns Ranked results with item IDs and distances.
|
||||
*/
|
||||
search(query: number[], k: number): HnswSearchResult[] {
|
||||
if (!this.isReady || !this.index) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const result = this.index.searchKnn(query, k);
|
||||
const results: HnswSearchResult[] = [];
|
||||
|
||||
for (let i = 0; i < result.neighbors.length; i++) {
|
||||
const label = result.neighbors[i];
|
||||
const itemId = this.labelToId.get(label);
|
||||
if (itemId !== undefined) {
|
||||
results.push({
|
||||
itemId,
|
||||
distance: result.distances[i],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Persists the index and ID mapping to disk.
|
||||
*
|
||||
* @param indexPath Path for the binary index file (without extension).
|
||||
*/
|
||||
saveIndex(indexPath: string): void {
|
||||
if (!this.isReady || !this.index) {
|
||||
return;
|
||||
}
|
||||
|
||||
// Ensure parent directory exists
|
||||
const dir = path.dirname(indexPath);
|
||||
if (!fs.existsSync(dir)) {
|
||||
fs.mkdirSync(dir, { recursive: true });
|
||||
}
|
||||
|
||||
// Save binary index
|
||||
this.index.writeIndexSync(indexPath);
|
||||
|
||||
// Save ID mapping as JSON companion file
|
||||
const metaPath = indexPath + ".meta.json";
|
||||
const meta = {
|
||||
dimensions: this.dimensions,
|
||||
nextLabel: this.nextLabel,
|
||||
maxElements: this.config.maxElements,
|
||||
m: this.config.m,
|
||||
efConstruction: this.config.efConstruction,
|
||||
ef: this.config.ef,
|
||||
labelToId: Array.from(this.labelToId.entries()),
|
||||
};
|
||||
fs.writeFileSync(metaPath, JSON.stringify(meta), "utf8");
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads a previously saved index from disk.
|
||||
*
|
||||
* @param indexPath Path for the binary index file (without extension).
|
||||
* @returns true if loaded successfully.
|
||||
*/
|
||||
loadIndex(indexPath: string): boolean {
|
||||
this.ensureHnswlib();
|
||||
|
||||
const binPath = indexPath;
|
||||
const metaPath = indexPath + ".meta.json";
|
||||
|
||||
if (!fs.existsSync(binPath) || !fs.existsSync(metaPath)) {
|
||||
return false;
|
||||
}
|
||||
|
||||
try {
|
||||
// Load metadata first
|
||||
const metaRaw = fs.readFileSync(metaPath, "utf8");
|
||||
const meta = JSON.parse(metaRaw) as {
|
||||
dimensions: number;
|
||||
nextLabel: number;
|
||||
maxElements: number;
|
||||
m: number;
|
||||
efConstruction: number;
|
||||
ef: number;
|
||||
labelToId: Array<[number, string]>;
|
||||
};
|
||||
|
||||
// Create and load index
|
||||
this.index = new (this.getHnswlib()).HierarchicalNSW("cosine", meta.dimensions);
|
||||
this.index.readIndexSync(binPath, true);
|
||||
this.index.setEf(meta.ef);
|
||||
|
||||
// Restore mappings
|
||||
this.labelToId.clear();
|
||||
this.idToLabel.clear();
|
||||
this.nextLabel = meta.nextLabel;
|
||||
this.config.maxElements = meta.maxElements;
|
||||
this.config.m = meta.m;
|
||||
this.config.efConstruction = meta.efConstruction;
|
||||
this.config.ef = meta.ef;
|
||||
|
||||
for (const [label, id] of meta.labelToId) {
|
||||
this.labelToId.set(label, id);
|
||||
this.idToLabel.set(id, label);
|
||||
}
|
||||
|
||||
this.loaded = true;
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the default index path for a given config.
|
||||
*
|
||||
* @param dbPath Database file path (used to derive index location).
|
||||
* @returns Absolute path for the HNSW index file.
|
||||
*/
|
||||
static getDefaultIndexPath(dbPath: string): string {
|
||||
const dir = path.dirname(dbPath);
|
||||
return path.join(dir, "hnsw.index");
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if an HNSW index file exists at the given path.
|
||||
*
|
||||
* @param indexPath Index file path.
|
||||
* @returns true if both index and metadata files exist.
|
||||
*/
|
||||
static indexExists(indexPath: string): boolean {
|
||||
return fs.existsSync(indexPath) && fs.existsSync(indexPath + ".meta.json");
|
||||
}
|
||||
|
||||
// ─── Private helpers ──────────────────────────────────────────────
|
||||
|
||||
/** Cached hnswlib module reference. */
|
||||
private static hnswlibModule: typeof import("hnswlib-node") | null = null;
|
||||
private static hnswlibAvailable: boolean | null = null;
|
||||
|
||||
/**
|
||||
* Returns the hnswlib-node module (lazy-loaded).
|
||||
* Returns null if unavailable.
|
||||
*/
|
||||
private getHnswlib(): typeof import("hnswlib-node") {
|
||||
if (HnswIndex.hnswlibModule) {
|
||||
return HnswIndex.hnswlibModule;
|
||||
}
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
HnswIndex.hnswlibModule = require("hnswlib-node");
|
||||
return HnswIndex.hnswlibModule!;
|
||||
}
|
||||
|
||||
/**
|
||||
* Throws if hnswlib-node is not available.
|
||||
*/
|
||||
private ensureHnswlib(): void {
|
||||
if (HnswIndex.hnswlibAvailable === false) {
|
||||
throw new Error("hnswlib-node is not available");
|
||||
}
|
||||
try {
|
||||
this.getHnswlib();
|
||||
HnswIndex.hnswlibAvailable = true;
|
||||
} catch {
|
||||
HnswIndex.hnswlibAvailable = false;
|
||||
throw new Error("hnswlib-node is not available. Install with: npm install hnswlib-node");
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a Buffer (float32 LE) to a number array.
|
||||
*/
|
||||
private bufferToNumberArray(buf: Buffer): number[] {
|
||||
const count = buf.length / 4;
|
||||
const arr = new Array<number>(count);
|
||||
for (let i = 0; i < count; i++) {
|
||||
arr[i] = buf.readFloatLE(i * 4);
|
||||
}
|
||||
return arr;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if hnswlib-node is available without throwing.
|
||||
*
|
||||
* @returns true if the native module can be loaded.
|
||||
*/
|
||||
export function isHnswAvailable(): boolean {
|
||||
try {
|
||||
// eslint-disable-next-line @typescript-eslint/no-require-imports
|
||||
require("hnswlib-node");
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
260
src/search/hybrid.ts
Normal file
260
src/search/hybrid.ts
Normal file
@@ -0,0 +1,260 @@
|
||||
/**
|
||||
* Hybrid search combining BM25 (FTS5) and Vector similarity via RRF.
|
||||
*
|
||||
* Implements Reciprocal Rank Fusion to merge results from two search methods:
|
||||
* - BM25: keyword matching via SQLite FTS5
|
||||
* - Vector: semantic similarity via fastembed embeddings
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import { searchItems } from "../db/items.js";
|
||||
import { searchByVector } from "./vector.js";
|
||||
import { embed } from "./embedder.js";
|
||||
import { enrichSearchResults, type EnrichedSearchResult } from "./context.js";
|
||||
import type { Item, ItemSource, SearchResult, WikiCategory } from "../types.js";
|
||||
|
||||
/** RRF constant k — prevents top ranks from dominating. */
|
||||
const RRF_K = 60;
|
||||
|
||||
/**
|
||||
* Source-based boost multipliers for RRF scoring.
|
||||
* Manual/wiki content is curated and higher quality — boost it.
|
||||
* MemPalace raw drawers are noisy — dampen them.
|
||||
*/
|
||||
const SOURCE_BOOST: Record<string, number> = {
|
||||
manual: 1.5,
|
||||
wiki: 1.3,
|
||||
mempalace: 0.7,
|
||||
};
|
||||
|
||||
/** Category-based boost — curated wiki pages get extra weight. */
|
||||
const CATEGORY_BOOST: Record<string, number> = {
|
||||
projects: 1.4,
|
||||
decisions: 1.3,
|
||||
guides: 1.2,
|
||||
topics: 1.1,
|
||||
people: 1.1,
|
||||
infrastructure: 1.1,
|
||||
};
|
||||
|
||||
/**
|
||||
* Compute a combined source + category boost multiplier for an item.
|
||||
*/
|
||||
function getBoost(source: string, category: string | null): number {
|
||||
const s = SOURCE_BOOST[source] ?? 1.0;
|
||||
const c = category ? (CATEGORY_BOOST[category] ?? 1.0) : 1.0;
|
||||
return s * c;
|
||||
}
|
||||
|
||||
/** Search mode options. */
|
||||
export type SearchMode = "keyword" | "semantic" | "hybrid";
|
||||
|
||||
/**
|
||||
* Result of a hybrid search.
|
||||
*/
|
||||
export interface HybridSearchResult {
|
||||
item: Item;
|
||||
score: number;
|
||||
snippet: string;
|
||||
sources: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Performs hybrid search combining BM25 and vector search with RRF fusion.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param query Search query string.
|
||||
* @param mode Search mode: keyword, semantic, or hybrid.
|
||||
* @param limit Max results.
|
||||
* @param dbPath Path to the database file (for HNSW index).
|
||||
* @param enrichContext When true, returns enriched results with context tree data.
|
||||
* @returns Ranked search results, optionally enriched with context.
|
||||
*/
|
||||
export async function hybridSearch(
|
||||
db: Database.Database,
|
||||
query: string,
|
||||
mode: SearchMode = "hybrid",
|
||||
limit: number = 20,
|
||||
dbPath?: string,
|
||||
enrichContext: boolean = false,
|
||||
): Promise<{
|
||||
results: HybridSearchResult[];
|
||||
enrichedResults?: EnrichedSearchResult[];
|
||||
mode: SearchMode;
|
||||
fallback?: string;
|
||||
}> {
|
||||
const candidateMultiplier = 3;
|
||||
const candidateLimit = limit * candidateMultiplier;
|
||||
|
||||
// 1. BM25 search (if keyword or hybrid)
|
||||
let bm25Results: SearchResult[] = [];
|
||||
if (mode === "keyword" || mode === "hybrid") {
|
||||
try {
|
||||
bm25Results = searchItems(db, { query, limit: candidateLimit });
|
||||
} catch {
|
||||
// FTS5 might fail on special characters
|
||||
bm25Results = [];
|
||||
}
|
||||
}
|
||||
|
||||
// 2. Vector search (if semantic or hybrid)
|
||||
let vectorResults: Array<{ itemId: string; score: number }> = [];
|
||||
let usedMode = mode;
|
||||
let fallback: string | undefined;
|
||||
|
||||
if (mode === "semantic" || mode === "hybrid") {
|
||||
try {
|
||||
const queryVector = await embed(query);
|
||||
vectorResults = searchByVector(db, queryVector, candidateLimit, dbPath);
|
||||
} catch {
|
||||
// Embedding failed — fall back to keyword
|
||||
if (mode === "semantic") {
|
||||
usedMode = "keyword";
|
||||
fallback = "Embedding unavailable, falling back to keyword search";
|
||||
bm25Results = searchItems(db, { query, limit: candidateLimit });
|
||||
}
|
||||
}
|
||||
|
||||
if (vectorResults.length === 0) {
|
||||
usedMode = "keyword" as SearchMode;
|
||||
fallback = "No vectors indexed, using keyword search only";
|
||||
if (bm25Results.length === 0) {
|
||||
bm25Results = searchItems(db, { query, limit: candidateLimit });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 3. If only one method has results, return directly
|
||||
if (bm25Results.length === 0 && vectorResults.length === 0) {
|
||||
return { results: [], mode: usedMode, fallback };
|
||||
}
|
||||
|
||||
if (vectorResults.length === 0) {
|
||||
// BM25 only — apply source/category boost
|
||||
const results = bm25Results.slice(0, limit).map((r) => ({
|
||||
item: r.item,
|
||||
score: r.rank * getBoost(r.item.source, r.item.category),
|
||||
snippet: r.snippet,
|
||||
sources: ["bm25"],
|
||||
}));
|
||||
if (enrichContext) {
|
||||
const enrichedResults = enrichSearchResults(db, bm25Results.slice(0, limit));
|
||||
return { results, enrichedResults, mode: usedMode, fallback };
|
||||
}
|
||||
return { results, mode: usedMode, fallback };
|
||||
}
|
||||
|
||||
if (bm25Results.length === 0) {
|
||||
// Vector only — need to load items, apply source/category boost, then re-sort
|
||||
const boostedResults = vectorResults.map((vr) => {
|
||||
const item = loadItemById(db, vr.itemId);
|
||||
if (!item) return null;
|
||||
return {
|
||||
item,
|
||||
score: vr.score * getBoost(item.source, item.category),
|
||||
snippet: item.content.slice(0, 200),
|
||||
sources: ["vector"],
|
||||
};
|
||||
}).filter((r): r is HybridSearchResult => r !== null);
|
||||
|
||||
// Re-sort by boosted score descending
|
||||
boostedResults.sort((a, b) => b.score - a.score);
|
||||
|
||||
// Normalize
|
||||
const maxScore = boostedResults[0]?.score ?? 1;
|
||||
const results: HybridSearchResult[] = boostedResults.slice(0, limit).map((r) => ({
|
||||
...r,
|
||||
score: Math.round((r.score / maxScore) * 100) / 100,
|
||||
}));
|
||||
|
||||
if (enrichContext) {
|
||||
const searchResults = boostedResults.slice(0, limit).map((r) => ({
|
||||
item: r.item, rank: r.score, snippet: r.snippet,
|
||||
}));
|
||||
const enrichedResults = enrichSearchResults(db, searchResults);
|
||||
return { results, enrichedResults, mode: usedMode, fallback };
|
||||
}
|
||||
return { results, mode: usedMode, fallback };
|
||||
}
|
||||
|
||||
// 4. RRF fusion with source/category boost
|
||||
const rrfScores = new Map<string, { score: number; item: Item; snippet: string; sources: Set<string> }>();
|
||||
|
||||
// BM25 ranks
|
||||
for (let rank = 0; rank < bm25Results.length; rank++) {
|
||||
const r = bm25Results[rank];
|
||||
const id = r.item.id;
|
||||
const boost = getBoost(r.item.source, r.item.category);
|
||||
const rrf = boost * (1 / (RRF_K + rank + 1));
|
||||
const existing = rrfScores.get(id);
|
||||
if (existing) {
|
||||
existing.score += rrf;
|
||||
existing.sources.add("bm25");
|
||||
} else {
|
||||
rrfScores.set(id, { score: rrf, item: r.item, snippet: r.snippet, sources: new Set(["bm25"]) });
|
||||
}
|
||||
}
|
||||
|
||||
// Vector ranks
|
||||
for (let rank = 0; rank < vectorResults.length; rank++) {
|
||||
const vr = vectorResults[rank];
|
||||
const item = loadItemById(db, vr.itemId);
|
||||
if (!item) continue;
|
||||
|
||||
const boost = getBoost(item.source, item.category);
|
||||
const rrf = boost * (1 / (RRF_K + rank + 1));
|
||||
const existing = rrfScores.get(vr.itemId);
|
||||
if (existing) {
|
||||
existing.score += rrf;
|
||||
existing.sources.add("vector");
|
||||
} else {
|
||||
rrfScores.set(vr.itemId, { score: rrf, item, snippet: item.content.slice(0, 200), sources: new Set(["vector"]) });
|
||||
}
|
||||
}
|
||||
|
||||
// Sort by RRF score descending
|
||||
const sorted = [...rrfScores.values()].sort((a, b) => b.score - a.score);
|
||||
|
||||
// Normalize scores to 0-1 range
|
||||
const maxScore = sorted[0]?.score ?? 1;
|
||||
const results: HybridSearchResult[] = sorted.slice(0, limit).map((r) => ({
|
||||
item: r.item,
|
||||
score: Math.round((r.score / maxScore) * 100) / 100,
|
||||
snippet: r.snippet,
|
||||
sources: [...r.sources],
|
||||
}));
|
||||
|
||||
// Optional context enrichment
|
||||
if (enrichContext) {
|
||||
const searchResults: SearchResult[] = sorted.slice(0, limit).map((r) => ({
|
||||
item: r.item,
|
||||
rank: r.score,
|
||||
snippet: r.snippet,
|
||||
}));
|
||||
const enrichedResults = enrichSearchResults(db, searchResults);
|
||||
return { results, enrichedResults, mode: usedMode, fallback };
|
||||
}
|
||||
|
||||
return { results, mode: usedMode, fallback };
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads a single item by ID.
|
||||
*/
|
||||
function loadItemById(db: Database.Database, itemId: string): Item | null {
|
||||
const row = db.prepare("SELECT * FROM items WHERE id = ?").get(itemId) as Record<string, unknown> | undefined;
|
||||
if (!row) return null;
|
||||
|
||||
return {
|
||||
id: String(row.id),
|
||||
title: String(row.title),
|
||||
content: String(row.content),
|
||||
source: String(row.source) as ItemSource,
|
||||
category: row.category ? String(row.category) as WikiCategory : null,
|
||||
tags: JSON.parse(String(row.tags ?? "[]")),
|
||||
createdAt: String(row.created_at),
|
||||
updatedAt: String(row.updated_at),
|
||||
metadata: JSON.parse(String(row.metadata ?? "{}")),
|
||||
};
|
||||
}
|
||||
308
src/search/vector.ts
Normal file
308
src/search/vector.ts
Normal file
@@ -0,0 +1,308 @@
|
||||
/**
|
||||
* Vector storage and search for HaBraid items.
|
||||
*
|
||||
* Stores and retrieves embedding vectors from SQLite.
|
||||
* Uses BLOB column for compact float32 array storage.
|
||||
*
|
||||
* Search strategy:
|
||||
* 1. HNSW ANN index (if available and built) — scalable to 10k+ vectors
|
||||
* 2. Brute-force cosine similarity — fallback for small datasets or missing index
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
import { HnswIndex, HnswSearchResult } from "./hnsw.js";
|
||||
|
||||
/** Vector dimension for multilingual-MiniLM-L12-v2. */
|
||||
export const VECTOR_DIM = 384;
|
||||
|
||||
/** Default embedding model name. */
|
||||
export const DEFAULT_MODEL = "paraphrase-multilingual-MiniLM-L12-v2";
|
||||
|
||||
/** Singleton HNSW index — lazy-loaded, reused across calls. */
|
||||
let hnswIndex: HnswIndex | null = null;
|
||||
|
||||
/**
|
||||
* Returns (or lazily loads) the HNSW index.
|
||||
*
|
||||
* @param dbPath Database path (used to derive index location).
|
||||
* @param db Connected database (for auto-rebuild if stale).
|
||||
* @returns HNSW index or null if unavailable.
|
||||
*/
|
||||
function getHnswIndex(dbPath: string, db: Database.Database): HnswIndex | null {
|
||||
if (hnswIndex) {
|
||||
// Check if index is stale (DB has more items than index)
|
||||
if (hnswIndex.isReady) {
|
||||
const dbCount = getVectorCount(db);
|
||||
if (dbCount > hnswIndex.size) {
|
||||
// Index is stale — rebuild silently
|
||||
try {
|
||||
hnswIndex.buildIndex(db);
|
||||
const indexPath = HnswIndex.getDefaultIndexPath(dbPath);
|
||||
hnswIndex.saveIndex(indexPath);
|
||||
} catch {
|
||||
// Rebuild failed — continue with stale index
|
||||
}
|
||||
}
|
||||
}
|
||||
return hnswIndex;
|
||||
}
|
||||
|
||||
const index = new HnswIndex();
|
||||
const indexPath = HnswIndex.getDefaultIndexPath(dbPath);
|
||||
|
||||
// Try to load existing index
|
||||
if (HnswIndex.indexExists(indexPath)) {
|
||||
if (index.loadIndex(indexPath)) {
|
||||
// Check if stale
|
||||
const dbCount = getVectorCount(db);
|
||||
if (dbCount > index.size) {
|
||||
try {
|
||||
index.buildIndex(db);
|
||||
index.saveIndex(indexPath);
|
||||
} catch {
|
||||
// Rebuild failed — use loaded (possibly stale) index
|
||||
}
|
||||
}
|
||||
hnswIndex = index;
|
||||
return hnswIndex;
|
||||
}
|
||||
}
|
||||
|
||||
// No existing index — try to build one if vectors exist
|
||||
const dbCount = getVectorCount(db);
|
||||
if (dbCount > 0) {
|
||||
try {
|
||||
const built = index.buildIndex(db);
|
||||
if (built > 0) {
|
||||
index.saveIndex(indexPath);
|
||||
hnswIndex = index;
|
||||
return hnswIndex;
|
||||
}
|
||||
} catch {
|
||||
// HNSW unavailable or build failed
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Invalidates the cached HNSW index, forcing rebuild on next access.
|
||||
*/
|
||||
export function invalidateHnswIndex(): void {
|
||||
hnswIndex = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores a vector for an item. Upserts on conflict.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param itemId Item ID.
|
||||
* @param vector Float array.
|
||||
* @param model Model name used for embedding.
|
||||
*/
|
||||
export function storeVector(
|
||||
db: Database.Database,
|
||||
itemId: string,
|
||||
vector: number[],
|
||||
model: string = DEFAULT_MODEL,
|
||||
): void {
|
||||
const buffer = floatArrayToBuffer(vector);
|
||||
db.prepare(`
|
||||
INSERT INTO item_vectors (item_id, vector, model, updated_at)
|
||||
VALUES (?, ?, ?, datetime('now'))
|
||||
ON CONFLICT(item_id) DO UPDATE SET
|
||||
vector = excluded.vector,
|
||||
model = excluded.model,
|
||||
updated_at = excluded.updated_at
|
||||
`).run(itemId, buffer, model);
|
||||
|
||||
// Update HNSW index if loaded
|
||||
if (hnswIndex?.isReady) {
|
||||
try {
|
||||
hnswIndex.addItem(itemId, vector);
|
||||
} catch {
|
||||
// Ignore HNSW update failures — index will be stale until rebuild
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves the vector for an item.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param itemId Item ID.
|
||||
* @returns Float array or undefined if not found.
|
||||
*/
|
||||
export function getVector(db: Database.Database, itemId: string): number[] | undefined {
|
||||
const row = db.prepare("SELECT vector FROM item_vectors WHERE item_id = ?").get(itemId) as { vector: Buffer } | undefined;
|
||||
if (!row) return undefined;
|
||||
return bufferToFloatArray(row.vector);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns all item IDs that have vectors.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @returns Array of item IDs.
|
||||
*/
|
||||
export function getIndexedItemIds(db: Database.Database): string[] {
|
||||
const rows = db.prepare("SELECT item_id FROM item_vectors").all() as Array<{ item_id: string }>;
|
||||
return rows.map((r) => r.item_id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Counts indexed items.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @returns Number of items with vectors.
|
||||
*/
|
||||
export function getVectorCount(db: Database.Database): number {
|
||||
const row = db.prepare("SELECT COUNT(*) as cnt FROM item_vectors").get() as { cnt: number };
|
||||
return row.cnt;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items whose vectors are most similar to the query vector (cosine similarity).
|
||||
*
|
||||
* Uses HNSW ANN index when available for O(log n) search.
|
||||
* Falls back to brute-force cosine similarity for small datasets or when HNSW is unavailable.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param queryVector Query embedding.
|
||||
* @param limit Max results.
|
||||
* @param dbPath Database path (for HNSW index location). Optional.
|
||||
* @returns Ranked results with similarity scores.
|
||||
*/
|
||||
export function searchByVector(
|
||||
db: Database.Database,
|
||||
queryVector: number[],
|
||||
limit: number = 20,
|
||||
dbPath?: string,
|
||||
): Array<{ itemId: string; score: number }> {
|
||||
// Try HNSW search first
|
||||
if (dbPath) {
|
||||
const index = getHnswIndex(dbPath, db);
|
||||
if (index?.isReady) {
|
||||
const hnswResults = index.search(queryVector, limit);
|
||||
if (hnswResults.length > 0) {
|
||||
// Convert HNSW distances (cosine distance) to similarity scores
|
||||
// cosine distance = 1 - cosine_similarity, so similarity = 1 - distance
|
||||
return hnswResults.map((r: HnswSearchResult) => ({
|
||||
itemId: r.itemId,
|
||||
score: 1 - r.distance,
|
||||
}));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback: brute-force search
|
||||
return searchByVectorBruteForce(db, queryVector, limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuilds the HNSW index from all vectors in the database.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param dbPath Database path (for index location).
|
||||
* @returns Number of vectors indexed.
|
||||
*/
|
||||
export function rebuildHnswIndex(db: Database.Database, dbPath: string): number {
|
||||
const index = new HnswIndex();
|
||||
const count = index.buildIndex(db);
|
||||
if (count > 0) {
|
||||
const indexPath = HnswIndex.getDefaultIndexPath(dbPath);
|
||||
index.saveIndex(indexPath);
|
||||
}
|
||||
// Update singleton
|
||||
hnswIndex = index;
|
||||
return count;
|
||||
}
|
||||
|
||||
// ─── Helpers ────────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Brute-force vector search using cosine similarity.
|
||||
* Loads all vectors from SQLite and computes similarity in-memory.
|
||||
*
|
||||
* @param db Connected database.
|
||||
* @param queryVector Query embedding.
|
||||
* @param limit Max results.
|
||||
* @returns Ranked results with similarity scores.
|
||||
*/
|
||||
function searchByVectorBruteForce(
|
||||
db: Database.Database,
|
||||
queryVector: number[],
|
||||
limit: number = 20,
|
||||
): Array<{ itemId: string; score: number }> {
|
||||
const rows = db.prepare("SELECT item_id, vector FROM item_vectors").all() as Array<{ item_id: string; vector: Buffer }>;
|
||||
|
||||
const queryNorm = norm(queryVector);
|
||||
if (queryNorm === 0) return [];
|
||||
|
||||
const results: Array<{ itemId: string; score: number }> = [];
|
||||
|
||||
for (const row of rows) {
|
||||
const vec = bufferToFloatArray(row.vector);
|
||||
const sim = cosineSimilarity(queryVector, vec, queryNorm);
|
||||
results.push({ itemId: row.item_id, score: sim });
|
||||
}
|
||||
|
||||
// Sort by score descending, take top limit
|
||||
results.sort((a, b) => b.score - a.score);
|
||||
return results.slice(0, limit);
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a float array to a compact Buffer (float32 LE).
|
||||
*/
|
||||
function floatArrayToBuffer(arr: number[]): Buffer {
|
||||
const buf = Buffer.alloc(arr.length * 4);
|
||||
for (let i = 0; i < arr.length; i++) {
|
||||
buf.writeFloatLE(arr[i], i * 4);
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts a Buffer to a float array.
|
||||
*/
|
||||
function bufferToFloatArray(buf: Buffer): number[] {
|
||||
const count = buf.length / 4;
|
||||
const arr = new Array<number>(count);
|
||||
for (let i = 0; i < count; i++) {
|
||||
arr[i] = buf.readFloatLE(i * 4);
|
||||
}
|
||||
return arr;
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes the L2 norm of a vector.
|
||||
*/
|
||||
function norm(vec: number[]): number {
|
||||
let sum = 0;
|
||||
for (let i = 0; i < vec.length; i++) {
|
||||
sum += vec[i] * vec[i];
|
||||
}
|
||||
return Math.sqrt(sum);
|
||||
}
|
||||
|
||||
/**
|
||||
* Computes cosine similarity between two vectors.
|
||||
*
|
||||
* @param a First vector.
|
||||
* @param b Second vector.
|
||||
* @param normA Pre-computed norm of a (optional optimization).
|
||||
* @returns Similarity score in [-1, 1].
|
||||
*/
|
||||
function cosineSimilarity(a: number[], b: number[], normA?: number): number {
|
||||
const na = normA ?? norm(a);
|
||||
const nb = norm(b);
|
||||
if (na === 0 || nb === 0) return 0;
|
||||
|
||||
let dot = 0;
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
dot += a[i] * b[i];
|
||||
}
|
||||
return dot / (na * nb);
|
||||
}
|
||||
283
src/setup.ts
Normal file
283
src/setup.ts
Normal file
@@ -0,0 +1,283 @@
|
||||
/**
|
||||
* Zero-config setup for HaBraid.
|
||||
*
|
||||
* On first run, automatically:
|
||||
* 1. Creates ~/.habraid/{app,data} directory structure
|
||||
* 2. Detects MemPalace installation
|
||||
* 3. Initializes database
|
||||
* 4. Generates default config
|
||||
* 5. Runs initial ingest + index
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { openDatabase } from "./db/database.js";
|
||||
import { getItemCount } from "./db/items.js";
|
||||
import type { WikiEngineConfig } from "./types.js";
|
||||
import { ensureDir, expandHomeDir, pathExists, resolvePath } from "./utils.js";
|
||||
|
||||
/** Base directory for internal HaBraid runtime data. */
|
||||
export const HABRAID_HOME = path.join(os.homedir(), ".habraid");
|
||||
|
||||
/** Hidden app repo path inside the HaBraid home. */
|
||||
export const HABRAID_APP_HOME = path.join(HABRAID_HOME, "app");
|
||||
|
||||
/** Hidden data path inside the HaBraid home. */
|
||||
export const HABRAID_DATA_HOME = path.join(HABRAID_HOME, "data");
|
||||
|
||||
/** Primary user-facing wiki vault path for one-click installs. */
|
||||
export const PRIMARY_VAULT_PATH = path.join(os.homedir(), "wiki");
|
||||
|
||||
/** Well-known MemPalace raw directory search paths. */
|
||||
const MEMPALACE_SEARCH_PATHS = [
|
||||
// MemPalace MCP standard location
|
||||
path.join(os.homedir(), ".local", "share", "mempalace", "raw"),
|
||||
// Hermes-integrated MemPalace
|
||||
path.join(os.homedir(), "wiki", "raw", "mempalace"),
|
||||
// Standalone MemPalace
|
||||
path.join(os.homedir(), "mempalace", "raw"),
|
||||
// OpenClaw legacy
|
||||
path.join(os.homedir(), ".openclaw", "raw"),
|
||||
];
|
||||
|
||||
/**
|
||||
* Searches for an existing MemPalace raw directory.
|
||||
*
|
||||
* @returns Absolute path to raw directory, or empty string if not found.
|
||||
*/
|
||||
export function detectMemPalacePath(): string {
|
||||
for (const candidate of MEMPALACE_SEARCH_PATHS) {
|
||||
try {
|
||||
if (fs.existsSync(candidate)) {
|
||||
const stat = fs.statSync(candidate);
|
||||
if (stat.isDirectory()) {
|
||||
// Verify it has at least one .md file (directly or nested)
|
||||
const hasMd = hasMarkdownFiles(candidate);
|
||||
if (hasMd) {
|
||||
return candidate;
|
||||
}
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively checks if a directory contains any .md files.
|
||||
*/
|
||||
function hasMarkdownFiles(dir: string): boolean {
|
||||
try {
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name.startsWith(".")) continue;
|
||||
if (entry.isFile() && entry.name.endsWith(".md")) return true;
|
||||
if (entry.isDirectory()) {
|
||||
if (hasMarkdownFiles(path.join(dir, entry.name))) return true;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the ~/.habraid/ directory structure.
|
||||
*/
|
||||
async function createDirectoryStructure(vaultPath: string): Promise<void> {
|
||||
const dirs = [
|
||||
HABRAID_HOME,
|
||||
HABRAID_APP_HOME,
|
||||
HABRAID_DATA_HOME,
|
||||
vaultPath,
|
||||
path.join(vaultPath, "raw"),
|
||||
path.join(vaultPath, "wiki"),
|
||||
path.join(HABRAID_DATA_HOME, "models"),
|
||||
path.join(HABRAID_DATA_HOME, "logs"),
|
||||
path.join(HABRAID_DATA_HOME, "backups"),
|
||||
];
|
||||
|
||||
for (const dir of dirs) {
|
||||
await ensureDir(dir);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a symlink from vault/raw/ to the detected MemPalace path.
|
||||
*
|
||||
* @param mempalacePath Detected MemPalace raw directory.
|
||||
*/
|
||||
function linkMemPalaceRaw(vaultPath: string, mempalacePath: string): void {
|
||||
const linkPath = path.join(vaultPath, "raw", "mempalace");
|
||||
|
||||
// Remove existing link/dir if it exists
|
||||
try {
|
||||
const stat = fs.lstatSync(linkPath);
|
||||
if (stat.isSymbolicLink()) {
|
||||
fs.unlinkSync(linkPath);
|
||||
}
|
||||
} catch {
|
||||
// doesn't exist, that's fine
|
||||
}
|
||||
|
||||
// Create symlink
|
||||
fs.symlinkSync(mempalacePath, linkPath, "junction");
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a default config.json with auto-detected paths.
|
||||
*
|
||||
* @param mempalacePath Auto-detected MemPalace path (may be empty).
|
||||
* @returns Config object.
|
||||
*/
|
||||
export function generateDefaultConfig(mempalacePath: string): WikiEngineConfig {
|
||||
return {
|
||||
vault: {
|
||||
path: PRIMARY_VAULT_PATH,
|
||||
branch: "main",
|
||||
},
|
||||
db: {
|
||||
path: path.join(HABRAID_DATA_HOME, "habraid.db"),
|
||||
},
|
||||
mempalace: {
|
||||
enabled: mempalacePath.length > 0,
|
||||
path: mempalacePath,
|
||||
},
|
||||
llm: {
|
||||
mode: "host",
|
||||
preferences: {
|
||||
priority: "balanced",
|
||||
},
|
||||
fallback: {
|
||||
provider: "zai",
|
||||
model: "glm-5.1",
|
||||
api_url: "https://api.example.com/v1",
|
||||
api_key_env: "GLM_API_KEY",
|
||||
max_tokens: 4096,
|
||||
},
|
||||
provider: "zai",
|
||||
model: "glm-5.1",
|
||||
api_url: "https://api.example.com/v1",
|
||||
api_key_env: "GLM_API_KEY",
|
||||
max_tokens: 4096,
|
||||
},
|
||||
sync: {
|
||||
timezone: Intl.DateTimeFormat().resolvedOptions().timeZone || "Asia/Seoul",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes config to ~/.habraid/data/config.json if it doesn't exist.
|
||||
*
|
||||
* @param config Config to write.
|
||||
*/
|
||||
async function writeConfigIfMissing(config: WikiEngineConfig): Promise<boolean> {
|
||||
const configPath = path.join(HABRAID_DATA_HOME, "config.json");
|
||||
|
||||
if (fs.existsSync(configPath)) {
|
||||
return false; // already exists
|
||||
}
|
||||
|
||||
const { writeTextFile } = await import("./utils.js");
|
||||
await writeTextFile(configPath, JSON.stringify(config, null, 2) + "\n");
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of the setup process.
|
||||
*/
|
||||
export interface SetupResult {
|
||||
/** Whether this was a first-time setup. */
|
||||
firstRun: boolean;
|
||||
/** Detected MemPalace path (empty if not found). */
|
||||
mempalacePath: string;
|
||||
/** Number of items already in DB (0 for first run). */
|
||||
existingItems: number;
|
||||
/** Messages to display to the user. */
|
||||
messages: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the zero-config setup process.
|
||||
*
|
||||
* Safe to call on every server start — only does work on first run.
|
||||
*
|
||||
* @param config Loaded config (may be defaults).
|
||||
* @returns Setup result with status info.
|
||||
*/
|
||||
export async function runSetup(config: WikiEngineConfig): Promise<SetupResult> {
|
||||
const messages: string[] = [];
|
||||
let firstRun = false;
|
||||
|
||||
// 1. Ensure directory structure exists
|
||||
const habraidExists = fs.existsSync(HABRAID_HOME);
|
||||
const vaultExists = fs.existsSync(resolvePath(config.vault.path));
|
||||
if (!habraidExists || !vaultExists) {
|
||||
await createDirectoryStructure(resolvePath(config.vault.path));
|
||||
messages.push(`Created HaBraid runtime home and primary vault directories.`);
|
||||
firstRun = true;
|
||||
}
|
||||
|
||||
// 2. Detect MemPalace
|
||||
const effectiveConfig = config.mempalace.path
|
||||
? config
|
||||
: { ...config, mempalace: { ...config.mempalace, path: "", enabled: config.mempalace.enabled } };
|
||||
const vaultPath = resolvePath(effectiveConfig.vault.path);
|
||||
let mempalacePath = config.mempalace.path
|
||||
? resolvePath(config.mempalace.path)
|
||||
: detectMemPalacePath();
|
||||
|
||||
if (mempalacePath) {
|
||||
messages.push(`MemPalace found at ${mempalacePath}`);
|
||||
// Create symlink if vault/raw/mempalace doesn't point there yet
|
||||
const linkPath = path.join(vaultPath, "raw", "mempalace");
|
||||
try {
|
||||
const existing = fs.readlinkSync(linkPath);
|
||||
if (existing !== mempalacePath) {
|
||||
linkMemPalaceRaw(vaultPath, mempalacePath);
|
||||
}
|
||||
} catch {
|
||||
linkMemPalaceRaw(vaultPath, mempalacePath);
|
||||
}
|
||||
} else {
|
||||
messages.push("MemPalace not found. You can set mempalace.path in config.json later.");
|
||||
}
|
||||
|
||||
// 3. Ensure config file exists
|
||||
const finalConfig = config.mempalace.path
|
||||
? { ...effectiveConfig, mempalace: { ...effectiveConfig.mempalace, path: mempalacePath, enabled: mempalacePath.length > 0 } }
|
||||
: { ...effectiveConfig, mempalace: { ...effectiveConfig.mempalace, path: mempalacePath, enabled: mempalacePath.length > 0 } };
|
||||
|
||||
const wroteConfig = await writeConfigIfMissing(finalConfig);
|
||||
if (wroteConfig) {
|
||||
messages.push("Created default config.json.");
|
||||
}
|
||||
|
||||
// 4. Initialize DB if needed
|
||||
let existingItems = 0;
|
||||
try {
|
||||
const dbPath = resolvePath(finalConfig.db.path);
|
||||
const db = openDatabase(dbPath);
|
||||
existingItems = getItemCount(db);
|
||||
db.close();
|
||||
messages.push(`DB ready (${existingItems} items).`);
|
||||
} catch (error) {
|
||||
messages.push(`DB init: ${error instanceof Error ? error.message : String(error)}`);
|
||||
}
|
||||
|
||||
// 5. First-run Obsidian hint
|
||||
if (firstRun) {
|
||||
messages.push("");
|
||||
messages.push("To view in Obsidian, open this folder as a vault:");
|
||||
messages.push(` ${vaultPath}`);
|
||||
messages.push("This is the primary user-facing wiki path for HaBraid one-click installs.");
|
||||
}
|
||||
|
||||
return { firstRun, mempalacePath, existingItems, messages };
|
||||
}
|
||||
57
src/sources/adapter.ts
Normal file
57
src/sources/adapter.ts
Normal file
@@ -0,0 +1,57 @@
|
||||
/**
|
||||
* Source adapter interface re-export.
|
||||
*
|
||||
* The actual interface is defined in types.ts for central type management.
|
||||
* This module provides the contract documentation and adapter registry.
|
||||
*/
|
||||
|
||||
import type { SourceAdapter, WikiEngineConfig } from "../types.js";
|
||||
|
||||
export type { SourceAdapter };
|
||||
|
||||
/**
|
||||
* Registry of available source adapters.
|
||||
* New adapters register themselves here.
|
||||
*/
|
||||
const adapterRegistry: Map<string, (config: WikiEngineConfig) => SourceAdapter> = new Map();
|
||||
|
||||
/**
|
||||
* Registers a source adapter factory.
|
||||
*
|
||||
* @param name Unique adapter name.
|
||||
* @param factory Factory function that creates the adapter from config.
|
||||
*/
|
||||
export function registerAdapter(name: string, factory: (config: WikiEngineConfig) => SourceAdapter): void {
|
||||
adapterRegistry.set(name, factory);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns all registered adapter names.
|
||||
*
|
||||
* @returns Adapter name list.
|
||||
*/
|
||||
export function getAdapterNames(): string[] {
|
||||
return [...adapterRegistry.keys()];
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates an adapter instance by name.
|
||||
*
|
||||
* @param name Adapter name.
|
||||
* @param config Wiki engine configuration.
|
||||
* @returns Source adapter instance.
|
||||
*/
|
||||
export function createAdapter(name: string, config: WikiEngineConfig): SourceAdapter | undefined {
|
||||
const factory = adapterRegistry.get(name);
|
||||
return factory ? factory(config) : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates all registered adapters and returns them.
|
||||
*
|
||||
* @param config Wiki engine configuration.
|
||||
* @returns Array of adapter instances.
|
||||
*/
|
||||
export function createAllAdapters(config: WikiEngineConfig): SourceAdapter[] {
|
||||
return [...adapterRegistry.values()].map((factory) => factory(config));
|
||||
}
|
||||
38
src/sources/manual.ts
Normal file
38
src/sources/manual.ts
Normal file
@@ -0,0 +1,38 @@
|
||||
/**
|
||||
* Manual source adapter.
|
||||
*
|
||||
* Allows direct item creation via MCP tools or CLI.
|
||||
* This adapter is always available — it represents user-initiated content.
|
||||
*/
|
||||
|
||||
import type { Item, SourceAdapter, WikiEngineConfig } from "../types.js";
|
||||
import { registerAdapter } from "./adapter.js";
|
||||
|
||||
/**
|
||||
* Manual source adapter — always available for direct item creation.
|
||||
*/
|
||||
export class ManualSource implements SourceAdapter {
|
||||
public readonly name = "manual";
|
||||
private readonly _config: WikiEngineConfig;
|
||||
|
||||
constructor(config: WikiEngineConfig) {
|
||||
this._config = config;
|
||||
}
|
||||
|
||||
/**
|
||||
* Manual source is always available.
|
||||
*/
|
||||
isAvailable(): boolean {
|
||||
return true;
|
||||
}
|
||||
|
||||
/**
|
||||
* Manual source does not fetch items — they are created explicitly.
|
||||
*/
|
||||
async fetchItems(): Promise<Item[]> {
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-register this adapter
|
||||
registerAdapter("manual", (config) => new ManualSource(config));
|
||||
152
src/sources/mempalace.ts
Normal file
152
src/sources/mempalace.ts
Normal file
@@ -0,0 +1,152 @@
|
||||
/**
|
||||
* MemPalace source adapter.
|
||||
*
|
||||
* Reads existing raw/ markdown files that were previously ingested from MemPalace
|
||||
* and converts them into Item objects. Does not require direct MemPalace DB access.
|
||||
*
|
||||
* This adapter is only available when:
|
||||
* 1. mempalace.enabled is true in config
|
||||
* 2. The vault's raw/mempalace/ directory exists
|
||||
*/
|
||||
|
||||
import path from "node:path";
|
||||
import { existsSync, readdirSync, readFileSync } from "node:fs";
|
||||
|
||||
import type { Item, SourceAdapter, WikiCategory, WikiEngineConfig } from "../types.js";
|
||||
import { toIsoTimestamp } from "../utils.js";
|
||||
import { parseFrontmatter } from "../utils.js";
|
||||
import { registerAdapter } from "./adapter.js";
|
||||
|
||||
/**
|
||||
* Maps MemPalace wing names to wiki categories.
|
||||
*/
|
||||
const WING_CATEGORY_MAP: Record<string, WikiCategory> = {
|
||||
code: "guides",
|
||||
projects: "projects",
|
||||
infrastructure: "infrastructure",
|
||||
people: "people",
|
||||
decisions: "decisions",
|
||||
topics: "topics",
|
||||
};
|
||||
|
||||
/**
|
||||
* MemPalace source adapter — reads from existing raw/mempalace/ files.
|
||||
*/
|
||||
export class MemPalaceSource implements SourceAdapter {
|
||||
public readonly name = "mempalace";
|
||||
private readonly config: WikiEngineConfig;
|
||||
|
||||
constructor(config: WikiEngineConfig) {
|
||||
this.config = config;
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if MemPalace integration is enabled and raw files exist.
|
||||
*/
|
||||
isAvailable(): boolean {
|
||||
if (!this.config.mempalace.enabled) {
|
||||
return false;
|
||||
}
|
||||
const rawDir = path.join(this.config.vault.path, "raw", "mempalace");
|
||||
return existsSync(rawDir);
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads all raw MemPalace files and converts them to Items.
|
||||
*
|
||||
* @param since Optional date filter (reads all if not provided).
|
||||
* @returns Array of Items from MemPalace drawers.
|
||||
*/
|
||||
async fetchItems(since?: Date): Promise<Item[]> {
|
||||
if (!this.isAvailable()) return [];
|
||||
|
||||
const rawDir = path.join(this.config.vault.path, "raw", "mempalace");
|
||||
const items: Item[] = [];
|
||||
|
||||
this.walkRawDir(rawDir, items, since);
|
||||
|
||||
return items;
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively walks the raw/mempalace/ directory and parses markdown files.
|
||||
*/
|
||||
private walkRawDir(dir: string, items: Item[], since?: Date): void {
|
||||
if (!existsSync(dir)) return;
|
||||
|
||||
const entries = readdirSync(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name.startsWith(".")) continue;
|
||||
|
||||
const fullPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
this.walkRawDir(fullPath, items, since);
|
||||
} else if (entry.name.endsWith(".md")) {
|
||||
const item = this.parseRawFile(fullPath);
|
||||
if (item && (!since || new Date(item.createdAt) > since)) {
|
||||
items.push(item);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a raw MemPalace markdown file into an Item.
|
||||
*/
|
||||
private parseRawFile(filePath: string): Item | null {
|
||||
try {
|
||||
const content = readFileSync(filePath, "utf-8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
const data = parsed.data as {
|
||||
type?: string;
|
||||
source?: string;
|
||||
wing?: string;
|
||||
room?: string;
|
||||
id?: string;
|
||||
drawer_id?: string;
|
||||
created?: string;
|
||||
filed_at?: string;
|
||||
agent?: string;
|
||||
added_by?: string;
|
||||
tags?: string[];
|
||||
};
|
||||
|
||||
// Accept both v2 (type=drawer, drawer_id) and v1 (id, wing, room) formats
|
||||
const drawerId = data.drawer_id ?? data.id;
|
||||
if (!drawerId || !data.wing) return null;
|
||||
|
||||
const category = data.wing ? (WING_CATEGORY_MAP[data.wing] ?? "topics") : null;
|
||||
const timestamp = data.filed_at ?? data.created ?? toIsoTimestamp();
|
||||
|
||||
return {
|
||||
id: `mempalace-${drawerId}`,
|
||||
title: this.extractTitle(parsed.content, data.room ?? "Untitled"),
|
||||
content: parsed.content.trim(),
|
||||
source: "mempalace",
|
||||
category,
|
||||
tags: data.tags ?? [],
|
||||
createdAt: timestamp,
|
||||
updatedAt: timestamp,
|
||||
metadata: {
|
||||
wing: data.wing,
|
||||
room: data.room,
|
||||
agent: data.agent ?? data.added_by,
|
||||
sourceFile: filePath,
|
||||
},
|
||||
};
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts a title from content (first heading or fallback).
|
||||
*/
|
||||
private extractTitle(content: string, fallback: string): string {
|
||||
const match = content.match(/^#\s+(.+)$/m);
|
||||
return match ? match[1].trim() : fallback;
|
||||
}
|
||||
}
|
||||
|
||||
// Auto-register this adapter
|
||||
registerAdapter("mempalace", (config) => new MemPalaceSource(config));
|
||||
159
src/sync/auto-sync.ts
Normal file
159
src/sync/auto-sync.ts
Normal file
@@ -0,0 +1,159 @@
|
||||
/**
|
||||
* Auto-sync: lazy sync logic that detects new items and processes them.
|
||||
*
|
||||
* Called by MCP handlers before each operation to ensure the DB is up-to-date
|
||||
* with the latest raw files.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import { openDatabase, closeDatabase } from "../db/database.js";
|
||||
import { getItemCount, getUngeneratedCount } from "../db/items.js";
|
||||
import { ingestMemPalace } from "../mempalace/ingest.js";
|
||||
import { createAllAdapters } from "../sources/adapter.js";
|
||||
import { upsertItem } from "../db/items.js";
|
||||
import type { WikiEngineConfig } from "../types.js";
|
||||
|
||||
// Auto-register adapters
|
||||
import "../sources/mempalace.js";
|
||||
import "../sources/manual.js";
|
||||
|
||||
/**
|
||||
* Result of a sync check.
|
||||
*/
|
||||
export interface SyncCheckResult {
|
||||
/** Whether any new items were found. */
|
||||
hadNewItems: boolean;
|
||||
/** Number of items ingested. */
|
||||
ingested: number;
|
||||
/** Whether sync is currently running (prevents re-entry). */
|
||||
wasAlreadyRunning: boolean;
|
||||
}
|
||||
|
||||
/** Singleton lock to prevent concurrent syncs. */
|
||||
let syncInProgress = false;
|
||||
|
||||
/**
|
||||
* Counts markdown files in a directory recursively.
|
||||
*/
|
||||
function countMdFiles(dir: string): number {
|
||||
let count = 0;
|
||||
try {
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (entry.name.startsWith(".")) continue;
|
||||
const fullPath = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
count += countMdFiles(fullPath);
|
||||
} else if (entry.isFile() && entry.name.endsWith(".md")) {
|
||||
count++;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// directory doesn't exist
|
||||
}
|
||||
return count;
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks if there are new raw files not yet in the DB.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @returns True if new items need ingestion.
|
||||
*/
|
||||
export function hasNewItems(config: WikiEngineConfig): boolean {
|
||||
// Quick check: compare raw file count vs DB item count
|
||||
const rawPath = config.mempalace.path
|
||||
? path.join(config.mempalace.path)
|
||||
: path.join(config.vault.path, "raw");
|
||||
|
||||
const rawCount = countMdFiles(rawPath);
|
||||
|
||||
let dbCount = 0;
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
dbCount = getItemCount(db);
|
||||
db.close();
|
||||
} catch {
|
||||
// DB not initialized yet
|
||||
}
|
||||
|
||||
return rawCount > dbCount;
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a lazy sync: ingest new items + index them.
|
||||
*
|
||||
* Does NOT run wiki generation (that's heavier — user calls hw_generate or
|
||||
* it runs via the watcher). Only ensures DB items are up to date.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @returns Sync check result.
|
||||
*/
|
||||
export async function ensureSync(config: WikiEngineConfig): Promise<SyncCheckResult> {
|
||||
if (syncInProgress) {
|
||||
return { hadNewItems: false, ingested: 0, wasAlreadyRunning: true };
|
||||
}
|
||||
|
||||
if (!hasNewItems(config)) {
|
||||
return { hadNewItems: false, ingested: 0, wasAlreadyRunning: false };
|
||||
}
|
||||
|
||||
syncInProgress = true;
|
||||
let ingested = 0;
|
||||
|
||||
try {
|
||||
// Ingest from MemPalace if enabled
|
||||
if (config.mempalace.enabled && config.mempalace.path) {
|
||||
try {
|
||||
const result = await ingestMemPalace(config, { full: false });
|
||||
ingested += result.drawersWritten;
|
||||
} catch {
|
||||
// MemPalace ingest failed, continue with adapters
|
||||
}
|
||||
}
|
||||
|
||||
// Sync source adapters → DB
|
||||
const db = openDatabase(config.db.path);
|
||||
try {
|
||||
const adapters = createAllAdapters(config);
|
||||
for (const adapter of adapters) {
|
||||
if (!adapter.isAvailable()) continue;
|
||||
|
||||
const items = await adapter.fetchItems();
|
||||
for (const item of items) {
|
||||
try {
|
||||
upsertItem(db, item);
|
||||
ingested++;
|
||||
} catch {
|
||||
// individual item failure, skip
|
||||
}
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
|
||||
return { hadNewItems: ingested > 0, ingested, wasAlreadyRunning: false };
|
||||
} finally {
|
||||
syncInProgress = false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Gets the count of items that need wiki generation.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @returns Count of ungenerated items.
|
||||
*/
|
||||
export function getPendingWikiCount(config: WikiEngineConfig): number {
|
||||
try {
|
||||
const db = openDatabase(config.db.path);
|
||||
const count = getUngeneratedCount(db);
|
||||
db.close();
|
||||
return count;
|
||||
} catch {
|
||||
return 0;
|
||||
}
|
||||
}
|
||||
114
src/sync/git.ts
Normal file
114
src/sync/git.ts
Normal file
@@ -0,0 +1,114 @@
|
||||
import { simpleGit, type SimpleGit } from "simple-git";
|
||||
|
||||
import { GitSyncError } from "../errors.js";
|
||||
|
||||
/**
|
||||
* Creates a simple-git client rooted at the vault path.
|
||||
*
|
||||
* @param vaultPath Vault repository path.
|
||||
* @returns Git client.
|
||||
*/
|
||||
function createGit(vaultPath: string): SimpleGit {
|
||||
return simpleGit(vaultPath);
|
||||
}
|
||||
|
||||
/**
|
||||
* Pulls from the configured branch if the vault is a git repository.
|
||||
*
|
||||
* @param vaultPath Vault path.
|
||||
* @param branch Branch name.
|
||||
*/
|
||||
export async function gitPull(vaultPath: string, branch: string): Promise<void> {
|
||||
try {
|
||||
const git = createGit(vaultPath);
|
||||
if (!(await git.checkIsRepo())) {
|
||||
return;
|
||||
}
|
||||
|
||||
const remotes = await git.getRemotes();
|
||||
if (!remotes.find((remote) => remote.name === "origin")) {
|
||||
return;
|
||||
}
|
||||
|
||||
await git.pull("origin", branch, { "--rebase": "false" });
|
||||
} catch (error) {
|
||||
throw new GitSyncError("Git pull failed.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Initializes a repository and optional remote if needed.
|
||||
*
|
||||
* @param vaultPath Vault path.
|
||||
* @param branch Branch name.
|
||||
* @param remoteUrl Optional remote URL.
|
||||
*/
|
||||
export async function ensureGitRepo(vaultPath: string, branch: string, remoteUrl?: string): Promise<void> {
|
||||
try {
|
||||
const git = createGit(vaultPath);
|
||||
if (!(await git.checkIsRepo())) {
|
||||
await git.init();
|
||||
await git.checkoutLocalBranch(branch);
|
||||
}
|
||||
|
||||
if (remoteUrl) {
|
||||
const remotes = await git.getRemotes(true);
|
||||
if (!remotes.find((remote) => remote.name === "origin")) {
|
||||
await git.addRemote("origin", remoteUrl);
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
throw new GitSyncError("Failed to initialize git repository.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Commits changed files if needed.
|
||||
*
|
||||
* @param vaultPath Vault path.
|
||||
* @param message Commit message.
|
||||
* @returns Whether a commit was created.
|
||||
*/
|
||||
export async function gitCommitIfNeeded(vaultPath: string, message: string): Promise<boolean> {
|
||||
try {
|
||||
const git = createGit(vaultPath);
|
||||
if (!(await git.checkIsRepo())) {
|
||||
return false;
|
||||
}
|
||||
|
||||
const status = await git.status();
|
||||
if (status.files.length === 0) {
|
||||
return false;
|
||||
}
|
||||
|
||||
await git.add(".");
|
||||
await git.commit(message);
|
||||
return true;
|
||||
} catch (error) {
|
||||
throw new GitSyncError("Git commit failed.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Pushes the current branch to origin.
|
||||
*
|
||||
* @param vaultPath Vault path.
|
||||
* @param branch Branch name.
|
||||
*/
|
||||
export async function gitPush(vaultPath: string, branch: string): Promise<void> {
|
||||
try {
|
||||
const git = createGit(vaultPath);
|
||||
if (!(await git.checkIsRepo())) {
|
||||
return;
|
||||
}
|
||||
|
||||
const remotes = await git.getRemotes();
|
||||
if (!remotes.find((remote) => remote.name === "origin")) {
|
||||
return;
|
||||
}
|
||||
|
||||
await git.push("origin", branch);
|
||||
} catch (error) {
|
||||
throw new GitSyncError("Git push failed.", error as Error);
|
||||
}
|
||||
}
|
||||
43
src/sync/pipeline.ts
Normal file
43
src/sync/pipeline.ts
Normal file
@@ -0,0 +1,43 @@
|
||||
import { GitSyncError } from "../errors.js";
|
||||
import type { WikiEngineConfig } from "../types.js";
|
||||
import { readSyncState, toIsoTimestamp, writeSyncState } from "../utils.js";
|
||||
import { ingestMemPalace } from "../mempalace/ingest.js";
|
||||
import { updateWiki } from "../wiki/generator.js";
|
||||
import { gitCommitIfNeeded, gitPull, gitPush } from "./git.js";
|
||||
|
||||
/**
|
||||
* Options for the top-level sync command.
|
||||
*/
|
||||
export interface SyncOptions {
|
||||
noWiki?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Executes the full pull -> ingest -> wiki -> commit -> push pipeline.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @param options Sync options.
|
||||
*/
|
||||
export async function runSyncPipeline(config: WikiEngineConfig, options: SyncOptions = {}): Promise<void> {
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
await gitPull(vaultPath, config.vault.branch);
|
||||
await ingestMemPalace(config, { full: false });
|
||||
|
||||
if (!options.noWiki) {
|
||||
await updateWiki(config);
|
||||
}
|
||||
|
||||
const committed = await gitCommitIfNeeded(vaultPath, "sync: update vault contents");
|
||||
if (committed) {
|
||||
await gitPush(vaultPath, config.vault.branch);
|
||||
const state = await readSyncState(vaultPath);
|
||||
await writeSyncState(vaultPath, {
|
||||
...state,
|
||||
last_git_push: toIsoTimestamp(),
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
throw new GitSyncError("Sync pipeline failed.", error as Error);
|
||||
}
|
||||
}
|
||||
145
src/sync/watcher.ts
Normal file
145
src/sync/watcher.ts
Normal file
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* File watcher for automatic sync on new MemPalace drawers.
|
||||
*
|
||||
* Watches the raw directory for new/changed .md files and triggers
|
||||
* ingest + index + wiki generation automatically.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import type { WikiEngineConfig } from "../types.js";
|
||||
|
||||
/** Callback type for sync trigger. */
|
||||
export type SyncCallback = (config: WikiEngineConfig) => Promise<void>;
|
||||
|
||||
/** Active watchers for cleanup. */
|
||||
const watchers: fs.FSWatcher[] = [];
|
||||
|
||||
/** Debounce timers per watched directory. */
|
||||
const debounceTimers = new Map<string, ReturnType<typeof setTimeout>>();
|
||||
|
||||
/** Debounce delay in ms. */
|
||||
const DEBOUNCE_MS = 1500;
|
||||
|
||||
/**
|
||||
* Creates a debounced sync callback.
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @param callback Function to call after debounce.
|
||||
* @returns Debounced event handler.
|
||||
*/
|
||||
function createDebouncedHandler(
|
||||
config: WikiEngineConfig,
|
||||
callback: SyncCallback,
|
||||
): (event: string, filename: string | null) => void {
|
||||
return (_event: string, filename: string | null) => {
|
||||
if (!filename || !filename.endsWith(".md")) return;
|
||||
|
||||
const key = config.mempalace.path || config.vault.path;
|
||||
const existing = debounceTimers.get(key);
|
||||
|
||||
if (existing) {
|
||||
clearTimeout(existing);
|
||||
}
|
||||
|
||||
debounceTimers.set(
|
||||
key,
|
||||
setTimeout(async () => {
|
||||
debounceTimers.delete(key);
|
||||
try {
|
||||
await callback(config);
|
||||
} catch {
|
||||
// Silent failure — watcher shouldn't crash
|
||||
}
|
||||
}, DEBOUNCE_MS),
|
||||
);
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Starts watching for file changes.
|
||||
*
|
||||
* On Linux, recursive fs.watch is not supported, so we manually watch
|
||||
* subdirectories one level deep (wing/room structure).
|
||||
*
|
||||
* @param config Effective config.
|
||||
* @param onSync Callback to run when changes are detected.
|
||||
*/
|
||||
export function startWatcher(config: WikiEngineConfig, onSync: SyncCallback): void {
|
||||
const watchPaths: string[] = [];
|
||||
|
||||
// Primary: MemPalace raw path
|
||||
if (config.mempalace.path) {
|
||||
watchPaths.push(config.mempalace.path);
|
||||
}
|
||||
|
||||
// Fallback: vault/raw/
|
||||
const vaultRaw = path.join(config.vault.path, "raw");
|
||||
if (!watchPaths.includes(vaultRaw)) {
|
||||
watchPaths.push(vaultRaw);
|
||||
}
|
||||
|
||||
for (const watchPath of watchPaths) {
|
||||
if (!fs.existsSync(watchPath)) continue;
|
||||
|
||||
const handler = createDebouncedHandler(config, onSync);
|
||||
|
||||
// Watch root
|
||||
try {
|
||||
const watcher = fs.watch(watchPath, { persistent: false }, handler);
|
||||
watchers.push(watcher);
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Watch subdirectories (wing/ level)
|
||||
try {
|
||||
const entries = fs.readdirSync(watchPath, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
if (!entry.isDirectory() || entry.name.startsWith(".")) continue;
|
||||
const subDir = path.join(watchPath, entry.name);
|
||||
try {
|
||||
const watcher = fs.watch(subDir, { persistent: false }, handler);
|
||||
watchers.push(watcher);
|
||||
} catch {
|
||||
// skip unreadable dirs
|
||||
}
|
||||
|
||||
// Watch room/ level (one more deep)
|
||||
try {
|
||||
const subEntries = fs.readdirSync(subDir, { withFileTypes: true });
|
||||
for (const subEntry of subEntries) {
|
||||
if (!subEntry.isDirectory() || subEntry.name.startsWith(".")) continue;
|
||||
const roomDir = path.join(subDir, subEntry.name);
|
||||
try {
|
||||
const watcher = fs.watch(roomDir, { persistent: false }, handler);
|
||||
watchers.push(watcher);
|
||||
} catch {
|
||||
// skip
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// skip
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// skip
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Stops all file watchers.
|
||||
*/
|
||||
export function stopWatcher(): void {
|
||||
for (const watcher of watchers) {
|
||||
watcher.close();
|
||||
}
|
||||
watchers.length = 0;
|
||||
|
||||
for (const timer of debounceTimers.values()) {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
debounceTimers.clear();
|
||||
}
|
||||
506
src/types.ts
Normal file
506
src/types.ts
Normal file
@@ -0,0 +1,506 @@
|
||||
/**
|
||||
* Supported wiki categories.
|
||||
*/
|
||||
export type WikiCategory =
|
||||
| "projects"
|
||||
| "topics"
|
||||
| "decisions"
|
||||
| "people"
|
||||
| "infrastructure"
|
||||
| "guides";
|
||||
|
||||
/**
|
||||
* Supported item source types.
|
||||
*/
|
||||
export type ItemSource = "manual" | "mempalace" | "cli" | "file" | "session";
|
||||
|
||||
/**
|
||||
* Supported LLM execution modes.
|
||||
*
|
||||
* "standalone" is the legacy mode name — still accepted for backwards compat.
|
||||
*/
|
||||
export type LlmMode = "host" | "standalone" | "openai" | "ollama" | "zai";
|
||||
|
||||
/**
|
||||
* Host-side routing preferences for generation.
|
||||
*/
|
||||
export interface HostLlmPreferences {
|
||||
priority?: "fast" | "balanced" | "smart";
|
||||
}
|
||||
|
||||
/**
|
||||
* Concrete backend configuration used for standalone mode or fallback.
|
||||
*/
|
||||
export interface StandaloneLlmConfig {
|
||||
provider: "openai" | "zai" | "glm" | "ollama";
|
||||
model: string;
|
||||
api_url: string;
|
||||
api_key_env: string;
|
||||
max_tokens: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Host backend config stored in llm.host.
|
||||
*/
|
||||
export interface HostLlmBackendConfig {
|
||||
command?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenAI-compatible backend config stored in llm.openai.
|
||||
*/
|
||||
export interface OpenAiLlmBackendConfig {
|
||||
apiKey?: string;
|
||||
apiKeyEnv?: string;
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
maxTokens?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ollama backend config stored in llm.ollama.
|
||||
*/
|
||||
export interface OllamaLlmBackendConfig {
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* ZAI backend config stored in llm.zai.
|
||||
*/
|
||||
export interface ZaiLlmBackendConfig {
|
||||
apiKey?: string;
|
||||
apiKeyEnv?: string;
|
||||
baseUrl?: string;
|
||||
model: string;
|
||||
maxTokens?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Metadata describing which LLM route actually handled a request.
|
||||
*/
|
||||
export interface LlmExecutionMetadata {
|
||||
route: "host" | "fallback" | "standalone";
|
||||
provider?: string;
|
||||
model?: string;
|
||||
fallbackUsed: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result returned from the active LLM route.
|
||||
*/
|
||||
export interface LlmGenerationResult {
|
||||
text: string;
|
||||
metadata: LlmExecutionMetadata;
|
||||
}
|
||||
|
||||
/**
|
||||
* Shared configuration model for the wiki engine.
|
||||
*/
|
||||
export interface WikiEngineConfig {
|
||||
vault: {
|
||||
path: string;
|
||||
git_remote?: string;
|
||||
branch: string;
|
||||
};
|
||||
db: {
|
||||
path: string;
|
||||
};
|
||||
mempalace: {
|
||||
enabled: boolean;
|
||||
path: string;
|
||||
};
|
||||
llm: {
|
||||
mode: LlmMode;
|
||||
preferences?: HostLlmPreferences;
|
||||
fallback?: StandaloneLlmConfig;
|
||||
/** New-style backend config blocks. */
|
||||
host?: HostLlmBackendConfig;
|
||||
openai?: OpenAiLlmBackendConfig;
|
||||
ollama?: OllamaLlmBackendConfig;
|
||||
zai?: ZaiLlmBackendConfig;
|
||||
/** Legacy flat fields — still supported for backwards compat. */
|
||||
provider: string;
|
||||
model: string;
|
||||
api_url: string;
|
||||
api_key_env: string;
|
||||
max_tokens: number;
|
||||
};
|
||||
sync: {
|
||||
timezone: string;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A single knowledge item stored in the local SQLite database.
|
||||
* Generalized from MemPalaceDrawer — the core unit of the wiki engine.
|
||||
*/
|
||||
export interface Item {
|
||||
id: string;
|
||||
title: string;
|
||||
content: string;
|
||||
source: ItemSource;
|
||||
category: WikiCategory | null;
|
||||
tags: string[];
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
metadata: Record<string, unknown>;
|
||||
wikiGeneratedAt?: string | null;
|
||||
wikiSlug?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parameters for creating a new item.
|
||||
*/
|
||||
export interface CreateItemParams {
|
||||
title: string;
|
||||
content: string;
|
||||
source?: ItemSource;
|
||||
category?: WikiCategory | null;
|
||||
tags?: string[];
|
||||
metadata?: Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Parameters for searching items via FTS5.
|
||||
*/
|
||||
export interface SearchParams {
|
||||
query: string;
|
||||
limit?: number;
|
||||
offset?: number;
|
||||
/** Optional source filter — only return items matching these sources. */
|
||||
sourceFilter?: string[];
|
||||
/** Optional category filter — only return items matching these categories. */
|
||||
categoryFilter?: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of an FTS5 search.
|
||||
*/
|
||||
export interface SearchResult {
|
||||
item: Item;
|
||||
rank: number;
|
||||
snippet: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Source adapter interface for pluggable data sources.
|
||||
* MemPalace is one implementation; more can be added later.
|
||||
*/
|
||||
export interface SourceAdapter {
|
||||
/** Human-readable adapter name. */
|
||||
readonly name: string;
|
||||
/** Whether this source is available on the current system. */
|
||||
isAvailable(): boolean;
|
||||
/** Fetch items from this source, optionally since a given date. */
|
||||
fetchItems(since?: Date): Promise<Item[]>;
|
||||
}
|
||||
|
||||
/**
|
||||
* A single drawer record loaded from MemPalace (legacy compat).
|
||||
*/
|
||||
export interface MemPalaceDrawer {
|
||||
id: string;
|
||||
wing: string;
|
||||
room: string;
|
||||
content: string;
|
||||
addedBy: string;
|
||||
sourceFile?: string;
|
||||
createdAt: string;
|
||||
updatedAt: string;
|
||||
tags: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A single fact loaded from the MemPalace knowledge graph.
|
||||
*/
|
||||
export interface KGFact {
|
||||
id: number;
|
||||
subject: string;
|
||||
predicate: string;
|
||||
object: string;
|
||||
validFrom?: string | null;
|
||||
validTo?: string | null;
|
||||
sourceCloset?: string | null;
|
||||
createdAt: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Frontmatter used for raw drawer markdown files.
|
||||
*/
|
||||
export interface RawDrawerFrontmatter {
|
||||
type: "drawer";
|
||||
source: "mempalace";
|
||||
wing: string;
|
||||
room: string;
|
||||
drawer_id: string;
|
||||
created: string;
|
||||
agent: string;
|
||||
tags: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Frontmatter used for generated wiki pages.
|
||||
*/
|
||||
export interface WikiFrontmatter {
|
||||
type: "wiki";
|
||||
category: WikiCategory;
|
||||
title: string;
|
||||
created: string;
|
||||
updated: string;
|
||||
sources: string[];
|
||||
tags: string[];
|
||||
status: "draft" | "stable" | "archived";
|
||||
agent: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Frontmatter for the vault index.
|
||||
*/
|
||||
export interface IndexFrontmatter {
|
||||
type: "index";
|
||||
vault_path: string;
|
||||
last_updated: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Frontmatter for the sync log.
|
||||
*/
|
||||
export interface LogFrontmatter {
|
||||
type: "log";
|
||||
}
|
||||
|
||||
/**
|
||||
* Persistent sync state stored in the vault.
|
||||
*/
|
||||
export interface SyncState {
|
||||
last_ingest?: string;
|
||||
last_wiki_update?: string;
|
||||
last_git_push?: string;
|
||||
last_llm_route?: string;
|
||||
last_llm_provider?: string;
|
||||
last_llm_model?: string;
|
||||
last_fallback_used?: boolean;
|
||||
ingested_drawers: string[];
|
||||
wiki_pages: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of an ingest run.
|
||||
*/
|
||||
export interface IngestResult {
|
||||
drawersProcessed: number;
|
||||
drawersWritten: number;
|
||||
drawerIds: string[];
|
||||
rawFilePaths: string[];
|
||||
skippedExisting: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A parseable wiki file emitted by the LLM orchestration layer.
|
||||
*/
|
||||
export interface GeneratedWikiFile {
|
||||
path: string;
|
||||
content: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Result of a wiki generation run.
|
||||
*/
|
||||
export interface WikiUpdateResult {
|
||||
filesWritten: number;
|
||||
pageSlugs: string[];
|
||||
filePaths: string[];
|
||||
/** Number of new (previously ungenerated) items processed. */
|
||||
newItems?: number;
|
||||
/** Number of changed items whose wiki pages were regenerated. */
|
||||
changedItems?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* A single structured log row for log.md.
|
||||
*/
|
||||
export interface SyncLogEntry {
|
||||
time: string;
|
||||
action: string;
|
||||
target: string;
|
||||
result: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stats returned by the status command.
|
||||
*/
|
||||
export interface VaultStatus {
|
||||
rawCount: number;
|
||||
wikiCount: number;
|
||||
dbItemCount: number;
|
||||
lastIngest?: string;
|
||||
lastWikiUpdate?: string;
|
||||
lastGitPush?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Lint results for the vault.
|
||||
*/
|
||||
export interface LintResult {
|
||||
ok: boolean;
|
||||
issues: string[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Supported wiki lint check types.
|
||||
*/
|
||||
export type LintCheckType =
|
||||
| "orphans"
|
||||
| "broken_links"
|
||||
| "stale"
|
||||
| "ungenerated"
|
||||
| "frontmatter"
|
||||
| "duplicates"
|
||||
| "contradictions";
|
||||
|
||||
/**
|
||||
* Options for wiki lint execution.
|
||||
*/
|
||||
export interface LintOptions {
|
||||
checks?: LintCheckType[];
|
||||
withLlm?: boolean;
|
||||
}
|
||||
|
||||
/**
|
||||
* Orphan wiki page lint result.
|
||||
*/
|
||||
export interface OrphanResult {
|
||||
file: string;
|
||||
title: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Broken wikilink lint result.
|
||||
*/
|
||||
export interface BrokenLinkResult {
|
||||
source: string;
|
||||
target: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stale wiki item result.
|
||||
*/
|
||||
export interface StaleItem {
|
||||
item_id: string;
|
||||
title: string;
|
||||
wiki_slug: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* A pair of semantically duplicate items detected by HNSW.
|
||||
*/
|
||||
export interface DuplicatePair {
|
||||
/** First item ID. */
|
||||
item_a_id: string;
|
||||
/** First item title (or slug). */
|
||||
item_a_title: string;
|
||||
/** Second item ID. */
|
||||
item_b_id: string;
|
||||
/** Second item title (or slug). */
|
||||
item_b_title: string;
|
||||
/** Cosine similarity score (0–1). 1.0 = identical. */
|
||||
similarity: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* A detected contradiction between wiki items or pages.
|
||||
*/
|
||||
export interface ContradictionLintResult {
|
||||
/** Unique contradiction ID from the DB. */
|
||||
id?: number;
|
||||
/** First item/page ID. */
|
||||
item_a_id: string;
|
||||
/** Second item/page ID. */
|
||||
item_b_id: string;
|
||||
/** First item slug (if applicable). */
|
||||
item_a_slug?: string;
|
||||
/** Second item slug (if applicable). */
|
||||
item_b_slug?: string;
|
||||
/** Conflicting field name. */
|
||||
field: string;
|
||||
/** Value from first item. */
|
||||
value_a: string;
|
||||
/** Value from second item. */
|
||||
value_b: string;
|
||||
/** Severity level. */
|
||||
severity: "low" | "medium" | "high";
|
||||
/** Contradiction status. */
|
||||
status: "open" | "resolved" | "false_positive";
|
||||
}
|
||||
|
||||
/**
|
||||
* Comprehensive wiki lint report.
|
||||
*/
|
||||
export interface LintReport {
|
||||
timestamp: string;
|
||||
total_checks: number;
|
||||
duration_ms: number;
|
||||
results: {
|
||||
orphans: OrphanResult[];
|
||||
broken_links: BrokenLinkResult[];
|
||||
stale: StaleItem[];
|
||||
ungenerated: {
|
||||
total: number;
|
||||
ungenerated: number;
|
||||
};
|
||||
frontmatter: string[];
|
||||
duplicates: DuplicatePair[];
|
||||
contradictions: ContradictionLintResult[];
|
||||
};
|
||||
summary: {
|
||||
critical: number;
|
||||
warnings: number;
|
||||
info: number;
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Graph node representation.
|
||||
*/
|
||||
export interface GraphNode {
|
||||
id: string;
|
||||
type: "entity";
|
||||
}
|
||||
|
||||
/**
|
||||
* Graph edge representation.
|
||||
*/
|
||||
export interface GraphEdge {
|
||||
source: string;
|
||||
target: string;
|
||||
predicate: string;
|
||||
sourceCloset?: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Graph export document.
|
||||
*/
|
||||
export interface GraphExport {
|
||||
generatedAt: string;
|
||||
nodes: GraphNode[];
|
||||
edges: GraphEdge[];
|
||||
}
|
||||
|
||||
/**
|
||||
* A detected contradiction between two items or wiki pages.
|
||||
*/
|
||||
export interface Contradiction {
|
||||
id?: number;
|
||||
item_a_id: string;
|
||||
item_b_id: string;
|
||||
item_a_slug?: string;
|
||||
item_b_slug?: string;
|
||||
field: string;
|
||||
value_a: string;
|
||||
value_b: string;
|
||||
severity: "low" | "medium" | "high";
|
||||
status: "open" | "resolved" | "false_positive";
|
||||
resolution?: string;
|
||||
detected_at: string;
|
||||
}
|
||||
313
src/utils.ts
Normal file
313
src/utils.ts
Normal file
@@ -0,0 +1,313 @@
|
||||
import matter from "gray-matter";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promises as fs } from "node:fs";
|
||||
|
||||
import type { SyncState, WikiEngineConfig } from "./types.js";
|
||||
|
||||
/**
|
||||
* Default runtime configuration.
|
||||
*/
|
||||
export const DEFAULT_CONFIG: WikiEngineConfig = {
|
||||
vault: {
|
||||
path: "~/wiki",
|
||||
branch: "main",
|
||||
},
|
||||
db: {
|
||||
path: "~/.habraid/data/habraid.db",
|
||||
},
|
||||
mempalace: {
|
||||
enabled: true,
|
||||
path: "", // auto-detected on first run
|
||||
},
|
||||
llm: {
|
||||
mode: "host",
|
||||
preferences: {
|
||||
priority: "balanced",
|
||||
},
|
||||
fallback: {
|
||||
provider: "zai",
|
||||
model: "glm-5.1",
|
||||
api_url: "https://api.example.com/v1",
|
||||
api_key_env: "GLM_API_KEY",
|
||||
max_tokens: 4096,
|
||||
},
|
||||
provider: "zai",
|
||||
model: "glm-5.1",
|
||||
api_url: "https://api.example.com/v1",
|
||||
api_key_env: "GLM_API_KEY",
|
||||
max_tokens: 4096,
|
||||
},
|
||||
sync: {
|
||||
timezone: "Asia/Seoul",
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Default persistent sync state.
|
||||
*/
|
||||
export const DEFAULT_SYNC_STATE: SyncState = {
|
||||
ingested_drawers: [],
|
||||
wiki_pages: [],
|
||||
};
|
||||
|
||||
/**
|
||||
* Expands a leading tilde into the current user's home directory.
|
||||
*
|
||||
* @param input Path that may begin with `~`.
|
||||
* @returns Expanded absolute-looking path string.
|
||||
*/
|
||||
export function expandHomeDir(input: string): string {
|
||||
if (input === "~") {
|
||||
return os.homedir();
|
||||
}
|
||||
|
||||
if (input.startsWith("~/")) {
|
||||
return path.join(os.homedir(), input.slice(2));
|
||||
}
|
||||
|
||||
return input;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves a path after expanding `~`.
|
||||
*
|
||||
* @param input Path to resolve.
|
||||
* @returns Absolute path.
|
||||
*/
|
||||
export function resolvePath(input: string): string {
|
||||
return path.resolve(expandHomeDir(input));
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts arbitrary text into a filesystem-safe kebab-case slug.
|
||||
*
|
||||
* @param input Source string.
|
||||
* @returns Slugified string.
|
||||
*/
|
||||
export function toKebabCase(input: string): string {
|
||||
return input
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9가-힣\s-]/g, "")
|
||||
.replace(/\s+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^-|-$/g, "") || "untitled";
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a normalized ISO timestamp.
|
||||
*
|
||||
* @param date Source date.
|
||||
* @returns ISO-8601 timestamp string.
|
||||
*/
|
||||
export function toIsoTimestamp(date: Date = new Date()): string {
|
||||
return date.toISOString();
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a `YYYY-MM-DD` date string.
|
||||
*
|
||||
* @param value Source date or date-like string.
|
||||
* @returns Calendar date string.
|
||||
*/
|
||||
export function toDateString(value: Date | string): string {
|
||||
const date = value instanceof Date ? value : new Date(value);
|
||||
return date.toISOString().slice(0, 10);
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts simple lowercase tags from a drawer content body.
|
||||
*
|
||||
* @param content Raw drawer content.
|
||||
* @returns Deduplicated tag list.
|
||||
*/
|
||||
export function extractTags(content: string): string[] {
|
||||
const matches = content.match(/#[\w-]+/g) ?? [];
|
||||
return [...new Set(matches.map((value) => value.slice(1).toLowerCase()))];
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a unique item ID from a prefix and timestamp.
|
||||
*
|
||||
* @param prefix Source prefix (e.g. "manual", "mempalace").
|
||||
* @returns Unique ID string.
|
||||
*/
|
||||
export function generateItemId(prefix: string): string {
|
||||
const ts = Date.now().toString(36);
|
||||
const rand = Math.random().toString(36).slice(2, 6);
|
||||
return `${prefix}-${ts}-${rand}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Ensures a directory exists.
|
||||
*
|
||||
* @param directoryPath Directory path to create.
|
||||
*/
|
||||
export async function ensureDir(directoryPath: string): Promise<void> {
|
||||
try {
|
||||
await fs.mkdir(directoryPath, { recursive: true });
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes a UTF-8 text file after ensuring its parent directory exists.
|
||||
*
|
||||
* @param filePath Destination file path.
|
||||
* @param content File content.
|
||||
*/
|
||||
export async function writeTextFile(filePath: string, content: string): Promise<void> {
|
||||
try {
|
||||
await ensureDir(path.dirname(filePath));
|
||||
await fs.writeFile(filePath, content, "utf8");
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads a UTF-8 file if present.
|
||||
*
|
||||
* @param filePath File path.
|
||||
* @returns File content or `undefined`.
|
||||
*/
|
||||
export async function readTextFileIfExists(filePath: string): Promise<string | undefined> {
|
||||
try {
|
||||
return await fs.readFile(filePath, "utf8");
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code === "ENOENT") {
|
||||
return undefined;
|
||||
}
|
||||
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks whether a path exists.
|
||||
*
|
||||
* @param targetPath File or directory path.
|
||||
* @returns Whether the path exists.
|
||||
*/
|
||||
export async function pathExists(targetPath: string): Promise<boolean> {
|
||||
try {
|
||||
await fs.access(targetPath);
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Recursively lists files below a root directory filtered by suffix.
|
||||
*
|
||||
* @param root Directory to scan.
|
||||
* @param suffix Optional filename suffix.
|
||||
* @returns Absolute file paths.
|
||||
*/
|
||||
export async function listFilesRecursive(root: string, suffix?: string): Promise<string[]> {
|
||||
try {
|
||||
if (!(await pathExists(root))) {
|
||||
return [];
|
||||
}
|
||||
|
||||
const entries = await fs.readdir(root, { withFileTypes: true });
|
||||
const files = await Promise.all(
|
||||
entries.map(async (entry) => {
|
||||
const fullPath = path.join(root, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
return listFilesRecursive(fullPath, suffix);
|
||||
}
|
||||
|
||||
if (!suffix || entry.name.endsWith(suffix)) {
|
||||
return [fullPath];
|
||||
}
|
||||
|
||||
return [];
|
||||
}),
|
||||
);
|
||||
|
||||
return files.flat().sort();
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Loads the sync state from the vault or returns defaults.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Loaded sync state.
|
||||
*/
|
||||
export async function readSyncState(vaultPath: string): Promise<SyncState> {
|
||||
try {
|
||||
const statePath = path.join(vaultPath, ".sync-state.json");
|
||||
const raw = await readTextFileIfExists(statePath);
|
||||
if (!raw) {
|
||||
return { ...DEFAULT_SYNC_STATE };
|
||||
}
|
||||
|
||||
const parsed = JSON.parse(raw) as Partial<SyncState>;
|
||||
return {
|
||||
...DEFAULT_SYNC_STATE,
|
||||
...parsed,
|
||||
ingested_drawers: parsed.ingested_drawers ?? [],
|
||||
wiki_pages: parsed.wiki_pages ?? [],
|
||||
};
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Persists the sync state JSON into the vault.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @param state State object to persist.
|
||||
*/
|
||||
export async function writeSyncState(vaultPath: string, state: SyncState): Promise<void> {
|
||||
try {
|
||||
const statePath = path.join(vaultPath, ".sync-state.json");
|
||||
await writeTextFile(statePath, `${JSON.stringify(state, null, 2)}\n`);
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Serializes markdown with YAML frontmatter.
|
||||
*
|
||||
* @param content Markdown body.
|
||||
* @param data Frontmatter data.
|
||||
* @returns Markdown document.
|
||||
*/
|
||||
export function stringifyFrontmatter<T extends object>(content: string, data: T): string {
|
||||
return matter.stringify(content.trimEnd() ? `${content.trimEnd()}\n` : "", data as Record<string, unknown>);
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses markdown frontmatter.
|
||||
*
|
||||
* @param content Markdown text.
|
||||
* @returns Parsed matter result.
|
||||
*/
|
||||
export function parseFrontmatter(content: string): matter.GrayMatterFile<string> {
|
||||
return matter(content);
|
||||
}
|
||||
|
||||
/**
|
||||
* Truncates long content for prompt construction.
|
||||
*
|
||||
* @param content Source text.
|
||||
* @param maxLength Maximum characters to keep.
|
||||
* @returns Truncated text.
|
||||
*/
|
||||
export function truncateForPrompt(content: string, maxLength: number): string {
|
||||
if (content.length <= maxLength) {
|
||||
return content;
|
||||
}
|
||||
|
||||
return `${content.slice(0, maxLength)}\n... (전문은 raw/ 참조)`;
|
||||
}
|
||||
417
src/vault/daily-log.ts
Normal file
417
src/vault/daily-log.ts
Normal file
@@ -0,0 +1,417 @@
|
||||
/**
|
||||
* Daily work log auto-generation for HaBraid.
|
||||
*
|
||||
* Generates date-based markdown summaries from wiki generation history,
|
||||
* written to `vault/daily/YYYY-MM-DD.md`. Each log lists new and updated
|
||||
* wiki pages grouped by category with Obsidian-compatible `[[backlinks]]`.
|
||||
*/
|
||||
|
||||
import fs from "node:fs";
|
||||
import path from "node:path";
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Item, WikiCategory } from "../types.js";
|
||||
import {
|
||||
ensureDir,
|
||||
parseFrontmatter,
|
||||
stringifyFrontmatter,
|
||||
toDateString,
|
||||
toIsoTimestamp,
|
||||
writeTextFile,
|
||||
} from "../utils.js";
|
||||
|
||||
/**
|
||||
* Row shape for daily-log queries.
|
||||
*/
|
||||
interface DailyLogRow {
|
||||
id: string;
|
||||
title: string;
|
||||
content: string;
|
||||
source: string;
|
||||
category: string | null;
|
||||
tags: string;
|
||||
created_at: string;
|
||||
updated_at: string;
|
||||
metadata: string;
|
||||
wiki_generated_at: string | null;
|
||||
wiki_slug: string | null;
|
||||
content_hash: string | null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts a one-line preview from item content (first non-empty, non-title line).
|
||||
*
|
||||
* @param content Item content string.
|
||||
* @returns Preview string.
|
||||
*/
|
||||
function contentPreview(content: string): string {
|
||||
const lines = content.split("\n");
|
||||
for (const line of lines) {
|
||||
const trimmed = line.trim();
|
||||
if (trimmed && !trimmed.startsWith("#")) {
|
||||
return trimmed.length > 80 ? `${trimmed.slice(0, 80)}…` : trimmed;
|
||||
}
|
||||
}
|
||||
return "";
|
||||
}
|
||||
|
||||
/**
|
||||
* Queries items whose `wiki_generated_at` falls on a given date.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param date ISO date string (`YYYY-MM-DD`).
|
||||
* @returns Matching items.
|
||||
*/
|
||||
function getItemsGeneratedOnDate(db: Database.Database, date: string): Item[] {
|
||||
const rows = db.prepare(`
|
||||
SELECT * FROM items
|
||||
WHERE wiki_generated_at IS NOT NULL
|
||||
AND wiki_generated_at >= ?
|
||||
AND wiki_generated_at < ?
|
||||
ORDER BY wiki_generated_at ASC
|
||||
`).all(`${date}T00:00:00`, `${date}T24:00:00`) as DailyLogRow[];
|
||||
|
||||
return rows.map((row) => ({
|
||||
id: row.id,
|
||||
title: row.title,
|
||||
content: row.content,
|
||||
source: row.source as Item["source"],
|
||||
category: row.category as Item["category"],
|
||||
tags: JSON.parse(row.tags || "[]"),
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
metadata: JSON.parse(row.metadata || "{}"),
|
||||
wikiGeneratedAt: row.wiki_generated_at,
|
||||
wikiSlug: row.wiki_slug,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Queries items whose `wiki_generated_at` falls within a date range (inclusive).
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param startDate Start date (`YYYY-MM-DD`).
|
||||
* @param endDate End date (`YYYY-MM-DD`).
|
||||
* @returns Matching items.
|
||||
*/
|
||||
function getItemsGeneratedInRange(db: Database.Database, startDate: string, endDate: string): Item[] {
|
||||
const rows = db.prepare(`
|
||||
SELECT * FROM items
|
||||
WHERE wiki_generated_at IS NOT NULL
|
||||
AND wiki_generated_at >= ?
|
||||
AND wiki_generated_at < ?
|
||||
ORDER BY wiki_generated_at ASC
|
||||
`).all(`${startDate}T00:00:00`, `${endDate}T24:00:00`) as DailyLogRow[];
|
||||
|
||||
return rows.map((row) => ({
|
||||
id: row.id,
|
||||
title: row.title,
|
||||
content: row.content,
|
||||
source: row.source as Item["source"],
|
||||
category: row.category as Item["category"],
|
||||
tags: JSON.parse(row.tags || "[]"),
|
||||
createdAt: row.created_at,
|
||||
updatedAt: row.updated_at,
|
||||
metadata: JSON.parse(row.metadata || "{}"),
|
||||
wikiGeneratedAt: row.wiki_generated_at,
|
||||
wikiSlug: row.wiki_slug,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Determines whether an item is "new" (first-time wiki generation on this date)
|
||||
* vs "updated" (had a previous wiki generation on an earlier date).
|
||||
*
|
||||
* For daily log purposes: if `created_at` date == target date, it's new;
|
||||
* otherwise it's an update.
|
||||
*
|
||||
* @param item Item to classify.
|
||||
* @param targetDate Date string being logged.
|
||||
* @returns Whether the item is newly created on this date.
|
||||
*/
|
||||
function isNewItem(item: Item, targetDate: string): boolean {
|
||||
const createdDate = item.createdAt.slice(0, 10);
|
||||
return createdDate === targetDate;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a daily work log as markdown for a specific date.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param date Target date in `YYYY-MM-DD` format (default: today).
|
||||
* @returns Formatted markdown string.
|
||||
*/
|
||||
export function generateDailyLog(db: Database.Database, date?: string): string {
|
||||
const targetDate = date ?? toDateString(new Date());
|
||||
const items = getItemsGeneratedOnDate(db, targetDate);
|
||||
|
||||
if (items.length === 0) {
|
||||
return generateEmptyLog(targetDate);
|
||||
}
|
||||
|
||||
const newItems: Item[] = [];
|
||||
const updatedItems: Item[] = [];
|
||||
|
||||
for (const item of items) {
|
||||
if (isNewItem(item, targetDate)) {
|
||||
newItems.push(item);
|
||||
} else {
|
||||
updatedItems.push(item);
|
||||
}
|
||||
}
|
||||
|
||||
// Group by category
|
||||
const categoryGroups = new Map<WikiCategory | string, Item[]>();
|
||||
for (const item of items) {
|
||||
const cat = item.category ?? "uncategorized";
|
||||
const group = categoryGroups.get(cat) ?? [];
|
||||
group.push(item);
|
||||
categoryGroups.set(cat, group);
|
||||
}
|
||||
|
||||
// Collect categories for frontmatter
|
||||
const categories = [...categoryGroups.keys()];
|
||||
|
||||
// Build markdown
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${targetDate} 작업 로그`);
|
||||
lines.push("");
|
||||
|
||||
// New pages section
|
||||
if (newItems.length > 0) {
|
||||
lines.push(`## 새로 생성된 페이지 (${newItems.length})`);
|
||||
for (const item of newItems) {
|
||||
const slug = item.wikiSlug ?? item.id;
|
||||
const preview = contentPreview(item.content);
|
||||
const suffix = preview ? ` — ${preview}` : "";
|
||||
lines.push(`- [[${slug}]]${suffix}`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
// Updated pages section
|
||||
if (updatedItems.length > 0) {
|
||||
lines.push(`## 갱신된 페이지 (${updatedItems.length})`);
|
||||
for (const item of updatedItems) {
|
||||
const slug = item.wikiSlug ?? item.id;
|
||||
const preview = contentPreview(item.content);
|
||||
const suffix = preview ? ` — ${preview}` : "";
|
||||
lines.push(`- [[${slug}]]${suffix}`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
// Category breakdown
|
||||
if (categories.length > 0) {
|
||||
lines.push("## 카테고리별");
|
||||
for (const cat of categories.sort()) {
|
||||
const catItems = categoryGroups.get(cat) ?? [];
|
||||
const displayName = cat.charAt(0).toUpperCase() + cat.slice(1);
|
||||
lines.push(`### ${displayName}`);
|
||||
for (const item of catItems) {
|
||||
const slug = item.wikiSlug ?? item.id;
|
||||
lines.push(`- [[${slug}]]`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
}
|
||||
|
||||
lines.push("---");
|
||||
lines.push(`> HaBraid 자동 생성 | 총 ${items.length}개 항목`);
|
||||
|
||||
// YAML frontmatter
|
||||
const frontmatter = {
|
||||
date: targetDate,
|
||||
type: "daily-log",
|
||||
items: items.length,
|
||||
categories,
|
||||
generated_at: toIsoTimestamp(),
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(lines.join("\n"), frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a daily log for a date range (summary).
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param startDate Start date (`YYYY-MM-DD`).
|
||||
* @param endDate End date (`YYYY-MM-DD`).
|
||||
* @returns Formatted markdown string.
|
||||
*/
|
||||
export function generateDailyLogRange(db: Database.Database, startDate: string, endDate: string): string {
|
||||
const items = getItemsGeneratedInRange(db, startDate, endDate);
|
||||
|
||||
if (items.length === 0) {
|
||||
return generateEmptyLogRange(startDate, endDate);
|
||||
}
|
||||
|
||||
// Group by date
|
||||
const dateGroups = new Map<string, Item[]>();
|
||||
for (const item of items) {
|
||||
const genDate = (item.wikiGeneratedAt ?? "").slice(0, 10);
|
||||
const group = dateGroups.get(genDate) ?? [];
|
||||
group.push(item);
|
||||
dateGroups.set(genDate, group);
|
||||
}
|
||||
|
||||
// Group by category
|
||||
const categoryGroups = new Map<string, Item[]>();
|
||||
for (const item of items) {
|
||||
const cat = item.category ?? "uncategorized";
|
||||
const group = categoryGroups.get(cat) ?? [];
|
||||
group.push(item);
|
||||
categoryGroups.set(cat, group);
|
||||
}
|
||||
|
||||
const categories = [...categoryGroups.keys()];
|
||||
|
||||
// Build markdown
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${startDate} ~ ${endDate} 작업 로그`);
|
||||
lines.push("");
|
||||
|
||||
// Summary
|
||||
lines.push(`총 **${items.length}**개 항목, **${dateGroups.size}**일간의 작업 기록`);
|
||||
lines.push("");
|
||||
|
||||
// Per-date breakdown
|
||||
const sortedDates = [...dateGroups.keys()].sort();
|
||||
for (const d of sortedDates) {
|
||||
const dateItems = dateGroups.get(d) ?? [];
|
||||
lines.push(`## ${d} (${dateItems.length}개)`);
|
||||
for (const item of dateItems) {
|
||||
const slug = item.wikiSlug ?? item.id;
|
||||
const preview = contentPreview(item.content);
|
||||
const suffix = preview ? ` — ${preview}` : "";
|
||||
lines.push(`- [[${slug}]]${suffix}`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
// Category summary
|
||||
lines.push("## 카테고리별");
|
||||
for (const cat of categories.sort()) {
|
||||
const catItems = categoryGroups.get(cat) ?? [];
|
||||
const displayName = cat.charAt(0).toUpperCase() + cat.slice(1);
|
||||
lines.push(`### ${displayName} (${catItems.length})`);
|
||||
for (const item of catItems) {
|
||||
const slug = item.wikiSlug ?? item.id;
|
||||
lines.push(`- [[${slug}]]`);
|
||||
}
|
||||
lines.push("");
|
||||
}
|
||||
|
||||
lines.push("---");
|
||||
lines.push(`> HaBraid 자동 생성 | ${startDate} ~ ${endDate} | 총 ${items.length}개 항목`);
|
||||
|
||||
const frontmatter = {
|
||||
date_range: `${startDate}~${endDate}`,
|
||||
type: "daily-log",
|
||||
items: items.length,
|
||||
days: dateGroups.size,
|
||||
categories,
|
||||
generated_at: toIsoTimestamp(),
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(lines.join("\n"), frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Writes a daily log file to the vault.
|
||||
*
|
||||
* Output path: `vault/daily/YYYY-MM-DD.md`
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param vaultPath Vault root path.
|
||||
* @param date Target date (default: today).
|
||||
* @returns Absolute path of the written file.
|
||||
*/
|
||||
export async function writeDailyLog(db: Database.Database, vaultPath: string, date?: string): Promise<string> {
|
||||
try {
|
||||
const targetDate = date ?? toDateString(new Date());
|
||||
const dailyDir = path.join(vaultPath, "daily");
|
||||
await ensureDir(dailyDir);
|
||||
|
||||
const markdown = generateDailyLog(db, targetDate);
|
||||
const filePath = path.join(dailyDir, `${targetDate}.md`);
|
||||
await writeTextFile(filePath, markdown);
|
||||
|
||||
return filePath;
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the path to the latest daily log file (or undefined if none exists).
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Path to the latest daily log or undefined.
|
||||
*/
|
||||
export async function getLatestDailyLogPath(vaultPath: string): Promise<string | undefined> {
|
||||
const dailyDir = path.join(vaultPath, "daily");
|
||||
if (!fs.existsSync(dailyDir)) return undefined;
|
||||
|
||||
const files = fs.readdirSync(dailyDir)
|
||||
.filter((f) => /^\d{4}-\d{2}-\d{2}\.md$/.test(f))
|
||||
.sort();
|
||||
|
||||
if (files.length === 0) return undefined;
|
||||
return path.join(dailyDir, files[files.length - 1]);
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates an empty daily log for a date with no activity.
|
||||
*
|
||||
* @param date Target date.
|
||||
* @returns Formatted markdown string.
|
||||
*/
|
||||
function generateEmptyLog(date: string): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${date} 작업 로그`);
|
||||
lines.push("");
|
||||
lines.push("오늘 생성된 위키 페이지가 없습니다.");
|
||||
lines.push("");
|
||||
lines.push("---");
|
||||
lines.push(`> HaBraid 자동 생성 | 총 0개 항목`);
|
||||
|
||||
const frontmatter = {
|
||||
date,
|
||||
type: "daily-log",
|
||||
items: 0,
|
||||
categories: [],
|
||||
generated_at: toIsoTimestamp(),
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(lines.join("\n"), frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates an empty range log for a date range with no activity.
|
||||
*
|
||||
* @param startDate Start date.
|
||||
* @param endDate End date.
|
||||
* @returns Formatted markdown string.
|
||||
*/
|
||||
function generateEmptyLogRange(startDate: string, endDate: string): string {
|
||||
const lines: string[] = [];
|
||||
lines.push(`# ${startDate} ~ ${endDate} 작업 로그`);
|
||||
lines.push("");
|
||||
lines.push("해당 기간에 생성된 위키 페이지가 없습니다.");
|
||||
lines.push("");
|
||||
lines.push("---");
|
||||
lines.push(`> HaBraid 자동 생성 | ${startDate} ~ ${endDate} | 총 0개 항목`);
|
||||
|
||||
const frontmatter = {
|
||||
date_range: `${startDate}~${endDate}`,
|
||||
type: "daily-log",
|
||||
items: 0,
|
||||
days: 0,
|
||||
categories: [],
|
||||
generated_at: toIsoTimestamp(),
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(lines.join("\n"), frontmatter);
|
||||
}
|
||||
72
src/vault/index.ts
Normal file
72
src/vault/index.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
import path from "node:path";
|
||||
|
||||
import type { SyncState } from "../types.js";
|
||||
import { listFilesRecursive, parseFrontmatter, stringifyFrontmatter, toDateString, toIsoTimestamp, writeTextFile } from "../utils.js";
|
||||
import { getLatestDailyLogPath } from "./daily-log.js";
|
||||
|
||||
/**
|
||||
* Rebuilds the vault `index.md` document from raw and wiki files.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @param state Current sync state.
|
||||
*/
|
||||
export async function updateVaultIndex(vaultPath: string, state?: SyncState): Promise<void> {
|
||||
try {
|
||||
const rawRoot = path.join(vaultPath, "raw");
|
||||
const wikiRoot = path.join(vaultPath, "wiki");
|
||||
const [rawFiles, wikiFiles] = await Promise.all([
|
||||
listFilesRecursive(rawRoot, ".md"),
|
||||
listFilesRecursive(wikiRoot, ".md"),
|
||||
]);
|
||||
|
||||
const recentLines: string[] = [];
|
||||
|
||||
for (const filePath of rawFiles.slice(-10).reverse()) {
|
||||
const relativePath = path.relative(vaultPath, filePath).replace(/\\/g, "/");
|
||||
const fileName = path.basename(filePath, ".md");
|
||||
recentLines.push(`- [[${relativePath.replace(/\.md$/, "")}|${fileName}]] — raw, ${toDateString(new Date())}`);
|
||||
}
|
||||
|
||||
for (const filePath of wikiFiles.slice(-10).reverse()) {
|
||||
const content = await parseTitle(filePath);
|
||||
const relativePath = path.relative(vaultPath, filePath).replace(/\\/g, "/");
|
||||
recentLines.push(`- [[${relativePath.replace(/\.md$/, "")}|${content.title}]] — ${content.updated}`);
|
||||
}
|
||||
|
||||
// Include link to latest daily log if available
|
||||
const latestLogPath = await getLatestDailyLogPath(vaultPath);
|
||||
const dailyLogSection = latestLogPath
|
||||
? `\n## Daily Log\n\n- [[${path.relative(vaultPath, latestLogPath).replace(/\\/g, "/").replace(/\.md$/, "")}|${path.basename(latestLogPath, ".md")}]]\n`
|
||||
: "";
|
||||
|
||||
const markdown = stringifyFrontmatter(
|
||||
`# Wiki Index\n\n## Recent\n\n${recentLines.join("\n")}\n${dailyLogSection}`,
|
||||
{
|
||||
type: "index",
|
||||
vault_path: vaultPath,
|
||||
last_updated: toIsoTimestamp(),
|
||||
state_snapshot: state?.last_wiki_update ?? state?.last_ingest ?? null,
|
||||
},
|
||||
);
|
||||
|
||||
await writeTextFile(path.join(vaultPath, "index.md"), markdown);
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Parses a wiki title and update date from a markdown file.
|
||||
*
|
||||
* @param filePath Markdown file path.
|
||||
* @returns Parsed title metadata.
|
||||
*/
|
||||
async function parseTitle(filePath: string): Promise<{ title: string; updated: string }> {
|
||||
const { readFile } = await import("node:fs/promises");
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
return {
|
||||
title: String(parsed.data.title ?? path.basename(filePath, ".md")),
|
||||
updated: String(parsed.data.updated ?? toDateString(new Date())),
|
||||
};
|
||||
}
|
||||
146
src/vault/init.ts
Normal file
146
src/vault/init.ts
Normal file
@@ -0,0 +1,146 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { VaultInitError } from "../errors.js";
|
||||
import type { WikiEngineConfig } from "../types.js";
|
||||
import { ensureDir, pathExists, stringifyFrontmatter, toIsoTimestamp, writeSyncState, writeTextFile } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Embedded vault schema document written to `SCHEMA.md`.
|
||||
*/
|
||||
const SCHEMA_BODY = `# Vault Schema
|
||||
|
||||
이 볼트는 habraid이 관리합니다.
|
||||
|
||||
## 디렉토리 구조
|
||||
|
||||
\`\`\`
|
||||
{vault}/
|
||||
├── SCHEMA.md
|
||||
├── index.md
|
||||
├── log.md
|
||||
├── overview.md
|
||||
├── raw/
|
||||
│ └── mempalace/
|
||||
├── wiki/
|
||||
│ ├── projects/
|
||||
│ ├── topics/
|
||||
│ ├── decisions/
|
||||
│ ├── people/
|
||||
│ ├── infrastructure/
|
||||
│ └── guides/
|
||||
└── graph/
|
||||
\`\`\`
|
||||
|
||||
## 핵심 규칙
|
||||
|
||||
- \`raw/\` 아래 파일은 원본 보관용이며 수정하지 않습니다.
|
||||
- 모든 마크다운은 YAML frontmatter를 포함합니다.
|
||||
- wiki 페이지는 Obsidian 백링크 \`[[]]\`를 사용합니다.
|
||||
- 한국어 본문을 기본으로 하되 코드, 경로, 명령어는 원문을 유지합니다.
|
||||
|
||||
## raw/ frontmatter
|
||||
|
||||
\`\`\`yaml
|
||||
---
|
||||
type: drawer
|
||||
source: mempalace
|
||||
wing: infrastructure
|
||||
room: server-config
|
||||
drawer_id: abc123
|
||||
created: "2026-04-15"
|
||||
agent: contributor
|
||||
tags: [hypervisor, container]
|
||||
---
|
||||
\`\`\`
|
||||
|
||||
## wiki/ frontmatter
|
||||
|
||||
\`\`\`yaml
|
||||
---
|
||||
type: wiki
|
||||
category: projects|topics|decisions|people|infrastructure|guides
|
||||
title: 페이지 제목
|
||||
created: "2026-04-15"
|
||||
updated: "2026-04-15"
|
||||
sources: [drawer_id]
|
||||
tags: [tag1]
|
||||
status: draft|stable|archived
|
||||
agent: contributor
|
||||
---
|
||||
\`\`\`
|
||||
`;
|
||||
|
||||
/**
|
||||
* Initializes the expected vault directory structure and seed documents.
|
||||
*
|
||||
* @param config Effective project config.
|
||||
*/
|
||||
export async function initializeVault(config: WikiEngineConfig): Promise<void> {
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
const directories = [
|
||||
vaultPath,
|
||||
path.join(vaultPath, "raw"),
|
||||
path.join(vaultPath, "raw", "mempalace"),
|
||||
path.join(vaultPath, "wiki"),
|
||||
path.join(vaultPath, "wiki", "projects"),
|
||||
path.join(vaultPath, "wiki", "topics"),
|
||||
path.join(vaultPath, "wiki", "decisions"),
|
||||
path.join(vaultPath, "wiki", "people"),
|
||||
path.join(vaultPath, "wiki", "infrastructure"),
|
||||
path.join(vaultPath, "wiki", "guides"),
|
||||
path.join(vaultPath, "graph"),
|
||||
];
|
||||
|
||||
await Promise.all(directories.map((directory) => ensureDir(directory)));
|
||||
|
||||
const schemaPath = path.join(vaultPath, "SCHEMA.md");
|
||||
const indexPath = path.join(vaultPath, "index.md");
|
||||
const logPath = path.join(vaultPath, "log.md");
|
||||
const overviewPath = path.join(vaultPath, "overview.md");
|
||||
|
||||
if (!(await pathExists(schemaPath))) {
|
||||
await writeTextFile(
|
||||
schemaPath,
|
||||
stringifyFrontmatter(SCHEMA_BODY, {
|
||||
type: "schema",
|
||||
updated: toIsoTimestamp(),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
if (!(await pathExists(indexPath))) {
|
||||
const content = stringifyFrontmatter("# Wiki Index\n\n## Recent\n", {
|
||||
type: "index",
|
||||
vault_path: vaultPath,
|
||||
last_updated: toIsoTimestamp(),
|
||||
});
|
||||
await writeTextFile(indexPath, content);
|
||||
}
|
||||
|
||||
if (!(await pathExists(logPath))) {
|
||||
const content = stringifyFrontmatter(
|
||||
"# Sync Log\n\n| 시간 | 작업 | 대상 | 결과 |\n|------|------|------|------|\n",
|
||||
{ type: "log" },
|
||||
);
|
||||
await writeTextFile(logPath, content);
|
||||
}
|
||||
|
||||
if (!(await pathExists(overviewPath))) {
|
||||
await writeTextFile(
|
||||
overviewPath,
|
||||
stringifyFrontmatter("# Overview\n\n초기화된 위키 볼트입니다. `wiki update` 실행 후 요약이 생성됩니다.\n", {
|
||||
type: "overview",
|
||||
updated: toIsoTimestamp(),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
await writeSyncState(vaultPath, {
|
||||
ingested_drawers: [],
|
||||
wiki_pages: [],
|
||||
});
|
||||
} catch (error) {
|
||||
throw new VaultInitError("Failed to initialize vault structure.", error as Error);
|
||||
}
|
||||
}
|
||||
83
src/vault/lint.ts
Normal file
83
src/vault/lint.ts
Normal file
@@ -0,0 +1,83 @@
|
||||
import path from "node:path";
|
||||
|
||||
import { LintError } from "../errors.js";
|
||||
import type { LintResult } from "../types.js";
|
||||
import { listFilesRecursive, parseFrontmatter } from "../utils.js";
|
||||
|
||||
/** Page types allowed inside wiki/ directory. */
|
||||
const VALID_WIKI_TYPES = new Set([
|
||||
"wiki", "entity", "concept", "comparison", "guide", "synthesis",
|
||||
]);
|
||||
|
||||
/** Page types allowed for root-level files (index.md, log.md, etc.). */
|
||||
const VALID_ROOT_TYPES = new Set([
|
||||
"index", "daily-log", "overview",
|
||||
]);
|
||||
|
||||
/** Files under wiki/_archive/ are exempt from type/title checks. */
|
||||
function isArchived(filePath: string): boolean {
|
||||
return filePath.includes(`${path.sep}_archive${path.sep}`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a basic vault integrity check.
|
||||
*
|
||||
* Validates frontmatter, page types, and required fields across
|
||||
* wiki/, raw/, and root-level files.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Lint result summary.
|
||||
*/
|
||||
export async function lintVault(vaultPath: string): Promise<LintResult> {
|
||||
try {
|
||||
const issues: string[] = [];
|
||||
const markdownFiles = await listFilesRecursive(vaultPath, ".md");
|
||||
|
||||
for (const filePath of markdownFiles) {
|
||||
// Skip archived pages
|
||||
if (isArchived(filePath)) continue;
|
||||
|
||||
const { readFile } = await import("node:fs/promises");
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
if (!parsed.matter) {
|
||||
issues.push(`${path.relative(vaultPath, filePath)}: missing YAML frontmatter`);
|
||||
continue;
|
||||
}
|
||||
|
||||
const relPath = path.relative(vaultPath, filePath);
|
||||
|
||||
const normalized = relPath.replace(/\\/g, "/");
|
||||
|
||||
// raw/ files must use type=drawer
|
||||
if (normalized.startsWith("raw/") && parsed.data.type !== "drawer") {
|
||||
issues.push(`${relPath}: raw file must use type=drawer`);
|
||||
}
|
||||
|
||||
// wiki/ subdirectory files: allow multiple page types, require title
|
||||
if (normalized.startsWith("wiki/")) {
|
||||
if (!VALID_WIKI_TYPES.has(parsed.data.type)) {
|
||||
issues.push(`${relPath}: wiki file has invalid type="${parsed.data.type}" (expected one of: ${[...VALID_WIKI_TYPES].join(", ")})`);
|
||||
}
|
||||
if (!parsed.data.title) {
|
||||
issues.push(`${relPath}: wiki file missing title`);
|
||||
}
|
||||
}
|
||||
|
||||
// Root-level / structural files (index.md, daily/, overview.md, etc.)
|
||||
const isStructuralFile = !normalized.startsWith("wiki/")
|
||||
&& !normalized.startsWith("raw/")
|
||||
&& !normalized.includes("/_archive/");
|
||||
if (isStructuralFile && parsed.data.type && !VALID_ROOT_TYPES.has(parsed.data.type)) {
|
||||
// Only warn if type is explicitly set but not a recognized root type
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
ok: issues.length === 0,
|
||||
issues,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new LintError("Vault lint failed.", error as Error);
|
||||
}
|
||||
}
|
||||
40
src/vault/log.ts
Normal file
40
src/vault/log.ts
Normal file
@@ -0,0 +1,40 @@
|
||||
import path from "node:path";
|
||||
|
||||
import type { SyncLogEntry } from "../types.js";
|
||||
import { parseFrontmatter, stringifyFrontmatter, writeTextFile } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Appends structured entries to `log.md`.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @param entries Entries to append.
|
||||
*/
|
||||
export async function appendSyncLog(vaultPath: string, entries: SyncLogEntry[]): Promise<void> {
|
||||
try {
|
||||
const filePath = path.join(vaultPath, "log.md");
|
||||
const { readFile } = await import("node:fs/promises");
|
||||
let existing = "";
|
||||
|
||||
try {
|
||||
existing = await readFile(filePath, "utf8");
|
||||
} catch (error) {
|
||||
if ((error as NodeJS.ErrnoException).code !== "ENOENT") {
|
||||
throw error;
|
||||
}
|
||||
}
|
||||
|
||||
const parsed = existing
|
||||
? parseFrontmatter(existing)
|
||||
: {
|
||||
data: { type: "log" },
|
||||
content: "# Sync Log\n\n| 시간 | 작업 | 대상 | 결과 |\n|------|------|------|------|\n",
|
||||
};
|
||||
|
||||
const rows = entries.map((entry) => `| ${entry.time} | ${entry.action} | ${entry.target} | ${entry.result} |`);
|
||||
const body = `${parsed.content.trimEnd()}\n${rows.join("\n")}\n`;
|
||||
const markdown = stringifyFrontmatter(body, parsed.data as Record<string, unknown>);
|
||||
await writeTextFile(filePath, markdown);
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
170
src/vault/render.ts
Normal file
170
src/vault/render.ts
Normal file
@@ -0,0 +1,170 @@
|
||||
import type { MemPalaceDrawer, RawDrawerFrontmatter, WikiCategory, WikiFrontmatter } from "../types.js";
|
||||
import { stringifyFrontmatter, toDateString, toKebabCase } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Renders a MemPalace drawer as immutable raw markdown.
|
||||
*
|
||||
* @param drawer Drawer to render.
|
||||
* @returns Markdown document.
|
||||
*/
|
||||
export function renderRawDrawerMarkdown(drawer: MemPalaceDrawer): string {
|
||||
const frontmatter: RawDrawerFrontmatter = {
|
||||
type: "drawer",
|
||||
source: "mempalace",
|
||||
wing: drawer.wing,
|
||||
room: drawer.room,
|
||||
drawer_id: drawer.id,
|
||||
created: toDateString(drawer.createdAt),
|
||||
agent: drawer.addedBy,
|
||||
tags: drawer.tags,
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(`# [${drawer.wing} / ${drawer.room}]\n\n${drawer.content}\n`, frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a deterministic fallback wiki page from a drawer.
|
||||
*
|
||||
* @param drawer Drawer source.
|
||||
* @returns Wiki markdown content.
|
||||
*/
|
||||
export function renderFallbackWikiMarkdown(drawer: MemPalaceDrawer): string {
|
||||
const title = `${humanizeSegment(drawer.room)} 정리`;
|
||||
const frontmatter: WikiFrontmatter = {
|
||||
type: "wiki",
|
||||
category: inferWikiCategory(drawer.wing, drawer.room),
|
||||
title,
|
||||
created: toDateString(drawer.createdAt),
|
||||
updated: toDateString(drawer.updatedAt),
|
||||
sources: [drawer.id],
|
||||
tags: drawer.tags,
|
||||
status: "draft",
|
||||
agent: drawer.addedBy,
|
||||
};
|
||||
|
||||
const body = [
|
||||
`# ${title}`,
|
||||
"",
|
||||
"## 개요",
|
||||
`${drawer.wing}/${drawer.room} 서랍의 원본 내용을 정리한 초안입니다.`,
|
||||
"",
|
||||
"## 핵심 내용",
|
||||
drawer.content,
|
||||
"",
|
||||
"## 원본",
|
||||
`- [[raw/mempalace/${drawer.wing}/${drawer.room}/${drawer.id}|${drawer.id}]]`,
|
||||
].join("\n");
|
||||
|
||||
return stringifyFrontmatter(body, frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a deterministic fallback wiki page by merging drawers from the same room.
|
||||
*
|
||||
* @param drawers Drawer sources belonging to the same room.
|
||||
* @returns Wiki markdown content.
|
||||
*/
|
||||
export function renderMergedFallbackWikiMarkdown(drawers: MemPalaceDrawer[]): string {
|
||||
if (drawers.length === 0) {
|
||||
return "";
|
||||
}
|
||||
|
||||
const firstDrawer = drawers[0];
|
||||
const title = `${humanizeSegment(firstDrawer.room)} 정리`;
|
||||
const allIds = drawers.map((drawer) => drawer.id);
|
||||
const allTags = [...new Set(drawers.flatMap((drawer) => drawer.tags))];
|
||||
const frontmatter: WikiFrontmatter = {
|
||||
type: "wiki",
|
||||
category: inferWikiCategory(firstDrawer.wing, firstDrawer.room),
|
||||
title,
|
||||
created: toDateString(firstDrawer.createdAt),
|
||||
updated: toDateString(new Date()),
|
||||
sources: allIds,
|
||||
tags: allTags,
|
||||
status: "draft",
|
||||
agent: firstDrawer.addedBy,
|
||||
};
|
||||
|
||||
const sections = drawers.map((drawer) => {
|
||||
const headerMatch = drawer.content.match(/^#\s+(.+)$/m);
|
||||
const sectionTitle = headerMatch?.[1]?.trim() || drawer.id;
|
||||
|
||||
return [`### ${sectionTitle}`, "", drawer.content, ""].join("\n");
|
||||
});
|
||||
|
||||
const body = [
|
||||
`# ${title}`,
|
||||
"",
|
||||
"## 개요",
|
||||
`${firstDrawer.wing}/${firstDrawer.room} 서랍의 원본 내용을 정리한 초안입니다. 총 ${drawers.length}개 서랍.`,
|
||||
"",
|
||||
"## 핵심 내용",
|
||||
"",
|
||||
...sections,
|
||||
"## 원본 서랍",
|
||||
...drawers.map((drawer) => `- [[raw/mempalace/${drawer.wing}/${drawer.room}/${drawer.id}|${drawer.id}]]`),
|
||||
].join("\n");
|
||||
|
||||
return stringifyFrontmatter(body, frontmatter);
|
||||
}
|
||||
|
||||
/**
|
||||
* Infers a wiki category from a wing and room pair.
|
||||
*
|
||||
* @param wing Wing name.
|
||||
* @param room Room name.
|
||||
* @returns Wiki category.
|
||||
*/
|
||||
export function inferWikiCategory(wing: string, room: string): WikiCategory {
|
||||
const value = `${wing}/${room}`.toLowerCase();
|
||||
if (value.includes("project")) {
|
||||
return "projects";
|
||||
}
|
||||
if (value.includes("decision")) {
|
||||
return "decisions";
|
||||
}
|
||||
if (value.includes("people") || value.includes("person") || value.includes("family")) {
|
||||
return "people";
|
||||
}
|
||||
if (value.includes("infra") || value.includes("network") || value.includes("server") || value.includes("ops")) {
|
||||
return "infrastructure";
|
||||
}
|
||||
if (value.includes("guide") || value.includes("workflow") || value.includes("runbook")) {
|
||||
return "guides";
|
||||
}
|
||||
return "topics";
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the default raw drawer file name.
|
||||
*
|
||||
* @param drawerId Drawer identifier.
|
||||
* @returns File name string.
|
||||
*/
|
||||
export function getRawDrawerFileName(drawerId: string): string {
|
||||
return `${drawerId}.md`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a wiki file slug from a title.
|
||||
*
|
||||
* @param title Wiki page title.
|
||||
* @returns Slug string.
|
||||
*/
|
||||
export function getWikiSlug(title: string): string {
|
||||
return toKebabCase(title);
|
||||
}
|
||||
|
||||
/**
|
||||
* Humanizes a path segment into a readable title-ish string.
|
||||
*
|
||||
* @param value Segment value.
|
||||
* @returns Humanized text.
|
||||
*/
|
||||
function humanizeSegment(value: string): string {
|
||||
return value
|
||||
.split(/[-_]/g)
|
||||
.filter(Boolean)
|
||||
.map((segment) => segment.charAt(0).toUpperCase() + segment.slice(1))
|
||||
.join(" ");
|
||||
}
|
||||
149
src/wiki/backends/host.ts
Normal file
149
src/wiki/backends/host.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
/**
|
||||
* Host-following backend — delegates inference to the host CLI (Hermes).
|
||||
*
|
||||
* This is the DEFAULT and PREFERRED backend. It shells out to the Hermes
|
||||
* CLI so that the host controls provider/model/routing policy.
|
||||
*
|
||||
* @module wiki/backends/host
|
||||
*/
|
||||
|
||||
import { execFile } from "node:child_process";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { promisify } from "node:util";
|
||||
|
||||
import { LlmCallError } from "../../errors.js";
|
||||
import type { ChatMessage, ChatOptions, HostBackendConfig } from "./types.js";
|
||||
|
||||
const execFileAsync = promisify(execFile);
|
||||
|
||||
/**
|
||||
* Resolves the Hermes CLI binary path.
|
||||
*/
|
||||
function getHermesCliPath(command?: string): string {
|
||||
return process.env.HERMES_CLI_PATH ?? command ?? "hermes";
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads the current Hermes model/provider from ~/.hermes/config.yaml.
|
||||
*/
|
||||
function readHermesModelMetadata(): { provider?: string; model?: string } {
|
||||
const configPath = path.join(os.homedir(), ".hermes", "config.yaml");
|
||||
if (!existsSync(configPath)) return {};
|
||||
|
||||
try {
|
||||
const lines = readFileSync(configPath, "utf-8").split("\n");
|
||||
let inModelBlock = false;
|
||||
let provider: string | undefined;
|
||||
let model: string | undefined;
|
||||
|
||||
for (const line of lines) {
|
||||
if (!inModelBlock) {
|
||||
if (line.trim() === "model:") {
|
||||
inModelBlock = true;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!line.startsWith(" ") && line.trim()) {
|
||||
break;
|
||||
}
|
||||
|
||||
const trimmed = line.trim();
|
||||
if (trimmed.startsWith("provider:")) {
|
||||
provider = trimmed.slice("provider:".length).trim();
|
||||
}
|
||||
if (trimmed.startsWith("default:")) {
|
||||
model = trimmed.slice("default:".length).trim();
|
||||
}
|
||||
}
|
||||
|
||||
return { provider, model };
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts the assistant answer from Hermes CLI output.
|
||||
*/
|
||||
function extractHermesChatText(output: string): string {
|
||||
const lines = output.split("\n");
|
||||
const cleaned = lines
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line.length > 0)
|
||||
.filter((line) => !line.startsWith("MemPalace MCP Server starting"))
|
||||
.filter((line) => !line.startsWith("session_id:"))
|
||||
.filter((line) => !line.startsWith("╭─"))
|
||||
.filter((line) => !line.startsWith("╰─"));
|
||||
|
||||
const text = cleaned.join("\n").trim();
|
||||
if (!text) {
|
||||
throw new LlmCallError("Hermes CLI bridge returned empty output.");
|
||||
}
|
||||
return text;
|
||||
}
|
||||
|
||||
/**
|
||||
* Host-following backend that delegates to the Hermes CLI.
|
||||
*/
|
||||
export class HostBackend {
|
||||
readonly name = "host";
|
||||
private readonly command: string;
|
||||
|
||||
constructor(config?: HostBackendConfig) {
|
||||
this.command = config?.command ?? "hermes";
|
||||
}
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
if (process.env.HABRAID_DISABLE_HERMES_BRIDGE === "1") {
|
||||
return false;
|
||||
}
|
||||
|
||||
try {
|
||||
await execFileAsync(getHermesCliPath(this.command), ["--version"], { timeout: 15_000 });
|
||||
return true;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async chat(messages: ChatMessage[], _options?: ChatOptions): Promise<string> {
|
||||
// Build a single prompt from system + user messages
|
||||
const systemMsg = messages.find((m) => m.role === "system")?.content ?? "";
|
||||
const userMsg = messages.find((m) => m.role === "user")?.content ?? "";
|
||||
const assistantMsg = messages.find((m) => m.role === "assistant")?.content;
|
||||
|
||||
const prompt = [
|
||||
"You are acting as the host-side wiki generation model for HaBraid.",
|
||||
"Do not use any tools.",
|
||||
"Return only the final markdown content with no preamble.",
|
||||
"",
|
||||
"[SYSTEM INSTRUCTIONS]",
|
||||
systemMsg,
|
||||
"",
|
||||
"[USER REQUEST]",
|
||||
userMsg,
|
||||
...(assistantMsg ? ["", "[ASSISTANT CONTEXT]", assistantMsg] : []),
|
||||
].join("\n");
|
||||
|
||||
try {
|
||||
const { stdout, stderr } = await execFileAsync(
|
||||
getHermesCliPath(this.command),
|
||||
["chat", "-q", prompt, "--toolsets", "", "--quiet"],
|
||||
{ timeout: 180_000, maxBuffer: 1024 * 1024 * 8 },
|
||||
);
|
||||
return extractHermesChatText(`${stdout}\n${stderr}`);
|
||||
} catch (error) {
|
||||
throw new LlmCallError("Hermes CLI host bridge failed.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Reads host model metadata for observability.
|
||||
*/
|
||||
getModelMetadata(): { provider?: string; model?: string } {
|
||||
return readHermesModelMetadata();
|
||||
}
|
||||
}
|
||||
141
src/wiki/backends/index.ts
Normal file
141
src/wiki/backends/index.ts
Normal file
@@ -0,0 +1,141 @@
|
||||
/**
|
||||
* LLM backend factory and registry.
|
||||
*
|
||||
* Creates the appropriate backend based on config, keeping host-following
|
||||
* as the DEFAULT and PREFERRED mode.
|
||||
*
|
||||
* @module wiki/backends
|
||||
*/
|
||||
|
||||
import { LlmCallError } from "../../errors.js";
|
||||
import type { WikiEngineConfig } from "../../types.js";
|
||||
import { HostBackend } from "./host.js";
|
||||
import { OpenAiBackend } from "./openai.js";
|
||||
import { OllamaBackend } from "./ollama.js";
|
||||
import { ZaiBackend } from "./zai.js";
|
||||
import type { LlmBackend } from "./types.js";
|
||||
|
||||
export type { LlmBackend, ChatMessage, ChatOptions, LlmConfig, HostBackendConfig, OpenAiBackendConfig, OllamaBackendConfig, ZaiBackendConfig } from "./types.js";
|
||||
export { HostBackend } from "./host.js";
|
||||
export { OpenAiBackend } from "./openai.js";
|
||||
export { OllamaBackend } from "./ollama.js";
|
||||
export { ZaiBackend } from "./zai.js";
|
||||
|
||||
/**
|
||||
* Determines the effective LLM mode from config.
|
||||
*
|
||||
* Legacy configs without explicit `mode` get "host" as default (per spec).
|
||||
* A config with `provider` but no `mode` is treated as legacy standalone,
|
||||
* but we still default to host with the provider as fallback.
|
||||
*/
|
||||
function resolveEffectiveMode(config: WikiEngineConfig): "host" | "openai" | "ollama" | "zai" {
|
||||
const mode = config.llm.mode;
|
||||
|
||||
// New explicit modes
|
||||
if (mode === "openai" || mode === "ollama" || mode === "zai" || mode === "host") {
|
||||
return mode;
|
||||
}
|
||||
|
||||
// Legacy "standalone" mode — route based on provider
|
||||
if (mode === "standalone") {
|
||||
const provider = (config.llm.provider ?? "").toLowerCase();
|
||||
if (provider === "ollama") return "ollama";
|
||||
if (provider === "zai" || provider === "glm") return "zai";
|
||||
return "openai"; // default standalone to openai-compatible
|
||||
}
|
||||
|
||||
// Default to host
|
||||
return "host";
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a backend from the new-style `llm.host` config block.
|
||||
*/
|
||||
function createHostBackend(config: WikiEngineConfig): HostBackend {
|
||||
return new HostBackend(config.llm.host);
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a backend from the new-style `llm.openai` config block or legacy fields.
|
||||
*/
|
||||
function createOpenAiBackend(config: WikiEngineConfig): OpenAiBackend {
|
||||
const openai = config.llm.openai;
|
||||
if (openai) {
|
||||
return new OpenAiBackend(openai);
|
||||
}
|
||||
|
||||
// Fallback: construct from legacy fields
|
||||
return new OpenAiBackend({
|
||||
baseUrl: config.llm.api_url || "https://api.openai.com/v1",
|
||||
model: config.llm.model || "gpt-4",
|
||||
apiKeyEnv: config.llm.api_key_env,
|
||||
maxTokens: config.llm.max_tokens,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a backend from the new-style `llm.ollama` config block or legacy fields.
|
||||
*/
|
||||
function createOllamaBackend(config: WikiEngineConfig): OllamaBackend {
|
||||
const ollama = config.llm.ollama;
|
||||
if (ollama) {
|
||||
return new OllamaBackend(ollama);
|
||||
}
|
||||
|
||||
// Fallback: construct from legacy fields
|
||||
return new OllamaBackend({
|
||||
baseUrl: config.llm.api_url || "http://localhost:11434",
|
||||
model: config.llm.model || "llama3",
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a backend from the new-style `llm.zai` config block or legacy fields.
|
||||
*/
|
||||
function createZaiBackend(config: WikiEngineConfig): ZaiBackend {
|
||||
const zai = config.llm.zai;
|
||||
if (zai) {
|
||||
return new ZaiBackend(zai);
|
||||
}
|
||||
|
||||
// Fallback: construct from legacy fields
|
||||
return new ZaiBackend({
|
||||
baseUrl: config.llm.api_url,
|
||||
model: config.llm.model || "glm-5.1",
|
||||
apiKeyEnv: config.llm.api_key_env || "GLM_API_KEY",
|
||||
maxTokens: config.llm.max_tokens,
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates the appropriate LLM backend based on config.
|
||||
*
|
||||
* Host-following is the DEFAULT. Other backends are fallback/independent only.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @param overrideMode Optional runtime override (e.g. from CLI --backend).
|
||||
* @returns The resolved LLM backend.
|
||||
*/
|
||||
export function createBackend(config: WikiEngineConfig, overrideMode?: string): LlmBackend {
|
||||
const mode = (overrideMode ?? resolveEffectiveMode(config)) as "host" | "openai" | "ollama" | "zai";
|
||||
|
||||
switch (mode) {
|
||||
case "host":
|
||||
return createHostBackend(config);
|
||||
case "openai":
|
||||
return createOpenAiBackend(config);
|
||||
case "ollama":
|
||||
return createOllamaBackend(config);
|
||||
case "zai":
|
||||
return createZaiBackend(config);
|
||||
default:
|
||||
throw new LlmCallError(`Unknown LLM backend mode: ${mode}`);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns the human-readable name of the active backend mode.
|
||||
*/
|
||||
export function getBackendModeName(config: WikiEngineConfig): string {
|
||||
return resolveEffectiveMode(config);
|
||||
}
|
||||
96
src/wiki/backends/ollama.ts
Normal file
96
src/wiki/backends/ollama.ts
Normal file
@@ -0,0 +1,96 @@
|
||||
/**
|
||||
* Ollama local LLM backend — no API key needed.
|
||||
*
|
||||
* @module wiki/backends/ollama
|
||||
*/
|
||||
|
||||
import { LlmCallError } from "../../errors.js";
|
||||
import type { ChatMessage, ChatOptions, OllamaBackendConfig } from "./types.js";
|
||||
|
||||
/**
|
||||
* Fetch wrapper with timeout.
|
||||
*/
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
options: RequestInit,
|
||||
timeoutMs: number,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
...options,
|
||||
signal: controller.signal,
|
||||
});
|
||||
return response;
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.name === "AbortError") {
|
||||
throw new LlmCallError(`Ollama request timed out after ${timeoutMs}ms.`);
|
||||
}
|
||||
throw new LlmCallError("Failed to call Ollama backend.", error as Error);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Ollama local LLM backend.
|
||||
*/
|
||||
export class OllamaBackend {
|
||||
readonly name = "ollama";
|
||||
private readonly baseUrl: string;
|
||||
private readonly defaultModel: string;
|
||||
|
||||
constructor(config: OllamaBackendConfig) {
|
||||
this.baseUrl = config.baseUrl;
|
||||
this.defaultModel = config.model;
|
||||
}
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
try {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 5_000);
|
||||
const response = await fetch(`${this.baseUrl}/api/tags`, {
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timeout);
|
||||
return response.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async chat(messages: ChatMessage[], options?: ChatOptions): Promise<string> {
|
||||
const model = options?.model ?? this.defaultModel;
|
||||
|
||||
const response = await fetchWithTimeout(
|
||||
`${this.baseUrl}/api/chat`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: { "Content-Type": "application/json" },
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
stream: false,
|
||||
messages: messages.map((m) => ({ role: m.role, content: m.content })),
|
||||
options: options?.temperature !== undefined
|
||||
? { temperature: options.temperature }
|
||||
: undefined,
|
||||
}),
|
||||
},
|
||||
120_000,
|
||||
);
|
||||
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new LlmCallError(`Ollama request failed with ${response.status}: ${body}`);
|
||||
}
|
||||
|
||||
const data = (await response.json()) as { message?: { content?: string } };
|
||||
const content = data.message?.content;
|
||||
if (!content) {
|
||||
throw new LlmCallError("Ollama response did not contain any message content.");
|
||||
}
|
||||
return content;
|
||||
}
|
||||
}
|
||||
133
src/wiki/backends/openai.ts
Normal file
133
src/wiki/backends/openai.ts
Normal file
@@ -0,0 +1,133 @@
|
||||
/**
|
||||
* OpenAI-compatible backend — works with OpenAI, DeepSeek, any compatible endpoint.
|
||||
*
|
||||
* @module wiki/backends/openai
|
||||
*/
|
||||
|
||||
import { LlmCallError } from "../../errors.js";
|
||||
import type { ChatMessage, ChatOptions, OpenAiBackendConfig } from "./types.js";
|
||||
|
||||
/**
|
||||
* Shared fetch wrapper with timeout and error handling.
|
||||
*/
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
options: RequestInit,
|
||||
timeoutMs: number,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
...options,
|
||||
signal: controller.signal,
|
||||
});
|
||||
return response;
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.name === "AbortError") {
|
||||
throw new LlmCallError(`LLM request timed out after ${timeoutMs}ms.`);
|
||||
}
|
||||
throw new LlmCallError("Failed to call the LLM backend.", error as Error);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts content from an OpenAI-compatible chat completions response.
|
||||
*/
|
||||
function extractContent(data: unknown): string {
|
||||
const resp = data as {
|
||||
choices?: Array<{ message?: { content?: string; reasoning_content?: string } }>;
|
||||
};
|
||||
const msg = resp.choices?.[0]?.message;
|
||||
const content = msg?.content || msg?.reasoning_content;
|
||||
if (!content) {
|
||||
throw new LlmCallError("LLM response did not contain any message content.");
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves API key from environment variable.
|
||||
*/
|
||||
function resolveApiKey(config: OpenAiBackendConfig): string {
|
||||
// Direct key takes precedence
|
||||
if (config.apiKey) {
|
||||
return config.apiKey;
|
||||
}
|
||||
|
||||
// Try env var
|
||||
if (config.apiKeyEnv) {
|
||||
const key = process.env[config.apiKeyEnv];
|
||||
if (key) return key;
|
||||
}
|
||||
|
||||
// Fallback to "no-key" for services that don't require auth
|
||||
return "no-key";
|
||||
}
|
||||
|
||||
/**
|
||||
* OpenAI-compatible API backend.
|
||||
*/
|
||||
export class OpenAiBackend {
|
||||
readonly name = "openai";
|
||||
private readonly apiKey: string;
|
||||
private readonly baseUrl: string;
|
||||
private readonly defaultModel: string;
|
||||
private readonly defaultMaxTokens: number;
|
||||
|
||||
constructor(config: OpenAiBackendConfig) {
|
||||
this.apiKey = resolveApiKey(config);
|
||||
this.baseUrl = config.baseUrl;
|
||||
this.defaultModel = config.model;
|
||||
this.defaultMaxTokens = config.maxTokens ?? 4096;
|
||||
}
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
try {
|
||||
// Attempt to reach the models endpoint as a liveness check
|
||||
const url = `${this.baseUrl}/models`;
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 10_000);
|
||||
const response = await fetch(url, {
|
||||
headers: { Authorization: `Bearer ${this.apiKey}` },
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timeout);
|
||||
return response.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async chat(messages: ChatMessage[], options?: ChatOptions): Promise<string> {
|
||||
const model = options?.model ?? this.defaultModel;
|
||||
|
||||
const response = await fetchWithTimeout(
|
||||
`${this.baseUrl}/chat/completions`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
max_tokens: options?.maxTokens ?? this.defaultMaxTokens,
|
||||
temperature: options?.temperature,
|
||||
messages: messages.map((m) => ({ role: m.role, content: m.content })),
|
||||
}),
|
||||
},
|
||||
60_000,
|
||||
);
|
||||
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new LlmCallError(`LLM request failed with ${response.status}: ${body}`);
|
||||
}
|
||||
|
||||
return extractContent(await response.json());
|
||||
}
|
||||
}
|
||||
95
src/wiki/backends/types.ts
Normal file
95
src/wiki/backends/types.ts
Normal file
@@ -0,0 +1,95 @@
|
||||
/**
|
||||
* Pluggable LLM backend interface and shared types.
|
||||
*
|
||||
* Every backend implements a simple chat-based contract that the wiki
|
||||
* generator uses to produce markdown content. Host-following remains the
|
||||
* DEFAULT and PREFERRED mode; all other backends are fallback/independent
|
||||
* execution only.
|
||||
*/
|
||||
|
||||
/** A single chat message in the format backends understand. */
|
||||
export interface ChatMessage {
|
||||
role: "system" | "user" | "assistant";
|
||||
content: string;
|
||||
}
|
||||
|
||||
/** Options that callers may pass to influence a single chat request. */
|
||||
export interface ChatOptions {
|
||||
model?: string;
|
||||
temperature?: number;
|
||||
maxTokens?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pluggable LLM backend contract.
|
||||
*
|
||||
* Backends are self-contained: each one knows how to reach its target
|
||||
* service and can report whether it is currently reachable.
|
||||
*/
|
||||
export interface LlmBackend {
|
||||
/** Human-readable backend identifier (e.g. "host", "openai", "ollama", "zai"). */
|
||||
readonly name: string;
|
||||
|
||||
/**
|
||||
* Sends a chat-completion request and returns the assistant text.
|
||||
*
|
||||
* @param messages Ordered conversation messages.
|
||||
* @param options Optional per-request overrides.
|
||||
* @returns The assistant's text response.
|
||||
*/
|
||||
chat(messages: ChatMessage[], options?: ChatOptions): Promise<string>;
|
||||
|
||||
/**
|
||||
* Probes whether this backend is reachable right now.
|
||||
*
|
||||
* @returns `true` when the backend can accept requests.
|
||||
*/
|
||||
isAvailable(): Promise<boolean>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Backend-specific configuration shapes stored in config.json under `llm`.
|
||||
*/
|
||||
export interface HostBackendConfig {
|
||||
command?: string; // default: "hermes"
|
||||
}
|
||||
|
||||
export interface OpenAiBackendConfig {
|
||||
apiKey?: string;
|
||||
apiKeyEnv?: string;
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
maxTokens?: number;
|
||||
}
|
||||
|
||||
export interface OllamaBackendConfig {
|
||||
baseUrl: string;
|
||||
model: string;
|
||||
}
|
||||
|
||||
export interface ZaiBackendConfig {
|
||||
apiKey?: string;
|
||||
apiKeyEnv?: string;
|
||||
baseUrl?: string;
|
||||
model: string;
|
||||
maxTokens?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* The full LLM section of config.json.
|
||||
*
|
||||
* Supports both the new explicit backend blocks and the legacy flat format.
|
||||
*/
|
||||
export interface LlmConfig {
|
||||
mode: "host" | "openai" | "ollama" | "zai";
|
||||
host?: HostBackendConfig;
|
||||
openai?: OpenAiBackendConfig;
|
||||
ollama?: OllamaBackendConfig;
|
||||
zai?: ZaiBackendConfig;
|
||||
/** Legacy flat fields — still supported for backwards compat. */
|
||||
provider?: string;
|
||||
model?: string;
|
||||
api_url?: string;
|
||||
api_key_env?: string;
|
||||
max_tokens?: number;
|
||||
}
|
||||
132
src/wiki/backends/zai.ts
Normal file
132
src/wiki/backends/zai.ts
Normal file
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* z.ai GLM API backend — OpenAI-compatible with reasoning_content fallback.
|
||||
*
|
||||
* @module wiki/backends/zai
|
||||
*/
|
||||
|
||||
import { LlmCallError } from "../../errors.js";
|
||||
import type { ChatMessage, ChatOptions, ZaiBackendConfig } from "./types.js";
|
||||
|
||||
/**
|
||||
* Fetch wrapper with timeout.
|
||||
*/
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
options: RequestInit,
|
||||
timeoutMs: number,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), timeoutMs);
|
||||
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
...options,
|
||||
signal: controller.signal,
|
||||
});
|
||||
return response;
|
||||
} catch (error) {
|
||||
if (error instanceof Error && error.name === "AbortError") {
|
||||
throw new LlmCallError(`LLM request timed out after ${timeoutMs}ms.`);
|
||||
}
|
||||
throw new LlmCallError("Failed to call the z.ai backend.", error as Error);
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts content from a GLM chat completions response.
|
||||
* Handles both standard `content` and GLM's `reasoning_content` fallback.
|
||||
*/
|
||||
function extractContent(data: unknown): string {
|
||||
const resp = data as {
|
||||
choices?: Array<{ message?: { content?: string; reasoning_content?: string } }>;
|
||||
};
|
||||
const msg = resp.choices?.[0]?.message;
|
||||
const content = msg?.content || msg?.reasoning_content;
|
||||
if (!content) {
|
||||
throw new LlmCallError("LLM response did not contain any message content.");
|
||||
}
|
||||
return content;
|
||||
}
|
||||
|
||||
/**
|
||||
* Resolves API key from config or environment.
|
||||
*/
|
||||
function resolveApiKey(config: ZaiBackendConfig): string {
|
||||
if (config.apiKey) {
|
||||
return config.apiKey;
|
||||
}
|
||||
|
||||
const envVar = config.apiKeyEnv ?? "GLM_API_KEY";
|
||||
const key = process.env[envVar];
|
||||
if (!key) {
|
||||
throw new LlmCallError(
|
||||
`Missing API key: tried env var ${envVar} and direct config. Set ${envVar} or configure llm.zai.apiKey.`,
|
||||
);
|
||||
}
|
||||
return key;
|
||||
}
|
||||
|
||||
/**
|
||||
* z.ai GLM API backend.
|
||||
*/
|
||||
export class ZaiBackend {
|
||||
readonly name = "zai";
|
||||
private readonly apiKey: string;
|
||||
private readonly baseUrl: string;
|
||||
private readonly defaultModel: string;
|
||||
private readonly defaultMaxTokens: number;
|
||||
|
||||
constructor(config: ZaiBackendConfig) {
|
||||
this.apiKey = resolveApiKey(config);
|
||||
this.baseUrl = config.baseUrl ?? "https://api.example.com/v1";
|
||||
this.defaultModel = config.model;
|
||||
this.defaultMaxTokens = config.maxTokens ?? 4096;
|
||||
}
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
try {
|
||||
// Simple liveness check — try models endpoint with a short timeout
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 10_000);
|
||||
const response = await fetch(`${this.baseUrl}/models`, {
|
||||
headers: { Authorization: `Bearer ${this.apiKey}` },
|
||||
signal: controller.signal,
|
||||
});
|
||||
clearTimeout(timeout);
|
||||
return response.ok;
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async chat(messages: ChatMessage[], options?: ChatOptions): Promise<string> {
|
||||
const model = options?.model ?? this.defaultModel;
|
||||
|
||||
const response = await fetchWithTimeout(
|
||||
`${this.baseUrl}/chat/completions`,
|
||||
{
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
Authorization: `Bearer ${this.apiKey}`,
|
||||
},
|
||||
body: JSON.stringify({
|
||||
model,
|
||||
max_tokens: options?.maxTokens ?? this.defaultMaxTokens,
|
||||
temperature: options?.temperature,
|
||||
messages: messages.map((m) => ({ role: m.role, content: m.content })),
|
||||
}),
|
||||
},
|
||||
60_000,
|
||||
);
|
||||
|
||||
if (!response.ok) {
|
||||
const body = await response.text();
|
||||
throw new LlmCallError(`LLM request failed with ${response.status}: ${body}`);
|
||||
}
|
||||
|
||||
return extractContent(await response.json());
|
||||
}
|
||||
}
|
||||
572
src/wiki/contradiction.ts
Normal file
572
src/wiki/contradiction.ts
Normal file
@@ -0,0 +1,572 @@
|
||||
/**
|
||||
* Contradiction/conflict detection for HaBraid.
|
||||
*
|
||||
* Rule-based detection of conflicts between knowledge items and wiki pages.
|
||||
* Detects status conflicts, date conflicts, and fact conflicts without LLM usage.
|
||||
* All detected contradictions are stored in the `contradictions` DB table.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Contradiction, Item } from "../types.js";
|
||||
import { listItems } from "../db/items.js";
|
||||
import { toIsoTimestamp } from "../utils.js";
|
||||
|
||||
// ─── Types ─────────────────────────────────────────────────────────────
|
||||
|
||||
/** Contradiction row from the database. */
|
||||
interface ContradictionRow {
|
||||
id: number;
|
||||
item_a_id: string;
|
||||
item_b_id: string;
|
||||
item_a_slug: string | null;
|
||||
item_b_slug: string | null;
|
||||
field: string;
|
||||
value_a: string;
|
||||
value_b: string;
|
||||
severity: string;
|
||||
status: string;
|
||||
resolution: string | null;
|
||||
detected_at: string;
|
||||
resolved_at: string | null;
|
||||
}
|
||||
|
||||
/** Parameters for creating a contradiction record. */
|
||||
export interface ContradictionInput {
|
||||
item_a_id: string;
|
||||
item_b_id: string;
|
||||
item_a_slug?: string;
|
||||
item_b_slug?: string;
|
||||
field: string;
|
||||
value_a: string;
|
||||
value_b: string;
|
||||
severity: "low" | "medium" | "high";
|
||||
}
|
||||
|
||||
/** Resolution info for marking a contradiction resolved. */
|
||||
export interface ContradictionResolution {
|
||||
resolution: string;
|
||||
status: "resolved" | "false_positive";
|
||||
}
|
||||
|
||||
// ─── Internal helpers ──────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Converts a DB row to a Contradiction object.
|
||||
*
|
||||
* @param row Database row.
|
||||
* @returns Contradiction object.
|
||||
*/
|
||||
function rowToContradiction(row: ContradictionRow): Contradiction {
|
||||
return {
|
||||
id: row.id,
|
||||
item_a_id: row.item_a_id,
|
||||
item_b_id: row.item_b_id,
|
||||
item_a_slug: row.item_a_slug ?? undefined,
|
||||
item_b_slug: row.item_b_slug ?? undefined,
|
||||
field: row.field,
|
||||
value_a: row.value_a,
|
||||
value_b: row.value_b,
|
||||
severity: row.severity as Contradiction["severity"],
|
||||
status: row.status as Contradiction["status"],
|
||||
resolution: row.resolution ?? undefined,
|
||||
detected_at: row.detected_at,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Checks whether a contradiction between two items on a given field already exists.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param itemAId First item ID.
|
||||
* @param itemBId Second item ID.
|
||||
* @param field Conflicting field name.
|
||||
* @returns Whether the contradiction already exists.
|
||||
*/
|
||||
function contradictionExists(db: Database.Database, itemAId: string, itemBId: string, field: string): boolean {
|
||||
const row = db.prepare(`
|
||||
SELECT COUNT(*) as count FROM contradictions
|
||||
WHERE item_a_id = ? AND item_b_id = ? AND field = ? AND status = 'open'
|
||||
`).get(itemAId, itemBId, field) as { count: number };
|
||||
return row.count > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes a string value for comparison (lowercase, trimmed).
|
||||
*
|
||||
* @param value Raw value.
|
||||
* @returns Normalized value.
|
||||
*/
|
||||
function normalize(value: string): string {
|
||||
return value.trim().toLowerCase();
|
||||
}
|
||||
|
||||
// ─── Detection rules ──────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Detects status contradictions: items referencing the same project
|
||||
* but with different status metadata values.
|
||||
*
|
||||
* @param existingItems All existing items from DB.
|
||||
* @param newItem Newly added item.
|
||||
* @returns Detected contradictions.
|
||||
*/
|
||||
function detectStatusConflicts(existingItems: Item[], newItem: Item): ContradictionInput[] {
|
||||
const results: ContradictionInput[] = [];
|
||||
const newMeta = newItem.metadata as Record<string, unknown>;
|
||||
const newProject = String(newMeta.project ?? newMeta.project_name ?? "");
|
||||
const newStatus = String(newMeta.status ?? "");
|
||||
|
||||
if (!newProject || !newStatus) return results;
|
||||
|
||||
for (const existing of existingItems) {
|
||||
const exMeta = existing.metadata as Record<string, unknown>;
|
||||
const exProject = String(exMeta.project ?? exMeta.project_name ?? "");
|
||||
const exStatus = String(exMeta.status ?? "");
|
||||
|
||||
if (!exProject || !exStatus) continue;
|
||||
if (normalize(exProject) !== normalize(newProject)) continue;
|
||||
if (normalize(exStatus) === normalize(newStatus)) continue;
|
||||
|
||||
results.push({
|
||||
item_a_id: existing.id,
|
||||
item_b_id: newItem.id,
|
||||
item_a_slug: existing.wikiSlug ?? undefined,
|
||||
item_b_slug: newItem.wikiSlug ?? undefined,
|
||||
field: "status",
|
||||
value_a: exStatus,
|
||||
value_b: newStatus,
|
||||
severity: "high",
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects date contradictions: items about the same event
|
||||
* but with different date metadata.
|
||||
*
|
||||
* @param existingItems All existing items from DB.
|
||||
* @param newItem Newly added item.
|
||||
* @returns Detected contradictions.
|
||||
*/
|
||||
function detectDateConflicts(existingItems: Item[], newItem: Item): ContradictionInput[] {
|
||||
const results: ContradictionInput[] = [];
|
||||
const newMeta = newItem.metadata as Record<string, unknown>;
|
||||
const newEvent = String(newMeta.event ?? newMeta.event_name ?? "");
|
||||
const newDate = String(newMeta.date ?? newMeta.event_date ?? "");
|
||||
|
||||
if (!newEvent || !newDate) return results;
|
||||
|
||||
for (const existing of existingItems) {
|
||||
const exMeta = existing.metadata as Record<string, unknown>;
|
||||
const exEvent = String(exMeta.event ?? exMeta.event_name ?? "");
|
||||
const exDate = String(exMeta.date ?? exMeta.event_date ?? "");
|
||||
|
||||
if (!exEvent || !exDate) continue;
|
||||
if (normalize(exEvent) !== normalize(newEvent)) continue;
|
||||
if (normalize(exDate) === normalize(newDate)) continue;
|
||||
|
||||
results.push({
|
||||
item_a_id: existing.id,
|
||||
item_b_id: newItem.id,
|
||||
item_a_slug: existing.wikiSlug ?? undefined,
|
||||
item_b_slug: newItem.wikiSlug ?? undefined,
|
||||
field: "date",
|
||||
value_a: exDate,
|
||||
value_b: newDate,
|
||||
severity: "medium",
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects fact contradictions: items about the same entity
|
||||
* but with conflicting tags or metadata values.
|
||||
*
|
||||
* Looks for items with the same category and overlapping title keywords
|
||||
* but different tag sets or conflicting metadata fields.
|
||||
*
|
||||
* @param existingItems All existing items from DB.
|
||||
* @param newItem Newly added item.
|
||||
* @returns Detected contradictions.
|
||||
*/
|
||||
function detectFactConflicts(existingItems: Item[], newItem: Item): ContradictionInput[] {
|
||||
const results: ContradictionInput[] = [];
|
||||
const newMeta = newItem.metadata as Record<string, unknown>;
|
||||
const newEntity = String(newMeta.entity ?? newMeta.entity_name ?? "");
|
||||
|
||||
if (!newEntity) return results;
|
||||
|
||||
for (const existing of existingItems) {
|
||||
const exMeta = existing.metadata as Record<string, unknown>;
|
||||
const exEntity = String(exMeta.entity ?? exMeta.entity_name ?? "");
|
||||
|
||||
if (!exEntity) continue;
|
||||
if (normalize(exEntity) !== normalize(newEntity)) continue;
|
||||
|
||||
// Check for conflicting metadata values on the same entity
|
||||
const conflictFields = ["version", "state", "type", "value", "count", "amount"];
|
||||
for (const field of conflictFields) {
|
||||
const exVal = String(exMeta[field] ?? "");
|
||||
const newVal = String(newMeta[field] ?? "");
|
||||
if (!exVal || !newVal) continue;
|
||||
if (normalize(exVal) === normalize(newVal)) continue;
|
||||
|
||||
results.push({
|
||||
item_a_id: existing.id,
|
||||
item_b_id: newItem.id,
|
||||
item_a_slug: existing.wikiSlug ?? undefined,
|
||||
item_b_slug: newItem.wikiSlug ?? undefined,
|
||||
field: `metadata.${field}`,
|
||||
value_a: exVal,
|
||||
value_b: newVal,
|
||||
severity: "low",
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects tag contradictions: items about the same entity with conflicting tags.
|
||||
*
|
||||
* @param existingItems All existing items from DB.
|
||||
* @param newItem Newly added item.
|
||||
* @returns Detected contradictions.
|
||||
*/
|
||||
function detectTagConflicts(existingItems: Item[], newItem: Item): ContradictionInput[] {
|
||||
const results: ContradictionInput[] = [];
|
||||
const newMeta = newItem.metadata as Record<string, unknown>;
|
||||
const newEntity = String(newMeta.entity ?? newMeta.entity_name ?? "");
|
||||
|
||||
if (!newEntity || newItem.tags.length === 0) return results;
|
||||
|
||||
for (const existing of existingItems) {
|
||||
const exMeta = existing.metadata as Record<string, unknown>;
|
||||
const exEntity = String(exMeta.entity ?? exMeta.entity_name ?? "");
|
||||
|
||||
if (!exEntity) continue;
|
||||
if (normalize(exEntity) !== normalize(newEntity)) continue;
|
||||
if (existing.tags.length === 0) continue;
|
||||
|
||||
// Find mutually exclusive tag pairs (e.g., "active"/"deprecated", "stable"/"experimental")
|
||||
const mutualExclusives: Array<[string, string]> = [
|
||||
["active", "deprecated"],
|
||||
["active", "inactive"],
|
||||
["stable", "experimental"],
|
||||
["stable", "unstable"],
|
||||
["complete", "incomplete"],
|
||||
["done", "todo"],
|
||||
["production", "development"],
|
||||
["production", "staging"],
|
||||
];
|
||||
|
||||
const newTagsLower = new Set(newItem.tags.map((t) => t.toLowerCase()));
|
||||
const exTagsLower = new Set(existing.tags.map((t) => t.toLowerCase()));
|
||||
|
||||
for (const [tagA, tagB] of mutualExclusives) {
|
||||
const exHasA = exTagsLower.has(tagA);
|
||||
const exHasB = exTagsLower.has(tagB);
|
||||
const newHasA = newTagsLower.has(tagA);
|
||||
const newHasB = newTagsLower.has(tagB);
|
||||
|
||||
if ((exHasA && newHasB) || (exHasB && newHasA)) {
|
||||
results.push({
|
||||
item_a_id: existing.id,
|
||||
item_b_id: newItem.id,
|
||||
item_a_slug: existing.wikiSlug ?? undefined,
|
||||
item_b_slug: newItem.wikiSlug ?? undefined,
|
||||
field: "tags",
|
||||
value_a: exHasA ? tagA : tagB,
|
||||
value_b: newHasA ? tagA : tagB,
|
||||
severity: "medium",
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
// ─── Wiki content detection ───────────────────────────────────────────
|
||||
|
||||
/** Parsed status line from wiki content. */
|
||||
interface WikiStatusEntry {
|
||||
entity: string;
|
||||
field: string;
|
||||
value: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts structured status entries from wiki markdown content.
|
||||
* Looks for patterns like `- **status**: active` or `- **date**: 2024-01-01`.
|
||||
*
|
||||
* @param content Wiki markdown content.
|
||||
* @param slug Wiki page slug.
|
||||
* @returns Parsed status entries.
|
||||
*/
|
||||
function extractWikiStatusEntries(content: string, slug: string): WikiStatusEntry[] {
|
||||
const entries: WikiStatusEntry[] = [];
|
||||
const lines = content.split("\n");
|
||||
|
||||
for (const line of lines) {
|
||||
// Match patterns like: - **field**: value or - field: value
|
||||
const match = line.match(/^\s*[-*]\s+\*\*(\w+)\*\*:\s*(.+?)$/);
|
||||
if (match) {
|
||||
const [, field, value] = match;
|
||||
const trackableFields = ["status", "date", "state", "version", "type", "stage"];
|
||||
if (trackableFields.includes(field.toLowerCase())) {
|
||||
entries.push({
|
||||
entity: slug,
|
||||
field: field.toLowerCase(),
|
||||
value: value.trim(),
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return entries;
|
||||
}
|
||||
|
||||
// ─── Public API ───────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Detects contradictions between a new item and all existing items.
|
||||
* Uses rule-based detection for status, date, fact, and tag conflicts.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param newItem The newly added/updated item.
|
||||
* @returns Array of detected contradictions (already stored in DB).
|
||||
*/
|
||||
export function detectContradictions(db: Database.Database, newItem: Item): Contradiction[] {
|
||||
const existingItems = listItems(db, 10000).filter((item) => item.id !== newItem.id);
|
||||
|
||||
const allInputs: ContradictionInput[] = [
|
||||
...detectStatusConflicts(existingItems, newItem),
|
||||
...detectDateConflicts(existingItems, newItem),
|
||||
...detectFactConflicts(existingItems, newItem),
|
||||
...detectTagConflicts(existingItems, newItem),
|
||||
];
|
||||
|
||||
const stored: Contradiction[] = [];
|
||||
for (const input of allInputs) {
|
||||
// Avoid duplicates
|
||||
if (contradictionExists(db, input.item_a_id, input.item_b_id, input.field)) continue;
|
||||
const contradiction = markContradiction(db, input);
|
||||
stored.push(contradiction);
|
||||
}
|
||||
|
||||
return stored;
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects contradictions within a wiki page by parsing structured content
|
||||
* and checking against the database.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param wikiContent The wiki page markdown content.
|
||||
* @param slug Wiki page slug.
|
||||
* @returns Array of detected contradictions.
|
||||
*/
|
||||
export function detectWikiContradictions(
|
||||
db: Database.Database,
|
||||
wikiContent: string,
|
||||
slug: string,
|
||||
): Contradiction[] {
|
||||
const entries = extractWikiStatusEntries(wikiContent, slug);
|
||||
if (entries.length === 0) return [];
|
||||
|
||||
const stored: Contradiction[] = [];
|
||||
|
||||
// Check for internal contradictions within the same page
|
||||
const fieldValues = new Map<string, WikiStatusEntry[]>();
|
||||
for (const entry of entries) {
|
||||
const existing = fieldValues.get(entry.field) ?? [];
|
||||
existing.push(entry);
|
||||
fieldValues.set(entry.field, existing);
|
||||
}
|
||||
|
||||
for (const [field, fieldEntries] of fieldValues) {
|
||||
const uniqueValues = new Set(fieldEntries.map((e) => normalize(e.value)));
|
||||
if (uniqueValues.size <= 1) continue;
|
||||
|
||||
// Multiple different values for the same field in one page
|
||||
const values = [...uniqueValues];
|
||||
for (let i = 0; i < values.length - 1; i++) {
|
||||
const input: ContradictionInput = {
|
||||
item_a_id: `wiki:${slug}`,
|
||||
item_b_id: `wiki:${slug}`,
|
||||
item_a_slug: slug,
|
||||
item_b_slug: slug,
|
||||
field: `wiki.${field}`,
|
||||
value_a: values[i],
|
||||
value_b: values[i + 1],
|
||||
severity: "medium",
|
||||
};
|
||||
|
||||
if (!contradictionExists(db, input.item_a_id, input.item_b_id, input.field)) {
|
||||
stored.push(markContradiction(db, input));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return stored;
|
||||
}
|
||||
|
||||
/**
|
||||
* Stores a contradiction record in the database.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param input Contradiction data to store.
|
||||
* @returns The stored Contradiction with assigned ID.
|
||||
*/
|
||||
export function markContradiction(db: Database.Database, input: ContradictionInput): Contradiction {
|
||||
const now = toIsoTimestamp();
|
||||
|
||||
const result = db.prepare(`
|
||||
INSERT INTO contradictions (item_a_id, item_b_id, item_a_slug, item_b_slug, field, value_a, value_b, severity, status, detected_at)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, 'open', ?)
|
||||
`).run(
|
||||
input.item_a_id,
|
||||
input.item_b_id,
|
||||
input.item_a_slug ?? null,
|
||||
input.item_b_slug ?? null,
|
||||
input.field,
|
||||
input.value_a,
|
||||
input.value_b,
|
||||
input.severity,
|
||||
now,
|
||||
);
|
||||
|
||||
const row = db.prepare("SELECT * FROM contradictions WHERE id = ?").get(Number(result.lastInsertRowid)) as ContradictionRow;
|
||||
return rowToContradiction(row);
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves contradictions from the database, optionally filtered by status.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param status Optional status filter ("open", "resolved", "false_positive").
|
||||
* @returns Array of matching contradictions.
|
||||
*/
|
||||
export function getContradictions(db: Database.Database, status?: string): Contradiction[] {
|
||||
let rows: ContradictionRow[];
|
||||
|
||||
if (status) {
|
||||
rows = db.prepare("SELECT * FROM contradictions WHERE status = ? ORDER BY detected_at DESC").all(status) as ContradictionRow[];
|
||||
} else {
|
||||
rows = db.prepare("SELECT * FROM contradictions ORDER BY detected_at DESC").all() as ContradictionRow[];
|
||||
}
|
||||
|
||||
return rows.map(rowToContradiction);
|
||||
}
|
||||
|
||||
/**
|
||||
* Retrieves a single contradiction by ID.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param id Contradiction ID.
|
||||
* @returns The contradiction or undefined if not found.
|
||||
*/
|
||||
export function getContradictionById(db: Database.Database, id: number): Contradiction | undefined {
|
||||
const row = db.prepare("SELECT * FROM contradictions WHERE id = ?").get(id) as ContradictionRow | undefined;
|
||||
return row ? rowToContradiction(row) : undefined;
|
||||
}
|
||||
|
||||
/**
|
||||
* Marks a contradiction as resolved or false positive.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param id Contradiction ID.
|
||||
* @param resolution Resolution info.
|
||||
* @returns Updated contradiction or undefined if not found.
|
||||
*/
|
||||
export function resolveContradiction(
|
||||
db: Database.Database,
|
||||
id: number,
|
||||
resolution: ContradictionResolution,
|
||||
): Contradiction | undefined {
|
||||
const existing = getContradictionById(db, id);
|
||||
if (!existing) return undefined;
|
||||
|
||||
const now = toIsoTimestamp();
|
||||
db.prepare(`
|
||||
UPDATE contradictions SET status = ?, resolution = ?, resolved_at = ? WHERE id = ?
|
||||
`).run(resolution.status, resolution.resolution, now, id);
|
||||
|
||||
return getContradictionById(db, id);
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs a full contradiction scan across all items in the database.
|
||||
* Useful for the lint --contradictions CLI command.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Total number of new contradictions found.
|
||||
*/
|
||||
export function scanAllContradictions(db: Database.Database): number {
|
||||
const items = listItems(db, 10000);
|
||||
let totalNew = 0;
|
||||
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const newItem = items[i];
|
||||
const remaining = items.slice(i + 1);
|
||||
if (remaining.length === 0) continue;
|
||||
|
||||
const inputs: ContradictionInput[] = [
|
||||
...detectStatusConflicts(remaining, newItem),
|
||||
...detectDateConflicts(remaining, newItem),
|
||||
...detectFactConflicts(remaining, newItem),
|
||||
...detectTagConflicts(remaining, newItem),
|
||||
];
|
||||
|
||||
for (const input of inputs) {
|
||||
if (contradictionExists(db, input.item_a_id, input.item_b_id, input.field)) continue;
|
||||
markContradiction(db, input);
|
||||
totalNew++;
|
||||
}
|
||||
}
|
||||
|
||||
return totalNew;
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns summary statistics about contradictions.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @returns Counts by status and severity.
|
||||
*/
|
||||
export function getContradictionStats(db: Database.Database): {
|
||||
total: number;
|
||||
open: number;
|
||||
resolved: number;
|
||||
falsePositive: number;
|
||||
bySeverity: Record<string, number>;
|
||||
} {
|
||||
const total = db.prepare("SELECT COUNT(*) as count FROM contradictions").get() as { count: number };
|
||||
const open = db.prepare("SELECT COUNT(*) as count FROM contradictions WHERE status = 'open'").get() as { count: number };
|
||||
const resolved = db.prepare("SELECT COUNT(*) as count FROM contradictions WHERE status = 'resolved'").get() as { count: number };
|
||||
const falsePositive = db.prepare("SELECT COUNT(*) as count FROM contradictions WHERE status = 'false_positive'").get() as { count: number };
|
||||
|
||||
const severityRows = db.prepare(
|
||||
"SELECT severity, COUNT(*) as count FROM contradictions GROUP BY severity",
|
||||
).all() as Array<{ severity: string; count: number }>;
|
||||
|
||||
const bySeverity: Record<string, number> = {};
|
||||
for (const row of severityRows) {
|
||||
bySeverity[row.severity] = row.count;
|
||||
}
|
||||
|
||||
return {
|
||||
total: total.count,
|
||||
open: open.count,
|
||||
resolved: resolved.count,
|
||||
falsePositive: falsePositive.count,
|
||||
bySeverity,
|
||||
};
|
||||
}
|
||||
860
src/wiki/generator.ts
Normal file
860
src/wiki/generator.ts
Normal file
@@ -0,0 +1,860 @@
|
||||
import path from "node:path";
|
||||
import fs from "node:fs";
|
||||
|
||||
import { LlmCallError } from "../errors.js";
|
||||
import type {
|
||||
GeneratedWikiFile,
|
||||
Item,
|
||||
MemPalaceDrawer,
|
||||
SyncLogEntry,
|
||||
SyncState,
|
||||
WikiEngineConfig,
|
||||
WikiUpdateResult,
|
||||
} from "../types.js";
|
||||
import { openDatabase } from "../db/database.js";
|
||||
import { getUngeneratedItems, markItemsGenerated, listItems, resetWikiGeneration, getItem as getItemFromDb } from "../db/items.js";
|
||||
import { computeItemHash, getChangedItems, markItemHash, getGeneratedSlugs } from "../db/hashing.js";
|
||||
import { addEntity, addRelation, getEntity, getRelations } from "../db/kg.js";
|
||||
import { appendSyncLog } from "../vault/log.js";
|
||||
import { updateVaultIndex } from "../vault/index.js";
|
||||
import { writeDailyLog } from "../vault/daily-log.js";
|
||||
import { inferWikiCategory, renderMergedFallbackWikiMarkdown } from "../vault/render.js";
|
||||
import { createLlmClient } from "./llm.js";
|
||||
import { buildIncrementalWikiPrompt, parseGeneratedWikiFiles, WIKI_SYSTEM_PROMPT } from "./prompts.js";
|
||||
import { injectMermaidDiagrams } from "./mermaid.js";
|
||||
import {
|
||||
listFilesRecursive,
|
||||
parseFrontmatter,
|
||||
pathExists,
|
||||
readSyncState,
|
||||
stringifyFrontmatter,
|
||||
toDateString,
|
||||
toIsoTimestamp,
|
||||
toKebabCase,
|
||||
writeSyncState,
|
||||
writeTextFile,
|
||||
} from "../utils.js";
|
||||
|
||||
const WIKI_GEN_LOG = "/tmp/habraid-wiki-gen.log";
|
||||
|
||||
function log(msg: string): void {
|
||||
const line = `${new Date().toISOString().slice(11, 19)} ${msg}`;
|
||||
try { fs.appendFileSync(WIKI_GEN_LOG, `${line}\n`); } catch { /* ignore */ }
|
||||
}
|
||||
|
||||
const ROOM_BATCH_SIZE = 10;
|
||||
const LLM_TIMEOUT_MS = 180_000;
|
||||
const BATCH_DELAY_MS = 3_000;
|
||||
|
||||
/**
|
||||
* Reads raw drawer markdown files from the vault and parses them into MemPalaceDrawer objects.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Parsed drawer objects.
|
||||
*/
|
||||
function readRawDrawers(vaultPath: string): MemPalaceDrawer[] {
|
||||
const rawDir = path.join(vaultPath, "raw", "mempalace");
|
||||
if (!fs.existsSync(rawDir)) return [];
|
||||
|
||||
const drawers: MemPalaceDrawer[] = [];
|
||||
|
||||
function walk(dir: string): void {
|
||||
const entries = fs.readdirSync(dir, { withFileTypes: true });
|
||||
for (const entry of entries) {
|
||||
const full = path.join(dir, entry.name);
|
||||
if (entry.isDirectory()) {
|
||||
walk(full);
|
||||
} else if (entry.name.endsWith(".md")) {
|
||||
try {
|
||||
const content = fs.readFileSync(full, "utf-8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
const data = parsed.data as Record<string, unknown>;
|
||||
drawers.push({
|
||||
id: String(data.id ?? entry.name.replace(".md", "")),
|
||||
wing: String(data.wing ?? "unknown"),
|
||||
room: String(data.room ?? "unknown"),
|
||||
content: parsed.content.trim(),
|
||||
addedBy: String(data.added_by ?? "mcp"),
|
||||
sourceFile: data.source_file ? String(data.source_file) : undefined,
|
||||
createdAt: String(data.filed_at ?? new Date().toISOString()),
|
||||
updatedAt: String(data.filed_at ?? new Date().toISOString()),
|
||||
tags: [],
|
||||
});
|
||||
} catch {
|
||||
// Skip malformed files
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
walk(rawDir);
|
||||
return drawers;
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts an Item to a MemPalaceDrawer for prompt building compatibility.
|
||||
*/
|
||||
function itemToDrawer(item: Item): MemPalaceDrawer {
|
||||
const meta = item.metadata as Record<string, unknown>;
|
||||
return {
|
||||
id: item.id,
|
||||
wing: String(meta.wing ?? "unknown"),
|
||||
room: String(meta.room ?? "unknown"),
|
||||
content: item.content,
|
||||
addedBy: String(meta.added_by ?? meta.agent ?? "habraid"),
|
||||
sourceFile: meta.source_file ? String(meta.source_file) : undefined,
|
||||
createdAt: item.createdAt,
|
||||
updatedAt: item.updatedAt,
|
||||
tags: item.tags,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates or updates wiki pages from unprocessed and changed items in the database.
|
||||
*
|
||||
* Handles two types of items:
|
||||
* a. New items (no wiki_generated_at) — generate new pages
|
||||
* b. Changed items (content_hash mismatch) — regenerate existing pages
|
||||
*
|
||||
* When force is true, all items are treated as needing (re)generation.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @param options Optional maxRooms limit and force flag.
|
||||
* @returns Wiki update summary.
|
||||
*/
|
||||
export async function updateWiki(config: WikiEngineConfig, options?: { maxRooms?: number; force?: boolean; backend?: string }): Promise<WikiUpdateResult> {
|
||||
try {
|
||||
const vaultPath = config.vault.path;
|
||||
const state = await readSyncState(vaultPath);
|
||||
const existingWikiFiles = await listFilesRecursive(path.join(vaultPath, "wiki"), ".md");
|
||||
const force = options?.force ?? false;
|
||||
|
||||
// Open DB and determine items to process
|
||||
const db = openDatabase(config.db.path);
|
||||
let newItems: Item[];
|
||||
let changedItems: Item[];
|
||||
|
||||
try {
|
||||
if (force) {
|
||||
// Force mode: treat all items as new — reset wiki generation for all
|
||||
const allItems = listItems(db, 10000);
|
||||
const allIds = allItems.map((item) => item.id);
|
||||
resetWikiGeneration(db, allIds);
|
||||
|
||||
// Delete old wiki files for all previously generated slugs
|
||||
const oldSlugs = getGeneratedSlugs(db);
|
||||
for (const slug of oldSlugs) {
|
||||
await deleteWikiFileForSlug(vaultPath, slug);
|
||||
}
|
||||
|
||||
newItems = allItems;
|
||||
changedItems = [];
|
||||
} else {
|
||||
newItems = getUngeneratedItems(db);
|
||||
changedItems = getChangedItems(db);
|
||||
}
|
||||
} finally {
|
||||
db.close();
|
||||
}
|
||||
|
||||
const totalNew = newItems.length;
|
||||
const totalChanged = changedItems.length;
|
||||
|
||||
if (totalNew === 0 && totalChanged === 0) {
|
||||
await rebuildOverview(vaultPath);
|
||||
return {
|
||||
filesWritten: 0,
|
||||
pageSlugs: [],
|
||||
filePaths: [],
|
||||
newItems: 0,
|
||||
changedItems: 0,
|
||||
};
|
||||
}
|
||||
|
||||
// Delete old wiki files for changed items (they'll be regenerated)
|
||||
if (totalChanged > 0) {
|
||||
const changedSlugs = [...new Set(changedItems.map((item) => item.wikiSlug).filter(Boolean))] as string[];
|
||||
for (const slug of changedSlugs) {
|
||||
await deleteWikiFileForSlug(vaultPath, slug);
|
||||
}
|
||||
}
|
||||
|
||||
// Combine new + changed items for processing
|
||||
const allItemsToProcess = [...newItems, ...changedItems];
|
||||
|
||||
// Convert items to drawer format for prompt building
|
||||
const sourceDrawers = allItemsToProcess.map(itemToDrawer);
|
||||
|
||||
const llmClient = createLlmClient(config, options?.backend);
|
||||
const roomGroups = groupDrawersByRoom(sourceDrawers);
|
||||
const writtenPaths: string[] = [];
|
||||
const pageSlugs: string[] = [];
|
||||
let lastLlmMetadata: {
|
||||
route?: string;
|
||||
provider?: string;
|
||||
model?: string;
|
||||
fallbackUsed?: boolean;
|
||||
} | undefined;
|
||||
|
||||
const roomLimit = options?.maxRooms ?? roomGroups.length;
|
||||
const roomsToProcess = roomGroups.slice(0, roomLimit);
|
||||
|
||||
log(`Processing ${roomsToProcess.length}/${roomGroups.length} rooms (${sourceDrawers.length} items: ${totalNew} new, ${totalChanged} changed)`);
|
||||
|
||||
const db2 = openDatabase(config.db.path);
|
||||
try {
|
||||
for (let roomIndex = 0; roomIndex < roomsToProcess.length; roomIndex += 1) {
|
||||
const [roomKey, roomDrawers] = roomGroups[roomIndex];
|
||||
log(`Room ${roomIndex + 1}/${roomsToProcess.length}: ${roomKey} (${roomDrawers.length} items)`);
|
||||
const usedSlugs = new Set<string>();
|
||||
|
||||
const subGroups = splitIntoSubGroups(roomDrawers, 30);
|
||||
|
||||
for (let sgIndex = 0; sgIndex < subGroups.length; sgIndex += 1) {
|
||||
const subGroup = subGroups[sgIndex];
|
||||
const batches = splitIntoBatches(subGroup, ROOM_BATCH_SIZE);
|
||||
const subGeneratedFiles: Array<GeneratedWikiFile & { sourceDrawers: MemPalaceDrawer[] }> = [];
|
||||
let subFailed = false;
|
||||
|
||||
for (let batchIndex = 0; batchIndex < batches.length; batchIndex += 1) {
|
||||
const batch = batches[batchIndex];
|
||||
|
||||
try {
|
||||
const prompt = await buildIncrementalWikiPrompt(batch, existingWikiFiles, vaultPath, {
|
||||
roomKey,
|
||||
batchIndex,
|
||||
totalBatches: batches.length,
|
||||
usedSlugs: [...usedSlugs],
|
||||
});
|
||||
log(` LLM call batch ${batchIndex + 1}/${batches.length} (${batch.length} items)...`);
|
||||
const response = await llmClient.generate(WIKI_SYSTEM_PROMPT, prompt, LLM_TIMEOUT_MS);
|
||||
lastLlmMetadata = {
|
||||
route: response.metadata.route,
|
||||
provider: response.metadata.provider,
|
||||
model: response.metadata.model,
|
||||
fallbackUsed: response.metadata.fallbackUsed,
|
||||
};
|
||||
log(` LLM response received (${response.text.length} chars) via ${response.metadata.route}`);
|
||||
const parsedFiles = parseGeneratedWikiFiles(response.text);
|
||||
|
||||
if (parsedFiles.length === 0) {
|
||||
subFailed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
for (const file of parsedFiles) {
|
||||
const dedupedPath = deduplicateSlug(file.path, usedSlugs);
|
||||
subGeneratedFiles.push({
|
||||
...file,
|
||||
path: dedupedPath,
|
||||
sourceDrawers: batch,
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
if (!(error instanceof LlmCallError)) {
|
||||
throw error;
|
||||
}
|
||||
|
||||
subFailed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
await sleep(BATCH_DELAY_MS);
|
||||
}
|
||||
|
||||
const filesToPersist = subFailed
|
||||
? [{
|
||||
...buildSubGroupFallbackFile(subGroup),
|
||||
sourceDrawers: subGroup,
|
||||
}]
|
||||
: subGeneratedFiles;
|
||||
|
||||
const persisted = await persistGeneratedFiles(db2, vaultPath, filesToPersist, subGroup);
|
||||
writtenPaths.push(...persisted.writtenPaths);
|
||||
pageSlugs.push(...persisted.pageSlugs);
|
||||
existingWikiFiles.push(...persisted.writtenPaths);
|
||||
log(` checkpointed subgroup ${sgIndex + 1}/${subGroups.length}: ${persisted.writtenPaths.length} files, ${persisted.itemIds.length} items`);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
db2.close();
|
||||
}
|
||||
|
||||
const nextState: SyncState = {
|
||||
...state,
|
||||
last_wiki_update: toIsoTimestamp(),
|
||||
last_llm_route: lastLlmMetadata?.route,
|
||||
last_llm_provider: lastLlmMetadata?.provider,
|
||||
last_llm_model: lastLlmMetadata?.model,
|
||||
last_fallback_used: lastLlmMetadata?.fallbackUsed,
|
||||
wiki_pages: [...new Set([...state.wiki_pages, ...pageSlugs])],
|
||||
};
|
||||
|
||||
await writeSyncState(vaultPath, nextState);
|
||||
await rebuildOverview(vaultPath);
|
||||
await updateVaultIndex(vaultPath, nextState);
|
||||
|
||||
const logEntry: SyncLogEntry = {
|
||||
time: new Date().toLocaleString("sv-SE", { timeZone: config.sync.timezone }).replace("T", " "),
|
||||
action: "wiki",
|
||||
target: "incremental -> wiki",
|
||||
result: `+${writtenPaths.length} pages (${totalNew} new, ${totalChanged} changed)`,
|
||||
};
|
||||
await appendSyncLog(vaultPath, [logEntry]);
|
||||
|
||||
// Auto-generate daily log after wiki batch completes
|
||||
try {
|
||||
const db3 = openDatabase(config.db.path);
|
||||
try {
|
||||
await writeDailyLog(db3, vaultPath);
|
||||
} finally {
|
||||
db3.close();
|
||||
}
|
||||
} catch {
|
||||
// Daily log generation failure must not block wiki generation result
|
||||
}
|
||||
|
||||
return {
|
||||
filesWritten: writtenPaths.length,
|
||||
pageSlugs,
|
||||
filePaths: writtenPaths,
|
||||
newItems: totalNew,
|
||||
changedItems: totalChanged,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new LlmCallError("Wiki generation failed.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Deletes a wiki file matching a given slug, searching recursively under wiki/.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @param slug Wiki page slug to delete.
|
||||
*/
|
||||
async function deleteWikiFileForSlug(vaultPath: string, slug: string): Promise<void> {
|
||||
try {
|
||||
const wikiDir = path.join(vaultPath, "wiki");
|
||||
if (!fs.existsSync(wikiDir)) return;
|
||||
|
||||
const files = await listFilesRecursive(wikiDir, ".md");
|
||||
for (const filePath of files) {
|
||||
const fileSlug = path.basename(filePath, ".md");
|
||||
if (fileSlug === slug) {
|
||||
await fs.promises.unlink(filePath);
|
||||
log(` deleted old wiki file: ${path.relative(vaultPath, filePath)}`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// Ignore deletion errors — file may not exist
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits drawers into sub-groups of fixed chunk size.
|
||||
*
|
||||
* Previously split by source_file, but MemPalace drawers don't have
|
||||
* source_file metadata — so every item became its own subgroup.
|
||||
* Now uses fixed chunk size for efficient batching.
|
||||
*
|
||||
* @param drawers Room drawers to split.
|
||||
* @param chunkSize Max drawers per subgroup.
|
||||
* @returns Array of sub-groups.
|
||||
*/
|
||||
function splitIntoSubGroups(drawers: MemPalaceDrawer[], chunkSize = 30): MemPalaceDrawer[][] {
|
||||
const groups: MemPalaceDrawer[][] = [];
|
||||
for (let i = 0; i < drawers.length; i += chunkSize) {
|
||||
groups.push(drawers.slice(i, i + chunkSize));
|
||||
}
|
||||
return groups;
|
||||
}
|
||||
|
||||
async function persistGeneratedFiles(
|
||||
db: ReturnType<typeof openDatabase>,
|
||||
vaultPath: string,
|
||||
files: Array<GeneratedWikiFile & { sourceDrawers: MemPalaceDrawer[] }>,
|
||||
groupDrawers: MemPalaceDrawer[],
|
||||
): Promise<{ writtenPaths: string[]; pageSlugs: string[]; itemIds: string[] }> {
|
||||
const writtenPaths: string[] = [];
|
||||
const pageSlugs: string[] = [];
|
||||
const generatedItemIds = new Set<string>();
|
||||
const groupDrawerIds = new Set(groupDrawers.map((drawer) => drawer.id));
|
||||
|
||||
for (const file of files) {
|
||||
const absolutePath = path.join(vaultPath, file.path);
|
||||
const normalizedContent = await normalizeGeneratedWikiFile(file.content, file.sourceDrawers);
|
||||
await writeTextFile(absolutePath, normalizedContent);
|
||||
|
||||
const slug = path.basename(absolutePath, ".md");
|
||||
const explicitSourceIds = extractSourceIdsFromWikiFile(normalizedContent)
|
||||
.filter((id) => groupDrawerIds.has(id));
|
||||
const itemIds = explicitSourceIds.length > 0
|
||||
? explicitSourceIds
|
||||
: file.sourceDrawers.map((drawer) => drawer.id);
|
||||
|
||||
markItemsGenerated(db, itemIds, slug);
|
||||
|
||||
// Update content_hash for all generated items
|
||||
for (const id of itemIds) {
|
||||
const item = getItemFromDb(db, id);
|
||||
if (item) {
|
||||
markItemHash(db, id, computeItemHash(item));
|
||||
}
|
||||
}
|
||||
|
||||
// 위키 콘텐츠에서 KG 엔티티/관계 추출
|
||||
extractEntitiesFromWiki(db, normalizedContent, slug);
|
||||
|
||||
// KG 관계가 있으면 Mermaid 다이어그램 주입
|
||||
const entity = getEntity(db, slug);
|
||||
if (entity) {
|
||||
const relations = getRelations(db, slug);
|
||||
if (relations.length > 0) {
|
||||
const contentWithDiagram = injectMermaidDiagrams(normalizedContent, db);
|
||||
if (contentWithDiagram !== normalizedContent) {
|
||||
await writeTextFile(absolutePath, contentWithDiagram);
|
||||
log(` injected mermaid diagram into ${file.path} (${relations.length} relations)`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 위키 콘텐츠에서 모순 감지 (non-blocking)
|
||||
try {
|
||||
const { detectWikiContradictions } = await import("./contradiction.js");
|
||||
const wikiContradictions = detectWikiContradictions(db, normalizedContent, slug);
|
||||
if (wikiContradictions.length > 0) {
|
||||
log(` detected ${wikiContradictions.length} contradictions in ${file.path}`);
|
||||
}
|
||||
} catch {
|
||||
// Contradiction detection failure must not block wiki generation
|
||||
}
|
||||
|
||||
itemIds.forEach((id) => generatedItemIds.add(id));
|
||||
writtenPaths.push(absolutePath);
|
||||
pageSlugs.push(slug);
|
||||
|
||||
log(` wrote ${file.path} (${itemIds.length} items)`);
|
||||
}
|
||||
|
||||
const remainingIds = groupDrawers
|
||||
.map((drawer) => drawer.id)
|
||||
.filter((id) => !generatedItemIds.has(id));
|
||||
if (remainingIds.length > 0 && pageSlugs.length > 0) {
|
||||
markItemsGenerated(db, remainingIds, pageSlugs[0]);
|
||||
remainingIds.forEach((id) => generatedItemIds.add(id));
|
||||
log(` backfilled ${remainingIds.length} items -> ${pageSlugs[0]}`);
|
||||
}
|
||||
|
||||
return {
|
||||
writtenPaths,
|
||||
pageSlugs,
|
||||
itemIds: [...generatedItemIds],
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a fallback file for a source_file sub-group.
|
||||
*
|
||||
* @param drawers Sub-group drawers sharing a source_file.
|
||||
* @returns Generated file definition.
|
||||
*/
|
||||
function buildSubGroupFallbackFile(drawers: MemPalaceDrawer[]): GeneratedWikiFile {
|
||||
const firstDrawer = drawers[0];
|
||||
const sourceName = firstDrawer.sourceFile
|
||||
? path.basename(firstDrawer.sourceFile, ".md")
|
||||
: firstDrawer.id;
|
||||
const content = renderMergedFallbackWikiMarkdown(drawers);
|
||||
const slug = toKebabCase(sourceName);
|
||||
const category = inferWikiCategory(firstDrawer.wing, firstDrawer.room);
|
||||
|
||||
return {
|
||||
path: path.join("wiki", category, `${slug}.md`),
|
||||
content,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Deduplicates a file path slug to prevent overwrites across batches.
|
||||
*
|
||||
* @param filePath Original file path (e.g. "wiki/topics/project-beta.md").
|
||||
* @param usedSlugs Set of already-used slugs; updated in place.
|
||||
* @returns Deduplicated file path.
|
||||
*/
|
||||
function deduplicateSlug(filePath: string, usedSlugs: Set<string>): string {
|
||||
const slug = path.basename(filePath, ".md");
|
||||
if (!usedSlugs.has(slug)) {
|
||||
usedSlugs.add(slug);
|
||||
return filePath;
|
||||
}
|
||||
|
||||
let suffix = 2;
|
||||
while (usedSlugs.has(`${slug}-${suffix}`)) {
|
||||
suffix += 1;
|
||||
}
|
||||
const newSlug = `${slug}-${suffix}`;
|
||||
usedSlugs.add(newSlug);
|
||||
const dir = path.dirname(filePath);
|
||||
return path.join(dir, `${newSlug}.md`);
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns a simple wiki status string summary.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Status summary object.
|
||||
*/
|
||||
export async function getWikiStatus(vaultPath: string): Promise<{ pageCount: number; categories: Record<string, number> }> {
|
||||
try {
|
||||
const files = await listFilesRecursive(path.join(vaultPath, "wiki"), ".md");
|
||||
const categories: Record<string, number> = {};
|
||||
|
||||
for (const filePath of files) {
|
||||
const category = path.basename(path.dirname(filePath));
|
||||
categories[category] = (categories[category] ?? 0) + 1;
|
||||
}
|
||||
|
||||
return {
|
||||
pageCount: files.length,
|
||||
categories,
|
||||
};
|
||||
} catch (error) {
|
||||
throw new LlmCallError("Failed to read wiki status.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Groups drawers by room key and sorts larger groups first.
|
||||
*
|
||||
* @param drawers Source drawers.
|
||||
* @returns Room groups keyed by `wing/room`.
|
||||
*/
|
||||
function groupDrawersByRoom(drawers: MemPalaceDrawer[]): Array<[string, MemPalaceDrawer[]]> {
|
||||
const groups = new Map<string, MemPalaceDrawer[]>();
|
||||
|
||||
for (const drawer of drawers) {
|
||||
const roomKey = `${drawer.wing}/${drawer.room}`;
|
||||
const roomGroup = groups.get(roomKey);
|
||||
|
||||
if (roomGroup) {
|
||||
roomGroup.push(drawer);
|
||||
continue;
|
||||
}
|
||||
|
||||
groups.set(roomKey, [drawer]);
|
||||
}
|
||||
|
||||
return [...groups.entries()].sort((left, right) => right[1].length - left[1].length);
|
||||
}
|
||||
|
||||
/**
|
||||
* Splits items into fixed-size batches.
|
||||
*
|
||||
* @param items Items to split.
|
||||
* @param batchSize Maximum batch size.
|
||||
* @returns Batches preserving input order.
|
||||
*/
|
||||
function splitIntoBatches<T>(items: T[], batchSize: number): T[][] {
|
||||
const batches: T[][] = [];
|
||||
|
||||
for (let index = 0; index < items.length; index += batchSize) {
|
||||
batches.push(items.slice(index, index + batchSize));
|
||||
}
|
||||
|
||||
return batches;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a single merged fallback file for a room.
|
||||
*
|
||||
* @param drawers Room drawers.
|
||||
* @returns Generated file definition.
|
||||
*/
|
||||
function buildMergedFallbackFile(drawers: MemPalaceDrawer[]): GeneratedWikiFile {
|
||||
const firstDrawer = drawers[0];
|
||||
const content = renderMergedFallbackWikiMarkdown(drawers);
|
||||
const slug = toKebabCase(`${firstDrawer.room}-summary`);
|
||||
const category = inferWikiCategory(firstDrawer.wing, firstDrawer.room);
|
||||
|
||||
return {
|
||||
path: path.join("wiki", category, `${slug}.md`),
|
||||
content,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Waits for a fixed duration between LLM batches.
|
||||
*
|
||||
* @param ms Delay duration in milliseconds.
|
||||
* @returns Promise that resolves after the delay.
|
||||
*/
|
||||
function sleep(ms: number): Promise<void> {
|
||||
return new Promise((resolve) => {
|
||||
setTimeout(resolve, ms);
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes generated wiki markdown to ensure required frontmatter fields.
|
||||
*
|
||||
* @param content Raw generated markdown.
|
||||
* @param drawers Source drawers.
|
||||
* @returns Normalized markdown.
|
||||
*/
|
||||
async function normalizeGeneratedWikiFile(content: string, drawers: MemPalaceDrawer[]): Promise<string> {
|
||||
try {
|
||||
const parsed = parseFrontmatter(content);
|
||||
const data = parsed.data as Record<string, unknown>;
|
||||
const title = String(data.title ?? extractTitleFromBody(parsed.content) ?? "Untitled");
|
||||
const sources = Array.isArray(data.sources)
|
||||
? data.sources.map((value) => String(value))
|
||||
: drawers.map((drawer) => drawer.id);
|
||||
const tags = Array.isArray(data.tags) ? data.tags.map((value) => String(value)) : [...new Set(drawers.flatMap((drawer) => drawer.tags))];
|
||||
const category = String(data.category ?? "topics");
|
||||
const normalized = {
|
||||
type: "wiki",
|
||||
category,
|
||||
title,
|
||||
created: String(data.created ?? toDateString(new Date())),
|
||||
updated: toDateString(new Date()),
|
||||
sources,
|
||||
tags,
|
||||
status: String(data.status ?? "draft"),
|
||||
agent: String(data.agent ?? drawers[0]?.addedBy ?? "habraid"),
|
||||
};
|
||||
return stringifyFrontmatter(parsed.content.trim(), normalized);
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts a title from the first markdown heading.
|
||||
*
|
||||
* @param content Markdown content body.
|
||||
* @returns Title string if found.
|
||||
*/
|
||||
function extractTitleFromBody(content: string): string | undefined {
|
||||
const match = content.match(/^#\s+(.+)$/m);
|
||||
return match?.[1]?.trim();
|
||||
}
|
||||
|
||||
function extractSourceIdsFromWikiFile(content: string): string[] {
|
||||
const parsed = parseFrontmatter(content);
|
||||
const data = parsed.data as Record<string, unknown>;
|
||||
if (!Array.isArray(data.sources)) {
|
||||
return [];
|
||||
}
|
||||
return data.sources.map((value) => String(value));
|
||||
}
|
||||
|
||||
// ─── KG 엔티티 추출 (결정론적) ──────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 위키 콘텐츠에서 [[wikilink]]를 추출한다.
|
||||
*
|
||||
* @param content 마크다운 콘텐츠.
|
||||
* @returns 위키링크 대상 배열 (표시 텍스트가 있으면 표시 텍스트 사용).
|
||||
*/
|
||||
function extractWikilinks(content: string): string[] {
|
||||
const matches = content.matchAll(/\[\[([^\]]+)\]\]/g);
|
||||
const links: string[] = [];
|
||||
for (const match of matches) {
|
||||
const inner = match[1];
|
||||
// [[path|display]] 형식이면 display 사용
|
||||
const pipeIndex = inner.indexOf("|");
|
||||
const target = pipeIndex >= 0 ? inner.slice(pipeIndex + 1).trim() : inner.trim();
|
||||
links.push(target);
|
||||
}
|
||||
return [...new Set(links)];
|
||||
}
|
||||
|
||||
/**
|
||||
* 위키 콘텐츠에서 마크다운 헤더를 추출한다.
|
||||
*
|
||||
* @param content 마크다운 콘텐츠.
|
||||
* @returns 헤더 텍스트 배열.
|
||||
*/
|
||||
function extractHeaders(content: string): string[] {
|
||||
const matches = content.matchAll(/^#{1,3}\s+(.+)$/gm);
|
||||
const headers: string[] = [];
|
||||
for (const match of matches) {
|
||||
headers.push(match[1].trim());
|
||||
}
|
||||
return [...new Set(headers)];
|
||||
}
|
||||
|
||||
/**
|
||||
* 텍스트를 KG 엔티티 slug로 변환한다.
|
||||
*
|
||||
* @param text 원본 텍스트.
|
||||
* @returns slug 형식 ID.
|
||||
*/
|
||||
function textToEntitySlug(text: string): string {
|
||||
return text
|
||||
.trim()
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9가-힣\s-]/g, "")
|
||||
.replace(/\s+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^-|-$/g, "") || toKebabCase(text);
|
||||
}
|
||||
|
||||
/**
|
||||
* 생성된 위키 파일에서 엔티티와 관계를 결정론적으로 추출하여 KG에 저장한다.
|
||||
*
|
||||
* 추출 규칙:
|
||||
* - 위키 페이지 자체 → concept/project 엔티티
|
||||
* - [[wikilink]] 대상 → concept 엔티티, related_to 관계
|
||||
* - YAML frontmatter tags → concept 엔티티, related_to 관계
|
||||
* - ## 헤더 → concept 엔티티, contains 관계
|
||||
*
|
||||
* @param db 데이터베이스 인스턴스.
|
||||
* @param content 생성된 위키 마크다운.
|
||||
* @param slug 위키 페이지 slug.
|
||||
*/
|
||||
function extractEntitiesFromWiki(db: ReturnType<typeof openDatabase>, content: string, slug: string): void {
|
||||
try {
|
||||
const parsed = parseFrontmatter(content);
|
||||
const data = parsed.data as Record<string, unknown>;
|
||||
const title = String(data.title ?? slug);
|
||||
const category = String(data.category ?? "topics");
|
||||
const tags = Array.isArray(data.tags) ? data.tags.map((v: unknown) => String(v)) : [];
|
||||
|
||||
// 위키 페이지 자체를 엔티티로 등록
|
||||
addEntity(db, {
|
||||
id: slug,
|
||||
name: title,
|
||||
type: "concept",
|
||||
wikiSlug: slug,
|
||||
});
|
||||
|
||||
// [[wikilink]]에서 엔티티 추출
|
||||
const wikilinks = extractWikilinks(parsed.content);
|
||||
for (const link of wikilinks) {
|
||||
const linkSlug = textToEntitySlug(link);
|
||||
if (linkSlug && linkSlug !== slug) {
|
||||
addEntity(db, { id: linkSlug, name: link, type: "concept" });
|
||||
addRelation(db, {
|
||||
subjectId: slug,
|
||||
predicate: "related_to",
|
||||
objectId: linkSlug,
|
||||
confidence: 0.8,
|
||||
source: "extracted",
|
||||
evidence: `위키링크 [[${link}]]`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// YAML tags에서 엔티티 추출
|
||||
for (const tag of tags) {
|
||||
const tagSlug = textToEntitySlug(tag);
|
||||
if (tagSlug && tagSlug !== slug) {
|
||||
addEntity(db, { id: tagSlug, name: tag, type: "concept" });
|
||||
addRelation(db, {
|
||||
subjectId: slug,
|
||||
predicate: "related_to",
|
||||
objectId: tagSlug,
|
||||
confidence: 0.9,
|
||||
source: "extracted",
|
||||
evidence: `태그: ${tag}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ## 헤더에서 하위 개념 추출
|
||||
const headers = extractHeaders(parsed.content);
|
||||
for (const header of headers) {
|
||||
if (header === title) continue; // H1과 동일하면 스킵
|
||||
const headerSlug = textToEntitySlug(header);
|
||||
if (headerSlug && headerSlug !== slug) {
|
||||
addEntity(db, { id: headerSlug, name: header, type: "concept" });
|
||||
addRelation(db, {
|
||||
subjectId: slug,
|
||||
predicate: "contains",
|
||||
objectId: headerSlug,
|
||||
confidence: 0.7,
|
||||
source: "extracted",
|
||||
evidence: `섹션: ${header}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// KG 추출 실패가 위키 생성을 중단시키지 않도록 무시
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Rebuilds `overview.md` from all wiki pages.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
*/
|
||||
async function rebuildOverview(vaultPath: string): Promise<void> {
|
||||
try {
|
||||
const wikiFiles = await listFilesRecursive(path.join(vaultPath, "wiki"), ".md");
|
||||
const lines = ["# Overview", "", "## Wiki Pages", ""];
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
const content = await pathExists(filePath) ? await (await import("node:fs/promises")).readFile(filePath, "utf8") : "";
|
||||
const parsed = parseFrontmatter(content);
|
||||
const relativePath = path.relative(vaultPath, filePath).replace(/\\/g, "/").replace(/\.md$/, "");
|
||||
const title = String(parsed.data.title ?? path.basename(filePath, ".md"));
|
||||
lines.push(`- [[${relativePath}|${title}]]`);
|
||||
}
|
||||
|
||||
await writeTextFile(
|
||||
path.join(vaultPath, "overview.md"),
|
||||
stringifyFrontmatter(`${lines.join("\n")}\n`, {
|
||||
type: "overview",
|
||||
updated: toIsoTimestamp(),
|
||||
}),
|
||||
);
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Archives a wiki page by moving it to wiki/_archive/.
|
||||
*
|
||||
* Sets frontmatter fields: status=archived, archived_at=<timestamp>.
|
||||
* The original file is removed from its location and recreated under _archive/.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @param wikiRelativePath Relative path under wiki/ (e.g. "topics/old-page.md").
|
||||
* @returns Absolute path of the archived file.
|
||||
*/
|
||||
export async function archiveWikiPage(
|
||||
vaultPath: string,
|
||||
wikiRelativePath: string,
|
||||
): Promise<string> {
|
||||
const wikiDir = path.join(vaultPath, "wiki");
|
||||
const srcPath = path.join(wikiDir, wikiRelativePath);
|
||||
const archiveDir = path.join(wikiDir, "_archive");
|
||||
|
||||
if (!await pathExists(srcPath)) {
|
||||
throw new Error(`Wiki page not found: ${srcPath}`);
|
||||
}
|
||||
|
||||
const content = await fs.promises.readFile(srcPath, "utf8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
|
||||
// Add archive metadata to frontmatter
|
||||
const archiveData = {
|
||||
...parsed.data,
|
||||
status: "archived",
|
||||
archived_at: toIsoTimestamp(),
|
||||
};
|
||||
|
||||
const archivedContent = stringifyFrontmatter(parsed.content, archiveData);
|
||||
|
||||
// Create _archive/ subdirectory structure
|
||||
const archiveSubDir = path.dirname(path.join(archiveDir, wikiRelativePath));
|
||||
await fs.promises.mkdir(archiveSubDir, { recursive: true });
|
||||
|
||||
const archivePath = path.join(archiveDir, wikiRelativePath);
|
||||
await writeTextFile(archivePath, archivedContent);
|
||||
await fs.promises.unlink(srcPath);
|
||||
|
||||
return archivePath;
|
||||
}
|
||||
503
src/wiki/linter.ts
Normal file
503
src/wiki/linter.ts
Normal file
@@ -0,0 +1,503 @@
|
||||
import path from "node:path";
|
||||
import { readFile } from "node:fs/promises";
|
||||
|
||||
import { closeDatabase, openDatabase } from "../db/database.js";
|
||||
import { getChangedItems, getWikiGenerationCounts } from "../db/hashing.js";
|
||||
import { listItems } from "../db/items.js";
|
||||
import { LintError } from "../errors.js";
|
||||
import { searchByVector } from "../search/vector.js";
|
||||
import type {
|
||||
BrokenLinkResult,
|
||||
ContradictionLintResult,
|
||||
DuplicatePair,
|
||||
LintCheckType,
|
||||
LintOptions,
|
||||
LintReport,
|
||||
OrphanResult,
|
||||
StaleItem,
|
||||
WikiEngineConfig,
|
||||
} from "../types.js";
|
||||
import {
|
||||
getContradictions,
|
||||
detectWikiContradictions,
|
||||
} from "./contradiction.js";
|
||||
import { listFilesRecursive, parseFrontmatter, toIsoTimestamp } from "../utils.js";
|
||||
import { lintVault } from "../vault/lint.js";
|
||||
|
||||
/** Tier 1 checks — no LLM needed. */
|
||||
const TIER_1_CHECKS: LintCheckType[] = [
|
||||
"orphans",
|
||||
"broken_links",
|
||||
"stale",
|
||||
"ungenerated",
|
||||
"frontmatter",
|
||||
];
|
||||
|
||||
/** Tier 2 checks — HNSW / vector-based, no LLM. */
|
||||
const TIER_2_CHECKS: LintCheckType[] = [
|
||||
"duplicates",
|
||||
];
|
||||
|
||||
/** Tier 3 checks — require LLM or heavy computation. */
|
||||
const TIER_3_CHECKS: LintCheckType[] = [
|
||||
"contradictions",
|
||||
];
|
||||
|
||||
const ALL_CHECKS: LintCheckType[] = [...TIER_1_CHECKS, ...TIER_2_CHECKS, ...TIER_3_CHECKS];
|
||||
|
||||
const EXCLUDED_ORPHAN_FILES = new Set(["index.md", "overview.md", "log.md"]);
|
||||
|
||||
/** Default similarity threshold for duplicate detection (cosine similarity). */
|
||||
const DUPLICATE_SIMILARITY_THRESHOLD = 0.95;
|
||||
|
||||
/**
|
||||
* Scans all wiki markdown files under the vault's wiki directory.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Absolute wiki markdown file paths.
|
||||
*/
|
||||
export async function scanWikiFiles(vaultPath: string): Promise<string[]> {
|
||||
try {
|
||||
const wikiPath = path.join(vaultPath, "wiki");
|
||||
return await listFilesRecursive(wikiPath, ".md");
|
||||
} catch (error) {
|
||||
throw new LintError("Failed to scan wiki files.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Extracts wikilink targets from markdown content.
|
||||
*
|
||||
* Supports aliases and heading references by normalizing `[[target|label]]`
|
||||
* and `[[target#section]]` to `target`.
|
||||
*
|
||||
* @param content Markdown content.
|
||||
* @returns Normalized wikilink targets.
|
||||
*/
|
||||
export function extractWikilinks(content: string): string[] {
|
||||
const matches = content.matchAll(/\[\[([^[\]]+)\]\]/g);
|
||||
const targets: string[] = [];
|
||||
|
||||
for (const match of matches) {
|
||||
const rawTarget = match[1]?.trim() ?? "";
|
||||
if (!rawTarget) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const target = normalizeWikilinkTarget(rawTarget);
|
||||
if (target) {
|
||||
targets.push(target);
|
||||
}
|
||||
}
|
||||
|
||||
return targets;
|
||||
}
|
||||
|
||||
/**
|
||||
* Builds a reverse backlink index from target slug to source wiki files.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Map of normalized target to source wiki files.
|
||||
*/
|
||||
export async function buildBacklinkIndex(vaultPath: string): Promise<Map<string, string[]>> {
|
||||
try {
|
||||
const wikiFiles = await scanWikiFiles(vaultPath);
|
||||
const wikiRoot = path.join(vaultPath, "wiki");
|
||||
const index = new Map<string, string[]>();
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const source = normalizeRelativeWikiPath(wikiRoot, filePath);
|
||||
const targets = extractWikilinks(content);
|
||||
|
||||
for (const target of targets) {
|
||||
const sources = index.get(target) ?? [];
|
||||
sources.push(source);
|
||||
index.set(target, sources);
|
||||
}
|
||||
}
|
||||
|
||||
return index;
|
||||
} catch (error) {
|
||||
throw new LintError("Failed to build wiki backlink index.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds wiki pages with zero incoming wikilinks.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Orphan wiki page results.
|
||||
*/
|
||||
export async function lintOrphans(vaultPath: string): Promise<OrphanResult[]> {
|
||||
try {
|
||||
const wikiFiles = await scanWikiFiles(vaultPath);
|
||||
const wikiRoot = path.join(vaultPath, "wiki");
|
||||
const backlinks = await buildBacklinkIndex(vaultPath);
|
||||
const results: OrphanResult[] = [];
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
const relativeFile = normalizeRelativeWikiPath(wikiRoot, filePath);
|
||||
if (EXCLUDED_ORPHAN_FILES.has(path.basename(relativeFile))) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const keys = getFileLookupKeys(wikiRoot, filePath);
|
||||
const incoming = keys.some((key) => (backlinks.get(key)?.length ?? 0) > 0);
|
||||
if (incoming) {
|
||||
continue;
|
||||
}
|
||||
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const parsed = parseFrontmatter(content);
|
||||
const fallbackTitle = path.basename(filePath, ".md");
|
||||
const title = typeof parsed.data.title === "string" && parsed.data.title.trim()
|
||||
? parsed.data.title.trim()
|
||||
: fallbackTitle;
|
||||
|
||||
results.push({
|
||||
file: relativeFile,
|
||||
title,
|
||||
});
|
||||
}
|
||||
|
||||
return results;
|
||||
} catch (error) {
|
||||
throw new LintError("Failed to lint orphan wiki pages.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds wikilinks that point to non-existent wiki files.
|
||||
*
|
||||
* @param vaultPath Vault root path.
|
||||
* @returns Broken wikilink results.
|
||||
*/
|
||||
export async function lintBrokenLinks(vaultPath: string): Promise<BrokenLinkResult[]> {
|
||||
try {
|
||||
const wikiFiles = await scanWikiFiles(vaultPath);
|
||||
const wikiRoot = path.join(vaultPath, "wiki");
|
||||
const existingTargets = new Set<string>();
|
||||
const results: BrokenLinkResult[] = [];
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
for (const key of getFileLookupKeys(wikiRoot, filePath)) {
|
||||
existingTargets.add(key);
|
||||
}
|
||||
}
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const source = normalizeRelativeWikiPath(wikiRoot, filePath);
|
||||
|
||||
for (const target of extractWikilinks(content)) {
|
||||
if (!existingTargets.has(target)) {
|
||||
results.push({ source, target });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
} catch (error) {
|
||||
throw new LintError("Failed to lint broken wiki links.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns items whose source content changed after wiki generation.
|
||||
*
|
||||
* @param dbPath SQLite database path.
|
||||
* @returns Stale wiki item results.
|
||||
*/
|
||||
export function lintStale(dbPath: string): StaleItem[] {
|
||||
const db = openDatabase(dbPath);
|
||||
|
||||
try {
|
||||
return getChangedItems(db)
|
||||
.filter((item) => Boolean(item.wikiSlug))
|
||||
.map((item) => ({
|
||||
item_id: item.id,
|
||||
title: item.title,
|
||||
wiki_slug: item.wikiSlug!,
|
||||
}));
|
||||
} finally {
|
||||
closeDatabase(db);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns total and ungenerated item counts from the DB.
|
||||
*
|
||||
* @param dbPath SQLite database path.
|
||||
* @returns Wiki generation count summary.
|
||||
*/
|
||||
export function lintUngenerated(dbPath: string): { total: number; ungenerated: number } {
|
||||
const db = openDatabase(dbPath);
|
||||
|
||||
try {
|
||||
const counts = getWikiGenerationCounts(db);
|
||||
return {
|
||||
total: counts.total,
|
||||
ungenerated: counts.ungenerated,
|
||||
};
|
||||
} finally {
|
||||
closeDatabase(db);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Runs the configured set of lint checks and returns a report.
|
||||
*
|
||||
* Supports Tier 1 (static), Tier 2 (HNSW duplicates), and
|
||||
* Tier 3 (contradiction) checks.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @param options Optional lint options.
|
||||
* @returns Full lint report.
|
||||
*/
|
||||
export async function lintWiki(config: WikiEngineConfig, options?: LintOptions): Promise<LintReport> {
|
||||
const startedAt = Date.now();
|
||||
|
||||
try {
|
||||
const requestedChecks = normalizeRequestedChecks(options?.checks, options?.withLlm);
|
||||
const report: LintReport = {
|
||||
timestamp: toIsoTimestamp(),
|
||||
total_checks: requestedChecks.length,
|
||||
duration_ms: 0,
|
||||
results: {
|
||||
orphans: [],
|
||||
broken_links: [],
|
||||
stale: [],
|
||||
ungenerated: { total: 0, ungenerated: 0 },
|
||||
frontmatter: [],
|
||||
duplicates: [],
|
||||
contradictions: [],
|
||||
},
|
||||
summary: {
|
||||
critical: 0,
|
||||
warnings: 0,
|
||||
info: 0,
|
||||
},
|
||||
};
|
||||
|
||||
// Tier 1: Static checks (no LLM, no vector search)
|
||||
if (requestedChecks.includes("orphans")) {
|
||||
report.results.orphans = await lintOrphans(config.vault.path);
|
||||
}
|
||||
|
||||
if (requestedChecks.includes("broken_links")) {
|
||||
report.results.broken_links = await lintBrokenLinks(config.vault.path);
|
||||
}
|
||||
|
||||
if (requestedChecks.includes("stale")) {
|
||||
report.results.stale = lintStale(config.db.path);
|
||||
}
|
||||
|
||||
if (requestedChecks.includes("ungenerated")) {
|
||||
report.results.ungenerated = lintUngenerated(config.db.path);
|
||||
}
|
||||
|
||||
if (requestedChecks.includes("frontmatter")) {
|
||||
const frontmatterResult = await lintVault(config.vault.path);
|
||||
report.results.frontmatter = frontmatterResult.issues;
|
||||
}
|
||||
|
||||
// Tier 2: HNSW semantic duplicate detection
|
||||
if (requestedChecks.includes("duplicates")) {
|
||||
report.results.duplicates = await lintDuplicates(config);
|
||||
}
|
||||
|
||||
// Tier 3: Contradiction detection (rule-based + wiki content)
|
||||
if (requestedChecks.includes("contradictions")) {
|
||||
report.results.contradictions = await lintContradictions(config);
|
||||
}
|
||||
|
||||
report.summary = {
|
||||
critical: report.results.broken_links.length
|
||||
+ report.results.stale.length
|
||||
+ report.results.frontmatter.length
|
||||
+ report.results.contradictions.filter((c) => c.severity === "high").length,
|
||||
warnings: report.results.orphans.length
|
||||
+ report.results.duplicates.length
|
||||
+ report.results.contradictions.filter((c) => c.severity === "medium").length,
|
||||
info: report.results.ungenerated.ungenerated
|
||||
+ report.results.contradictions.filter((c) => c.severity === "low").length,
|
||||
};
|
||||
report.duration_ms = Date.now() - startedAt;
|
||||
|
||||
return report;
|
||||
} catch (error) {
|
||||
throw new LintError("Wiki lint failed.", error as Error);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects semantically duplicate items using vector similarity.
|
||||
*
|
||||
* Compares every item's embedding against all others. Pairs with
|
||||
* cosine similarity above the threshold are reported as duplicates.
|
||||
* Items with the same content hash are skipped (exact dupes are
|
||||
* expected for versioned content).
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @param threshold Minimum cosine similarity to flag (default: 0.92).
|
||||
* @returns Array of duplicate pairs with similarity scores.
|
||||
*/
|
||||
export async function lintDuplicates(
|
||||
config: WikiEngineConfig,
|
||||
threshold: number = DUPLICATE_SIMILARITY_THRESHOLD,
|
||||
): Promise<DuplicatePair[]> {
|
||||
const db = openDatabase(config.db.path);
|
||||
|
||||
try {
|
||||
const items = listItems(db, 100000);
|
||||
if (items.length < 2) return [];
|
||||
|
||||
const results: DuplicatePair[] = [];
|
||||
const seen = new Set<string>();
|
||||
|
||||
for (let i = 0; i < items.length; i++) {
|
||||
const item = items[i];
|
||||
|
||||
// Search for similar items using vector search
|
||||
const vector = await import("../search/vector.js").then((m) => m.getVector(db, item.id));
|
||||
if (!vector) continue;
|
||||
|
||||
const neighbors = searchByVector(db, vector, 10, config.db.path);
|
||||
|
||||
for (const neighbor of neighbors) {
|
||||
// Skip self
|
||||
if (neighbor.itemId === item.id) continue;
|
||||
|
||||
// Skip if already seen (avoid A-B and B-A duplicates)
|
||||
const pairKey = [item.id, neighbor.itemId].sort().join("::");
|
||||
if (seen.has(pairKey)) continue;
|
||||
seen.add(pairKey);
|
||||
|
||||
if (neighbor.score >= threshold) {
|
||||
const otherItem = items.find((it) => it.id === neighbor.itemId);
|
||||
results.push({
|
||||
item_a_id: item.id,
|
||||
item_a_title: item.title,
|
||||
item_b_id: neighbor.itemId,
|
||||
item_b_title: otherItem?.title ?? neighbor.itemId,
|
||||
similarity: Math.round(neighbor.score * 10000) / 10000,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
} catch (error) {
|
||||
throw new LintError("Duplicate detection failed.", error as Error);
|
||||
} finally {
|
||||
closeDatabase(db);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Detects contradictions across items and wiki pages.
|
||||
*
|
||||
* Runs rule-based contradiction detection against all wiki pages
|
||||
* and returns open contradictions from the database.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @returns Array of detected contradictions.
|
||||
*/
|
||||
export async function lintContradictions(config: WikiEngineConfig): Promise<ContradictionLintResult[]> {
|
||||
const db = openDatabase(config.db.path);
|
||||
|
||||
try {
|
||||
// Run wiki content contradiction detection
|
||||
const wikiPath = path.join(config.vault.path, "wiki");
|
||||
const wikiFiles = await listFilesRecursive(wikiPath, ".md");
|
||||
|
||||
for (const filePath of wikiFiles) {
|
||||
const content = await readFile(filePath, "utf8");
|
||||
const slug = path.relative(wikiPath, filePath).replace(/\.md$/i, "").replace(/\\/g, "/");
|
||||
detectWikiContradictions(db, content, slug);
|
||||
}
|
||||
|
||||
// Return all open contradictions
|
||||
const openContradictions = getContradictions(db, "open");
|
||||
|
||||
return openContradictions.map((c) => ({
|
||||
id: c.id,
|
||||
item_a_id: c.item_a_id,
|
||||
item_b_id: c.item_b_id,
|
||||
item_a_slug: c.item_a_slug,
|
||||
item_b_slug: c.item_b_slug,
|
||||
field: c.field,
|
||||
value_a: c.value_a,
|
||||
value_b: c.value_b,
|
||||
severity: c.severity,
|
||||
status: c.status,
|
||||
}));
|
||||
} catch (error) {
|
||||
throw new LintError("Contradiction detection failed.", error as Error);
|
||||
} finally {
|
||||
closeDatabase(db);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Normalizes a wikilink target for lookup.
|
||||
*
|
||||
* @param rawTarget Raw target inside `[[...]]`.
|
||||
* @returns Normalized target slug/path.
|
||||
*/
|
||||
function normalizeWikilinkTarget(rawTarget: string): string {
|
||||
return rawTarget
|
||||
.split("|")[0]
|
||||
.split("#")[0]
|
||||
.trim()
|
||||
.replace(/\\/g, "/")
|
||||
.replace(/^\//, "")
|
||||
.replace(/\.md$/i, "")
|
||||
.toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* Returns lookup keys for a wiki file using both relative path and basename.
|
||||
*
|
||||
* @param wikiRoot Wiki root directory.
|
||||
* @param filePath Absolute wiki file path.
|
||||
* @returns Lookup keys for wikilink resolution.
|
||||
*/
|
||||
function getFileLookupKeys(wikiRoot: string, filePath: string): string[] {
|
||||
const relativePath = normalizeRelativeWikiPath(wikiRoot, filePath).replace(/\.md$/i, "");
|
||||
const fileName = path.basename(relativePath);
|
||||
|
||||
return [...new Set([relativePath.toLowerCase(), fileName.toLowerCase()])];
|
||||
}
|
||||
|
||||
/**
|
||||
* Converts an absolute wiki file path to a normalized wiki-relative path.
|
||||
*
|
||||
* @param wikiRoot Wiki root directory.
|
||||
* @param filePath Absolute wiki file path.
|
||||
* @returns Wiki-relative path using `/`.
|
||||
*/
|
||||
function normalizeRelativeWikiPath(wikiRoot: string, filePath: string): string {
|
||||
return path.relative(wikiRoot, filePath).replace(/\\/g, "/");
|
||||
}
|
||||
|
||||
/**
|
||||
* Filters requested checks down to supported check types.
|
||||
*
|
||||
* When no checks specified, runs Tier 1 + Tier 2 by default.
|
||||
* Tier 3 (contradictions) requires explicit opt-in via `--with-llm` or
|
||||
* explicit check list.
|
||||
*
|
||||
* @param checks Optional requested checks.
|
||||
* @param withLlm Whether LLM-based checks are enabled.
|
||||
* @returns Checks to execute.
|
||||
*/
|
||||
function normalizeRequestedChecks(checks?: LintCheckType[], withLlm?: boolean): LintCheckType[] {
|
||||
if (!checks || checks.length === 0) {
|
||||
// Default: Tier 1 + Tier 2. Tier 3 only with --with-llm
|
||||
return withLlm ? [...ALL_CHECKS] : [...TIER_1_CHECKS, ...TIER_2_CHECKS];
|
||||
}
|
||||
|
||||
return checks.filter((check): check is LintCheckType => ALL_CHECKS.includes(check));
|
||||
}
|
||||
290
src/wiki/llm.ts
Normal file
290
src/wiki/llm.ts
Normal file
@@ -0,0 +1,290 @@
|
||||
import "dotenv/config";
|
||||
import { existsSync, readFileSync } from "node:fs";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import { LlmCallError } from "../errors.js";
|
||||
import type { LlmGenerationResult, StandaloneLlmConfig, WikiEngineConfig } from "../types.js";
|
||||
import { createBackend, type LlmBackend } from "./backends/index.js";
|
||||
import { HostBackend } from "./backends/host.js";
|
||||
import { OpenAiBackend } from "./backends/openai.js";
|
||||
import { OllamaBackend } from "./backends/ollama.js";
|
||||
import { ZaiBackend } from "./backends/zai.js";
|
||||
|
||||
/**
|
||||
* Well-known .env file locations to search for API keys.
|
||||
* Checked in order; first match wins.
|
||||
*/
|
||||
const ENV_SEARCH_PATHS = [
|
||||
path.join(os.homedir(), ".hermes", ".env"),
|
||||
path.join(os.homedir(), ".openclaw", ".env"),
|
||||
path.join(process.cwd(), ".env"),
|
||||
];
|
||||
|
||||
/**
|
||||
* Loads environment variables from well-known .env files.
|
||||
* Does not overwrite variables already set in the environment.
|
||||
*/
|
||||
function loadEnvFromSearchPaths(): void {
|
||||
for (const envPath of ENV_SEARCH_PATHS) {
|
||||
if (!existsSync(envPath)) continue;
|
||||
|
||||
try {
|
||||
const content = readFileSync(envPath, "utf-8");
|
||||
for (const line of content.split("\n")) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed || trimmed.startsWith("#")) continue;
|
||||
|
||||
const eqIndex = trimmed.indexOf("=");
|
||||
if (eqIndex === -1) continue;
|
||||
|
||||
const key = trimmed.slice(0, eqIndex).trim();
|
||||
const value = trimmed.slice(eqIndex + 1).trim();
|
||||
if (!process.env[key]) {
|
||||
process.env[key] = value;
|
||||
}
|
||||
}
|
||||
return;
|
||||
} catch {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
loadEnvFromSearchPaths();
|
||||
|
||||
/**
|
||||
* LLM client contract — pluggable backend interface.
|
||||
*/
|
||||
export interface LLMClient {
|
||||
generate(systemPrompt: string, userPrompt: string, timeoutMs?: number): Promise<LlmGenerationResult>;
|
||||
}
|
||||
|
||||
/**
|
||||
* Contract for host-side delegated inference.
|
||||
*/
|
||||
export interface HostInferenceBridge {
|
||||
isAvailable(): Promise<boolean>;
|
||||
generate(request: {
|
||||
systemPrompt: string;
|
||||
userPrompt: string;
|
||||
timeoutMs?: number;
|
||||
maxTokens?: number;
|
||||
preference?: "fast" | "balanced" | "smart";
|
||||
}): Promise<{ text: string; provider?: string; model?: string }>;
|
||||
}
|
||||
|
||||
// Re-export the backend system for direct use
|
||||
export { createBackend } from "./backends/index.js";
|
||||
export type { LlmBackend } from "./backends/index.js";
|
||||
|
||||
/**
|
||||
* Creates the appropriate LLM client based on config.
|
||||
* Host-following is preferred; standalone remains available for fallback/legacy mode.
|
||||
*
|
||||
* When mode is one of the new backend modes ("openai", "ollama", "zai"),
|
||||
* creates a BackendAdapterClient that wraps the pluggable backend.
|
||||
*
|
||||
* @param config Effective runtime config.
|
||||
* @param overrideBackend Optional runtime backend override (e.g. from CLI --backend).
|
||||
*/
|
||||
export function createLlmClient(config: WikiEngineConfig, overrideBackend?: string): LLMClient {
|
||||
const mode = config.llm.mode;
|
||||
|
||||
// New backend modes use the pluggable backend system directly
|
||||
if (mode === "openai" || mode === "ollama" || mode === "zai") {
|
||||
const backend = createBackend(config, overrideBackend);
|
||||
return new BackendAdapterClient(backend);
|
||||
}
|
||||
|
||||
// Host mode (default) — with optional fallback
|
||||
if (mode === "host" || mode === undefined) {
|
||||
const backend = createBackend(config, overrideBackend);
|
||||
return new HostDelegatingClient(config, backend);
|
||||
}
|
||||
|
||||
// Legacy standalone mode — route through new backend system
|
||||
const backend = createBackend(config, overrideBackend);
|
||||
return new BackendAdapterClient(backend);
|
||||
}
|
||||
|
||||
/**
|
||||
* Adapter that wraps an LlmBackend to satisfy the LLMClient interface.
|
||||
*
|
||||
* Translates generate() calls into chat() calls.
|
||||
*/
|
||||
class BackendAdapterClient implements LLMClient {
|
||||
constructor(private readonly backend: LlmBackend) {}
|
||||
|
||||
async generate(systemPrompt: string, userPrompt: string, _timeoutMs?: number): Promise<LlmGenerationResult> {
|
||||
const text = await this.backend.chat(
|
||||
[
|
||||
{ role: "system", content: systemPrompt },
|
||||
{ role: "user", content: userPrompt },
|
||||
],
|
||||
);
|
||||
|
||||
return {
|
||||
text,
|
||||
metadata: {
|
||||
route: "standalone",
|
||||
provider: this.backend.name,
|
||||
model: undefined,
|
||||
fallbackUsed: false,
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Host-following client. Delegates to the host backend when available,
|
||||
* otherwise falls back to the configured standalone backend.
|
||||
*/
|
||||
export class HostDelegatingClient implements LLMClient {
|
||||
constructor(
|
||||
private readonly config: WikiEngineConfig,
|
||||
private readonly hostBackend: LlmBackend,
|
||||
) {}
|
||||
|
||||
async generate(systemPrompt: string, userPrompt: string, _timeoutMs: number = 60_000): Promise<LlmGenerationResult> {
|
||||
// Try host backend first
|
||||
if (await this.hostBackend.isAvailable()) {
|
||||
try {
|
||||
const text = await this.hostBackend.chat(
|
||||
[
|
||||
{ role: "system", content: systemPrompt },
|
||||
{ role: "user", content: userPrompt },
|
||||
],
|
||||
{ maxTokens: this.config.llm.max_tokens },
|
||||
);
|
||||
|
||||
// Try to get host model metadata for observability
|
||||
let provider: string | undefined;
|
||||
let model: string | undefined;
|
||||
if (this.hostBackend instanceof HostBackend) {
|
||||
const meta = this.hostBackend.getModelMetadata();
|
||||
provider = meta.provider;
|
||||
model = meta.model;
|
||||
}
|
||||
|
||||
return {
|
||||
text,
|
||||
metadata: {
|
||||
route: "host",
|
||||
provider,
|
||||
model,
|
||||
fallbackUsed: false,
|
||||
},
|
||||
};
|
||||
} catch (hostError) {
|
||||
// Host failed — try fallback if configured
|
||||
const fallback = this.config.llm.fallback;
|
||||
if (!fallback) {
|
||||
throw hostError;
|
||||
}
|
||||
// Fall through to fallback below
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback to standalone backend
|
||||
const fallback = this.config.llm.fallback;
|
||||
if (!fallback) {
|
||||
throw new LlmCallError("Host LLM route unavailable and no fallback is configured.");
|
||||
}
|
||||
|
||||
const fallbackBackend = createFallbackBackend(fallback);
|
||||
const text = await fallbackBackend.chat(
|
||||
[
|
||||
{ role: "system", content: systemPrompt },
|
||||
{ role: "user", content: userPrompt },
|
||||
],
|
||||
{ maxTokens: fallback.max_tokens },
|
||||
);
|
||||
|
||||
return {
|
||||
text,
|
||||
metadata: {
|
||||
route: "fallback",
|
||||
provider: fallback.provider,
|
||||
model: fallback.model,
|
||||
fallbackUsed: true,
|
||||
},
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a backend from the legacy StandaloneLlmConfig (fallback config).
|
||||
*/
|
||||
function createFallbackBackend(runtime: StandaloneLlmConfig): LlmBackend {
|
||||
const provider = runtime.provider.toLowerCase();
|
||||
|
||||
switch (provider) {
|
||||
case "ollama":
|
||||
return new OllamaBackend({
|
||||
baseUrl: runtime.api_url || "http://localhost:11434",
|
||||
model: runtime.model,
|
||||
});
|
||||
case "zai":
|
||||
case "glm":
|
||||
return new ZaiBackend({
|
||||
baseUrl: runtime.api_url,
|
||||
model: runtime.model,
|
||||
apiKeyEnv: runtime.api_key_env,
|
||||
maxTokens: runtime.max_tokens,
|
||||
});
|
||||
case "openai":
|
||||
default:
|
||||
return new OpenAiBackend({
|
||||
baseUrl: runtime.api_url,
|
||||
model: runtime.model,
|
||||
apiKeyEnv: runtime.api_key_env,
|
||||
maxTokens: runtime.max_tokens,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ─── Legacy bridge classes (kept for backwards compatibility) ────────
|
||||
|
||||
/**
|
||||
* Bridge that delegates generation to the local Hermes CLI using the current host config.
|
||||
*
|
||||
* @deprecated Use HostBackend from ./backends/host.js instead.
|
||||
*/
|
||||
export class HermesCliHostInferenceBridge implements HostInferenceBridge {
|
||||
private readonly backend = new HostBackend();
|
||||
|
||||
async isAvailable(): Promise<boolean> {
|
||||
return this.backend.isAvailable();
|
||||
}
|
||||
|
||||
async generate(request: {
|
||||
systemPrompt: string;
|
||||
userPrompt: string;
|
||||
timeoutMs?: number;
|
||||
maxTokens?: number;
|
||||
preference?: "fast" | "balanced" | "smart";
|
||||
}): Promise<{ text: string; provider?: string; model?: string }> {
|
||||
const text = await this.backend.chat([
|
||||
{ role: "system", content: request.systemPrompt },
|
||||
{ role: "user", content: request.userPrompt },
|
||||
]);
|
||||
const meta = this.backend.getModelMetadata();
|
||||
return { text, provider: meta.provider, model: meta.model };
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Placeholder bridge for when host is unavailable.
|
||||
*
|
||||
* @deprecated Use createBackend() from ./backends/index.js instead.
|
||||
*/
|
||||
export class UnavailableHostInferenceBridge implements HostInferenceBridge {
|
||||
async isAvailable(): Promise<boolean> {
|
||||
return false;
|
||||
}
|
||||
|
||||
async generate(): Promise<{ text: string; provider?: string; model?: string }> {
|
||||
throw new LlmCallError("Host inference bridge is not available.");
|
||||
}
|
||||
}
|
||||
350
src/wiki/mermaid.ts
Normal file
350
src/wiki/mermaid.ts
Normal file
@@ -0,0 +1,350 @@
|
||||
/**
|
||||
* Mermaid 다이어그램 자동 생성 모듈.
|
||||
*
|
||||
* KG 관계를 기반으로 Mermaid 그래프/플로우차트를 생성하고
|
||||
* 위키 페이지에 주입한다.
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import { getEntity, getRelations, getEntityNeighbors } from "../db/kg.js";
|
||||
import type { EntityWithStats, RelationInfo } from "../db/kg.js";
|
||||
|
||||
// ─── 타입 정의 ────────────────────────────────────────────────────────
|
||||
|
||||
/** Mermaid 다이어그램 생성 옵션 */
|
||||
export interface MermaidOptions {
|
||||
/** 최대 노드 수 (기본값: 20) */
|
||||
maxNodes?: number;
|
||||
/** 순회 깊이 (기본값: 2) */
|
||||
depth?: number;
|
||||
}
|
||||
|
||||
// ─── 엔티티 유형별 색상 매핑 ──────────────────────────────────────────
|
||||
|
||||
const TYPE_COLORS: Record<string, string> = {
|
||||
project: "#4A90D9",
|
||||
person: "#27AE60",
|
||||
concept: "#E67E22",
|
||||
tool: "#8E44AD",
|
||||
event: "#E74C3C",
|
||||
decision: "#F39C12",
|
||||
};
|
||||
|
||||
const DEFAULT_COLOR = "#95A5A6";
|
||||
|
||||
// ─── 내부 유틸 ────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* Mermaid 노드 ID로 안전한 문자열을 생성한다.
|
||||
* 특수문자를 제거하고 하이픈을 언더스코어로 변환한다.
|
||||
*
|
||||
* @param id 원본 엔티티 ID.
|
||||
* @returns Mermaid-safe 노드 ID.
|
||||
*/
|
||||
function safeNodeId(id: string): string {
|
||||
return id
|
||||
.replace(/[^a-zA-Z0-9가-힣_-]/g, "_")
|
||||
.replace(/^-+|-+$/g, "")
|
||||
|| "node_unknown";
|
||||
}
|
||||
|
||||
/**
|
||||
* Mermaid 라벨로 안전한 문자열을 생성한다.
|
||||
* 따옴표 내부에서 문제가 될 수 있는 문자를 이스케이프한다.
|
||||
*
|
||||
* @param text 원본 텍스트.
|
||||
* @returns 이스케이프된 라벨.
|
||||
*/
|
||||
function safeLabel(text: string): string {
|
||||
return text
|
||||
.replace(/"/g, "'")
|
||||
.replace(/\n/g, " ")
|
||||
.slice(0, 40);
|
||||
}
|
||||
|
||||
/**
|
||||
* 엔티티 유형에 따른 색상을 반환한다.
|
||||
*
|
||||
* @param type 엔티티 유형.
|
||||
* @returns HEX 색상 코드.
|
||||
*/
|
||||
function getColor(type: string): string {
|
||||
return TYPE_COLORS[type] ?? DEFAULT_COLOR;
|
||||
}
|
||||
|
||||
/**
|
||||
* 관계 predicate를 Mermaid 엣지 라벨로 변환한다.
|
||||
*
|
||||
* @param predicate 관계 predicate.
|
||||
* @returns 짧은 라벨.
|
||||
*/
|
||||
function shortenPredicate(predicate: string): string {
|
||||
const map: Record<string, string> = {
|
||||
related_to: "관련",
|
||||
depends_on: "의존",
|
||||
uses: "사용",
|
||||
implements: "구현",
|
||||
contains: "포함",
|
||||
fixes: "해결",
|
||||
creates: "생성",
|
||||
blocks: "차단",
|
||||
derived_from: "파생",
|
||||
answers: "답변",
|
||||
};
|
||||
return map[predicate] ?? predicate;
|
||||
}
|
||||
|
||||
// ─── 공개 함수 ─────────────────────────────────────────────────────────
|
||||
|
||||
/**
|
||||
* 특정 엔티티를 중심으로 KG 관계에서 Mermaid 그래프를 생성한다.
|
||||
*
|
||||
* BFS로 이웃을 순회하여 최대 ~20개 노드의 그래프를 만든다.
|
||||
* 엔티티 유형별로 색상을 구분한다.
|
||||
*
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @param entityId 중심 엔티티 ID (slug).
|
||||
* @param options 생성 옵션 (maxNodes, depth).
|
||||
* @returns Mermaid 그래프 문자열. 관계가 없으면 빈 문자열.
|
||||
*/
|
||||
export function generateRelationDiagram(
|
||||
db: Database.Database,
|
||||
entityId: string,
|
||||
options?: MermaidOptions,
|
||||
): string {
|
||||
const maxNodes = options?.maxNodes ?? 20;
|
||||
const depth = options?.depth ?? 2;
|
||||
|
||||
// 엔티티 존재 확인
|
||||
const entity = getEntity(db, entityId);
|
||||
if (!entity) return "";
|
||||
|
||||
// BFS로 이웃 순회
|
||||
const { entities, relations } = getEntityNeighbors(db, entityId, depth);
|
||||
|
||||
if (relations.length === 0) return "";
|
||||
|
||||
// 노드 수 제한
|
||||
const limitedEntities = entities.slice(0, maxNodes);
|
||||
const entityIds = new Set(limitedEntities.map((e) => e.id));
|
||||
|
||||
// 엣지 필터링: 양쪽 노드 모두 포함된 것만
|
||||
const limitedRelations = relations.filter(
|
||||
(r) => entityIds.has(r.subjectId) && entityIds.has(r.objectId),
|
||||
);
|
||||
|
||||
if (limitedRelations.length === 0) return "";
|
||||
|
||||
const lines: string[] = ["graph LR"];
|
||||
|
||||
// 노드 정의 + 스타일
|
||||
const nodeStyles: string[] = [];
|
||||
for (const ent of limitedEntities) {
|
||||
const nodeId = safeNodeId(ent.id);
|
||||
const label = safeLabel(ent.name);
|
||||
const color = getColor(ent.type);
|
||||
lines.push(` ${nodeId}["${label}"]`);
|
||||
nodeStyles.push(` style ${nodeId} fill:${color},color:#fff,stroke:#333`);
|
||||
}
|
||||
|
||||
lines.push("");
|
||||
|
||||
// 엣지 정의
|
||||
for (const rel of limitedRelations) {
|
||||
const sourceId = safeNodeId(rel.subjectId);
|
||||
const targetId = safeNodeId(rel.objectId);
|
||||
const predLabel = shortenPredicate(rel.predicate);
|
||||
lines.push(` ${sourceId} -->|"${predLabel}"| ${targetId}`);
|
||||
}
|
||||
|
||||
// 스타일 적용
|
||||
lines.push("");
|
||||
for (const style of nodeStyles) {
|
||||
lines.push(style);
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* 마크다운 콘텐츠에서 "A → B" 패턴을 추출하여 Mermaid 플로우차트를 생성한다.
|
||||
*
|
||||
* 감지 패턴:
|
||||
* - `A -> B`, `A → B` (화살표)
|
||||
* - "A uses B", "A depends on B" 등의 문장
|
||||
* - "A는 B를 사용한다", "A는 B에 의존한다" 등 한국어 문장
|
||||
*
|
||||
* @param content 마크다운 콘텐츠.
|
||||
* @returns Mermaid 플로우차트 문자열. 패턴이 없으면 빈 문자열.
|
||||
*/
|
||||
export function generateArchitectureDiagram(content: string): string {
|
||||
const edges: Array<{ source: string; target: string; label: string }> = [];
|
||||
|
||||
// 영어 패턴: "A -> B", "A → B", "A --> B"
|
||||
const arrowPattern = /\[([^\]]+)\]\s*(?:--?>|→)\s*\[?([^\[\]]+)\]?/g;
|
||||
let match: RegExpExecArray | null;
|
||||
while ((match = arrowPattern.exec(content)) !== null) {
|
||||
const source = match[1].trim();
|
||||
const target = match[2].trim().replace(/\]+$/, "");
|
||||
if (source && target && source !== target) {
|
||||
edges.push({ source, target, label: "" });
|
||||
}
|
||||
}
|
||||
|
||||
// 영어 동사 패턴: "A uses B", "A depends on B", "A implements B"
|
||||
const verbPatterns = [
|
||||
{ regex: /(\w[\w\s]{1,30}?)\s+uses?\s+(\w[\w\s]{1,30}?)(?:\s*[.\n,])/gi, label: "사용" },
|
||||
{ regex: /(\w[\w\s]{1,30}?)\s+depends?\s+on\s+(\w[\w\s]{1,30}?)(?:\s*[.\n,])/gi, label: "의존" },
|
||||
{ regex: /(\w[\w\s]{1,30}?)\s+implements?\s+(\w[\w\s]{1,30}?)(?:\s*[.\n,])/gi, label: "구현" },
|
||||
{ regex: /(\w[\w\s]{1,30}?)\s+calls?\s+(\w[\w\s]{1,30}?)(?:\s*[.\n,])/gi, label: "호출" },
|
||||
];
|
||||
|
||||
for (const { regex, label } of verbPatterns) {
|
||||
while ((match = regex.exec(content)) !== null) {
|
||||
const source = match[1].trim();
|
||||
const target = match[2].trim();
|
||||
if (source && target && source !== target) {
|
||||
edges.push({ source, target, label });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 한국어 패턴: "A는 B를 사용한다", "A는 B에 의존한다"
|
||||
const krPatterns = [
|
||||
{ regex: /([가-힣\w]{1,20}?)은?는\s+([가-힣\w]{1,20}?)을?를\s+사용/g, label: "사용" },
|
||||
{ regex: /([가-힣\w]{1,20}?)은?는\s+([가-힣\w]{1,20}?)에\s+의존/g, label: "의존" },
|
||||
{ regex: /([가-힣\w]{1,20}?)은?는\s+([가-힣\w]{1,20}?)을?를\s+구현/g, label: "구현" },
|
||||
{ regex: /([가-힣\w]{1,20}?)에서\s+([가-힣\w]{1,20}?)으로/g, label: "" },
|
||||
];
|
||||
|
||||
for (const { regex, label } of krPatterns) {
|
||||
while ((match = regex.exec(content)) !== null) {
|
||||
const source = match[1].trim();
|
||||
const target = match[2].trim();
|
||||
if (source && target && source !== target) {
|
||||
edges.push({ source, target, label });
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (edges.length === 0) return "";
|
||||
|
||||
// 중복 제거
|
||||
const seen = new Set<string>();
|
||||
const uniqueEdges = edges.filter((e) => {
|
||||
const key = `${e.source}|${e.target}`;
|
||||
if (seen.has(key)) return false;
|
||||
seen.add(key);
|
||||
return true;
|
||||
});
|
||||
|
||||
// 최대 15개 엣지
|
||||
const limitedEdges = uniqueEdges.slice(0, 15);
|
||||
|
||||
// 노드 수집
|
||||
const nodeSet = new Set<string>();
|
||||
for (const edge of limitedEdges) {
|
||||
nodeSet.add(edge.source);
|
||||
nodeSet.add(edge.target);
|
||||
}
|
||||
|
||||
// 노드가 너무 많으면 축소
|
||||
const nodes = [...nodeSet].slice(0, 20);
|
||||
const nodeSetLimited = new Set(nodes);
|
||||
const finalEdges = limitedEdges.filter(
|
||||
(e) => nodeSetLimited.has(e.source) && nodeSetLimited.has(e.target),
|
||||
);
|
||||
|
||||
const lines: string[] = ["graph TD"];
|
||||
|
||||
for (const edge of finalEdges) {
|
||||
const sourceId = safeNodeId(edge.source);
|
||||
const targetId = safeNodeId(edge.target);
|
||||
const sourceLabel = safeLabel(edge.source);
|
||||
const targetLabel = safeLabel(edge.target);
|
||||
|
||||
lines.push(` ${sourceId}["${sourceLabel}"]`);
|
||||
lines.push(` ${targetId}["${targetLabel}"]`);
|
||||
|
||||
if (edge.label) {
|
||||
lines.push(` ${sourceId} -->|"${edge.label}"| ${targetId}`);
|
||||
} else {
|
||||
lines.push(` ${sourceId} --> ${targetId}`);
|
||||
}
|
||||
}
|
||||
|
||||
return lines.join("\n");
|
||||
}
|
||||
|
||||
/**
|
||||
* 위키 마크다운 콘텐츠에 Mermaid 다이어그램 섹션을 주입한다.
|
||||
*
|
||||
* KG 관계가 있으면 "## 📊 관계도" 섹션을 추가한다.
|
||||
* 이미 섹션이 존재하면 교체한다.
|
||||
*
|
||||
* @param content 원본 위키 마크다운.
|
||||
* @param db 연결된 데이터베이스 인스턴스.
|
||||
* @returns 다이어그램이 주입된 마크다운.
|
||||
*/
|
||||
export function injectMermaidDiagrams(content: string, db: Database.Database): string {
|
||||
// frontmatter 파싱
|
||||
const fmMatch = content.match(/^---\n([\s\S]*?)\n---\n/);
|
||||
if (!fmMatch) return content;
|
||||
|
||||
const frontmatter = fmMatch[0];
|
||||
const body = content.slice(frontmatter.length);
|
||||
|
||||
// frontmatter에서 slug/title 추출
|
||||
const titleMatch = frontmatter.match(/^title:\s*["']?(.+?)["']?\s*$/m);
|
||||
const slugMatch = body.match(/^#\s+(.+)$/m);
|
||||
const title = titleMatch?.[1] ?? slugMatch?.[1] ?? "";
|
||||
|
||||
// slug 추론: frontmatter의 title 또는 파일명
|
||||
const slug = title
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9가-힣\s-]/g, "")
|
||||
.replace(/\s+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.replace(/^-|-$/g, "");
|
||||
|
||||
if (!slug) return content;
|
||||
|
||||
// KG에서 관계 다이어그램 생성
|
||||
const relationDiagram = generateRelationDiagram(db, slug, { maxNodes: 20, depth: 2 });
|
||||
|
||||
// 콘텐츠에서 아키텍처 다이어그램 생성
|
||||
const archDiagram = generateArchitectureDiagram(body);
|
||||
|
||||
// 둘 다 없으면 주입하지 않음
|
||||
if (!relationDiagram && !archDiagram) return content;
|
||||
|
||||
// 기존 관계도 섹션 제거
|
||||
const sectionPattern = /\n## 📊 관계도[\s\S]*?(?=\n## |\n*$)/;
|
||||
let cleanBody = body.replace(sectionPattern, "");
|
||||
|
||||
// 다이어그램 섹션 구성
|
||||
const diagramParts: string[] = [];
|
||||
|
||||
if (relationDiagram) {
|
||||
diagramParts.push("### 엔티티 관계\n");
|
||||
diagramParts.push("```mermaid");
|
||||
diagramParts.push(relationDiagram);
|
||||
diagramParts.push("```\n");
|
||||
}
|
||||
|
||||
if (archDiagram) {
|
||||
diagramParts.push("### 아키텍처\n");
|
||||
diagramParts.push("```mermaid");
|
||||
diagramParts.push(archDiagram);
|
||||
diagramParts.push("```\n");
|
||||
}
|
||||
|
||||
const diagramSection = `\n## 📊 관계도\n\n${diagramParts.join("\n")}`;
|
||||
|
||||
// 본문 끝에 주입 (trailing whitespace 정리)
|
||||
cleanBody = cleanBody.trimEnd();
|
||||
const result = frontmatter + cleanBody + "\n" + diagramSection + "\n";
|
||||
|
||||
return result;
|
||||
}
|
||||
246
src/wiki/prompts.ts
Normal file
246
src/wiki/prompts.ts
Normal file
@@ -0,0 +1,246 @@
|
||||
import path from "node:path";
|
||||
|
||||
import type { GeneratedWikiFile, MemPalaceDrawer } from "../types.js";
|
||||
import { parseFrontmatter, truncateForPrompt } from "../utils.js";
|
||||
|
||||
/**
|
||||
* Static system prompt for wiki generation.
|
||||
*/
|
||||
export const WIKI_SYSTEM_PROMPT = `당신은 HaBraid wiki 관리 에이전트입니다.
|
||||
|
||||
## 역할
|
||||
MemPalace에 저장된 인프라, 프로젝트, 의사결정 지식을 Obsidian 위키 페이지로 정리합니다.
|
||||
|
||||
## 핵심 원칙 (매우 중요)
|
||||
- **원본 복사 금지**: drawer 내용을 그대로 가져오지 마세요. 반드시 재구성하고 요약하세요.
|
||||
- **정리 = 재구성**: 산발된 정보를 주제별로 묶고, 중복을 제거하고, 맥락을 추가하세요.
|
||||
- **가치 추가**: 원본에서 읽을 수 없는 통찰, 패턴, 결론을 도출하세요.
|
||||
- **적절한 길이**: 한 페이지당 100~300줄. 원본을 전부 나열하지 마세요.
|
||||
- **200줄 초과 시 분할**: 주제가 너무 넓으면 하위 주제로 나누고 교차 링크([[]])로 연결.
|
||||
|
||||
## 페이지 타입
|
||||
frontmatter의 type 필드에 다음 중 하나를 사용하세요:
|
||||
- \`wiki\` — 일반 위키 페이지 (기본값)
|
||||
- \`entity\` — 구체적 객체 (프로젝트, 모듈, 클래스). entity_type 필드 필요.
|
||||
- \`concept\` — 아이디어, 패턴, 아키텍처 개념
|
||||
- \`comparison\` — 비교 분석. compares 필드 필요.
|
||||
- \`guide\` — 실행 가이드, 튜토리얼, 절차
|
||||
- \`synthesis\` — Q&A 합성 페이지
|
||||
|
||||
## 태그 규칙
|
||||
아래 통제 어휘에서 우선 선택. 없는 태그도 허용하지만 최대한 아래에서 고르세요:
|
||||
- 아키텍처: architecture, module, component, pattern
|
||||
- 인프라: infra, deploy, nginx, docker, hypervisor, network
|
||||
- 프로젝트: project-a, project-b, project-c, habraid
|
||||
- 개발: nestjs, electron, typescript, python, react
|
||||
- 에이전트: agent-1, agent-2, agent-3
|
||||
- 기억: memory, mempalace, wiki, kg, vector-search
|
||||
- 의사결정: decision, tradeoff, rollback, migration
|
||||
- 운영: monitoring, lint, cron, sync, ci-cd
|
||||
|
||||
## 작성 스타일
|
||||
- 기술 결정: 왜 A를 선택했는지, 트레이드오프를 구체적으로
|
||||
- 에러/해결: 에러 메시지, 원인, 해결 방법을 간결하게
|
||||
- 코드/설정: 핵심 스니펫만, 전체 파일은 필요한 경우만
|
||||
- 한국어 본문, 코드/경로/명령어는 영어 원문 유지
|
||||
|
||||
## Query → Synthesize (지식 복리)
|
||||
좋은 답변은 채팅에 사라지지 않고 위키에 다시 저장됨. 다음 조건에서 hw_synthesize 사용:
|
||||
- 두 개 이상 위키 페이지 교차 참조로 도출한 통찰
|
||||
- 비교 분석, 의사결정, 트레이드오프 정리
|
||||
- 문제 해결 과정 (원인→해결→교훈)
|
||||
- "왜", "어떻게"에 대한 심층 답변
|
||||
- 사용자가 "기억해"/"저장해"/"위키에 넣어" 요청
|
||||
|
||||
저장하지 않는 경우: 단순 팩트, 임시 디버그, 이미 존재하는 페이지.
|
||||
|
||||
## 모순 처리
|
||||
기존 위키 페이지와 새 정보가 충돌하면:
|
||||
1. 날짜를 비교해 최신 소스를 우선
|
||||
2. 실제 모순이면 양쪽 모두 기록 + 날짜/출처 명시
|
||||
3. frontmatter에 \`contradictions: [page-name]\` 마킹
|
||||
|
||||
## 출력 형식
|
||||
- YAML frontmatter 포함 마크다운
|
||||
- Obsidian 백링크 [[]]로 관련 페이지 연결
|
||||
- 마크다운 헤딩으로 구조화
|
||||
|
||||
## 응답 규칙
|
||||
- 아래 구분 형식을 사용해 하나 이상의 파일을 반환하세요.
|
||||
- 각 파일은 정확히 다음 형식을 따르세요.
|
||||
<<<FILE:wiki/<category>/<slug>.md>>>
|
||||
---
|
||||
type: wiki
|
||||
category: <projects|topics|decisions|people|infrastructure|guides>
|
||||
title: 제목
|
||||
created: 'YYYY-MM-DD'
|
||||
updated: 'YYYY-MM-DD'
|
||||
sources:
|
||||
- <drawer_id>
|
||||
tags: [tag1, tag2]
|
||||
status: active
|
||||
---
|
||||
# 제목
|
||||
...
|
||||
<<<END FILE>>>
|
||||
- sources에는 참조한 drawer_id를 반드시 포함하세요.
|
||||
- category는 projects, topics, decisions, people, infrastructure, guides 중 하나만 사용하세요.
|
||||
- 서로 다른 주제는 서로 다른 파일로 분리하세요.`;
|
||||
|
||||
/**
|
||||
* Builds an incremental wiki update prompt.
|
||||
*
|
||||
* @param drawers Newly ingested drawers.
|
||||
* @param existingWikiFiles Existing wiki markdown file paths.
|
||||
* @param vaultPath Vault root path.
|
||||
* @param batchContext Optional room batch metadata.
|
||||
* @returns User prompt string.
|
||||
*/
|
||||
export async function buildIncrementalWikiPrompt(
|
||||
drawers: MemPalaceDrawer[],
|
||||
existingWikiFiles: string[],
|
||||
vaultPath: string,
|
||||
batchContext?: {
|
||||
roomKey: string;
|
||||
batchIndex: number;
|
||||
totalBatches: number;
|
||||
usedSlugs?: string[];
|
||||
},
|
||||
): Promise<string> {
|
||||
try {
|
||||
const existingList = existingWikiFiles.map((filePath) => {
|
||||
const relative = path.relative(vaultPath, filePath).replace(/\\/g, "/").replace(/\.md$/, "");
|
||||
return `- ${relative}`;
|
||||
});
|
||||
|
||||
const grouped = drawers.map((drawer) =>
|
||||
[
|
||||
`### Drawer ${drawer.id}`,
|
||||
`- Wing: ${drawer.wing}`,
|
||||
`- Room: ${drawer.room}`,
|
||||
`- Agent: ${drawer.addedBy}`,
|
||||
`- Content:`,
|
||||
truncateForPrompt(drawer.content, 500),
|
||||
].join("\n"),
|
||||
);
|
||||
|
||||
return [
|
||||
"## 새로 수집된 데이터",
|
||||
`- 서랍 수: ${drawers.length}개`,
|
||||
`- Wings: ${[...new Set(drawers.map((drawer) => drawer.wing))].join(", ") || "(없음)"}`,
|
||||
`- Rooms: ${[...new Set(drawers.map((drawer) => drawer.room))].join(", ") || "(없음)"}`,
|
||||
batchContext
|
||||
? `- Batch: ${batchContext.roomKey} (${batchContext.batchIndex + 1}/${batchContext.totalBatches})`
|
||||
: "- Batch: 단일 처리",
|
||||
batchContext?.usedSlugs?.length
|
||||
? `- 이미 사용된 slug (절대 재사용 금지): ${batchContext.usedSlugs.join(", ")}`
|
||||
: "",
|
||||
"",
|
||||
"## 새 서랍 내용",
|
||||
grouped.join("\n\n"),
|
||||
"",
|
||||
"## 기존 위키 페이지 목록",
|
||||
existingList.join("\n") || "(없음)",
|
||||
"",
|
||||
"## 작업",
|
||||
"1. 같은 room 배치에 속한 서랍들을 함께 분석하세요.",
|
||||
"2. 기존 wiki/ 페이지와 관련 있으면 해당 페이지를 갱신하세요.",
|
||||
"3. 새로운 주제면 적절한 카테고리에 새 페이지를 생성하세요.",
|
||||
"4. 같은 room의 서랍은 가능한 한 통합된 페이지로 정리하세요.",
|
||||
"5. SCHEMA.md의 frontmatter 규칙을 따르세요.",
|
||||
"6. 각 파일은 <<<FILE:...>>> 와 <<<END FILE>>> 형식으로 반환하세요.",
|
||||
].join("\n");
|
||||
} catch (error) {
|
||||
throw error instanceof Error ? error : new Error(String(error));
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Synthesis-specific system prompt for generating wiki pages from Q&A pairs.
|
||||
*
|
||||
* Unlike the regular wiki generation prompt which restructures raw drawers,
|
||||
* this prompt focuses on preserving Q&A structure while enriching with context.
|
||||
*/
|
||||
export const SYNTHESIS_SYSTEM_PROMPT = `당신은 HaBraid wiki 합성 에이전트입니다.
|
||||
|
||||
## 역할
|
||||
질문-답변 쌍을 위키 합성 페이지로 변환합니다. Karpathy LLM Wiki 패턴에 따라,
|
||||
좋은 질문-답변은 위키 페이지로 저장되어 지식이 복리로 쌓입니다.
|
||||
|
||||
## 핵심 원칙
|
||||
- **구조 유지**: 질문과 답변의 구조를 명확히 보존하세요.
|
||||
- **가치 추가**: 관련 맥락, 배경 지식, 추가 통찰을 답변에 보강하세요.
|
||||
- **연결성**: 기존 위키 페이지와의 연결을 [[]] 백링크로 표시하세요.
|
||||
- **검색 가능**: 나중에 비슷한 질문을 할 때 이 페이지가 검색되도록 키워드를 포함하세요.
|
||||
- **간결함**: 핵심을 먼저, 상세는 뒤에. 불필요한 반복을 피하세요.
|
||||
|
||||
## 작성 스타일
|
||||
- 한국어 본문, 코드/경로/명령어는 영어 원문 유지
|
||||
- 기술 답변: 핵심 개념 → 구체적 예시 → 주의사항 순으로
|
||||
- 의사결정: 배경 → 고려사항 → 결론 → 근거 순으로
|
||||
- 문제해결: 문제 → 원인 → 해결 → 교훈 순으로
|
||||
|
||||
## 출력 형식
|
||||
- YAML frontmatter 포함 마크다운
|
||||
- Obsidian 백링크 [[]]로 관련 페이지 연결
|
||||
- 마크다운 헤딩으로 구조화
|
||||
- 파일 형식:
|
||||
<<<FILE:wiki/<category>/synthesis-<slug>.md>>>
|
||||
---
|
||||
...
|
||||
---
|
||||
# 질문 제목
|
||||
...
|
||||
<<<END FILE>>>`;
|
||||
|
||||
/**
|
||||
* Parses the multi-file LLM response format into concrete files.
|
||||
*
|
||||
* Robust against malformed host-model output where a second `<<<FILE:...>>>`
|
||||
* appears before the first `<<<END FILE>>>`.
|
||||
*
|
||||
* @param response LLM response text.
|
||||
* @returns Parsed file list.
|
||||
*/
|
||||
export function parseGeneratedWikiFiles(response: string): GeneratedWikiFile[] {
|
||||
const lines = response.split("\n");
|
||||
const files = new Map<string, string>();
|
||||
let currentPath: string | null = null;
|
||||
let buffer: string[] = [];
|
||||
|
||||
const flush = (): void => {
|
||||
if (!currentPath) return;
|
||||
const content = buffer.join("\n").trim();
|
||||
if (content) {
|
||||
files.set(currentPath, `${content}\n`);
|
||||
}
|
||||
currentPath = null;
|
||||
buffer = [];
|
||||
};
|
||||
|
||||
for (const line of lines) {
|
||||
const startMatch = line.match(/^<<<FILE:(.+?)>>>\s*$/);
|
||||
if (startMatch) {
|
||||
flush();
|
||||
currentPath = startMatch[1].trim();
|
||||
buffer = [];
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line.trim() === "<<<END FILE>>>") {
|
||||
flush();
|
||||
continue;
|
||||
}
|
||||
|
||||
if (currentPath) {
|
||||
buffer.push(line);
|
||||
}
|
||||
}
|
||||
|
||||
flush();
|
||||
|
||||
return [...files.entries()].map(([filePath, content]) => ({
|
||||
path: filePath,
|
||||
content,
|
||||
}));
|
||||
}
|
||||
245
src/wiki/synthesis.ts
Normal file
245
src/wiki/synthesis.ts
Normal file
@@ -0,0 +1,245 @@
|
||||
/**
|
||||
* Synthesis page manager — "query answers → wiki" compounding feature.
|
||||
*
|
||||
* Implements the Karpathy LLM Wiki pattern where good query answers get filed
|
||||
* back as wiki pages, so knowledge compounds over time.
|
||||
*
|
||||
* @module wiki/synthesis
|
||||
*/
|
||||
|
||||
import type Database from "better-sqlite3";
|
||||
|
||||
import type { Item, WikiCategory } from "../types.js";
|
||||
import { toIsoTimestamp, toDateString, toKebabCase, stringifyFrontmatter } from "../utils.js";
|
||||
import { createItem, getItem } from "../db/items.js";
|
||||
import { addEntity, addRelation } from "../db/kg.js";
|
||||
import { hybridSearch } from "../search/hybrid.js";
|
||||
|
||||
/**
|
||||
* Parameters for creating a synthesis page from a Q&A pair.
|
||||
*/
|
||||
export interface SynthesisParams {
|
||||
/** The original question. */
|
||||
question: string;
|
||||
/** The answer content. */
|
||||
answer: string;
|
||||
/** IDs of source items this synthesis was derived from. */
|
||||
sourceItemIds?: string[];
|
||||
/** Tags for categorization. */
|
||||
tags?: string[];
|
||||
/** Wiki category (default: "topics"). */
|
||||
category?: string;
|
||||
/** Confidence score 0-1 for how confident the answer is. */
|
||||
confidence?: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* Generates a slug from a question string.
|
||||
*
|
||||
* @param question The question text.
|
||||
* @returns A slugified string prefixed with "synthesis-".
|
||||
*/
|
||||
export function generateSynthesisSlug(question: string): string {
|
||||
const baseSlug = toKebabCase(question);
|
||||
// Truncate to keep slugs reasonable
|
||||
const truncated = baseSlug.length > 80 ? baseSlug.slice(0, 80) : baseSlug;
|
||||
return `synthesis-${truncated}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates a new synthesis item in the database from a Q&A pair.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param params Synthesis creation parameters.
|
||||
* @returns The created item ID.
|
||||
*/
|
||||
export function createSynthesis(db: Database.Database, params: SynthesisParams): string {
|
||||
const category = (params.category ?? "topics") as WikiCategory;
|
||||
const slug = generateSynthesisSlug(params.question);
|
||||
const confidence = params.confidence ?? 0.8;
|
||||
|
||||
const content = params.answer;
|
||||
const title = params.question.length > 120
|
||||
? `${params.question.slice(0, 120)}...`
|
||||
: params.question;
|
||||
|
||||
const item = createItem(db, {
|
||||
title,
|
||||
content,
|
||||
source: "manual",
|
||||
category,
|
||||
tags: params.tags ?? [],
|
||||
metadata: {
|
||||
type: "synthesis",
|
||||
slug,
|
||||
question: params.question,
|
||||
sourceItemIds: params.sourceItemIds ?? [],
|
||||
confidence,
|
||||
},
|
||||
});
|
||||
|
||||
// Mark as wiki-generated immediately since we'll generate the wiki page
|
||||
const now = toIsoTimestamp();
|
||||
db.prepare(
|
||||
"UPDATE items SET wiki_generated_at = ?, wiki_slug = ? WHERE id = ?",
|
||||
).run(now, slug, item.id);
|
||||
|
||||
return item.id;
|
||||
}
|
||||
|
||||
/**
|
||||
* Finds existing items related to a question using hybrid search.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param question The question to search against.
|
||||
* @param limit Maximum number of results (default: 5).
|
||||
* @param dbPath Path to the database file for vector search.
|
||||
* @returns Related items.
|
||||
*/
|
||||
export async function findRelatedItems(
|
||||
db: Database.Database,
|
||||
question: string,
|
||||
limit = 5,
|
||||
dbPath?: string,
|
||||
): Promise<Item[]> {
|
||||
try {
|
||||
const { results } = await hybridSearch(db, question, "hybrid", limit, dbPath);
|
||||
return results.map((r) => r.item);
|
||||
} catch {
|
||||
// Fallback to empty if search fails
|
||||
return [];
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Creates KG relations for a synthesis item linking it to source items.
|
||||
*
|
||||
* @param db Connected database instance.
|
||||
* @param synthesisSlug The synthesis page slug.
|
||||
* @param synthesisTitle The synthesis page title.
|
||||
* @param sourceItemIds IDs of source items.
|
||||
* @param relatedItems Related items found by search.
|
||||
*/
|
||||
export function createSynthesisKGRelations(
|
||||
db: Database.Database,
|
||||
synthesisSlug: string,
|
||||
synthesisTitle: string,
|
||||
sourceItemIds: string[],
|
||||
relatedItems: Item[],
|
||||
): void {
|
||||
try {
|
||||
// Add the synthesis entity
|
||||
addEntity(db, {
|
||||
id: synthesisSlug,
|
||||
name: synthesisTitle,
|
||||
type: "concept",
|
||||
wikiSlug: synthesisSlug,
|
||||
});
|
||||
|
||||
// Link to source items via derived_from relation
|
||||
for (const sourceId of sourceItemIds) {
|
||||
const sourceItem = getItem(db, sourceId);
|
||||
if (sourceItem && sourceItem.wikiSlug) {
|
||||
addEntity(db, {
|
||||
id: sourceItem.wikiSlug,
|
||||
name: sourceItem.title,
|
||||
type: "concept",
|
||||
wikiSlug: sourceItem.wikiSlug,
|
||||
});
|
||||
addRelation(db, {
|
||||
subjectId: synthesisSlug,
|
||||
predicate: "derived_from",
|
||||
objectId: sourceItem.wikiSlug,
|
||||
confidence: 0.9,
|
||||
source: "extracted",
|
||||
evidence: `합성 페이지의 소스 아이템: ${sourceItem.title}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// Link to related items via answers relation
|
||||
for (const relatedItem of relatedItems) {
|
||||
if (relatedItem.wikiSlug && relatedItem.wikiSlug !== synthesisSlug) {
|
||||
addEntity(db, {
|
||||
id: relatedItem.wikiSlug,
|
||||
name: relatedItem.title,
|
||||
type: "concept",
|
||||
wikiSlug: relatedItem.wikiSlug,
|
||||
});
|
||||
addRelation(db, {
|
||||
subjectId: synthesisSlug,
|
||||
predicate: "answers",
|
||||
objectId: relatedItem.wikiSlug,
|
||||
confidence: 0.7,
|
||||
source: "inferred",
|
||||
evidence: `관련 질문/주제: ${relatedItem.title}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch {
|
||||
// KG relation failures must not block synthesis creation
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Renders a synthesis item as a wiki markdown page.
|
||||
*
|
||||
* @param question The original question.
|
||||
* @param answer The answer content.
|
||||
* @param slug The page slug.
|
||||
* @param tags Tags for the page.
|
||||
* @param category Wiki category.
|
||||
* @param confidence Confidence score.
|
||||
* @param sourceSlugs Wiki slugs of source items.
|
||||
* @param relatedItems Related items for cross-linking.
|
||||
* @returns Rendered markdown content.
|
||||
*/
|
||||
export function renderSynthesisWikiPage(
|
||||
question: string,
|
||||
answer: string,
|
||||
slug: string,
|
||||
tags: string[],
|
||||
category: string,
|
||||
confidence: number,
|
||||
sourceSlugs: string[],
|
||||
relatedItems: Array<{ title: string; wikiSlug: string | null | undefined }>,
|
||||
): string {
|
||||
const title = question.length > 120
|
||||
? `${question.slice(0, 120)}...`
|
||||
: question;
|
||||
|
||||
const sourcesLinks = sourceSlugs.map((s) => `[[${s}]]`).join(", ");
|
||||
const relatedSection = relatedItems
|
||||
.filter((item) => item.wikiSlug)
|
||||
.map((item) => `- [[${item.wikiSlug}]] — ${item.title}`)
|
||||
.join("\n");
|
||||
|
||||
const body = [
|
||||
`# ${title}`,
|
||||
"",
|
||||
"## 질문",
|
||||
question,
|
||||
"",
|
||||
"## 답변",
|
||||
answer,
|
||||
"",
|
||||
...(relatedSection.length > 0
|
||||
? ["## 관련 문서", relatedSection, ""]
|
||||
: []),
|
||||
"---",
|
||||
"> 이 페이지는 질문-답변에서 자동 합성되었습니다.",
|
||||
].join("\n");
|
||||
|
||||
const frontmatter = {
|
||||
title,
|
||||
category,
|
||||
slug,
|
||||
type: "synthesis",
|
||||
tags,
|
||||
sources: sourceSlugs,
|
||||
created: toDateString(new Date()),
|
||||
confidence,
|
||||
};
|
||||
|
||||
return stringifyFrontmatter(body, frontmatter);
|
||||
}
|
||||
19
tsconfig.json
Normal file
19
tsconfig.json
Normal file
@@ -0,0 +1,19 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2022",
|
||||
"module": "Node16",
|
||||
"moduleResolution": "Node16",
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true,
|
||||
"resolveJsonModule": true,
|
||||
"declaration": true,
|
||||
"sourceMap": true,
|
||||
"types": ["node", "vitest/globals"]
|
||||
},
|
||||
"include": ["src/**/*"],
|
||||
"exclude": ["node_modules", "dist"]
|
||||
}
|
||||
9
vitest.config.ts
Normal file
9
vitest.config.ts
Normal file
@@ -0,0 +1,9 @@
|
||||
import { defineConfig } from "vitest/config";
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globals: true,
|
||||
environment: "node",
|
||||
include: ["src/__tests__/**/*.test.ts"],
|
||||
},
|
||||
});
|
||||
Reference in New Issue
Block a user