HaBraid/README.md

<div align="center">

# HaBraid

**Host-following memory visualization engine for Obsidian**

Turn long-term memory backends into a curated, graph-friendly wiki for humans and AI.

<p>
  <img alt="status" src="https://img.shields.io/badge/status-operational%20alpha-7c3aed">
  <img alt="runtime" src="https://img.shields.io/badge/runtime-node.js-339933">
  <img alt="transport" src="https://img.shields.io/badge/integration-MCP-1f6feb">
  <img alt="vault" src="https://img.shields.io/badge/output-Obsidian%20Wiki-8b5cf6">
  <img alt="llm" src="https://img.shields.io/badge/llm-host--following-0f766e">
  <img alt="loc" src="https://img.shields.io/badge/loc-12%2C000%2B-blueviolet">
  <img alt="files" src="https://img.shields.io/badge/source-57%20TS%20files-orange">
</p>

[Architecture](docs/architecture.md) ·
[Config Reference](docs/config-reference.md) ·
[Wiki Generation](docs/wiki-generation.md) ·
[Plans](docs/plans/README.md)

</div>

---

## Table of Contents

- [What is HaBraid?](#what-is-habraid)
- [Why HaBraid exists](#why-habraid-exists)
- [Core ideas](#core-ideas)
- [At a glance](#at-a-glance)
- [Demo workflow](#demo-workflow)
- [Features](#features)
- [Architecture diagram](#architecture-diagram)
- [How it works](#how-it-works)
- [Directory layout](#directory-layout)
- [Quick start](#quick-start)
- [MCP tools](#mcp-tools)
- [Example MCP flows](#example-mcp-flows)
- [Current status](#current-status)
- [Roadmap](#roadmap)
- [Documentation](#documentation)

---

## What is HaBraid?

HaBraid is a **memory visualization engine**.

It does **not** try to replace your memory backend. Instead, it sits between:

1. a long-term memory store such as **MemPalace**
2. a curated **Obsidian wiki layer**
3. a host agent such as **Hermes / Codex / OpenClaw**

Its job is to turn raw memory fragments—drawers, notes, logs, decisions, infrastructure facts, session traces—into a cleaner knowledge surface that is:

- readable by humans
- re-usable by agents
- navigable in Obsidian
- suitable for graph view without raw-noise pollution

In one line:

> **HaBraid turns memory backends into a curated, host-following Obsidian wiki.**

---

## Why HaBraid exists

Raw memory systems are great at **retaining information**, but not always great at **presenting relationships**.

Typical problems:

- too many small raw records
- duplicated context across sessions
- hidden connections between decisions, people, projects, and infra
- noisy graph view when raw files live in the same vault
- backend data that agents can search, but humans can't comfortably read

HaBraid exists to solve that gap.

It takes a backend that is optimized for retention and query, and adds a layer optimized for:

- structure
- linking
- summarization
- graph clarity
- handoff between people and AI systems

---

## Core ideas

### 1. Source of truth stays in the memory backend

MemPalace, mem0, or any future memory backend remains the canonical store.

HaBraid is intentionally **not** the source of truth.

### 2. The wiki is the expression layer

The wiki is where information becomes understandable.

That means HaBraid focuses on:

- grouping scattered facts into topics
- linking related pages with wikilinks
- preserving important provenance in metadata
- exposing concepts, decisions, people, infra, and projects as readable pages

### 3. Raw should be hidden, wiki should be visible

To keep Obsidian graph view useful:

- raw/source/debug material belongs in hidden runtime storage
- curated wiki pages belong in the user vault

That means the long-term target structure is:

- `~/.habraid/data/raw` → internal source material
- `~/wiki` → curated user-facing vault

### 4. Inference should follow the host when possible

HaBraid prefers **host-following generation**.

Instead of forcing its own internal model, it can delegate generation to the current host environment. In practice this means:

- HaBraid orchestrates prompts, batching, parsing, and persistence
- the host agent chooses the actual model/policy
- standalone fallback remains available when needed

---

## At a glance

| Area | What HaBraid does |
|------|--------------------|
| Source layer | Reads memory backends through adapters |
| Storage layer | Maintains a local DB for item tracking and generation state |
| Search layer | Hybrid BM25 + HNSW vector search with context enrichment |
| Graph layer | Knowledge graph with entities, relations, and community detection |
| Generation layer | Uses host-following or fallback LLM routes to create curated pages |
| Lint layer | 3-tier vault validation (static, HNSW duplicates, contradiction detection) |
| Vault layer | Writes graph-friendly wiki pages for Obsidian |
| Integration layer | Exposes 16 MCP tools for agent-driven workflows |

### Good fit for

- long-term memory systems that are strong at storage but weak at presentation
- Obsidian users who want cleaner graph structure
- agent systems that need a readable shared knowledge layer
- infra / project / decision archives that have too many raw records

### Not trying to be

- a replacement for MemPalace or other memory backends
- a generic note-taking app
- a fully standalone model-serving platform
- a raw log browser inside the visible vault

---

## Demo workflow

A typical HaBraid loop looks like this:

1. new memory items arrive in the backend
2. HaBraid ingests them into the local item DB
3. `hw_generate` batches ungenerated items
4. the host agent generates or updates curated wiki pages
5. Obsidian opens `~/wiki` and shows the cleaned graph

### Example result

```text
raw memory fragments
  ├─ deploy validation logs
  ├─ infra decisions
  ├─ session summaries
  └─ people / project facts

        ↓ HaBraid

curated wiki pages
  ├─ wiki/decisions/deploy-gate-signals-for-mvp-rollouts.md
  ├─ wiki/guides/session-bootstrap-and-deploy-manager-prompts.md
  ├─ wiki/projects/project-alpha.md
  ├─ wiki/projects/mijung-ai-foodtech-platform.md
  └─ wiki/decisions/nestjs-circular-dependency-response.md
```

The important change is not just format conversion.
It is **semantic compression + linkage**.

---

## Features

### Core engine

- **Host-following LLM route**
  - host-first generation path
  - fallback provider support (OpenAI, Ollama, ZAI)
  - route/provider/model observability in runtime state

- **Incremental wiki generation**
  - only ungenerated items are processed
  - DB-backed generation tracking via `wiki_generated_at`
  - restart-friendly flow

- **Checkpointed long-running generation**
  - subgroup-level persistence during generation
  - partial progress survives failures
  - large rebuilds no longer wait until the very end to write files

### Search & Retrieval

- **Hybrid search (BM25 + HNSW)**
  - BM25 keyword search via SQLite FTS5
  - Semantic vector search with ONNX local embeddings
  - HNSW ANN index for fast approximate nearest neighbor
  - Context enrichment mode with KG relations and wiki link re-ranking

### Knowledge Graph

- **Entity and relation management**
  - Typed entities: concept, project, person, tool, event, decision
  - Predicate-based relations with confidence scores
  - Source tracking (extracted, inferred, ambiguous)

- **Community detection**
  - Label Propagation Algorithm (LPA) clustering
  - Automatic community assignment for graph visualization
  - Mermaid diagram export for KG subgraphs

### Quality & Validation

- **3-tier lint system**
  - Tier 1: Static checks (orphans, broken links, frontmatter, ungenerated items)
  - Tier 2: HNSW duplicate detection (semantic near-duplicate identification)
  - Tier 3: LLM-powered contradiction detection across items

- **Contradiction tracking**
  - Automatic conflict detection between knowledge items
  - Resolution workflow (resolve / false positive)
  - Open/resolved status management

### Content

- **Q&A Synthesis**
  - Karpathy LLM Wiki pattern: question-answer pairs become wiki pages
  - Compound knowledge accumulation from source items

- **Daily work log**
  - Automatic generation activity logging
  - Date-range summaries for retrospectives

### Integration

- **MCP server**
  - 16 tools covering the full operational loop
  - Hermes config.yaml registration
  - tsx runtime for TypeScript MCP server

- **SQLite-backed item store**
  - local item database (500+ items in production)
  - schema integrity auto-repair on startup
  - generation metadata and sync state

- **Obsidian-friendly output**
  - YAML frontmatter
  - wikilinks
  - category-based page layout
  - graph-oriented curated pages

---

## Architecture diagram

```mermaid
flowchart TD
    A[Memory backend\nMemPalace / future sources] --> B[Ingest adapters]
    B --> C[Local item DB\nSQLite + FTS5 + HNSW vectors]
    C --> D[Prompt builder + batching]
    D --> E[LLM gateway]
    E -->|preferred| F[Host-following generation]
    E -->|fallback| G[Standalone backend]
    F --> H[Curated wiki output]
    G --> H
    H --> I[~/wiki in Obsidian]
    C --> J[MCP tools\nhw_status / hw_search / hw_generate / ...]
    C --> K[Knowledge Graph\nentities + relations + communities]
    K --> J
    C --> L[3-tier Lint\nstatic + HNSW dup + contradictions]
    L --> J
```

### Separation of concerns

- **backend** keeps durable memory
- **HaBraid** performs orchestration, transformation, and graph extraction
- **host** owns model selection when available
- **Obsidian** becomes the readable interface layer

---

## How it works

```text
memory backend (MemPalace / future sources)
        │
        ▼
   ingest / adapters
        │
        ▼
   local item DB (SQLite)
   ├── FTS5 keyword index
   ├── HNSW vector index
   └── KG entity/relation tables
        │
        ├──→ hybrid search (BM25 + semantic)
        ├──→ knowledge graph (entities + communities)
        ├──→ 3-tier lint (static + dup + contradiction)
        │
        ▼
 prompt builder + batching
        │
        ▼
 host-following or fallback generation
        │
        ▼
 curated Obsidian wiki
```

The important design split is:

- **backend keeps memory**
- **HaBraid structures memory**
- **Obsidian presents memory**

---

## Directory layout

Current runtime-oriented structure:

```text
~/.habraid/
├── app/                     # source repo (57 TS files, 12K+ LOC)
│   ├── src/
│   │   ├── mcp-server.ts    # MCP server entry point
│   │   ├── db/              # SQLite DB, migrations, schema
│   │   ├── search/          # Hybrid BM25 + HNSW search
│   │   ├── kg/              # Knowledge graph + community detection
│   │   ├── lint/            # 3-tier lint system
│   │   ├── wiki/            # Wiki generation engine
│   │   └── tools/           # MCP tool handlers
│   └── dist/                # compiled output
└── data/                    # runtime state
    ├── config.json
    ├── habraid.db           # SQLite (items, vectors, KG, FTS5)
    ├── logs/
    ├── models/              # ONNX embedding model cache
    ├── backups/
    └── raw/                 # internal raw cache / source material

~/wiki/
├── index.md
├── overview.md
├── log.md
└── wiki/                    # curated pages visible to the user
    ├── projects/
    ├── topics/
    ├── decisions/
    ├── guides/
    └── infrastructure/
```

### Intended separation

| Path | Purpose |
|------|---------|
| `~/.habraid/app` | application source code |
| `~/.habraid/data` | runtime DB, config, logs, models, raw cache |
| `~/wiki` | curated user-facing Obsidian vault |

This split is intentional:

- runtime state stays hidden and operational
- user-facing graph remains cleaner
- repo, DB, and generated output do not all fight for the same surface

---

## Quick start

### Requirements

- Node.js 18+
- npm
- Obsidian (optional but recommended for browsing the vault)
- Hermes / host environment if using host-following mode
- MemPalace if you want live backend ingestion

### Install

```bash
cd ~/.habraid/app
npm install
npm run build
```

### Config

Primary config path:

```text
~/.habraid/data/config.json
```

Related references:

- `docs/config-reference.md`
- legacy fallbacks still exist for compatibility, but the runtime target is `~/.habraid/data/config.json`

### Run locally

Development:

```bash
cd ~/.habraid/app
npm run dev
```

Built server:

```bash
cd ~/.habraid/app
node dist/mcp-server.js
```

### Register with Hermes

Add to `~/.hermes/config.yaml`:

```yaml
mcp_servers:
  habraid:
    command: npx
    args: ["tsx", "/path/to/.habraid/app/src/mcp-server.ts"]
```

### Open the curated vault

```text
~/wiki
```

That is the path intended for Obsidian.

---

## MCP tools

HaBraid exposes 16 MCP tools under the `hw_*` namespace.

### Tool summary

| Tool | Purpose |
|------|---------|
| `hw_status` | Show vault, DB, and generation route status |
| `hw_ingest` | Ingest source material into raw cache and local DB |
| `hw_index` | Build vector embeddings and HNSW ANN index |
| `hw_search` | Search items using keyword / semantic / hybrid modes |
| `hw_add` | Add a knowledge item directly to the DB |
| `hw_read` | Read files inside the vault |
| `hw_generate` | Generate or update wiki pages incrementally |
| `hw_sync` | Run the full pipeline: ingest → index → generate |
| `hw_lint` | 3-tier vault validation (static + HNSW dup + contradictions) |
| `hw_graph` | Knowledge graph query, entity subgraph, community detection, Mermaid export |
| `hw_entity_add` | Add typed entities to the knowledge graph |
| `hw_entity_search` | Search entities by name, ID, or type |
| `hw_relation_add` | Add relations between entities |
| `hw_contradictions` | Detect and manage contradictions across knowledge items |
| `hw_daily_log` | Generate or read daily work log summaries |
| `hw_synthesize` | Save Q&A as synthesized wiki pages (Karpathy LLM Wiki pattern) |

### Operational loop

```text
ingest → index → generate → lint → search → graph → synthesize
```

These cover the main workflows:

- **inspect**: status, daily log
- **ingest & index**: bring data in, build embeddings
- **generate & sync**: create curated pages
- **search**: retrieve items with BM25 + semantic hybrid
- **graph**: explore entities, relations, communities
- **lint**: validate quality (orphans, broken links, duplicates, contradictions)
- **synthesize**: accumulate compound knowledge

---

## Example MCP flows

### Inspect current runtime state

```text
User: "HaBraid 상태 봐줘"
→ hw_status
→ raw files / wiki pages / DB items / last LLM route 확인
```

### Ingest then generate

```text
User: "새 memory 반영하고 위키 생성해줘"
→ hw_ingest
→ hw_index
→ hw_generate
```

### Search before opening notes

```text
User: "project-alpha 관련 의사결정 찾아줘"
→ hw_search(query="project-alpha", mode="hybrid", context=true)
→ KG relations, wiki links, re-ranked results
```

### Full pipeline

```text
User: "전체 sync 돌려줘"
→ hw_sync
```

### Knowledge graph exploration

```text
User: "nestjs 순환참조 관련 엔티티와 관계 보여줘"
→ hw_entity_search(query="nestjs")
→ hw_graph(entity="nestjs-circular-dep", depth=2, format="mermaid")
```

### Lint and fix issues

```text
User: "위키 품질 검사해줘"
→ hw_lint(checks=["orphans", "broken_links", "frontmatter"])
→ hw_lint(checks=["duplicates"])          # Tier 2: HNSW-based
→ hw_lint(checks=["contradictions"], withLlm=true)  # Tier 3: LLM-based
```

### Synthesize compound knowledge

```text
User: "이 질문에 대한 답변을 위키로 남겨줘"
→ hw_synthesize(question="...", answer="...", tags=["..."])
→ wiki/topics/ 생성
```

These flows are the intended operator experience: HaBraid should feel like a structured memory engine behind a clean MCP interface.

---

## Current status

HaBraid is already usable in production with real data.

### Production metrics

- **500+** knowledge items in DB
- **500+** vector embeddings indexed
- **69** curated wiki pages generated
- **57** TypeScript source files, **12K+** LOC
- **16** MCP tools

### Stable enough to use

- hidden runtime layout (`~/.habraid/`)
- MCP-driven workflow with 16 tools
- host-following generation path
- checkpointed regeneration
- hybrid BM25 + HNSW search
- knowledge graph with entities, relations, and community detection
- 3-tier lint system (static + HNSW dup + contradictions)
- Q&A synthesis (Karpathy LLM Wiki pattern)
- curated wiki output in `~/wiki`
- schema integrity auto-repair

### Not final yet

- hidden raw isolation is still being tightened
- generation quality and category/slug consistency still need iteration
- direct MemPalace ingest remains a separate bugfix track
- contradiction detection LLM integration is early stage

Think of the current state as:

> **operational alpha with real output, real infra value, and active architectural cleanup**

---

## Roadmap

### Near term

- move raw fully out of the user vault
- improve graph-friendly page relationships
- finish runtime path cleanup around hidden raw storage
- harden continuation behavior for partial generation failures
- expand lint auto-fix capabilities

### Mid term

- improve page deduplication and merge quality
- tighten category placement and slug stability
- better quality control for generated frontmatter and links
- improve wiki index / overview / navigation surfaces
- richer contradiction detection with cross-source verification

### Longer term

- richer hybrid retrieval with re-ranking
- backend abstraction beyond MemPalace
- stronger entity / concept / decision graph extraction
- more productized onboarding and one-click setup
- real-time sync with live memory backends

---

## Documentation

### Product / architecture docs

- `docs/architecture.md`
- `docs/config-reference.md`
- `docs/wiki-generation.md`
- `docs/specs/host-following-llm.md`

### Planning docs

- `docs/plans/README.md`
- `docs/plans/PLAN.md`
- `docs/plans/PLAN-v2.md`
- `docs/plans/PLAN-MCP.md`
- `docs/plans/PLAN-VECTOR.md`
- `docs/plans/PLAN-WIKI-LINT.md` — 3-tier lint system design

### Session-scoped plans

- `.hermes/plans/`

---

## Design stance

HaBraid is built around a specific product stance:

- memory systems should remain good at memory
- host agents should remain in charge of inference
- the wiki should become the shared interpretation layer
- graph quality matters more than raw completeness inside the visible vault
- knowledge should compound over time through synthesis

If the backend is the archive, HaBraid is the map.