The Mess#

Figure 1 — Three years of accumulated entropy: the Obsidian graph shows scattered, disconnected clusters while the tag pane reveals 1,280 chaotic tags — including a hex color code (#3498db) and duplicate variations like #AI (118), #aiAgents (5), and #agents (11).

Three years of Obsidian notes. 1,280 tags. Including a hex color code that somehow became a category. We fixed all of it in 30 minutes for $1.50.

Here’s what we were working with:

1
#ai
2
#AI
3
#artificialIntelligence
4
#agentic-systems
5
#agenticFrameworks
6
#3498db  <- A hex color somehow became a tag
7
#5year   <- What does this even mean?

The inconsistencies ran deeper than tags. Our YAML frontmatter was a mess of uppercase keys, ISO timestamps that broke tooling, and duplicate entries:

1
---
2
Title: Some Note          # Should be lowercase
3
Created: 2024-12-27T07:02:00  # ISO format breaks some tools
4
tags:
5
  - ai
6
  - AI                    # Duplicate!
7
---

Some older notes used Obsidian’s legacy inline property format. About 80 notes had no frontmatter at all — just raw markdown floating in the void.

The manual fix would have taken weeks. We had 1,028 files across three different formats, each needing analysis, tag assignment from a curated taxonomy, and formatting corrections. Nobody was going to do that by hand.

The Result#

After the batch completed and all files were updated, we opened Obsidian.

Figure 4 — From flat chaos to nested hierarchy: Claude Code analyzed 1,280 tags, identified logical categories, and designed a four-level taxonomy. The new structure enables drill-down navigation at every level.

The tag pane, which had been an unusable wall of 1,280 unrelated tags, was now a cleanly nested hierarchy. Claude Code had analyzed the full tag list, identified logical categories, designed a four-level hierarchy, and generated a complete mapping from old tags to new — all in minutes. The 1,280 chaotic tags became 1,040 hierarchical ones:

1
#ai/                              # Level 1: Top-level category
2
#ai/agents/                       # Level 2: Subcategory
3
#ai/agents/frameworks/            # Level 3: Specific area
4
#ai/agents/frameworks/crewai      # Level 4: Individual item
5
#ai/agents/frameworks/langgraph
6
#ai/agents/frameworks/autogen
7

8
#trading/                         # Another top-level
9
#trading/patterns/                # Trading patterns
10
#trading/patterns/abcd            # Specific pattern
11
#trading/patterns/harmonics
12
#trading/psychology/              # Trading psychology
13
#trading/psychology/discipline
14

15
#infrastructure/                  # DevOps & Infrastructure
16
#infrastructure/docker/           # Container tech
17
#infrastructure/docker/compose
18
#infrastructure/kubernetes/
19
#infrastructure/servers/proxmox

Click #ai/agents/frameworks/ and see exactly the 34 notes about agent frameworks. No more hunting through #ai vs #AI vs #artificialIntelligence. Click #trading/patterns/ and every pattern note is right there. The hierarchy enables drill-down at every level: #ai/ surfaces all 200+ AI-related notes, while #ai/rag/embeddings narrows to exactly the 15 notes about embedding strategies.

Three years of accumulated tag chaos, fixed in under an hour.

KEY INSIGHT: AI-assisted taxonomy design is a force multiplier — what would take days of manual categorization, Claude Code did in minutes, producing a cleaner result than we would have designed ourselves.

The Approach#

We built a three-stage Python pipeline: prepare, submit, and apply. The pipeline scanned every markdown file, built format-specific prompts, sent them through Anthropic’s Batch API, and applied the results back to the source files.

Figure 2 — The three-stage pipeline: each markdown file flows through analysis, format-specific prompting, and Batch API execution to produce a clean, standardized output.

1
prompt = f"""This file uses OLD Obsidian inline property format.
2

3
## Current Inline Properties

1
## Task
2
Convert to proper YAML frontmatter, preserving:
3
- Emoji values exactly
4
- Wikilinks as quoted strings
5
- ALL existing properties (lowercase keys)
6

7
Add: title, created, description, and curated tags from the approved taxonomy.
8
"""

The Transformation#

Figure 5 — Standardized metadata and frontmatter: keys lowercased, dates standardized, wikilinks preserved, descriptions added, and flat tags replaced with a hierarchical taxonomy — all in a single automated pass.

The most dramatic transformations were the inline property conversions, where the entire metadata format changed while preserving every relationship:

Before: Inline Properties

1
Type:: #note/seedling
2
Tags:: #area/personal , #area/health
3
Links:: [[2022-11-23]]
4
up:: [[Try This Before You Get Angry]]
5

6
Content starts here...

After: Clean YAML Frontmatter

1
---
2
title: Anger Only Makes Things Worse
3
created: 2022-11-23 10:30
4
type: "#note/seedling"
5
links: "[[2022-11-23]]"
6
up: "[[Try This Before You Get Angry]]"
7
tags:
8
  - "personal/health"
9
  - "philosophy/stoicism"
10
description: A reflection on managing anger and its negative consequences.
11
---
12

13
Content starts here...

Every wikilink preserved. Every emoji intact. Every property key lowercased. And now the note has a proper title, a description for search, and hierarchical tags that connect it to related notes across the vault.

What Clean Data Enabled#

Figure 6 — A structured knowledge graph: after curation, the vault transformed from scattered noise into an interconnected web of signal — the foundation for semantic search and retrieval.

Vault curation was never the end goal — it was the foundation. With clean, consistent metadata across every note, the rest of the system could do its job.

The semantic linking system (covered in Part 3) analyzed each note’s content, found related notes using vector similarity, and added bidirectional wiki links — 2,757 of them across the vault, turning isolated notes into a connected knowledge graph. Clean tags meant the embeddings captured real semantic meaning instead of noise from inconsistent metadata.

And the RAG chatbot (covered in Part 5) could finally deliver accurate answers because every note had a proper title, description, and hierarchical tags for the retrieval system to work with.

KEY INSIGHT: Data curation has compound returns. Clean tags enabled semantic linking, which enabled the chatbot. Each layer depends on the one before it — skip the foundation and everything downstream suffers.

Figure 7 — Curation was the foundation, not the goal: clean metadata enabled semantic linking (2,757 auto-generated links), which enabled the RAG chatbot (2.5s latency). Each layer depends on the one before it.

The Series#

This is Part 4 of a 5-part series on building an AI-powered knowledge management system:

1. From YouTube to Knowledge Graph — Turning 1,000+ videos into an interconnected knowledge base for $1.50
2. Anthropic Batch API in Production — 50% cost savings at scale, and the bug that almost corrupted everything
3. Building a Semantic Note Network — Vector search turned 1,024 isolated notes into a dense knowledge graph
4. Obsidian Vault Curation at Scale (this article) — Three years of tag chaos, fixed in 30 minutes for $1.50
5. Ask Your Vault Anything — A RAG chatbot that answers from your notes in 2.5 seconds

Next: Ask Your Vault Anything — The foundation is clean, the index is built. Now what if you could just ask your notes a question?