First Ingestion and Semantic Search

First Ingestion & Semantic Search

This guide walks through your first end-to-end ingestion and semantic search workflow using the Hedera Guardian AI Toolkit.

---

Overview

At this stage, your infrastructure services should already be running:

• Qdrant (vector database)
• MCP Server

You will complete the end-to-end workflow in five steps.

1

Step 1 — Add Your Documents

Place your methodology files into:

data/input/documents/

Supported formats:

• PDF
• DOCX

You may add:

• Full methodologies (100+ pages)
• Templates
• Supporting documentation

Each file will be processed independently and indexed into the vector database.

2

Step 2 — Run Document Ingestion

From the root of the repository:

Standard profile:

docker compose run --rm document-ingestion-worker

GPU profile (if configured):

docker compose -f docker-compose.yml -f docker-compose.gpu.yml run --rm document-ingestion-worker

Low-memory profile:

docker compose -f docker-compose.yml -f docker-compose.low-memory.yml run --rm document-ingestion-worker

3

Step 3 — Verify Ingestion

Once ingestion completes:

• A Qdrant collection named methodology_documents will exist
• The collection will contain one point per chunk

You can verify via:

http://localhost:6333/dashboard

Each stored record contains:

• The chunk text
• Embedding vectors
• Metadata
• LaTeX formulas (when applicable)

4

Step 4 — Connect Your MCP Client

Your MCP Server should already be running at:

http://localhost:9000

To verify:

npx @modelcontextprotocol/inspector --server-url http://localhost:9000/mcp

You should see available tools including:

• Semantic search tools
• Schema builder tools

5

Step 5 — Perform Your First Semantic Search

Using an MCP-compatible AI client (e.g., Claude Desktop):

Ask a grounded question such as:

• What are the applicability conditions defined in VM42?
• What quantification approaches are defined in this methodology?
• What data parameters must be monitored during the crediting period?

The AI will:

1. Call the semantic search tool
2. Retrieve relevant chunks
3. Filter by metadata if necessary
4. Generate a response grounded in document content

Responses may:

• Cite sections
• Extract structured data from tables
• Present formulas
• Suggest next steps (such as schema generation)

---

What Happens During Ingestion

When ingestion starts, the system:

1. Discovers documents in the input folder
2. Creates a collection in Qdrant (if it does not exist)
3. Processes each document in parallel
4. Extracts:

   • Text
   • Section structure
   • Tables
   • Formulas
5. Performs post-processing
6. Splits documents into contextual chunks
7. Converts chunks into vector embeddings
8. Stores embeddings and metadata in Qdrant

---

Advanced Processing Steps

The ingestion pipeline includes:

Formula Recognition

• Formulas are detected
• Converted to LaTeX
• Repaired if split across layout blocks
• Processed using OCR when necessary

Table Normalization

• Multi-page tables are detected
• Split tables are merged into single logical units

Metadata Enrichment

Each stored chunk includes metadata such as:

• Chunk ID
• Heading
• Document structure path
• Page number
• Source filename
• has_formula flag
• has_table flag

These metadata flags allow targeted semantic filtering during search.

---

How Semantic Search Works

The toolkit uses hybrid retrieval:

• Dense embeddings (semantic meaning)
• Sparse retrieval (keyword matching)

Chunks are ranked using reciprocal rank fusion.

The result:

• High recall
• High precision
• Context-aware retrieval
• Reduced hallucination risk

The AI does not use internet knowledge. It operates only on your ingested documents.

---

What Success Looks Like

You have successfully completed this stage when:

• Documents are indexed
• Qdrant contains chunk records
• The MCP server exposes tools
• Your AI client can answer methodology-specific questions
• Responses reflect actual document content

At this point, you have transformed static PDFs into a searchable knowledge base.

---

What Comes Next

Now that you can search methodologies semantically, the next step is:

• Generating Guardian-compatible schemas
• Extracting formulas into structured definitions
• Beginning structured digitization workflows

Proceed to:

Schema & Formula Generation