Knowledge Graph Proof of Concept

Context

Engaged through Earley Information Science to build a proof of concept demonstrating the value of knowledge graph-based retrieval versus traditional vector search for a global management consultancy. The client wanted to improve how their consultants discovered and leveraged internal research, case studies, and methodologies.

Problem

The consultancy had a massive internal knowledge base spanning thousands of research documents, case studies, and methodology frameworks. Their existing Elastic search-based system returned too many irrelevant results, and consultants often couldn't find the specific knowledge they needed to prepare for client engagements. Simple keyword and vector search approaches couldn't capture the semantic relationships between concepts, industries, and methodologies.

Approach

• Collaborated with subject matter experts to design an ontology capturing the consultancy's domain knowledge: industries, capabilities, methodologies, concepts, and their interrelationships
• Built a knowledge graph that encoded these relationships, enabling traversal-based retrieval that vector search cannot replicate
• Developed MCP (Model Context Protocol) agents that could query the knowledge graph, reason over relationships, and synthesize answers from multiple connected nodes
• Implemented LLM-based classification to automatically tag new documents against the ontology as they entered the knowledge base
• Designed a rigorous evaluation framework comparing knowledge graph retrieval against baseline Elastic search and vector search, using MRR (Mean Reciprocal Rank) and Precision@K metrics

Key Technologies

• Knowledge graph (Neo4j)
• MCP agents for graph-augmented retrieval
• LLM classification pipeline for automated ontology tagging
• Elasticsearch (baseline comparison)
• Evaluation framework: MRR, Precision@K, expert relevance judgments

Results

• Knowledge graph retrieval outperformed vector search baseline on Precision@K and MRR metrics
• MCP agents could answer complex multi-hop questions ("Which methodologies have we used for digital transformation in financial services?") that pure search systems couldn't address
• Automated LLM classification achieved strong agreement with expert-tagged documents, enabling sustainable knowledge graph maintenance

Lessons Learned

Knowledge graphs shine when the domain has rich, structured relationships that vector similarity can't capture. The key insight was that ontology quality matters more than graph size — a well-designed ontology with hundreds of carefully defined concepts outperformed a larger but noisier graph. The MCP agent architecture proved particularly powerful for multi-hop reasoning, where the agent could traverse the graph across relationship types to synthesize answers that no single document contained.