Entity graphs / provisional comparison

Entity Graph Extraction Leaderboard

MINE Cache-Based Comparison v1 — Provisional

Document-to-graph extraction · Four selected candidates · Fact retention, not NER F1

Four measured systems. ATOM now covers all 85 articles alongside the adopted KGGen, GraphRAG and OpenIE scores. Graphiti and AutoSchemaKG remain unmeasured. ATOM uses a different judge configuration; see Protocol.

Graph extraction results

ProvisionalFact retention (%)

Measured systems ordered by fact retention; approaches awaiting measurement follow. Scores are actual measurements, not paper claims. This is a derivative comparison, not an official MINE leaderboard.

Complete 85-case cohort: 1,275 reference facts per measured system; evidence and judge settings differ by row.
System	Fact retention	Scored cases	Supported / total facts	Evidence	Extraction configuration
KGGen	65.73%	85	838 / 1,275	Released cached judgments	Historical model/settings unverified
ATOM / iText2KG	54.20%	85	691 / 1,275	Completed full85 run (9 November 2026)	Native ATOM / iText2KG 1.1.0; Codex gpt-6.1-sol extraction and judging, provider defaults; local BGE extraction embeddings and MiniLM retrieval.
GraphRAG	49.10%	85	626 / 1,275	Released cached judgments	Historical model/settings unverified
OpenIE	29.33%	85	374 / 1,275	Released cached judgments	Historical model/settings unverified
Graphiti	Not measured	0	Not measured	Awaiting evaluation	Existing integration: Graphiti 0.31.2; new graph-run configuration not executed.
AutoSchemaKG / Atlas-RAG	Not measured	0	Not measured	Awaiting evaluation	Recovered cached graphs; historical chat model and settings unverified.

Focus the table and use arrow keys to scroll for all columns.

Reproducibility details remain open with KGGen and TRACE-KG: historical judge settings, some artifact pairings and reuse rights are unresolved. These gaps do not erase the measured scores.

What the score measures

Fact retention: Percentage of original reference facts marked supported by the extracted graph in binary judgments. Not entity F1, relation F1, temporal truth or general graph correctness.
Measured coverage: Each scored row covers the same 85 cases and 1,275 facts, matched by ordered reference content. No confidence interval or significance result is available.
Remaining approaches: Graphiti and AutoSchemaKG need full-cohort judgments. The earlier 79-case full-context preparation is separate from the completed ATOM released-retrieval result. No missing result is treated as zero.

Contenders

Selected extraction systems, not interchangeable chat models. Linked projects do not endorse this comparison.

Graphiti

LLM-based temporal graph construction and entity resolution. This track evaluates retained document assertions, not temporal memory behavior.

Original implementation

Existing integration: Graphiti 0.31.2; new graph-run configuration not executed.

No applicable judgments yet; extraction or cache availability is not a score. 0 graph outputs available in the prepared 79-case cohort.

ATOM / iText2KG

Document-to-graph extraction with stored entity and relationship consolidation; an experimental integration is available.

Original implementation

Native ATOM / iText2KG 1.1.0; Codex gpt-6.1-sol extraction and judging, provider defaults; local BGE extraction embeddings and MiniLM retrieval.

Measured: 54.20%, 691 / 1,275 facts over all 85 cases. Released retrieval and original prompts; Codex judge defaults differ from the released GPT-4 configuration. All 85 native graphs completed; no successful-subset denominator.

KGGen

Text-to-triple extraction, graph aggregation and separate semantic entity/relation consolidation.

Original implementation

Released cached graphs; historical chat model and settings unverified.

Measured: 65.73% over 85 cases from released cached judgments. 79 graph outputs available in the prepared 79-case cohort.

AutoSchemaKG / Atlas-RAG

Corpus graph construction with entity/event extraction and concept-based schema induction. Concept grouping does not prove identity resolution.

Original implementation

Recovered cached graphs; historical chat model and settings unverified.

No applicable judgments yet; extraction or cache availability is not a score. 19 graph outputs available in the prepared 79-case cohort.

GraphRAG

Measured reference baseline: 49.10% over 85 cases. Released graph outputs and cached judgments; historical model/configuration unverified.

GraphRAG project

OpenIE

Measured reference baseline: 29.33% over 85 cases. Exact historical implementation/configuration unverified; evidence supplied through the KGGen comparison.

Released comparison resources

GraphRAG and OpenIE are measured reference baselines alongside the selected Graphiti, ATOM, KGGen and AutoSchemaKG approaches. knwler is a previously identified candidate outside the selected increment.

Measurement protocol & reproducibility

Every measured row covers the same adopted 85 articles and 1,275 original reference facts. Scores use the complete denominator. This measures reference-fact retention, not entity F1, temporal truth or isolated graph correctness.

Adopted baseline scores: mine-released-cache-unique-common-v1. KGGen, GraphRAG and OpenIE use released cached judgments. Historical chat/judge settings and runtime versions remain partly unverified. KGGen's 62_resultsN.json zero-score row is retained, not excluded to improve the result.
ATOM full85: mine-released-retrieval-codex-v1. Native iText2KG 1.1.0 unchanged. Released name/triple projection, local MiniLM top-eight node retrieval, depth-two incoming/outgoing context, original judge messages and strict binary votes. One fact per request; all 85 graphs and 1,275 valid judgments complete.
Judge difference: ATOM uses gpt-6.1-sol with provider defaults instead of the released GPT-4 / temperature-zero / one-token judge. Historical runtime and set iteration order remain unconfirmed. Shared coverage and scorer alignment do not establish an identical-judge controlled ranking.
Evidence and audit: ATOM completed 9 November 2026: 691 supported facts, 54.20%. Offline audit matched all 1,275 unique saved provider receipts, checked all 85 native graph hashes, and verified source/fact/graph bindings for 1,260 final-stage votes; 15 calibration votes retain their original admission. Retrieval contexts were not independently recomputed. Independent code/result review approved; raw data stays private.
Execution identity: ATOM source revision b383d779191a5a87cb98132da549bb810a2b8e35. Its reviewed sanitized-result digest and aggregate configuration are in evidence JSON. No upstream author endorsement is implied.
Historical preparation: The 2026-11-02 preparation at 6de8dda877edc44c0b2397def32cdd771ce0b4cb screened 85 cases into a separate 79-case full-context track (mine-cached-graphs-full-context-v1). Those six screening exclusions are not applied to the adopted full85 scores. Its private bundle digest f7615f4b9e3a5af8aed89976ecd724ccd828e77af40663e524572145068c53e6 is provenance metadata, not a public dataset.

Reproduce & inspect

Download aggregate result and preparation evidence or this build receipt and file digests. These are counts, configuration notes and hashes; they do not contain essays, reference facts, graphs, predictions or cached judge responses.

Inspect the track identity, preparation revision and source hashes before interpreting coverage.
Review the selected extraction configurations and common scorer. Cached extraction backends remain partly unknown.
Inspect the completed ATOM receipt identity, actual judge configuration and declared historical differences before comparing systems.
Reuse the adopted cached baseline measurements. Missing candidates need complete comparable coverage; never publish a successful-subset score as a full result.

Page generation is offline and does not reproduce extraction or judging. Source artifacts and preparation code remain private; upstream requests track remaining end-to-end reproduction gaps. Original upstream implementations are linked in Contenders. Dataset reuse licensing and historical protocol clarification remain unresolved.

Local page build

uv sync --locked
uv run --locked hf-workspace-build-graph-space
uv run --locked python -m http.server 8766 --bind 127.0.0.1 --directory .local/entity-graph-space

Run in the owning workspace on Python 3.12. Open localhost:8766. Choose a fresh ANVIL_HF_GRAPH_SPACE_OUTPUT_DIRECTORY for subsequent builds. Stop the server with Ctrl+C. No model calls or upload occurs.