Fidelity of the corpus text extraction pipeline

See also

Sources

1. 1
   10 of the pages behind tranche 006 and tranche 007 claims carried Mermaid diagrams that the corpus's own extraction dropped, totalling roughly 11,000 characters of unread evidence.field report

   Model Context Protocol specification 2026-07-28, Server Features: Tools · re-extraction of every stored source body on 2026-09-05 · Checked 2026-09-05
2. 2
   The Caching page diagram draws the list_changed notification with the '-->>' arrow, whereas the Tools page diagram of the same specification revision draws it with the '--)' arrow.

   Model Context Protocol specification 2026-07-28, Caching · Interaction with Notifications, mermaid block, line 'Server-->>Client: notifications/tools/list_changed' · Checked 2026-09-30
3. 3
   The 2026-07-28 Caching page carries a mermaid sequenceDiagram under Interaction with Notifications that works through a timeline: a tools/list result with ttlMs 300000 is used from cache at two minutes, re-fetched after expiry, and invalidated immediately by a list_changed notification.

   Model Context Protocol specification 2026-07-28, Caching · Interaction with Notifications, mermaid block · Checked 2026-09-30
4. 4
   The Caching page's Freshness Calculation subsection contains a fenced code block with no language tag whose content is the single expression 'now < t_received + ttlMs'.

   Model Context Protocol specification 2026-07-28, Caching · Cacheable Model > Time-to-Live (TTL) Field > Freshness Calculation · Checked 2026-09-30
5. 5
   CommonMark 0.31.2 defines a code fence as a sequence of at least three consecutive backtick characters or tildes.

   CommonMark Spec Version 0.31.2 · section 4.5 Fenced code blocks · Checked 2026-09-30
6. 6
   CommonMark 0.31.2 specifies that if the end of the containing block or document is reached without a closing code fence, the code block contains all lines after the opening fence up to that end.

   CommonMark Spec Version 0.31.2 · section 4.5 Fenced code blocks · Checked 2026-09-30
7. 7
   In the Markdown served for MCP specification pages, opening code fences carry the suffix 'theme={null}' after the language tag, for example on json, jsonc and mermaid blocks.

   Model Context Protocol specification 2026-07-28, Server Features: Tools · any fenced block, e.g. Capabilities and Message Flow · Checked 2026-09-30
8. 8
   Because the Message Flow section of the Tools page is diagram-only in both 2025-11-25 and 2026-07-28, an extraction that removes mermaid blocks leaves that section empty in both revisions and a comparison of the two extractions shows no change there, although the diagrams differ.inferred

   Model Context Protocol specification 2026-07-28, Server Features: Tools · Message Flow (compared with the same section of mcp-spec-2025-11-25-server-tools) · Checked 2026-09-30
9. 9
   GitHub documents that a Mermaid diagram is created by placing Mermaid syntax inside a fenced code block with the mermaid language identifier.

   Creating diagrams · Creating Mermaid diagrams · Checked 2026-09-30
10. 10
    The Stateful Tools example on the 2026-07-28 Tools page is a jsonc block containing comments and three separate objects, so it is complete as published yet is not a single valid JSON document.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · Stateful Tools, jsonc block · Checked 2026-09-30
11. 11
    The MCP site's llms.txt lists documentation pages as links ending in .md, including the 2026-07-28 Tools, Subscriptions and Resources specification pages.

    Model Context Protocol (llms.txt documentation index) · link list; entries for specification/2026-07-28/server/tools.md, basic/patterns/subscriptions.md, server/resources.md · Checked 2026-09-30
12. 12
    The llms.txt proposal states that converting an HTML page back into clean text is difficult and imprecise.

    The /llms.txt file · Background · Checked 2026-09-30
13. 13
    The llms.txt proposal recommends that pages useful to agents offer a clean Markdown version at the same URL with .md appended or with the extension replaced by .md.

    The /llms.txt file · Proposal · Checked 2026-09-30
14. 14
    The README of the modelcontextprotocol/modelcontextprotocol repository states that the project is licensed under the MIT License.

    modelcontextprotocol/modelcontextprotocol README · README > License · Checked 2026-09-30
15. 15
    The MCP repository README says the schema is defined in TypeScript at schema/2026-07-28/schema.ts and is also made available as JSON Schema at schema/2026-07-28/schema.json.

    modelcontextprotocol/modelcontextprotocol README · README, schema paragraph · Checked 2026-09-30
16. 16
    In Mermaid sequence diagrams '->>' is a solid line with arrowhead, '-->>' is a dotted line with arrowhead, and '--)' is a dotted line with an open arrow at the end, labelled async.

    Sequence diagrams · Messages, arrow type table · Checked 2026-09-30
17. 17
    Mermaid describes itself as a JavaScript-based diagramming tool that renders Markdown-inspired text definitions into diagrams.

    About Mermaid · About Mermaid, opening description · Checked 2026-09-30
18. 18
    In Mermaid sequence diagrams an 'opt' block denotes an optional sequence, described as an if without an else.

    Sequence diagrams · Alt (alternative paths and optional) · Checked 2026-09-30
19. 19
    Mermaid's documentation says its parser looks for div or pre elements with class 'mermaid', reads the diagram definitions from them and renders them into SVG charts.

    About Mermaid · deployment / API usage passage (heading not returned) · Checked 2026-09-30
20. 20
    Mintlify documents that llms-full.txt combines an entire documentation site into a single file.

    llms.txt · section on llms-full.txt (heading not returned) · Checked 2026-09-30
21. 21
    Mintlify documents that page links in a site's llms.txt carry a .md extension so that AI tools can fetch the Markdown version of each page directly.

    llms.txt · section on llms.txt structure (heading not returned) · Checked 2026-09-30
22. 22
    Mintlify documents that a Mermaid diagram is authored by writing the diagram definition inside a Mermaid code block.

    Mermaid · opening instruction (heading not returned) · Checked 2026-09-30
23. 23
    Anthropic documents PDF requirements of a 32 MB maximum request size and a maximum of 600 pages per request, reduced to 100 pages when the request's context window is under 1M tokens.

    PDF support · Before you begin > Check PDF requirements, table · Checked 2026-09-30
24. 24
    The 'Documentation Index' preamble is text added by the documentation host for machine readers and is not part of the specification, so it should be excluded from excerpts and content hashes of the specification text.inferred

    Model Context Protocol specification 2026-07-28, Server Features: Tools · first three lines of the served document · Checked 2026-09-30
25. 25
    Where a documentation page delivers Mermaid diagrams by rendering them to SVG in the browser, a fetcher that does not execute JavaScript and works from the HTML rather than from Markdown source is likely to return no usable diagram text.inferred

    Web fetch tool · How web fetch works, Note after the numbered list · Checked 2026-09-30
26. 26
    The 2026-07-28 Resources page Message Flow section is a mermaid sequenceDiagram with no prose, and it shows subscriptions/listen (resourceSubscriptions) followed by notifications/subscriptions/acknowledged and then notifications/resources/updated.

    Model Context Protocol specification 2026-07-28, Resources · Message Flow · Checked 2026-09-30
27. 27
    The 2026-07-28 Resources page embeds a PNG image (resource-picker.png) under User Interaction Model whose only textual representation in the served Markdown is the alt text 'Example of resource context picker'.

    Model Context Protocol specification 2026-07-28, Resources · User Interaction Model, img element after the bullet list · Checked 2026-09-30
28. 28
    Each specification page served to WebFetch began with a three-line 'Documentation Index' blockquote directing the reader to https://modelcontextprotocol.io/llms.txt, placed before the page's own title.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · first three lines of the served document · Checked 2026-09-30
29. 29
    On 2026-09-30 modelcontextprotocol.io returned Markdown source rather than rendered HTML to WebFetch for specification pages, with mermaid blocks present as fenced diagram source.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · served document as a whole; mermaid fence under Message Flow · Checked 2026-09-30
30. 30
    The Subscriptions page states in prose that the server MUST send notifications/subscriptions/acknowledged as the first message for a subscription and MUST NOT send any notification on that subscription before it.

    Model Context Protocol specification 2026-07-28, Subscriptions · Acknowledgment, first paragraph · Checked 2026-09-30
31. 31
    The repository source file for the 2026-07-28 Subscriptions page was reported to contain no mermaid block and four JSON code blocks, the same inventory as the published page.

    docs/specification/2026-07-28/basic/patterns/subscriptions.mdx · whole file · Checked 2026-09-30
32. 32
    The MCP 2026-07-28 Subscriptions page, as served, contains four json code blocks and one table and contains no mermaid block and no image.

    Model Context Protocol specification 2026-07-28, Subscriptions · whole page · Checked 2026-09-30
33. 33
    In the MCP 2025-11-25 Tools page, the Message Flow section consists of a single mermaid sequenceDiagram code block and contains no prose.

    Model Context Protocol specification 2025-11-25, Server Features: Tools · Message Flow · Checked 2026-09-30
34. 34
    The 2025-11-25 Tools Message Flow diagram shows the update steps without an opt block and without any subscriptions/listen or acknowledgement message: the server sends tools/list_changed and the client then calls tools/list.

    Model Context Protocol specification 2025-11-25, Server Features: Tools · Message Flow, mermaid block, 'Updates' note · Checked 2026-09-30
35. 35
    As served on 2026-09-30 the 2026-07-28 Tools page contained 23 json blocks, 1 jsonc block and 1 mermaid block, every fence was closed, and the document ended with the list item 'Log tool usage for audit purposes'.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · whole page; final line under Security Considerations · Checked 2026-09-30
36. 36
    In the MCP 2026-07-28 Tools page, the Message Flow section consists of a single mermaid sequenceDiagram code block and contains no prose.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · Message Flow · Checked 2026-09-30
37. 37
    The 2026-07-28 Tools Message Flow diagram places the update steps inside an 'opt listChanged' block in which the client first sends subscriptions/listen (toolsListChanged: true) and the server sends notifications/subscriptions/acknowledged before notifications/tools/list_changed.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · Message Flow, mermaid block, 'opt listChanged' group · Checked 2026-09-30
38. 38
    On the 2026-07-28 Tools page the string notifications/subscriptions/acknowledged appears only inside the Message Flow mermaid block; the page's prose does not mention the acknowledgement.

    Model Context Protocol specification 2026-07-28, Server Features: Tools · whole page; sole occurrence under Message Flow · Checked 2026-09-30
39. 39
    Turndown's codeBlockStyle option accepts 'indented' or 'fenced' and defaults to 'indented'.

    mixmark-io/turndown README · Options table, codeBlockStyle · Checked 2026-09-30
40. 40
    Turndown's README says nodes not recognised by any rule are handled by a default rule that outputs the node's text content.

    mixmark-io/turndown README · Special Rules > Default rule · Checked 2026-09-30
41. 41
    Turndown's README describes remove rules, under which designated elements are removed entirely from the output.

    mixmark-io/turndown README · Special Rules > Remove rules · Checked 2026-09-30
42. 42
    An extracted Markdown document that ends inside an open code fence is a usable signal that the text was cut mid-block, because under CommonMark an unclosed fence silently extends the code block to the end of the document rather than raising an error.inferred

    CommonMark Spec Version 0.31.2 · section 4.5 Fenced code blocks · Checked 2026-09-30
43. 43
    Anthropic documents that the Claude API web fetch tool caches results, that returned content may not reflect the latest version at the URL, and that setting use_cache to false (tool version web_fetch_20260309 or later) fetches fresh content.

    Web fetch tool · Response > Fetch results, Note; and Tool definition > Cache bypass · Checked 2026-09-30
44. 44
    Anthropic documents that the Claude API web fetch tool supports only text, HTML and PDF content and returns the error code unsupported_content_type for anything else.

    Web fetch tool · Response > Errors, error code list · Checked 2026-09-30
45. 45
    Anthropic documents that the max_content_tokens limit applies to text content and not to binary content such as PDFs, and that the limit is approximate.

    Web fetch tool · Tool definition > Content limits, body and Note · Checked 2026-09-30
46. 46
    Anthropic documents that the Claude API web fetch tool does not currently support websites dynamically rendered with JavaScript.

    Web fetch tool · How web fetch works, Note after the numbered list · Checked 2026-09-30
47. 47
    The documented web_fetch_result carries url, content and retrieved_at and no documented field reports that truncation occurred, so a consumer cannot tell from the result structure alone whether the content was cut.inferred

    Web fetch tool · Response > Fetch results, field list · Checked 2026-09-30
48. 48
    Anthropic documents that for PDFs the Claude API web fetch tool returns the content as base64-encoded data and processes it like a directly attached PDF document.

    Web fetch tool · How web fetch works, numbered list item 3 · Checked 2026-09-30
49. 49
    Anthropic's example token figures for the web fetch tool are about 2,500 tokens for a 10 kB web page, about 25,000 tokens for a 100 kB documentation page and about 125,000 tokens for a 500 kB research paper PDF.

    Web fetch tool · Usage and pricing, 'Example token usage for typical content' · Checked 2026-09-30
50. 50
    Anthropic documents that when fetched content exceeds the max_content_tokens limit, the Claude API web fetch tool truncates it.

    Web fetch tool · Tool definition > Content limits · Checked 2026-09-30
51. 51
    The caching page's freshness formula survives only as its first token in this corpus's extracted text and must be read from the raw payload.field report

    Model Context Protocol specification 2026-07-28, Caching · Caching, freshness · Checked 2026-09-05

Documents

Cite this page

WarmTransfer. (2026, September 30). Fidelity of the corpus text extraction pipeline. WarmTransfer. https://warmtransfer.net/knowledge/corpus-extraction-fidelity

@misc{warmtransfer-corpus-extraction-fidelity,
  title  = {Fidelity of the corpus text extraction pipeline},
  author = {{WarmTransfer}},
  year   = {2026},
  url    = {https://warmtransfer.net/knowledge/corpus-extraction-fidelity},
  note   = {Verified 2026-09-30}
}