Proposal: `llms-tldr.txt` - Token-Efficient Knowledge Summary

[Alex9001]

Proposal: llms-tldr.txt - Token-Efficient Knowledge Summary

Summary

Add an optional third file to the llms.txt family: /llms-tldr.txt.

It delivers substantive site knowledge in one HTTP request, within a declared token budget — without replacing llms.txt (navigation) or llms-full.txt (exhaustive listings).

Reference implementation: Cybermaps WordPress Plugin v3.1.9 (protocol llms-tldr.txt v1.1).

---

Problem

File	Strength	Failure mode on large sites
llms.txt	Fast orientation (~20 links + YAML)	Agent must crawl each URL for real knowledge
llms-full.txt	Deep per-post text (~100+ entries)	Token count explodes (500+ posts → millions of tokens)

Modern models commonly offer 128K–200K context windows. A multi-million-token llms-full.txt is truncated, skimmed, or processed with degraded citation quality.

Gap: a single, budget-bounded file that is more than links and less than everything.

---

Proposed Solution

Path: /llms-tldr.txt at site root (same discovery pattern as llms.txt).

MIME: text/plain; charset=utf-8

Body: Markdown structured for machine ingestion (headers + bullet metadata, not long prose).

Protocol header (v1.1, recommended):

# {Site Name} | Knowledge TL;DR
> Protocol: llms-tldr.txt v1.1
> Generated-by: {generator name}
> Generated-at: {ISO8601 UTC}
> Token-estimate: {integer}
> License: {optional, e.g. CC-BY-4.0}

---

Generation Pipeline (Normative Stages)

Generators MAY implement additional logic; these stages define interoperable expectations.

Stage 1 — Pinned content (optional)

Human-selected URLs or post IDs included first, bypassing automated filters. For pricing, policies, and docs the operator knows agents must see.

Stage 2 — Candidate pool

Fetch a bounded set of recent content (implementations often default to 100 posts from configurable types). Respect site-wide and per-item exclusion flags consistent with llms.txt.

Stage 3 — Quality filtering

Exclude items that are:

• Empty or shortcode-only (no readable text)
• Demo/placeholder (e.g. "Lorem ipsum", "Sample page", "Hello world")
• Below a minimum substantive length (reference: 50 characters)

Stage 4 — Scoring, deduplication, and layout

Scoring: rank remaining items with explainable signals (reference implementation uses AI metadata: info gain, depth, entities, plus optional substance/structure bonuses). Drop items below a configurable threshold (reference default: 0.6).

Deduplication: if two items share ≥2 taxonomy term slugs (categories or tags), treat as one topic cluster; keep the highest-scoring item. Optionally emit a cluster note: > N posts in this cluster. Top-scoring entry shown.

Layout (hybrid, recommended):

• Section headers by primary topic label (reference: primary category, else first tag, else General) — e.g. ## Topic: WordPress Development
• Intent (informational, transactional, …) as per-entry metadata, not necessarily as the only sectioning model — preserves compatibility with intent-aware llms.txt and search APIs

Stage 5 — Token budgeting

Stop adding automated entries when estimated tokens would exceed a configurable cap (reference default: 80,000). Pinned content is typically exempt from the cap. If truncated, append a machine-readable footer listing omitted topic labels.

Token estimate: implementations MAY use strlen(content) / 4 or another documented heuristic; expose the estimate in the protocol header when possible.

---

Entry Format (Reference)

## Topic: WordPress Development
> 12 posts in this cluster. Top-scoring entry shown.

### Building Custom Gutenberg Blocks
- **URL**: https://example.com/gutenberg-blocks/
- **Core Intelligence**: {dense snippet — entities, outcomes, or excerpt}
- **Topic**: WordPress Development
- **Intent**: informational
- **Score**: 0.84
- **Key Entities**: Gutenberg, block-editor, React

Field	Required	Notes
### title	Yes	Post or resource title
URL	Yes	Canonical URL
Core Intelligence	Yes	Primary knowledge payload
Topic	Recommended	Section/taxonomy label
Intent	Recommended	Agent routing (aligned with llms.txt silos)
Score	Optional	Generator confidence / density (0.0–1.0)
Key Entities	Optional	Comma-separated when extractable

---

Discovery & Caching

• Discovery: link from llms.txt Optional section (e.g. tldr: /llms-tldr.txt)
• Caching: recommend 1-hour HTTP or application cache with invalidation on content change
• Static mirror: optional on-disk copy for zero-PHP delivery (reference: Cybermaps Static File Engine)

---

Relationship to Other Files

File	Role
llms.txt	Structure, links, YAML sitemap, intent silos for navigation
llms-tldr.txt	Budgeted knowledge extract for one-shot ingestion
llms-full.txt	Exhaustive listings when token budget allows

None replaces the others.

---

Reference Results (Cybermaps Test Site)

~150 posts, 8 categories:

	llms-full.txt	llms-tldr.txt
Estimated tokens	~450,000	~28,000
Distinct topics represented	All posts (diluted)	~14 clusters
Quality filtering	No	Yes
Fits 128K window	No	Yes

Figures vary by site; the pattern — full vs. budgeted — is the point.

---

Alternatives Considered

Approach	Why not sufficient alone
Vector DB / RAG	Multiple round-trips; loses single-file simplicity of llms.txt
Chunking llms-full.txt	Agent state across chunks; per-chunk attention still limited
llms.txt only	Requires per-URL crawl for substance

---

Reference Implementation (Cybermaps v3.1.9)

Item	Location
HTTP handler	src/Discovery/LLMSTLDR.php
Pipeline	src/Discovery/LLMSTLDR/LLMSTLDRGenerator.php
Quality	LLMSTLDRQuality.php
Scoring	LLMSTLDRScorer.php
Clustering / dedup	LLMSTLDRCluster.php
Rendering	LLMSTLDRRenderer.php
Token budget	TokenBudget.php
Transient cache	cybermaps_tldr_cache (1h)
REST mirror	GET /wp-json/cybermaps/v1/llms-tldr
Multilingual static	{lang}/llms-tldr.txt when multilingual hub enabled

Further reading:

• Technical deep-dive
• Blog post / rationale

Scoring, filters, and dedup rules are plain heuristics — no external API required to reproduce the idea.

---

Specification Request

Consider adding /llms-tldr.txt as an optional third file in the llms.txt specification:

1. Optional protocol header block (v1.1 fields above)
2. Five-stage generation model (pinned → pool → quality → score/dedup/layout → token budget)
3. Hybrid layout guidance (taxonomy sections + per-entry intent)
4. Recommended default token cap (80,000, generator-configurable)
5. Link from llms.txt Optional section

Cybermaps provides a working, GPL-licensed reference. Feedback on field names, dedup rules (2+ shared terms), and header semantics is welcome.