Max/docs-diataxis-ify (#6494)
* Diatixisify! * First pass at Typedoc generation for TS SDK * Remove overview pages * Fix typos and remove codebase references from docs Fix typos across network and developer docs: Quorum, available, cryptosystem, transaction, proportional, Standalone. Remove TODO placeholder from dVPN protocol page. Strip GitHub source links from network docs to decouple documentation from repo structure. * Expand thin landing pages across network and developer docs - Add intro content to network overview, infrastructure, and reference landing pages - Expand developer index with "where to start" guide - Add usage instructions and explanations to all five TS playground pages - Expand WebSocket client page with setup and message format examples * Restructure Rust SDK developer docs - Delete redundant mixnet example, message-helpers, and message-types subpages - Delete client-pool architecture and example subpages (content folded into landing) - Delete tcpproxy troubleshooting (folded into landing page) - Add deprecation notices to TcpProxy pages, pointing to Stream module - Add stream module docs: landing page, architecture, tutorial, and 4 example pages - Add mixnet and client-pool tutorials - Add SDK tour page - Update navigation and landing pages with docs.rs links * Restructure TS SDK developer docs - Merge overview, installation, and getting started into TS SDK landing page - Fold FAQ content into bundling/troubleshooting section - Delete redundant overview, installation, start, and FAQ pages - Update internal links in browsers.mdx and native.mdx - Update navigation and example page imports * Flatten and expand APIs section - Collapse nested API subpages into single pages with inline Redoc embeds - Rewrite introduction as landing page with decision table - Add endpoint categories, quick curl examples to each API page - Mark Explorer API as deprecated - Move NS API deployment guide to operators/performance-and-testing - Fix dangling /apis/nym-api/mainnet link in network-components - Remove sandbox endpoints from all API pages * Add redirects for moved and deleted pages - Add 25 redirects covering TS SDK, Rust SDK, APIs, and network sections - Fix dangling /developers/typescript/start link in operators changelog * Replace individual example doc pages with GitHub-linked tables, expand tutorials - replace individual example doc pages with GitHub-linked tables - expand mixnet tutorial with persistent identity and split_sender sections - add tcpproxy tutorial - rename "API Reference" to "TypeDoc Reference" in TS SDK sidebar - rename "Misc" to "Extras" in developer sidebar, move VPN CLI up - remove echo server from tools - update message-queue callout to reference actual modules - fix mixnet/examples redirect collision * Add SEO frontmatter, validate encryption standards, clean up URLs - add title/description/schemaType/section/lastUpdated frontmatter to 48 pages across developers, network, and APIs sections - remove network/.archive/ directory (compare against develop instead) - update nymtech.net → nym.com for website/blog links (keep infra URLs) - add native proxy "in progress" callout for Rust/C/Go * API-scraper update (#6598) * read nodes and locations * update python-prebuild.sh * Address PR #6494 review feedback - Use "mode" consistently instead of "role" on nym-nodes page - Replace "staking" with "bonding" for NYM token collateral - Wire up auto-scraped node counts via TimeNow + nodes-count.json - Fix broken licensing images: download CC icons locally, replace inline HTML - Fix 9 stale redirects pointing through deleted /network/architecture path * Fix linkcheck errors - Fix stale cross-links: /network/concepts/ → /network/mixnet-mode/ - Replace README.md references with globals.md in TypeDoc output - Add entryFileName: globals to typedoc.json configs to prevent recurrence * Fix remaining stale /network/architecture links - zk-nym-overview: architecture/nyx#nym-api → /network/infrastructure/nyx#nym-api - setup: network/architecture → /network/overview * Remove accidentally re-included architecture.md file from rebase * Standardize tutorials, document examples, add llms.txt, apply tone fixes - Expand Rust SDK tutorials with step-by-step structure; document all SDK examples across mixnet, client-pool, and tcpproxy pages - Add llms.txt generation script, wire into build and CI workflows - Apply tone/style fixes: deduplicate callouts, vary sentence structure, standardize voice consistency across changed pages * Consolidate redundant network overview docs * Trim dev docs: git-first imports, stream notice, collapse TcpProxy * Update tutorial * Refresh auto-generated API and command outputs * Update network section docs * Update developer and API docs: reusable components, stream protocol, conventions, tutorial fixes * Fix Rust SDK tutorial bugs: setup_env, port conflicts, logging, open_stream race condition * Update stream.mdx * Remove docs.rs link from Stream overview for the moment * add llms.txt and llms-full.txt note to readme --------- Co-authored-by: import this <97586125+serinko@users.noreply.github.com>
This commit is contained in:
mfahampshire
@@ -0,0 +1,176 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
/**
|
||||
* Generates public/llms-full.txt by walking pages/, reading _meta.json
|
||||
* for ordering, and concatenating all MDX/MD content as clean Markdown
|
||||
* with per-page frontmatter (Next.js llms-full.txt format).
|
||||
*
|
||||
* Run from repo root or documentation/docs/:
|
||||
* node documentation/scripts/next-scripts/generate-llms-txt.mjs
|
||||
*/
|
||||
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { fileURLToPath } from 'url';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const PAGES_DIR = path.resolve(__dirname, '../../docs/pages');
|
||||
const OUTPUT_FILE = path.resolve(__dirname, '../../docs/public/llms-full.txt');
|
||||
const SITE_URL = 'https://nym.com/docs';
|
||||
|
||||
// Directories to skip entirely (auto-generated API reference, archives, etc.)
|
||||
const SKIP_DIRS = new Set(['api', 'archive', 'playground']);
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Helpers
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
/** Read _meta.json for ordered keys; fall back to alphabetical. */
|
||||
function getPageOrder(dir) {
|
||||
const metaPath = path.join(dir, '_meta.json');
|
||||
if (fs.existsSync(metaPath)) {
|
||||
try {
|
||||
const meta = JSON.parse(fs.readFileSync(metaPath, 'utf-8'));
|
||||
return Object.keys(meta);
|
||||
} catch { /* fall through */ }
|
||||
}
|
||||
return fs.readdirSync(dir)
|
||||
.filter(f => !f.startsWith('_') && !f.startsWith('.'))
|
||||
.map(f => f.replace(/\.mdx?$/, ''))
|
||||
.filter((v, i, a) => a.indexOf(v) === i)
|
||||
.sort();
|
||||
}
|
||||
|
||||
/** Extract title from frontmatter or first H1. */
|
||||
function extractTitle(content, fallback) {
|
||||
const fm = content.match(/^---[\s\S]*?title:\s*["']?(.+?)["']?\s*$/m);
|
||||
if (fm) return fm[1];
|
||||
const h1 = content.match(/^#\s+(.+)$/m);
|
||||
if (h1) return h1[1];
|
||||
return fallback.replace(/[-_]/g, ' ');
|
||||
}
|
||||
|
||||
/** Extract description from frontmatter. */
|
||||
function extractDescription(content) {
|
||||
const fm = content.match(/^---[\s\S]*?description:\s*["']?(.+?)["']?\s*$/m);
|
||||
return fm ? fm[1] : '';
|
||||
}
|
||||
|
||||
/** Strip frontmatter, imports, and JSX from MDX, leaving clean Markdown. */
|
||||
function stripMdx(content) {
|
||||
let s = content;
|
||||
|
||||
// Frontmatter
|
||||
s = s.replace(/^---[\s\S]*?---\n*/m, '');
|
||||
|
||||
// Import statements
|
||||
s = s.replace(/^import\s+.*$/gm, '');
|
||||
|
||||
// Self-closing JSX tags: <Component ... />
|
||||
s = s.replace(/^\s*<\w[\w.-]*(?:\s[^>]*)?\s*\/>\s*$/gm, '');
|
||||
|
||||
// JSX block tags on their own line: <Callout type="info">, </Callout>, etc.
|
||||
// Keep the children — only remove the tag lines themselves.
|
||||
s = s.replace(/^\s*<\/?\w[\w.-]*(?:\s[^>]*)?\s*>\s*$/gm, '');
|
||||
|
||||
// Collapse 3+ blank lines → 2
|
||||
s = s.replace(/\n{3,}/g, '\n\n');
|
||||
|
||||
return s.trim();
|
||||
}
|
||||
|
||||
/** Convert file path to URL. */
|
||||
function fileToUrl(filePath) {
|
||||
let rel = path.relative(PAGES_DIR, filePath)
|
||||
.replace(/\.mdx?$/, '')
|
||||
.replace(/\/index$/, '');
|
||||
return `${SITE_URL}/${rel}`;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Recursive page collector
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
function collectPages(dir) {
|
||||
const pages = [];
|
||||
const order = getPageOrder(dir);
|
||||
|
||||
for (const key of order) {
|
||||
const subDir = path.join(dir, key);
|
||||
|
||||
// Skip excluded directories
|
||||
if (SKIP_DIRS.has(key) && fs.existsSync(subDir) && fs.statSync(subDir).isDirectory()) {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Find the page file (.mdx preferred over .md)
|
||||
let filePath = null;
|
||||
for (const ext of ['.mdx', '.md']) {
|
||||
const p = path.join(dir, `${key}${ext}`);
|
||||
if (fs.existsSync(p)) { filePath = p; break; }
|
||||
}
|
||||
|
||||
// Or an index file inside a subdirectory
|
||||
if (!filePath && fs.existsSync(subDir) && fs.statSync(subDir).isDirectory()) {
|
||||
for (const ext of ['.mdx', '.md']) {
|
||||
const p = path.join(subDir, `index${ext}`);
|
||||
if (fs.existsSync(p)) { filePath = p; break; }
|
||||
}
|
||||
}
|
||||
|
||||
if (filePath) {
|
||||
const raw = fs.readFileSync(filePath, 'utf-8');
|
||||
const title = extractTitle(raw, key);
|
||||
const description = extractDescription(raw);
|
||||
const body = stripMdx(raw);
|
||||
|
||||
if (body.length > 0) {
|
||||
pages.push({ title, description, url: fileToUrl(filePath), body });
|
||||
}
|
||||
}
|
||||
|
||||
// Recurse into subdirectory
|
||||
if (fs.existsSync(subDir) && fs.statSync(subDir).isDirectory()) {
|
||||
pages.push(...collectPages(subDir));
|
||||
}
|
||||
}
|
||||
|
||||
return pages;
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Main
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
console.log(`Scanning ${PAGES_DIR} ...`);
|
||||
const pages = collectPages(PAGES_DIR);
|
||||
|
||||
const lines = [];
|
||||
|
||||
// Global header
|
||||
lines.push(`# Nym Documentation\n`);
|
||||
lines.push(`@version: 1.20.4`);
|
||||
lines.push(`@generated: ${new Date().toISOString().split('T')[0]}`);
|
||||
lines.push(`@pages: ${pages.length}`);
|
||||
lines.push(`@source: https://github.com/nymtech/nym/tree/develop/documentation/docs`);
|
||||
lines.push('');
|
||||
|
||||
// Per-page blocks
|
||||
for (const page of pages) {
|
||||
lines.push('---');
|
||||
lines.push(`title: ${page.title}`);
|
||||
if (page.description) {
|
||||
lines.push(`description: ${page.description}`);
|
||||
}
|
||||
lines.push(`url: ${page.url}`);
|
||||
lines.push('---');
|
||||
lines.push('');
|
||||
lines.push(page.body);
|
||||
lines.push('');
|
||||
}
|
||||
|
||||
const output = lines.join('\n');
|
||||
fs.writeFileSync(OUTPUT_FILE, output);
|
||||
|
||||
const sizeKb = (Buffer.byteLength(output, 'utf-8') / 1024).toFixed(0);
|
||||
console.log(`Wrote ${pages.length} pages to ${OUTPUT_FILE} (${sizeKb} KB)`);
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
#!/bin/bash
|
||||
|
||||
# Generates TypeDoc markdown API reference for TypeScript SDK packages.
|
||||
# Output goes into docs/pages/developers/typescript-sdk/api/
|
||||
#
|
||||
# Prerequisites: typedoc and typedoc-plugin-markdown must be installed globally
|
||||
# pnpm add -g typedoc@0.25.13 typedoc-plugin-markdown@4.0.3
|
||||
#
|
||||
# Usage: run from the documentation/ directory, or it will cd there automatically.
|
||||
|
||||
set -o errexit
|
||||
set -o nounset
|
||||
set -o pipefail
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "$0")/../.." && pwd)"
|
||||
SDK_PACKAGES="$REPO_ROOT/sdk/typescript/packages"
|
||||
|
||||
# packages to generate docs for (name = directory name under packages/)
|
||||
PACKAGES=("sdk" "mix-fetch")
|
||||
|
||||
for pkg in "${PACKAGES[@]}"; do
|
||||
echo "Generating TypeDoc for @nymproject/${pkg}..."
|
||||
cd "$SDK_PACKAGES/$pkg"
|
||||
typedoc --skipErrorChecking
|
||||
done
|
||||
|
||||
echo "TypeDoc generation complete."
|
||||
echo "Output: documentation/docs/pages/developers/typescript-sdk/api/"
|
||||
@@ -25,6 +25,8 @@ python3 api_targets.py validator --api mainnet --endpoint epoch/reward_params --
|
||||
|
||||
python3 api_targets.py validator --api mainnet --endpoint epoch/reward_params --value interval epoch_reward_budget --format markdown --separator _ > ../../docs/components/outputs/api-scraping-outputs/nyx-outputs/epoch-reward-budget.md &&
|
||||
|
||||
python3 api_targets.py described_nodes > ../../docs/components/outputs/api-scraping-outputs/nodes-count.json
|
||||
|
||||
python3 api_targets.py time_now > ../../docs/components/outputs/api-scraping-outputs/time-now.md &&
|
||||
|
||||
python3 api_targets.py calculate --staking_target --separator _ > ../../docs/components/outputs/api-scraping-outputs/nyx-outputs/staking-target.md &&
|
||||
|
||||
Reference in New Issue
Block a user