No description
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Robear Selwans 5b35631592 v1.0
2026-05-12 14:01:25 +03:00
src v1.0 2026-05-12 14:01:25 +03:00
.gitignore v0.1 2026-05-11 14:57:47 +03:00
Cargo.lock v1.0 2026-05-12 14:01:25 +03:00
Cargo.toml v1.0 2026-05-12 14:01:25 +03:00
README.md v1.0 2026-05-12 14:01:25 +03:00

sdo2

Slopware notice: this is slopware. It is a personal, fast-moving documentation TUI built to be useful first and tidy second. Expect sharp edges, changing keybindings, and implementation decisions optimized for immediate use rather than polish.

sdo2 is a standalone terminal documentation search and reading tool. It indexes local Markdown files and imported web documentation, provides fuzzy search, renders Markdown previews in the terminal, and includes a companion sdo2-explorer TUI for managing imported documentation and crawling documentation sites.

The project currently produces two binaries:

sdo2           search/read documentation
sdo2-explorer  manage imported docs, crawl sites, batch import pages

Highlights

  • Fast fuzzy search over Markdown documentation using neo_frizbee.
  • XDG config/data paths:
    • ~/.config/sdo2/config.toml
    • ~/.local/share/sdo2/imported
    • ~/.local/share/sdo2/index
  • Local Markdown discovery from configured roots or command-line paths.
  • Imported web docs stored as Markdown under the imported-docs directory.
  • URL import through local defuddle CLI when available, with hosted defuddle.md API fallback.
  • Markdown preview with headings, lists, blockquotes, tables, links, inline styles, syntax-highlighted fenced code blocks, and enhanced Kitty/WezTerm rendering.
  • Preview-focused reading workflow with interactive links and link hints.
  • sdo2-explorer for imported-doc management, safe deletion, crawling, selective include/exclude, and batch import.

Installation / build

From the repo root:

cargo build --release --bins

Run directly during development:

cargo run --bin sdo2
cargo run --bin sdo2-explorer

Or install locally:

cargo install --path .

Basic usage

Launch the search TUI

sdo2

This indexes configured doc roots plus imported docs.

Use explicit roots for one session:

sdo2 ~/docs ./project/docs

Explicit paths replace configured roots for that run, but imported docs are still included.

Add a local documentation root

sdo2 add ~/docs

Show config/storage paths

sdo2 config

Import one URL

sdo2 import https://example.com/docs/page

Import prefers a local defuddle CLI when available:

defuddle parse --markdown <url>

If the CLI is not found, or if it fails, import falls back to the online defuddle.md API. Imported Markdown keeps sdo2 frontmatter. Defuddle/Defuddle API frontmatter is stripped before writing.

Plain text-like URLs such as .txt, .md, .rst, .adoc, .csv, and .tsv are fetched directly and written as Markdown without passing through Defuddle.

--selector is accepted for compatibility but ignored by the current import pipeline.

Markdown rendering

sdo2 auto-enables an enhanced Markdown profile when running inside Kitty or WezTerm. The enhanced profile keeps rendering terminal-native, but uses richer Unicode markers, subtle background panels for code/table headers/callouts, and nicer quote/list/table styling.

Force the generic renderer if needed:

SDO2_GENERIC_MARKDOWN=1 sdo2

sdo2 keybindings

Global

Ctrl+c / Ctrl+q   quit
Ctrl+t            toggle search mode: files ↔ text
Ctrl+p            maximize/restore preview pane
Tab               switch focus between Results and Preview
Esc               cancel preview modes / clear link highlight; does not quit

Results focus

printable chars   edit global search query; defaults to file/title/path search
Backspace         delete query char
Ctrl+w            delete previous query word
j/k or ↑/↓        move result selection
Home/End          first/last result
Enter             focus preview

Preview focus

j/k or ↑/↓        scroll preview line-by-line
Ctrl+d/Ctrl+u     scroll preview by larger steps
PageDown/PageUp   page scroll
g/G               jump top/bottom
h/l or ←/→        horizontal scroll
/                 preview-local search
n/N               next/previous preview-search match
] / [             next/previous screen-space link
f                 show inline visible-link hints
e                 open current document in $EDITOR/configured editor
Enter             open/jump selected link
{ / }             back/forward in preview jump history
Esc               cancel search/hints or clear selected link

In Preview focus, press f to show inline hints for visible links. Hints use two-letter home-row combinations:

aa as ad af ag ah aj ak al a;
sa ss sd ...

Type the two-letter hint to select that link, then press Enter to open or jump it.

External links open through xdg-open. Internal/local/imported documentation links are resolved inside the preview before falling back to the browser. Internal anchors jump inside the preview when a matching heading/anchor can be found. Preview link jumps are recorded in history, so { returns to the previous preview location and } moves forward again. Forward history is cleared whenever a new link jump is made.

sdo2-explorer

Launch:

sdo2-explorer

explorer manages imported docs only. It is where deleting and crawling live.

Imported-doc view

↑/↓ or j/k        move selection
h/l or ←/→        collapse/expand hierarchy
i                 import a new URL
c                 crawl from URL
o                 open selected source URL
d                 delete selected doc or hierarchy
r                 reload list
q/Esc             quit explorer

Imported docs are shown hierarchically. Directories show document counts. Deleting a directory deletes that imported hierarchy, but deletion is restricted to ~/.local/share/sdo2/imported and refuses to delete the imported root itself.

Crawl view

↑/↓ or j/k        move selection
h/l or ←/→        collapse/expand hierarchy
Space             include/exclude selected URL
o                 open selected crawled URL
a                 include all discovered URLs
x                 exclude all discovered URLs
Enter             import included URLs
Esc               return to imported docs; abort import if importing
c/C               abort running batch import

Crawl results are shown hierarchically by URL host/path. Crawling is same-origin, deduplicated, async/concurrent, has no page or depth limit, and filters obvious non-document/code resources. Batch imports run concurrently and can be aborted; queued imports are skipped while already-active imports are allowed to finish.

Storage safety

Deletion is intentionally conservative:

  • only imported docs under ~/.local/share/sdo2/imported may be deleted;
  • external configured doc roots are never deleted;
  • deleting the imported-docs root itself is refused;
  • files and directory hierarchies are deleted only through sdo2-explorer.

Version history

v0.1

Initial usable version of sdo2.

Major changes/features:

  • Created the Rust project and sdo2 binary.
  • Added CLI commands:
    • sdo2 [PATHS...]
    • sdo2 config
    • sdo2 add <PATH>
    • sdo2 import <URL>
  • Implemented XDG config/storage layout.
  • Implemented Markdown discovery, indexing, and heading-based chunking.
  • Added synthetic document-level chunks so empty search shows one document entry per doc.
  • Changed heading chunk previews to include the full subtree under a heading.
  • Added fuzzy search using neo_frizbee.
  • Added a ratatui/crossterm TUI with colored theme.
  • Added Markdown preview rendering with:
    • headings,
    • inline styles,
    • lists,
    • blockquotes,
    • horizontal rules,
    • links,
    • tables,
    • syntax-highlighted fenced code blocks.
  • Improved table rendering with wrapping cells and aligned borders.
  • Added loose-table normalization for preview-only rendering.
  • Implemented URL import through the online defuddle.md API.
  • Stripped Defuddle API frontmatter and wrote sdo2 frontmatter.
  • Removed the experimental built-in HTML/Markdown converter.
  • Removed image rendering and image dependencies.
  • Removed delete/index controls from the main sdo2 TUI/CLI.
  • Added sdo2-explorer as a second binary for imported-doc management.
  • Added imported-doc listing, import, URL opening, and safe deletion in sdo2-explorer.
  • Added editor handoff fixes for WezTerm/Neovim during early preview workflow.
  • Removed q as a quit binding in sdo2 so it can be typed as search text.

Tag:

v0.1  4af4a6b

v0.2

Focused on sdo2-explorer crawling and hierarchy management.

Changes since v0.1:

  • Added hierarchical imported-doc explorer with directory rows and document counts.
  • Added safe hierarchy deletion for imported docs, including recursive directory deletion under the imported storage root only.
  • Added crawler workflow to sdo2-explorer:
    • seed URL crawling,
    • same-origin discovery,
    • include/exclude selection,
    • opening crawled URLs,
    • batch import of included URLs.
  • Added background crawling with live progress/spinner.
  • Added batched imports with progress reporting.
  • Sped up crawling with:
    • higher concurrency,
    • shorter request/connect timeouts,
    • connection reuse,
    • compressed HTTP response support.
  • Added hierarchical crawl results grouped by URL host/path with case-insensitive hierarchy keys.
  • Preserved crawler internal ordering in the rendered crawl view.
  • Added collapse/expand bindings for imported-doc and crawl hierarchies via h/l and Left/Right.
  • Improved explorer list rendering:
    • dim source URLs beside titles,
    • visible include/exclude glyphs,
    • cleaner single-line document rows.
  • Added explorer vim-style movement bindings and status/help text updates.

Tag:

v0.2  63f98b6

v0.3

Focused on making the preview pane useful enough to replace editor handoff.

Changes since v0.2:

  • Added focus-based main TUI interaction with separate Results and Preview focus states.
  • Removed normal editor handoff from sdo2; Enter now focuses/uses the preview instead of opening files in the editor.
  • Changed quit bindings to Ctrl+c and Ctrl+q, leaving Esc for cancelling preview modes and link selection.
  • Added independent preview navigation:
    • line scrolling,
    • page scrolling,
    • top/bottom jumps,
    • horizontal scrolling.
  • Added preview-local search mode with match highlighting and next/previous match navigation.
  • Added line numbers to syntax-highlighted fenced code blocks.
  • Kept images ignored in Markdown preview.
  • Added interactive preview links with:
    • selection,
    • highlighting,
    • external xdg-open support,
    • internal anchor jumps.
  • Improved internal anchor matching for decoded, slugified, and compact heading forms.
  • Added selected-link highlighting for:
    • normal links,
    • inline-code links,
    • links inside table cells.
  • Added rendered-position link metadata from the Markdown renderer so screen-space link navigation uses preview line positions instead of source-line approximations.
  • Added viewport-aware link navigation with [ and ], preferring visible links and jumping to offscreen links only at boundaries.
  • Added inline two-letter home-row link hint mode for visible links via f, with Esc cancellation and Enter to open/jump the selected link.

Tag:

v0.3  cd37ca1

v0.4

Focused on traversible documentation, faster crawling/importing, and import robustness.

Changes since v0.3:

  • Added preview document traversal for local/indexed/imported documentation links before falling back to xdg-open.
  • Added support for preview jumps across:
    • same-page anchors,
    • relative page links,
    • relative links with anchors,
    • absolute same/source-site URLs,
    • imported docs with matching source_url frontmatter.
  • Added preview jump history; current bindings are { / } for backward/forward traversal.
  • Added independent traversed preview state so following documentation links does not require changing the search query.
  • Fixed preview bottom scrolling by avoiding double-wrapping between the Markdown renderer and ratatui::Paragraph.
  • Improved held-key preview scrolling responsiveness by draining queued terminal key events more aggressively.
  • Clamped preview downward scrolling to the rendered document bottom.
  • Reworked sdo2-explorer crawling to a pipelined async crawler using tokio, async reqwest, and FuturesUnordered.
  • Removed crawler page and depth limits; crawling remains bounded by same-origin deduplication and resource filtering.
  • Split crawl progress into found/queued/active/scanned counts.
  • Raised crawl concurrency and kept the fetch pipeline full as pages complete.
  • Filtered obvious code/resource URLs from crawl results and from fetch/scanning.
  • Added local Defuddle CLI support for importing, with online defuddle.md API fallback.
  • Added direct plain-text import for text documentation/data files without passing them through Defuddle.
  • Added concurrent batch imports from crawl results, configurable with SDO2_IMPORT_CONCURRENCY.
  • Added batch-import abort via Esc, c, or C; queued imports are skipped and active imports finish.

Tag:

v0.4

v0.5

Focused on fast default search, lower idle CPU, Ayu Dark UI polish, and better terminal Markdown rendering.

Changes since v0.4:

  • Added persistent docs/search caches and background search-index loading for faster startup.
  • Changed default search mode to fast file/title/path search.
  • Added Ctrl+t to toggle between file search and text search.
  • Moved search execution off the UI thread and limited background search workers to keep the TUI responsive.
  • Reduced idle CPU usage by redrawing only when state changes.
  • Re-enabled fuzzy fallback with constrained metadata fuzzy by default and opt-in full/always fuzzy modes.
  • Added fff-style prechunked fuzzy matching and bounded search cache memory usage.
  • Simplified result rows to one line: title plus source URL.
  • Changed heading-result previews to open the full document and scroll to the selected heading.
  • Applied an Ayu Dark-inspired palette to sdo2 and sdo2-explorer.
  • Added preview maximize/restore via Ctrl+p.
  • Added preview padding.
  • Improved generic-terminal Markdown rendering with stronger headings, nested bullets, callouts, task lists, tables, external-link markers, math/footnote fallbacks, and better code-block borders.
  • Expanded syntax highlighting aliases, including better C/C++/GLSL/HLSL/Metal/WGSL support.
  • Changed import cleanup so the first imported heading is always an H1, inserting one from the title when needed.

Tag:

v0.5  3a685ea

v1.0

First feature-complete release for the current documentation reading workflow.

Changes since v0.5:

  • Added Kitty/WezTerm-enhanced Markdown rendering with richer heading markers, quote bars, panel-style code/table/callout backgrounds, and a generic fallback via SDO2_GENERIC_MARKDOWN=1.
  • Added full-width code-block panel backgrounds for richer terminals.
  • Improved wide table rendering by turning very wide terminal-hostile tables into labeled card rows.
  • Added preview-focused e binding to open the current document in the configured editor / $EDITOR / vi, temporarily leaving and restoring the TUI.
  • Reworked preview jump history into backward/forward traversal:
    • { goes backward,
    • } goes forward,
    • forward history is cleared on new link jumps.
  • Added visible preview history as stacked pane edges above and below the current preview pane.
  • Added a small rendered-preview cache to make history traversal faster.
  • Suppressed Defuddle CLI fallback warnings inside sdo2-explorer imports so stderr logs no longer corrupt or scroll the TUI.

Tag:

v1.0  8ceb91c

Development validation

Common validation commands:

cargo fmt
cargo test --quiet
cargo check --quiet --bins

These were run before the v0.1, v0.2, v0.3, v0.4, v0.5, and v1.0 tags.