This is the full developer documentation for Otso # Otso > Archive your digital life. Search everything locally. Own your data. What's Working Now Otso can index 23+ data sources and search them locally via a web UI. Clone the repo, build the indexer, and start searching in minutes. ## Available Now [Section titled “Available Now”](#available-now) Run Local Search Index your data and search it locally. * [Set up the indexer and search UI](/tutorials/run-local-search/) * [Configure data sources](/reference/sources-toml/) * Browse Twitter archives, bookmarks, notes, and more 23+ Data Sources Pre-configured extractors for common platforms. * **Reading**: Pinboard, Readwise * **Music**: Spotify, Last.fm, Apple Podcasts * **Social**: Twitter/X archives, LinkedIn * **Notes**: Apple Notes, Notion * **Code**: GitHub activity * [Full source list](/reference/sources-toml/#configured-sources) ## Learn More [Section titled “Learn More”](#learn-more) Overview What Otso is and why it matters. – [What is Otso?](/overview/what-is-otso/) – [Key capabilities](/overview/key-capabilities/) – [Architecture](/overview/architecture/) Reference Configuration and internals. – [sources.toml format](/reference/sources-toml/) – [Event model](/reference/event-model/) – [Schema mappings](/reference/data-schema-mappings/) Explanations Concepts and trade‑offs. – [POSSE vs PESOS](/explanations/posse-vs-pesos/) – [Local‑first choices](/explanations/local-first-choices/) – [Event‑driven design](/explanations/event-driven-design/) Ecosystem Alternatives and inspirations. – [Related projects](/explanations/ecosystem-alternatives-inspirations/) ## Roadmap [Section titled “Roadmap”](#roadmap) These features are planned but not yet implemented: Publishing Planned: CLI and web UI for publishing. * `otso publish` command * Micropub support * POSSE to Mastodon/Bluesky Live Import Planned: Pull from live APIs. * PESOS from Mastodon * Automatic sync jobs * Webmention backfeed # Ecosystem — Alternatives & Inspirations > How Otso compares to adjacent projects, with gratitude and links for deeper dives. Otso lives happily in an ecosystem of IndieWeb and local‑first tools. This page summarizes how Otso differs, when you might choose something else, and how they can play together. We link out with admiration—these projects are great. ## Prominent adjacent projects [Section titled “Prominent adjacent projects”](#prominent-adjacent-projects) * [DogSheep](https://github.com/dogsheep) — a suite of small, sharp CLIs that export your data into per‑source SQLite databases, usually explored with [Datasette](https://datasette.io/). DogSheep’s philosophy—simple CLIs, SQLite everywhere—influences Otso’s import layer and “rebuildable projections.” * [Chronicle](https://github.com/chronicle-app/chronicle-etl) — a CLI‑first personal data framework built around declarative pipelines, append‑only events, and clear staging→canonical flows. Chronicle inspired Otso’s idempotent imports and event/log mindset. * [HPI (Human Programming Interface)](https://github.com/karlicoss/HPI) — a Python library that treats your whole “personal data lake” as one interface (messaging, quantified‑self, PKM, logs, etc.). HPI broadened our view of sources beyond social/bookmarks. * [Data Transfer Initiative](https://datatransferinitiative.org/) — a nonprofit championing data portability; a kindred spirit for Otso’s “own your data” ethos. * [parallel-flickr](https://github.com/straup/parallel-flickr) — Aaron Straup Cope’s tool for backing up your Flickr photos; an early “own your data” inspiration. * [Omnivore](https://github.com/omnivore-app/omnivore) — an open-source, self-hosted, cross-platform read later platform. ## How Otso differs (at a glance) [Section titled “How Otso differs (at a glance)”](#how-otso-differs-at-a-glance) * Otso is an event‑driven, local‑first toolkit: import (PESOS), publish (Micropub/POSSE), keep a durable archive (SQLite/Postgres), and build fast projections you can rebuild when needed. * Many adjacent tools are excellent at one slice (e.g., Micropub server, hosted site, file‑based notes). Otso complements them when you want import/sync + ownership in one place. ## Micropub servers and hosted platforms [Section titled “Micropub servers and hosted platforms”](#micropub-servers-and-hosted-platforms) * [Indiekit](https://indiekit.indieweb.org/) — a lovely Micropub server you can self‑host. Use it when you want a slim server that speaks Micropub, handles media, and connects to services. Otso adds an event store, import/sync, and projections; you can still publish to an Indiekit‑powered site. * [Micro.blog](https://micro.blog/) — generous hosted IndieWeb blogging with Micropub, feeds, and export. If you want a turnkey home, Micro.blog is a joy. Otso can publish to it and import your archive so you retain your history. * [WordPress](https://wordpress.org/) and [Ghost](https://ghost.org/) — both are fantastic publishing platforms. Keep them as your public site while Otso maintains a local archive, runs imports, and syndicates out. Best of both worlds. ## Local‑first notes and static content [Section titled “Local‑first notes and static content”](#localfirst-notes-and-static-content) * [Obsidian](https://obsidian.md/)/Markdown (Git repos, static sites) — plain files are comforting and powerful. Keep them! Otso can export Markdown projections or read Markdown sources. When you need consistent kinds, imports from APIs, search, or snapshots, Otso’s small database helps without taking that simplicity away. ## Sources and targets (friends, not competitors) [Section titled “Sources and targets (friends, not competitors)”](#sources-and-targets-friends-not-competitors) Mastodon, Bluesky, Twitter/X, GitHub, Readwise, Pinboard/Raindrop/Pocket—these are where your stuff already lives. Otso treats them as import and/or publish endpoints: * Mastodon — POSSE and import via APIs/archives. * Bluesky — app‑password auth + cursor pagination. * Twitter/X — tight API limits; archives remain valuable. * GitHub — activity feeds map cleanly to kinds. * Readwise — highlight sync from API/CSV. The focus is idempotent upserts, clear mappings (notes, replies, bookmarks, photos), and safe re‑runs. ## Search and scheduling [Section titled “Search and scheduling”](#search-and-scheduling) * [Pagefind](https://pagefind.app/) — fast, local static search (Starlight’s default). Great for docs. * [Algolia DocSearch](https://docsearch.algolia.com/) — hosted search for docs; easy if you’re in the DocSearch program. * [Vercel Cron](https://vercel.com/docs/cron-jobs) and [Upstash QStash](https://upstash.com/qstash) — nice ways to automate sync/retries without heavy infra. ## Local‑first sync inspirations [Section titled “Local‑first sync inspirations”](#localfirst-sync-inspirations) * [Litestream](https://litestream.io/) / [LiteFS](https://fly.io/docs/litefs/) — pragmatic SQLite replication. Lovely for “SQLite as source of truth.” * [ElectricSQL](https://electric-sql.com/) — CRDT‑powered sync (SQLite ⇄ Postgres) for true multi‑device, offline editing. Reach for it when you need collaborative editing beyond simple imports. * [Datasette](https://datasette.io/) — a delight for exploring/querying SQLite. Inspires Otso’s “queryable archive” feel. ## Standards we share (the glue) [Section titled “Standards we share (the glue)”](#standards-we-share-the-glue) * [Micropub](https://www.w3.org/TR/micropub/) — write API for your site. * [IndieAuth](https://indieauth.net/) — authentication for IndieWeb services. * [Webmention](https://www.w3.org/TR/webmention/) — cross‑site replies/likes. * [Microformats2](https://microformats.org/) — semantic HTML for posts. Otso speaks these, so you can mix tools without losing ownership. ## When to reach for what [Section titled “When to reach for what”](#when-to-reach-for-what) * “I just need Micropub.” Use [Indiekit](https://indiekit.indieweb.org/) or a hosted Micropub‑friendly home like [Micro.blog](https://micro.blog/). * “I want an owned archive + imports.” Use Otso for ingest, snapshots, projections, and optional POSSE. * “I want both.” Keep your public site (WordPress, Ghost, Micro.blog) and let Otso import history, publish new posts, and keep everything searchable. If we missed a favorite project, open an issue with a link and a quick note—happy to add it. # Emulator Plugins > Browse local backups through platform-like interfaces. Emulator plugins let Otso display exported data with a UI that mirrors the service it came from. Each plugin interprets a platform’s layout and metadata so that browsing a local archive feels like using the original site. ## How it works [Section titled “How it works”](#how-it-works) 1. **Local backup** – content and metadata are stored offline. 2. **Source-specific plugin** – knows the platform’s structure and reproduces its look and behavior. 3. **Uniform shell** – multiple plugins run in the same host so you can switch between emulated interfaces. 4. **Interactive browsing** – search, filter, and follow links entirely on local data. ## Advantages [Section titled “Advantages”](#advantages) * Familiar interface for archived content. * Backups remain accessible even if the original service disappears. * All browsing happens locally for privacy and control. * Developers can add plugins for new platforms. ## Inspirations [Section titled “Inspirations”](#inspirations) * **parallel-flickr** – a reconstruction of Flickr’s photo pages that demonstrates how a backup can feel like the live site. * **N64 emulators** – recreate console hardware in software so classic games remain playable on modern machines. * **Rhizome Web Recorder** – captures entire browsing sessions for later playback, preserving not just files but the experience. ## Why Emulate? [Section titled “Why Emulate?”](#why-emulate) Plain archives keep data, but they lose context. Emulating an interface revives the look, behavior, and cultural feel of a platform so that a backup becomes something you can inhabit, not just inspect. # Event‑Driven Design > Benefits, drawbacks, and a practical mental model. Otso treats every change as an immutable event. This makes history auditable and projections easy to rebuild. ## Why events work for Otso [Section titled “Why events work for Otso”](#why-events-work-for-otso) * Simplifies imports from unreliable APIs. * Enables replay and migration to new projections. * Encourages small, composable plugins. ## Drawbacks and mitigations [Section titled “Drawbacks and mitigations”](#drawbacks-and-mitigations) * Event logs grow forever → use snapshots or pruning for old events. * Higher initial complexity → start with high level helpers like `createEvent`. ## Practical rules of thumb [Section titled “Practical rules of thumb”](#practical-rules-of-thumb) 1. Capture the smallest meaningful fact as an event. 2. Keep events immutable; derive views elsewhere. 3. Name projections after the query they serve (e.g., `latest_notes`). # IndieWeb Alignment > How Otso supports IndieWeb principles. Otso builds on IndieWeb protocols and encourages publishing on your own domain first. ## Principles embraced [Section titled “Principles embraced”](#principles-embraced) * **Own your data** – everything lives in a local database you control. * **Publish first** – your site is the source of truth; POSSE pushes copies out. * **PESOS optional** – you can import from silos without surrendering ownership. ## Protocol support [Section titled “Protocol support”](#protocol-support) * **Micropub** for publishing. * **Webmention** for replies, likes, and reposts. * **Microformats2** for semantic HTML. ## Extending the IndieWeb [Section titled “Extending the IndieWeb”](#extending-the-indieweb) By keeping sources, enrichments, and publishing logic pluggable, Otso allows experimentation while staying interoperable with existing IndieWeb tools. # Local‑First Choices > Why Otso prefers local databases and offline‑first workflows. Running locally keeps your data fast and private. ## Benefits [Section titled “Benefits”](#benefits) * Works offline and syncs when convenient. * Easy to inspect and back up the entire database. * No vendor lock‑in. ## Trade‑offs [Section titled “Trade‑offs”](#tradeoffs) * You manage backups and updates yourself. * Collaborative editing requires extra tooling. ## Practices [Section titled “Practices”](#practices) * Use Git to version configuration and content. * Sync databases with Litestream, LiteFS, or regular snapshots. * Export and re‑import when moving between machines. # Platform Constraints — Twitter/X > Dealing with API limits and archive quirks. Twitter’s shifting APIs and export formats require extra care. ## API limits [Section titled “API limits”](#api-limits) * The v2 API enforces tight rate limits; use cursor‑based pagination and retry with backoff. * OAuth tokens expire—store refresh tokens securely. ## Archive quirks [Section titled “Archive quirks”](#archive-quirks) * Exports arrive as a ZIP containing a `data/` directory with tweet JSON. * Media files may need to be fetched separately; URLs can expire. ## Recommendations [Section titled “Recommendations”](#recommendations) * Prefer the official archive when available; it’s more complete than scraping. * Cache responses locally to avoid hitting limits. * Map retweets and quoted tweets to `repost` and `mention` kinds respectively. # POSSE vs PESOS > When to syndicate out vs. import back. Two common IndieWeb workflows: * **POSSE** – Publish on your Own Site, Syndicate Elsewhere. * **PESOS** – Publish Elsewhere, Syndicate to Own Site. ## When to use POSSE [Section titled “When to use POSSE”](#when-to-use-posse) Use when you control the canonical post and want copies on silos. Otso publishes to your site first and then cross‑posts. ## When to use PESOS [Section titled “When to use PESOS”](#when-to-use-pesos) Useful when a silo offers tools you need. Import the content and keep a canonical copy in Otso. ## Hybrid workflows [Section titled “Hybrid workflows”](#hybrid-workflows) You can mix both: POSSE your own posts while PESOS likes or replies made on other networks. Otso tracks origin for each event so projections stay consistent. # Configure Visibility > Use public, unlisted, private, and secret visibility. Events carry a `visibility` property to control where content appears. ## Visibility options [Section titled “Visibility options”](#visibility-options) * `public` – visible in feeds and search. * `unlisted` – not listed but accessible via permalink. * `private` – kept in local tools only. * `secret` – encrypted or restricted projections. ## Set defaults [Section titled “Set defaults”](#set-defaults) Define a default in `otsorc.yml`: ```yaml defaults: visibility: unlisted ``` ## Override per item [Section titled “Override per item”](#override-per-item) Pass a flag to the CLI when creating or importing: ```bash otso publish note.md --visibility private ``` ## Effects [Section titled “Effects”](#effects) * Only `public` items are syndicated or exposed via APIs. * Search indexes exclude `private` and `secret` content. * Changing visibility emits a new event, allowing audit of who saw what. # Import from Bluesky > Import from Bluesky with cursor-based pagination. Otso’s Bluesky adapter pulls posts via the public API. ## Obtain credentials [Section titled “Obtain credentials”](#obtain-credentials) 1. Create an app password in Bluesky settings. 2. Set `BSKY_IDENTIFIER` and `BSKY_PASSWORD` in your environment. ## Run the import [Section titled “Run the import”](#run-the-import) ```bash otso import bluesky --cursor latest.json ``` The adapter stores the last cursor so subsequent runs resume where they left off. ## Mapping [Section titled “Mapping”](#mapping) * Posts become `note` events with AT URIs stored as `external_id`. * Images are downloaded and attached as media. * Embeds to external URLs become `bookmark` events. # Import from GitHub > Bring GitHub activity into Otso. Track your commits, issues, and stars. ## Token and permissions [Section titled “Token and permissions”](#token-and-permissions) Create a personal access token with `repo` and `read:user` scopes and store it in `GITHUB_TOKEN`. ## Fetch events [Section titled “Fetch events”](#fetch-events) ```bash otso import github ``` The importer walks the Events API and upserts each action. ## Mapping [Section titled “Mapping”](#mapping) * Push and merge events → `entry` with commit URLs. * Issues and PRs → `entry` with labels mapped to tags. * Stars → `bookmark` events. ## Rate limits [Section titled “Rate limits”](#rate-limits) GitHub allows 5k requests/hour. The importer sleeps and resumes when limits are reached. # Import from Mastodon > Import your Mastodon timeline or account archive. ## Using the API [Section titled “Using the API”](#using-the-api) 1. Create an access token in Mastodon preferences. 2. Set `MASTODON_TOKEN` and `MASTODON_INSTANCE` in your environment. 3. Run `otso import mastodon --since 2024-01-01`. ## Using the archive [Section titled “Using the archive”](#using-the-archive) Download your account archive, unzip, and run: ```bash otso import mastodon --archive ~/Downloads/mastodon-export ``` ## Run as a PESOS job [Section titled “Run as a PESOS job”](#run-as-a-pesos-job) `otso import mastodon` is idempotent, so you can run it regularly to keep your site and local archive up to date. Execute it on demand or deploy it to a cron or serverless worker to poll Mastodon on a schedule and sync the results back to your local environment. Use `otso run` to deploy the import on a timer, for example every 15 minutes on Vercel: ```bash otso run --every 15m --on vercel import mastodon --include likes,replies ``` ## Mapping [Section titled “Mapping”](#mapping) * Toots → `note` * Replies → `reply` with `inReplyTo` URL. * Boosts → `repost` * Favorites → `like` Media attachments are downloaded and referenced as `media` entries. # Import Twitter/X Archive > Import your Twitter/X export and map to Otso kinds. ## Prepare the archive [Section titled “Prepare the archive”](#prepare-the-archive) 1. Request your archive from Twitter and unzip it. 2. Locate `data/tweets.js` and remove the JavaScript assignment to get pure JSON. ## Run the import [Section titled “Run the import”](#run-the-import) ```bash otso import twitter --archive ~/Downloads/twitter ``` ## Mapping [Section titled “Mapping”](#mapping) * Tweets → `note` * Retweets → `repost` * Favorites → `like` * Media URLs are resolved and downloaded. ## Rate limits [Section titled “Rate limits”](#rate-limits) If using the API, supply `TWITTER_BEARER_TOKEN`. Otso respects rate limit headers and retries with backoff. # Interactive CLI > Key features of the Otso CLI's text user interface. Otso includes an interactive CLI built with a text user interface (TUI) for common tasks: * Guided imports from other platforms. * Duplicate scanning and merge review. * Publishing drafts to sites and social networks. * Full-text search and tagging. * Configuration and plugin management. * View building, event tails, data cleanup, and encrypted backups. ## Import [Section titled “Import”](#import) ```bash $ otso import ``` ```text ┌──────────────────────────────────────────────────────────────────────┐ │ Otso • Import │ │----------------------------------------------------------------------│ │ Choose a platform (↑↓/Enter, type to search): │ │ ▸ Twitter / X │ │ Instagram │ │ Google Photos │ │ Slack │ │ YouTube │ │ Reddit │ │ GitHub │ │ ChatGPT │ │ Back │ └──────────────────────────────────────────────────────────────────────┘ ``` ## Dedupe & Merge [Section titled “Dedupe & Merge”](#dedupe--merge) ```bash $ otso dedupe resolve ``` ```text ┌──────────────────────────────────────────────────────────────────────┐ │ Resolve Duplicates & Merges (↑↓ navigate • Enter open • / search) │ │----------------------------------------------------------------------│ │ 1) [DUP] 0.98 p_9f3 "Sunny day at the lake" twitter ←→ mastodon │ │ 2) [MER] 0.92 p_a1c Photo + caption + alt text (two variants) │ │ 3) [CNF] — p_77 Visibility mismatch: public vs private │ │----------------------------------------------------------------------│ │ [A]ccept suggestion [M]anual merge [K]eep both [S]kip [Q]uit │ └──────────────────────────────────────────────────────────────────────┘ ``` ## Publish [Section titled “Publish”](#publish) ```bash $ otso publish ``` ```text ┌──────────────────────────────────────────────────────────────────────┐ │ Publish • Choose a target │ │----------------------------------------------------------------------│ │ ▸ My Site (Micropub, blog view) │ │ Mastodon (@casey@mastodon.social) │ │ Bluesky (@casey.bsky.social) │ │ WordPress (casey.blog) │ │----------------------------------------------------------------------│ │ [ Back ] │ └──────────────────────────────────────────────────────────────────────┘ ``` ## Search [Section titled “Search”](#search) ```bash $ otso search "sunny day lake" ``` ```text ┌──────────────────────────────────────────────────────────────────────┐ │ Search: "sunny day lake" [↑↓ to scroll • Enter to open • / refine] │ │----------------------------------------------------------------------│ │ 1) p_9f3 note 2025-09-08 "Sunny day at the lake" tags: life,photo│ │ 2) p_b42 photo 2025-08-21 lake.jpg alt:"mountain lake" │ │ 3) p_a1c article 2025-07-30 "Lakes I love" │ │----------------------------------------------------------------------│ │ [ Open ] [ Tag ] [ Publish ] [ Copy link ] [ Back ] │ └──────────────────────────────────────────────────────────────────────┘ ``` ## Config [Section titled “Config”](#config) ```bash $ otso config get ``` ```text ┌──────────────────────────────────────────────────────────────────────┐ │ Config │ │----------------------------------------------------------------------│ │ storage.path ~/.otso │ │ visibility.default private │ │ publish.defaultTargets [site] │ │ connect.pollInterval 15m │ │ web.port 5173 │ │----------------------------------------------------------------------│ │ [ Edit ] [ Reset ] [ Export ] [ Back ] │ └──────────────────────────────────────────────────────────────────────┘ ``` # Manage Media, Alt‑text, and Tags > Handle assets, generate alt‑text, and organize with tags. ## Media storage [Section titled “Media storage”](#media-storage) Imported media is saved under `media/` with hashed filenames. Use `otso media ls` to list attachments. ## Alt‑text [Section titled “Alt‑text”](#alttext) * Auto‑generate descriptions via `otso media alt --auto`. * Edit manually: ```bash otso media alt IMG_1234.png "Sunset over the lake" ``` ## Tagging [Section titled “Tagging”](#tagging) Attach tags when importing or editing: ```bash otso tag add 1234 travel photo ``` Tags feed into projections and search filters. # Publish to Bluesky > POSSE to Bluesky with app passwords. ## Setup [Section titled “Setup”](#setup) 1. Create an app password in Bluesky. 2. Set `BSKY_IDENTIFIER` and `BSKY_PASSWORD`. ## Publish a post [Section titled “Publish a post”](#publish-a-post) ```bash otso publish bluesky 1234 ``` The command takes an existing event ID and posts it to Bluesky, including a link to the canonical URL. ## Limits [Section titled “Limits”](#limits) * Posts max 300 characters; longer posts are truncated. * Attach up to four images. * Bluesky does not yet support alt text via API, so Otso embeds descriptions in the post body. # Publish to Mastodon > POSSE to Mastodon while keeping your site as the source. ## Connect account [Section titled “Connect account”](#connect-account) 1. Create an application on your Mastodon instance. 2. Set `MASTODON_TOKEN` and `MASTODON_INSTANCE`. ## Configure syndication [Section titled “Configure syndication”](#configure-syndication) Add a `posse` section in `otsorc.yml`: ```yaml posse: mastodon: kinds: [note] ``` ## Publish [Section titled “Publish”](#publish) ```bash otso publish mastodon 1234 ``` Otso posts the note and includes a link to the canonical URL. ## Verify [Section titled “Verify”](#verify) Visit Mastodon to ensure the link points back to your site. # Publish via Micropub > Configure Micropub and publish to your site. ## IndieAuth [Section titled “IndieAuth”](#indieauth) 1. Ensure your site advertises IndieAuth endpoints. 2. Run `otso login ` to obtain a token. ## Configure endpoint [Section titled “Configure endpoint”](#configure-endpoint) Set the token and endpoint in `otsorc.yml`: ```yaml micropub: endpoint: https://example.com/micropub token: $INDIEAUTH_TOKEN ``` ## Publish [Section titled “Publish”](#publish) ```bash otso publish note.md ``` Otso sends a Micropub create request with content and media. Responses include the canonical URL which becomes the event’s permalink. # Schedule Sync (Vercel Cron / Upstash) > Run periodic sync jobs in production. ## Vercel cron [Section titled “Vercel cron”](#vercel-cron) 1. Add a `vercel.json` cron entry: ```json { "crons": [{ "path": "/api/sync", "schedule": "0 * * * *" }] } ``` 2. Deploy and verify the function triggers hourly. ## Upstash QStash [Section titled “Upstash QStash”](#upstash-qstash) For retries and dedupe: 1. Create a QStash URL targeting `/api/sync`. 2. Set the schedule in the Upstash console. 3. Provide `QSTASH_CURRENT_SIGNING_KEY` environment variables. ## Track progress [Section titled “Track progress”](#track-progress) Otso records the last `synced_at` for each source so reruns are idempotent. # Architecture at a Glance > High‑level architecture and how pieces fit together. ``` flowchart LR subgraph Clients CLI Desktop iOS end subgraph Otso["Otso core"] SDK["SDK & runtime"] API["HTTP API"] DB[("Event store")] PR["Projections"] Plugins["Plugins"] SDK --> DB DB --> PR SDK --> API Plugins --> SDK end CLI --> SDK Desktop --> SDK iOS --> SDK Sources["Sources"] -- import --> Plugins SDK -- publish --> Publishers["Publishers / networks"] PR --> Site["Your site & dashboards"] API --> Emulators["Emulators"] classDef store fill:#2b5,stroke:#185,stroke-width:1,color:#fff; class DB,PR store; ``` ## Core concepts [Section titled “Core concepts”](#core-concepts) * Events: append‑only records of actions (imported, enriched, published). Easy to audit and replay. * Projections: derived tables/JSON/search indexes for fast reads. Rebuildable on demand. * Adapters: thin plugins for sources (imports) and publishers (POSSE, Micropub). ## Storage: SQLite or Postgres [Section titled “Storage: SQLite or Postgres”](#storage-sqlite-or-postgres) * Start with SQLite: fast, simple, great for local use and CLIs. * Move to Postgres when you want concurrent writers, external services, or hosted ops. * Replication and backup: use Litestream/LiteFS for SQLite; standard tools for Postgres. ## Import flow (any API or archive) [Section titled “Import flow (any API or archive)”](#import-flow-any-api-or-archive) ``` sequenceDiagram autonumber participant Src as Source API/Archive participant CLI as Otso CLI participant DB as Store (SQLite/Postgres) participant PR as Projections Src->>CLI: Fetch page (cursor / ETag / since‑ID) CLI->>CLI: Normalize → kinds + media CLI->>DB: Upsert with idempotency key DB-->>CLI: Insert/Update (or No‑op) CLI->>PR: Emit events → update projections ``` ## Publish flow (your site is the source) [Section titled “Publish flow (your site is the source)”](#publish-flow-your-site-is-the-source) ``` sequenceDiagram autonumber participant You as You participant CLI as Otso CLI participant Site as Your Site (Micropub) participant Net as Networks (POSSE) participant WM as Webmention You->>CLI: Compose post CLI->>Site: Micropub create Site-->>CLI: 201 Created + canonical URL CLI->>Net: Cross‑post with canonical link Site->>WM: Send/Receive webmentions ``` ## Privacy & visibility [Section titled “Privacy & visibility”](#privacy--visibility) ``` flowchart LR Item[Content item] -->|public| Pub[Public projections & feeds] Item -->|unlisted| Unl[Permalink only no index/feed] Item -->|private| Priv[Private tools only] Item -->|secret| Sec[Secure projection or encrypted vault] Idx[[Search Index]] -. excludes .-> Priv Idx -. excludes .-> Sec ``` * Credentials live in env vars or OS keychain; never in the repo or search indexes. ## What lives where [Section titled “What lives where”](#what-lives-where) * Core SDK: stable data model, command surface, and utilities (rate limits, media). * Plugins: per‑service logic (API/format quirks) and small mapping functions. * Site/UI: thin layers over projections; easy to swap or customize. This separation keeps Otso fast to use, easy to reason about, and safe to extend. # Contributing > Ways to participate across roles and skills. We welcome contributions of all kinds and from contributors of every background. Whether you’re writing code, improving documentation, shaping user experience, reporting issues, offering translations, or fostering community discussions, your help moves the project forward. If you’re new to open source, you’re welcome here—start with our issue tracker, ask questions, and share your ideas. ## Open Source Philosophy [Section titled “Open Source Philosophy”](#open-source-philosophy) Otso is built in the open to empower individuals and communities. We believe software should be transparent, forkable, and shaped by those who use it. By sharing our work and collaborating publicly, we strive to create tools that respect privacy, encourage portability, and grow through collective stewardship. # Key Capabilities > What you can do with Otso. ## Available Now [Section titled “Available Now”](#available-now) ### Full-text search across sources [Section titled “Full-text search across sources”](#full-text-search-across-sources) Search your entire digital history from one place. The FTS5-powered search index handles: * **Instant results** — Type and see matches immediately * **Snippet highlighting** — See where your query matched * **Ranked results** — Most relevant items first * **Empty query browsing** — See recent items when no search term ### Timeline visualization [Section titled “Timeline visualization”](#timeline-visualization) See your activity over time with an interactive histogram: * **Monthly buckets** — Aggregated view of when things happened * **Source breakdown** — Color-coded by data source * **Date range selection** — Click to filter by time period * **Category stacking** — See which sources dominated each period ### Multi-source indexing [Section titled “Multi-source indexing”](#multi-source-indexing) Index data from 23+ configured sources: | Category | Sources | | ------------ | ------------------------------------- | | Reading | Pinboard, Readwise | | Music | Spotify, Last.fm, Apple Podcasts | | Social | Twitter/X archives, LinkedIn messages | | Productivity | Things tasks, Timing app usage | | Browse | Safari, Chrome, Google Search | | Notes | Apple Notes, Notion | | Photos | Apple Photos | | Location | Foursquare check-ins | | Code | GitHub activity | | AI | Claude conversations | | Curation | Are.na blocks and channels | | Video | YouTube watch history | | Calendar | Google Calendar events | ### Category and source filtering [Section titled “Category and source filtering”](#category-and-source-filtering) Filter search results by: * **Categories** — Click to show only music, or only notes * **Individual sources** — Drill down to just Pinboard or just Spotify * **Exclusion mode** — Hide specific sources from results ### Configuration-driven extraction [Section titled “Configuration-driven extraction”](#configuration-driven-extraction) Add new sources without code changes: ```toml [[sources]] name = "my-source" enabled = true db_path = "my-source/chronicle/chronicle.db" [[sources.queries]] sql = "SELECT id, title, content, created_at FROM items" ``` *** ## On the Roadmap [Section titled “On the Roadmap”](#on-the-roadmap) Note These features are planned but not yet implemented. ### Compose & publish [Section titled “Compose & publish”](#compose--publish) Create notes, articles, photos, replies, and more. Publish to your site via Micropub. Cross‑post (POSSE) to networks like Mastodon or Bluesky. ### Live API import (PESOS) [Section titled “Live API import (PESOS)”](#live-api-import-pesos) Import your existing history from live APIs (Mastodon, Bluesky, GitHub). Run on a schedule to keep your archive current. ### Enrich & organize [Section titled “Enrich & organize”](#enrich--organize) Run enrichment pipelines to clean web pages (Defuddle‑style readability), generate alt‑text, extract tags, and create summaries. ### Webmention & backfeed [Section titled “Webmention & backfeed”](#webmention--backfeed) Send and receive webmentions for replies, likes, and reposts. Show conversations on your own site. ### CLI commands [Section titled “CLI commands”](#cli-commands) Full command-line interface for: * `otso import ` — Pull from APIs and archives * `otso publish ` — Send to your site and syndicate * `otso search ` — Search from the terminal * `otso snapshot` — Backup before risky operations # Roadmap & Philosophy > Near‑term roadmap and guiding principles. ## Near‑term roadmap [Section titled “Near‑term roadmap”](#nearterm-roadmap) * M1 · Basics: CLI quickstart, SQLite store, Micropub publish to a minimal site, snapshots. * M2 · Imports: GitHub, Mastodon, Bluesky, bookmarks/NDJSON; checkpointed pagination; media handling. * M3 · POSSE: publish to Mastodon/Bluesky with canonical backlinks; simple retry/backoff. * M4 · Webmention: send/receive and show conversations on your site. * M5 · Enrichment: Defuddle‑style readability, alt‑text, tags, summaries with provenance. * M6 · Search & projections: fast listings and full‑text with strict privacy rules. * M7 · Plugins & SDK: stable interfaces, example plugins, testing and docs. These steps are informed by real platform constraints (e.g., Twitter/X limits, Mastodon app passwords) and a focus on idempotency and safety. ## Guiding principles [Section titled “Guiding principles”](#guiding-principles) * Local‑first by default: keep a copy you control; sync/replicate when useful. * Standards‑first: Micropub, IndieAuth, Webmention, Microformats2. * Small core, strong edges: plugins add service‑specific logic; the core stays tiny and boring. * Idempotent and observable: every operation is safe to retry; events are auditable. * Privacy by design: explicit visibility; no secrets in indexes; credentials in env/keychain. * Portability: projections you can export (JSON, Markdown) without losing fidelity. * Performance: prefer rebuildable projections over complicated live queries. ## What this means for you [Section titled “What this means for you”](#what-this-means-for-you) You can start small—import a few sources, publish a note—and grow into more. Because Otso is modular and event‑driven, you can swap storage, add plugins, or change your site without repainting your whole house. This page will outline the short‑term roadmap and the principles guiding Otso’s design. Planned sections: * Milestones and priorities * Local‑first trade‑offs * IndieWeb alignment and interoperability # What Is Otso > A concise introduction to Otso and its core components. Otso is a local‑first toolkit that helps you own your content and activity across the web. It indexes your digital trails from dozens of sources and makes them searchable on your machine. ## Current Components [Section titled “Current Components”](#current-components) These components are working today: ### otso-indexer (Rust) [Section titled “otso-indexer (Rust)”](#otso-indexer-rust) The indexer reads data from SQLite databases and builds a unified, searchable event store. * **Configuration-driven** — Add sources via `sources.toml`, no code changes needed * **Parallel processing** — Indexes multiple sources concurrently * **Transaction-safe** — Batch operations with automatic rollback on failure * **Robust parsing** — Handles 15+ timestamp formats automatically ### otso-search (SvelteKit) [Section titled “otso-search (SvelteKit)”](#otso-search-sveltekit) A web UI for searching your indexed data. * **Full-text search** — FTS5-powered search across all sources * **Timeline histogram** — Visual overview of your activity over time * **Category filters** — Filter by source type (reading, music, social, etc.) * **Keyboard navigation** — Arrow keys, Enter to open, Escape to close ```plaintext ┌─────────────────────────────────────────────────┐ │ SOURCE DATA (SQLite databases) │ │ Twitter, Pinboard, Spotify, GitHub, etc. │ └───────────────────────┬─────────────────────────┘ ▼ ┌─────────────────────────────────────────────────┐ │ otso-indexer (Rust) │ │ Configurable via sources.toml │ └───────────────────────┬─────────────────────────┘ ▼ ┌─────────────────────────────────────────────────┐ │ Event Store + FTS5 Search Index (SQLite) │ └───────────────────────┬─────────────────────────┘ ▼ ┌─────────────────────────────────────────────────┐ │ otso-search (SvelteKit) │ │ Timeline · Filters · Full-text search │ └─────────────────────────────────────────────────┘ ``` ## Planned Components [Section titled “Planned Components”](#planned-components) Note These features are on the roadmap but not yet implemented. * **CLI** — Commands for import, publish, and maintenance (`otso import`, `otso publish`) * **Web dashboard** — Compose posts, review imports, manage accounts * **Site generator** — Render posts with Microformats2 and Webmention support * **Live API connectors** — PESOS import from Mastodon, Bluesky, and more ## Supported content types [Section titled “Supported content types”](#supported-content-types) The indexer extracts events matching these types: | Type | Examples | | --------- | -------------------------------------- | | Bookmarks | Pinboard saves, Readwise highlights | | Listens | Spotify plays, Last.fm scrobbles | | Notes | Apple Notes, Notion pages | | Posts | Tweets, toots, skeets | | Photos | Apple Photos with location/description | | Check-ins | Foursquare visits | | Tasks | Things to-dos | | Events | Calendar entries | | Code | GitHub commits, issues, PRs | ## Visibility & privacy [Section titled “Visibility & privacy”](#visibility--privacy) Every item can carry a visibility: `public`, `unlisted`, `private`, or `secret`. * **Public**: included in search results * **Unlisted**: searchable but not in feeds * **Private**: visible only in local tools * **Secret**: excluded from search indexes entirely The search projection enforces these rules—private and secret items are never indexed. ## Quick mental model [Section titled “Quick mental model”](#quick-mental-model) Think of Otso as an append‑only event log plus projections: 1. **Sources** (SQLite databases with your data) 2. **Events** (normalized records in a unified store) 3. **Projections** (FTS5 search index, timeline views) Projections are rebuildable. If something goes wrong, rebuild from the event store. # Why Otso > The value of owning your content and using open standards. You create across many places, but platforms change, rate limits move, and export tools are an afterthought. Otso helps you keep a durable, portable archive you control, while still meeting people where they are. ## Own the source, share widely [Section titled “Own the source, share widely”](#own-the-source-share-widely) * Your site holds the canonical copy. POSSE cross‑posts link back to it. * PESOS imports bring old posts home so you can search and reuse them. * Webmention connects conversations without ceding ownership. ## Local‑first reliability [Section titled “Local‑first reliability”](#localfirst-reliability) * Works offline with SQLite and file assets; easy to back up or sync. * Snapshots and idempotent commands make big operations safe and reversible. * Projections are rebuildable, so damage is contained and recoverable. ## Standards and interoperability [Section titled “Standards and interoperability”](#standards-and-interoperability) * Micropub + IndieAuth for publishing and auth; Microformats2 for semantic HTML. * Plays nicely with IndieWeb tools and other ecosystems. ## Understand and control your data [Section titled “Understand and control your data”](#understand-and-control-your-data) * Transparent, event‑driven internals—see what happened and why. * Clear privacy rules: public, unlisted, private, secret. * No secrets in indexes; credentials live in env or OS keychain. ## Why not “just Markdown in git”? [Section titled “Why not “just Markdown in git”?”](#why-not-just-markdown-in-git) Git + Markdown is great for static content. Otso handles streams of activity, imports from APIs, media management, visibility rules, and cross‑posting—with a small database that makes queries and enrichment fast. You can still keep Markdown projections for portability, but a real store gives you power and safety. * Own your data in a durable local‑first store. * Publish anywhere while keeping your site as the source. * Built on open standards (Micropub, IndieAuth, Webmention). * Extensible plugins and safe operations. # APIs & Endpoints > Micropub, Webmention, and internal HTTP endpoints. ## Micropub [Section titled “Micropub”](#micropub) * `POST /micropub` – create or update posts. * `GET /micropub?q=syndicate-to` – list POSSE targets. * `GET /micropub?q=source&id=…` – fetch stored content. ## Webmention [Section titled “Webmention”](#webmention) * `POST /webmention` – receive mentions. * Outgoing pings are queued when you publish; run `otso webmention send` to deliver. ## Internal endpoints [Section titled “Internal endpoints”](#internal-endpoints) When running the Vercel app, Otso exposes: * `GET /api/health` – health check. * `POST /api/sync` – trigger scheduled imports. # AI > Optional helpers for summaries, tags, and cleaned reading views. AI features in Otso are opt-in. You can run local models or connect to hosted ones to enrich your archive. * Generate summaries, keywords, or alt-text for imports. * Produce cleaned reading views for saved links. * Store provenance so you can redo enrichment with better models later. * Completely optional—the core works without any AI. # CLI > Command-line interface for imports, publishing, and maintenance. The `otso` CLI lets you script and automate everything: * Import archives or APIs with idempotent commands. * Publish posts and cross‑post to other networks. * Run enrichers, manage visibility, or snapshot the store. It is the foundation on which other apps build. # Desktop > Tauri-based desktop shell for drag-and-drop imports and local control. A lightweight desktop shell built with Tauri brings Otso’s tooling to macOS, Windows, and Linux. * Drag and drop archive ZIPs to import. * Local-first storage with access to system keychains. * Reuses the Web interface and talks to the core through Tauri commands. * Small footprint and auto-updating releases. # iOS > Fast Micropub client for quick capture and offline posting. The Otso iOS interface focuses on one-thumb publishing. It stores drafts locally, syncs in the background, and supports the full Micropub post kinds. * IndieAuth login with multi-account support. * Compose notes, articles, photos, replies, reposts, likes, bookmarks, check-ins, and RSVPs. * Local-first engine with SQLite drafts and an outbox queue for retryable sync. * Built with Swift and modern SwiftUI patterns. ## Share Sheet extension [Section titled “Share Sheet extension”](#share-sheet-extension) A Share Sheet extension lets any app hand content to Otso. From Safari, Photos, or a reader, choose Otso to queue a draft instantly. * Accepts text, URLs, images, or files. * Prompts for post kind before queuing the item. * Works with Shortcuts and App Intents for quick capture flows. # Web > Browser dashboard for composing and hosting your site. Otso ships a web interface that doubles as a dashboard and the default site theme. * Compose posts and browse your archive in a lightweight interface. * Manage accounts and syndication targets. * Generate a clean personal site using Microformats2. * Runs locally or deploys as a static site. # CLI Commands > Command reference with flags and examples. The `otso` CLI orchestrates imports, publishing, and maintenance. ## Global flags [Section titled “Global flags”](#global-flags) * `--config ` – path to config file. * `--db ` – override database connection. * `-v` / `--verbose` – increase logging. ## Subcommands [Section titled “Subcommands”](#subcommands) ### `import` [Section titled “import”](#import) Run a source adapter. ```bash otso import github ``` ### `publish` [Section titled “publish”](#publish) Publish an existing event. ```bash otso publish mastodon 1234 ``` ### `webmention send` [Section titled “webmention send”](#webmention-send) Send queued Webmentions. ## Exit codes [Section titled “Exit codes”](#exit-codes) * `0` – success * `1` – generic error * `2` – configuration or validation error # Data Schema & Mappings > Schema sketches and mappings (e.g., GitHub → mf2). Otso keeps the storage model deliberately small. All items are stored as **events** with normalized content and optional media references. ## Minimal schemas [Section titled “Minimal schemas”](#minimal-schemas) | Table | Key fields | Purpose | | -------- | ----------------------------------------------- | -------------------------------------------------- | | `events` | `id`, `kind`, `source`, `created_at`, `content` | Append‑only log of everything Otso learns or does. | | `media` | `id`, `event_id`, `url`, `alt` | References to local or remote media assets. | | `tags` | `event_id`, `name` | Free‑form tagging for search and organization. | ## Mapping examples [Section titled “Mapping examples”](#mapping-examples) ### GitHub [Section titled “GitHub”](#github) * `commit` → `entry` with `content.text`, `url`, and `author`. * `issue` / `discussion` → `entry` with labels mapped to tags. ### Mastodon [Section titled “Mastodon”](#mastodon) * `status` → `note` with HTML content, media attachments and visibility flag. ### Bluesky [Section titled “Bluesky”](#bluesky) * `post` → `note` with AT URI as `external_id` and facets mapped to tags. ## Query snippets [Section titled “Query snippets”](#query-snippets) List the latest public notes: ```sql SELECT content->>'text', created_at FROM events WHERE kind = 'note' AND visibility = 'public' ORDER BY created_at DESC LIMIT 20; ``` Fetch media for a given event: ```sql SELECT m.url, m.alt FROM media m JOIN events e ON m.event_id = e.id WHERE e.id = $1; ``` # Deployment Recipes > Vercel, self‑hosted, and scheduling. Otso runs as a small Node app and does not require special infrastructure. These deployment recipes cover common setups. ## Vercel [Section titled “Vercel”](#vercel) 1. Fork the repository and push your content. 2. In the Vercel dashboard, import the repo. 3. Add environment variables (`OTSO_DB_URL`, `OTSO_SECRET`, etc.). 4. Enable the schedule function if using QStash. 5. Trigger a deploy and verify the site responds. ## Self‑hosted [Section titled “Self‑hosted”](#selfhosted) * Install Node and a database on your server. * Clone the repo and run `pnpm install && pnpm build`. * Use systemd or pm2 to run `pnpm start` as a service. * Configure a reverse proxy (Nginx/Caddy) with HTTPS. ## Scheduling [Section titled “Scheduling”](#scheduling) Use cron or QStash to run syncs periodically: ```bash # crontab -e 0 * * * * /usr/local/bin/otso sync >> ~/otso.log 2>&1 ``` On Vercel with Upstash QStash: 1. Create a QStash URL that hits `https://your-app.vercel.app/api/sync`. 2. Set the schedule (e.g., hourly). 3. Monitor invocation logs to ensure sync jobs complete. # API > Exposing Otso's commands and plugin hooks without the UI. Beyond the bundled apps, Otso exposes its Micropub implementation and plugin system as headless services. * `@otso/protocol-micropub` provides a Micropub client and optional server. * Plugins register capabilities such as storage, publishers, or enrichers. * Capabilities are discoverable so custom interfaces can adapt dynamically. * Build custom extensions without touching the core. # Otso SDK > Composable core and plugin API for building on top of Otso. Otso’s heart is a small TypeScript SDK that exposes the event log, domain models, and typed commands. Interfaces like the CLI, web, or iOS all consume the same package, while plugins register storage adapters, importers, publishers, and enrichers. * Tiny core with commands such as `createPost`, `import`, and `publish`. * Typed event bus so plugins react to changes instead of poking internals. * Shared across surfaces: CLI, Web, iOS, or future desktop builds. # Event Model > Event categories, fields, and before/after policy. Events capture every action Otso takes. ## Categories [Section titled “Categories”](#categories) * **import** – data pulled from a source. * **publish** – sent to external services. * **enrich** – tags, alt-text, or other processing. ## Schema [Section titled “Schema”](#schema) ```json { "id": "uuid", "kind": "note", "created_at": "2024-01-01T00:00:00Z", "content": {}, "visibility": "public", "before": {}, "after": {} } ``` `before` and `after` allow reversible operations like updates. ## Guidelines [Section titled “Guidelines”](#guidelines) * Use `before`/`after` only when an action changes data. * Keep `content` small; large blobs should live in media tables. * Prefer new events over mutation when possible. # Emulators > Interfaces that mimic other apps using Otso's extension hooks. Emulators let new surfaces talk to Otso without reimplementing the core. Examples include the iOS Share Sheet and headless Micropub services that bridge existing tools. They rely on the same plugin system and queue as the main apps. # Outputs > Export content beyond a website. Otso is not limited to generating a public site. Planned export targets include: * Zipped archives of your full event log or site. * Markdown or JSONL dumps for long-term storage. * PDF or print-ready bundles for portable collections. These exports let you move or share your data on your own terms. # Protocols > Standard adapters like Micropub, Webmention, ActivityPub, and more. Otso treats network protocols as pluggable modules. A protocol plugin defines how the core speaks a standard like Micropub or Webmention, and multiple publishers can build on top of it. Current and planned protocol plugins include: * Micropub client and optional server. * Webmention send/receive. * WebSub for push feeds. * ActivityPub bridge for the Fediverse. * Experimental ideas such as AT Protocol, Nostr, or IPFS. Protocol plugins power publishers: an ActivityPub plugin can serve both Mastodon importers and posting adapters. # Sources > Import, sync, and syndication coverage for common services. Otso connects to many services. The table below summarizes which actions are possible for each source. * `✅` – supported * `🟡` – partial or limited support * `❌` – not available | Source | Category | Import | Sync (PESOS) | Syndication (POSSE) | | ----------------------- | ----------- | ------ | ------------ | ------------------- | | 23andMe | Health | ✅ | ❌ | ❌ | | Apple Health | Health | ✅ | ❌ | ❌ | | Apple Notes | Notes | ✅ | ❌ | ❌ | | Apple Photos | Photos | ✅ | ❌ | ❌ | | Apple Podcasts | Podcast | ✅ | ❌ | ❌ | | Bluesky (ATProto) | Social | 🟡 | ✅ | ✅ | | Browser History | Browser | ✅ | ❌ | ❌ | | Discord | Messaging | ✅ | 🟡 | ❌ | | Email (IMAP/mbox) | Email | ✅ | ❌ | ❌ | | Evernote | Notes | ✅ | ❌ | ❌ | | Fitbit | Fitness | ✅ | ❌ | ❌ | | Flickr | Photos | ✅ | 🟡 | ✅ | | Foursquare / Swarm | Location | ✅ | 🟡 | ❌ | | Ghost | Blog | ✅ | 🟡 | ✅ | | GitHub | Code | ✅ | ✅ | ✅ | | GitLab | Code | ✅ | 🟡 | 🟡 | | Gmail | Email | ✅ | ❌ | ❌ | | Google Calendar | Calendar | ✅ | ❌ | ❌ | | Google Drive | Files | ✅ | ❌ | ❌ | | Google Location History | Location | ✅ | ❌ | ❌ | | Google Photos | Photos | ✅ | ❌ | ❌ | | Google Takeout | Export | ✅ | ❌ | ❌ | | Goodreads | Books | ✅ | ❌ | ❌ | | Hacker News | Social | ✅ | ❌ | ❌ | | Hypothesis | Annotations | ✅ | ❌ | ❌ | | iCal / ICS | Calendar | ✅ | ✅ | ❌ | | iMessage | Messaging | ✅ | ❌ | ❌ | | iNaturalist | Science | ✅ | ❌ | ❌ | | Instapaper | Reading | ✅ | ❌ | ❌ | | Instagram | Social | ✅ | ❌ | 🟡 | | Kindle | Books | ✅ | ❌ | ❌ | | Kobo | Books | ✅ | ❌ | ❌ | | Last.fm | Music | ✅ | ❌ | ❌ | | Matrix | Messaging | 🟡 | ✅ | ✅ | | Mastodon / Fediverse | Social | 🟡 | ✅ | ✅ | | Medium | Blog | ✅ | ❌ | ✅ | | Micro.blog | Blog | ✅ | 🟡 | ✅ | | Notion | Notes | ✅ | 🟡 | ❌ | | Obsidian / Markdown | Notes | ✅ | ✅ | ✅ | | Overcast | Podcast | ✅ | ❌ | ❌ | | Pinboard | Bookmarks | ✅ | ✅ | ❌ | | Pocket | Bookmarks | ✅ | 🟡 | ❌ | | Polar | Fitness | ✅ | ❌ | ❌ | | Raindrop.io | Bookmarks | ✅ | ✅ | ❌ | | Readwise | Reading | ✅ | ✅ | ❌ | | Reddit | Social | ✅ | ❌ | ❌ | | Safari | Browser | ✅ | ❌ | ❌ | | Shell | System | ✅ | ❌ | ❌ | | Signal | Messaging | ✅ | ❌ | ❌ | | Slack | Messaging | ✅ | 🟡 | ❌ | | Spotify | Music | ✅ | ❌ | ❌ | | Stack Overflow | Q\&A | ✅ | 🟡 | ❌ | | Strava | Fitness | ✅ | ✅ | ✅ | | Telegram | Messaging | ✅ | ❌ | ❌ | | Threads (Meta) | Social | ❌ | ❌ | 🟡 | | Twitter / X | Social | ✅ | ❌ | 🟡 | | Vimeo | Video | ✅ | 🟡 | ✅ | | WhatsApp | Messaging | ✅ | ❌ | ❌ | | WordPress | Blog | ✅ | ✅ | ✅ | | YouTube | Video | ✅ | 🟡 | ✅ | | Zulip | Messaging | ✅ | ❌ | ❌ | # Glossary & FAQ > Common terms and answers to frequent questions. ## Glossary [Section titled “Glossary”](#glossary) * **Adapter** – Plugin that knows how to talk to an external service. * **Event** – An immutable record representing an import, publish, or enrichment. * **Projection** – A derived view built from events for fast reads. * **POSSE** – Publish on your Own Site, Syndicate Elsewhere. * **PESOS** – Publish Elsewhere, Syndicate to Own Site. ## FAQ [Section titled “FAQ”](#faq) ### Is Otso open source? [Section titled “Is Otso open source?”](#is-otso-open-source) Yes. The code is available under the MIT license. ### Do I need a server? [Section titled “Do I need a server?”](#do-i-need-a-server) No. You can run Otso locally with SQLite and deploy later if desired. ### Can I mix SQLite and Postgres? [Section titled “Can I mix SQLite and Postgres?”](#can-i-mix-sqlite-and-postgres) Not in the same project, but you can migrate using the export/import commands. ### How are credentials stored? [Section titled “How are credentials stored?”](#how-are-credentials-stored) Secrets live in environment variables or the OS keychain. They are never committed to your repo. # Plugin Architecture > Plugin manifest, lifecycle, and examples. Otso plugins extend imports, publishers, and enrichers. A plugin is a Node module exporting a manifest and hooks. ## Discovery and manifest [Section titled “Discovery and manifest”](#discovery-and-manifest) Plugins are discovered via `package.json` using the `"otso"` field or by loading from the local `plugins/` directory. Example manifest: ```ts export const manifest = { name: 'example', version: '1.0.0', hooks: ['import', 'publish'] }; ``` ## Minimal plugin interface [Section titled “Minimal plugin interface”](#minimal-plugin-interface) ```ts import type { ImportHook } from 'otso'; export const importHook: ImportHook = async (ctx) => { // fetch data and yield events }; ``` Hook functions receive a context with logging, storage, and config helpers. ## Examples and testing [Section titled “Examples and testing”](#examples-and-testing) * Use `pnpm link` during development. * Provide fixtures for APIs and run `pnpm test` inside your plugin repo. * See `@otso/import-github` for a real world example. # Projections & Search > Policies for projections, indexing, and full‑text search. Projections provide fast read models derived from events. ## What to index [Section titled “What to index”](#what-to-index) * Public and unlisted items. * Selected fields like `content.text`, `tags`, and `published_at`. * Exclude secrets and private content. ## Search adapters [Section titled “Search adapters”](#search-adapters) Otso ships with a SQLite FTS5 adapter and can emit JSON for external search services. Rebuild projections when schemas change: ```bash otso projections rebuild ``` ## Privacy considerations [Section titled “Privacy considerations”](#privacy-considerations) Never index API keys or other sensitive values. Visibility rules are enforced when building search indexes. # Security & Privacy > Managing secrets, storage boundaries, and privacy rules. Otso is designed to keep sensitive data local and under your control. ## Secrets management [Section titled “Secrets management”](#secrets-management) * Load API keys from environment variables or the OS keychain. * Never commit secrets to the repository. * Use `.env` for local development and hosting dashboards for production. ## Visibility rules [Section titled “Visibility rules”](#visibility-rules) Each event has a `visibility` field: * `public` – included in feeds and search. * `unlisted` – accessible via permalink but excluded from lists. * `private` – visible only in local tools. * `secret` – stored encrypted; not exported. ## Backup and encryption [Section titled “Backup and encryption”](#backup-and-encryption) * Keep regular database snapshots. * For SQLite, pair backups with tools like Litestream or LiteFS. * Encrypt archives at rest using `age` or similar tools when storing off‑device. # sources.toml Reference > Complete reference for configuring data sources. The `sources.toml` file defines which data sources the indexer processes. It’s located at `data/config/sources.toml`. ## File Structure [Section titled “File Structure”](#file-structure) ```toml [defaults] schema_version = "1.0" data_dir = "data/fixtures" # or "data/sources" for production [[sources]] name = "pinboard" enabled = true db_path = "pinboard/chronicle/chronicle.db" category = "reading" [[sources.queries]] table = "events" entry_type = "bookmark" action_type = "BookmarkAction" object_type = "WebPage" sql = """ SELECT source_id as external_id, json_extract(object, '$.name') as title, json_extract(object, '$.description') as content, end_time as occurred_at, json_extract(object, '$.url') as url FROM events WHERE type = 'BookmarkAction' """ ``` ## Defaults Section [Section titled “Defaults Section”](#defaults-section) | Field | Required | Description | | ---------------- | -------- | ------------------------------------------------------------------------------------------------- | | `schema_version` | No | Config format version (currently “1.0”) | | `data_dir` | No | Base directory for source databases. Defaults to `data/sources`. Use `data/fixtures` for testing. | ## Source Configuration [Section titled “Source Configuration”](#source-configuration) Each `[[sources]]` block defines a data source. | Field | Required | Description | | -------------------- | -------- | ---------------------------------------------------------------------------------------------------- | | `name` | Yes | Unique identifier for the source | | `enabled` | No | Whether to process this source. Defaults to `true` | | `db_path` | Yes | Path to SQLite database, relative to `data_dir` | | `category` | No | Category for filtering. See [categories](#categories) below | | `default_visibility` | No | Visibility for items without explicit visibility. Options: `public`, `unlisted`, `private`, `secret` | ### Categories [Section titled “Categories”](#categories) | Category | Use for | | -------------- | ---------------------------- | | `reading` | Bookmarks, highlights, RSS | | `music` | Listening history, scrobbles | | `social` | Posts, messages, replies | | `comms` | Email, chat, DMs | | `productivity` | Tasks, time tracking | | `browse` | Browser history, searches | | `notes` | Notes, documents | | `photos` | Images, screenshots | | `location` | Check-ins, GPS logs | | `code` | Commits, issues, PRs | | `ai` | AI conversations | | `curation` | Collections, boards | | `video` | Watch history | | `calendar` | Events, meetings | ## Query Configuration [Section titled “Query Configuration”](#query-configuration) Each source can have one or more `[[sources.queries]]` blocks. | Field | Required | Description | | ------------- | -------- | -------------------------------------------------- | | `table` | No | Table name for validation (optional) | | `entry_type` | Yes | Type of entry (e.g., “bookmark”, “listen”, “note”) | | `action_type` | No | Schema.org action type (e.g., “BookmarkAction”) | | `object_type` | No | Schema.org object type (e.g., “WebPage”) | | `sql` | Yes | SQL query to extract events | ### Required SQL Columns [Section titled “Required SQL Columns”](#required-sql-columns) Your SQL query should return these columns: | Column | Required | Description | | ------------- | -------- | ----------------------------------------------------------------------------- | | `external_id` | Yes | Unique ID within the source. Also accepts `source_id` or `id` | | `title` | Yes | Display title | | `content` | No | Full text for search indexing | | `occurred_at` | Yes | When the event happened. Also accepts `timestamp`, `created_at`, `date`, etc. | | `url` | No | Link to original item | | `visibility` | No | Item visibility level | ### Timestamp Handling [Section titled “Timestamp Handling”](#timestamp-handling) The indexer automatically parses many timestamp formats: * **Unix timestamps**: `1704067200` (seconds) or `1704067200000` (milliseconds) * **ISO 8601**: `2024-01-01T00:00:00Z` * **Date strings**: `2024-01-01` * **Various formats**: `Jan 1, 2024`, `01/01/2024`, etc. If parsing fails, the event uses the current time with a warning. ## Aggregation [Section titled “Aggregation”](#aggregation) For high-volume sources (like time tracking), you can enable aggregation: ```toml [[sources]] name = "timing" enabled = true db_path = "timing/chronicle/chronicle.db" [sources.aggregation] strategy = "daily" key_fields = ["title"] ``` Available strategies: * **daily** — Group events by date + key fields (e.g., 2.2M records → \~50K daily aggregates) * **hourly** — Group events by hour + key fields * **session** — Group events with gaps less than `time_bucket` seconds (default 30 minutes) | Field | Description | | ------------- | ---------------------------------------------------------------------- | | `strategy` | Aggregation period: `daily`, `hourly`, `session`, or `none` | | `key_fields` | Fields to group by | | `time_bucket` | For session strategy: gap threshold in seconds (default 1800 = 30 min) | ## Configured Sources [Section titled “Configured Sources”](#configured-sources) The default configuration includes 23 sources: ### Reading [Section titled “Reading”](#reading) * **pinboard** — Bookmarks from Pinboard * **readwise** — Highlights from Readwise ### Music [Section titled “Music”](#music) * **spotify** — Listening history from Spotify * **lastfm** — Scrobbles from Last.fm * **apple-podcasts** — Podcast episodes ### Messaging [Section titled “Messaging”](#messaging) * **imessage** — iMessage conversations * **linkedin** — LinkedIn messages ### Productivity [Section titled “Productivity”](#productivity) * **things** — Tasks from Things app * **timing** — App usage from Timing ### Browse [Section titled “Browse”](#browse) * **safari** — Safari browsing history * **chrome** — Chrome history (from Google Takeout) * **google-search** — Search queries ### Notes [Section titled “Notes”](#notes) * **apple-notes** — Notes from Apple Notes * **notion** — Pages from Notion ### Photos [Section titled “Photos”](#photos) * **apple-photos** — Photos with metadata ### Location [Section titled “Location”](#location) * **foursquare** — Check-ins ### Code [Section titled “Code”](#code) * **github** — GitHub activity ### AI [Section titled “AI”](#ai) * **claude** — Claude conversation exports ### Curation [Section titled “Curation”](#curation) * **arena** — Are.na blocks and channels ### Social [Section titled “Social”](#social) * **twitter-**\*\* — Twitter/X archive (supports multiple accounts) ### Video [Section titled “Video”](#video) * **youtube** — YouTube watch history ### Calendar [Section titled “Calendar”](#calendar) * **gcal** — Google Calendar events ## Example: Adding a Custom Source [Section titled “Example: Adding a Custom Source”](#example-adding-a-custom-source) Here’s a complete example for adding a custom notes database: ```toml [[sources]] name = "my-notes" enabled = true db_path = "my-notes/notes.db" category = "notes" default_visibility = "private" [[sources.queries]] table = "notes" entry_type = "note" action_type = "CreateAction" object_type = "NoteDigitalDocument" sql = """ SELECT id as external_id, title, body as content, strftime('%s', created_at) as occurred_at, NULL as url FROM notes WHERE deleted_at IS NULL ORDER BY created_at DESC """ ``` Key points: * Use `strftime('%s', ...)` to convert datetime columns to Unix timestamps * Filter out deleted items in the WHERE clause * Return `NULL` for optional columns you don’t have * Set `default_visibility` to control indexing ## Validating Configuration [Section titled “Validating Configuration”](#validating-configuration) Check your configuration before indexing: ```bash cd packages/otso-indexer cargo run --release -- validate ``` This checks: * TOML syntax * Required fields * Table existence * SQL syntax (via EXPLAIN) ## CLI Reference [Section titled “CLI Reference”](#cli-reference) ```bash # Build all enabled sources cargo run --release -- build # Build a specific source cargo run --release -- build --source pinboard # Rebuild search index only (skip event store) cargo run --release -- build --meili-only # Show statistics cargo run --release -- stats # Validate configuration cargo run --release -- validate # Full rebuild from event store cargo run --release -- rebuild ``` # Storage Adapters > FS‑Markdown, JSONL, SQLite, and Postgres adapters. Otso ships with multiple storage backends. ## Adapter interfaces [Section titled “Adapter interfaces”](#adapter-interfaces) All adapters expose `put(event)`, `get(id)`, and `query(...)` methods. Custom adapters implement this interface. ## Built-in adapters [Section titled “Built-in adapters”](#built-in-adapters) * **SQLite** – default local store; supports FTS and transactions. * **Postgres** – for multi-user or hosted setups. * **FS-Markdown** – writes events as Markdown files for simple sites. * **JSONL** – append events to a newline-delimited log. ## Trade-offs [Section titled “Trade-offs”](#trade-offs) | Adapter | Pros | Cons | | ----------- | ------------------ | ---------------- | | SQLite | Simple, fast | Single writer | | Postgres | Scales, concurrent | Requires server | | FS-Markdown | Human-readable | Limited querying | # Backup & Restore (Snapshots) > Take a snapshot before big operations and restore if needed. Snapshots let you rewind if a migration goes wrong. ## Create a snapshot [Section titled “Create a snapshot”](#create-a-snapshot) ```bash otso db snapshot before-migration ``` ## List and inspect [Section titled “List and inspect”](#list-and-inspect) ```bash otso db snapshots otso db inspect before-migration ``` ## Restore [Section titled “Restore”](#restore) ```bash otso db restore before-migration ``` Verify the database state and rerun any failed operations. # Build a Simple Site with Microformats > Create a basic site that renders posts with mf2 semantics. This tutorial uses Astro but the concepts apply to any framework. ## Scaffold [Section titled “Scaffold”](#scaffold) ```bash pnpm create astro@latest otso-site cd otso-site ``` ## Template [Section titled “Template”](#template) Render a note: ```html ``` ## Webmention display [Section titled “Webmention display”](#webmention-display) Fetch mentions from the projections and list them under each post. # Import Bookmarks & Enrich with Defuddle > Bring in bookmarks and clean them up with Defuddle. ## Import bookmarks [Section titled “Import bookmarks”](#import-bookmarks) ```bash otso import bookmarks ~/exports/pinboard.ndjson ``` ## Run Defuddle [Section titled “Run Defuddle”](#run-defuddle) ```bash otso enrich defuddle --query "kind=bookmark and content.html is null" ``` This pulls article text and metadata. ## Add tags and alt text [Section titled “Add tags and alt text”](#add-tags-and-alt-text) ```bash otso tag add 42 readlater webdev ``` # Quickstart — Archive, Sync, Harmonize, and Publish > Install Otso then try a real‑world task. In this tutorial you’ll install Otso and pick a simple place to begin. ## Install prerequisites [Section titled “Install prerequisites”](#install-prerequisites) * Node.js 18+ * pnpm Then install: ```bash pnpm install -g otso ``` ## Initialize workspace [Section titled “Initialize workspace”](#initialize-workspace) ```bash otso init mysite cd mysite ``` ## Choose your path [Section titled “Choose your path”](#choose-your-path) After setup, try one of these common tasks: ### Archive a Twitter/X export and publish to your own site [Section titled “Archive a Twitter/X export and publish to your own site”](#archive-a-twitterx-export-and-publish-to-your-own-site) Save a local copy of your tweets then publish them on your site. ```bash otso import twitter --archive ~/Downloads/twitter otso publish site ``` See [Import Twitter/X Archive](/guides/import-twitter-archives/) for details. ### Automatically sync your Mastodon posts, likes, and replies to your website and local storage [Section titled “Automatically sync your Mastodon posts, likes, and replies to your website and local storage”](#automatically-sync-your-mastodon-posts-likes-and-replies-to-your-website-and-local-storage) Run this as a PESOS job—like Bridgy or Echofeed—to pull your Mastodon activity into your local archive and mirror it to your site. You can run it locally on demand or deploy it as a scheduled job on a serverless or cron service. ```bash otso import mastodon --include likes,replies --since 7d otso publish site ``` Deploy a recurring import every 15 minutes on Vercel: ```bash otso run --every 15m --on vercel import mastodon --include likes,replies ``` More options in [Import from Mastodon](/guides/import-mastodon/). ### Publish a note on your website and cross-post it to Bluesky and Mastodon [Section titled “Publish a note on your website and cross-post it to Bluesky and Mastodon”](#publish-a-note-on-your-website-and-cross-post-it-to-bluesky-and-mastodon) Send a post to your site, Bluesky, and Mastodon at once. ```bash otso publish hello.md --to bluesky,mastodon ``` Follow [Publish via Micropub](/guides/publish-micropub/), [Publish to Bluesky](/guides/publish-bluesky/), and [Publish to Mastodon](/guides/publish-mastodon/) to configure endpoints. ### Tag your bookmarks from Pinboard and Readwise using a local AI model [Section titled “Tag your bookmarks from Pinboard and Readwise using a local AI model”](#tag-your-bookmarks-from-pinboard-and-readwise-using-a-local-ai-model) Import bookmarks then enrich them with tags from a local model. ```bash otso import pinboard --since 30d otso import readwise --since 30d otso enrich tags --model local ``` See [Import Bookmarks & Enrich with Defuddle](/tutorials/import-bookmarks-defuddle/) for more on cleaning up bookmarks. ### Generate searchable descriptions from your screenshots folder [Section titled “Generate searchable descriptions from your screenshots folder”](#generate-searchable-descriptions-from-your-screenshots-folder) Pull images from a folder, auto‑caption them, then search by description. ```bash otso import folder ~/Pictures/Screenshots otso enrich describe --model local ``` Search later and see matches highlighted: ```bash otso search "concert ticket" --images ``` > \~/Pictures/Screenshots/live-show\.png — “July 4 **concert ticket**” ### Visualize your music listening across sites [Section titled “Visualize your music listening across sites”](#visualize-your-music-listening-across-sites) Combine listening history from multiple services into one timeline. ```bash otso import lastfm --since 30d otso import spotify --since 30d otso view music-timeline ``` ### Ask questions of your Notion docs using a local AI model [Section titled “Ask questions of your Notion docs using a local AI model”](#ask-questions-of-your-notion-docs-using-a-local-ai-model) Pull your Notion pages and ask them questions offline. ```bash otso import notion --database my-notes otso ask "When is the next team meeting?" --model local ``` ### Migrate Pinboard bookmarks to Raindrop with a searchable local copy [Section titled “Migrate Pinboard bookmarks to Raindrop with a searchable local copy”](#migrate-pinboard-bookmarks-to-raindrop-with-a-searchable-local-copy) Move your links to Raindrop while keeping everything indexed locally. ```bash otso import pinboard --all otso export raindrop otso search "pasta recipe" --bookmarks ``` Pick another path anytime—Otso lets you roam. # Run Local Search > Set up the Otso indexer and search UI on your machine. This tutorial walks you through setting up Otso’s indexer and search UI on your machine. By the end, you’ll have a local search interface for your digital history. ## Prerequisites [Section titled “Prerequisites”](#prerequisites) Before starting, ensure you have: * **Rust 1.70+** — Install from [rustup.rs](https://rustup.rs) * **Node.js 18+** — Install from [nodejs.org](https://nodejs.org) * **pnpm** — Install with `npm install -g pnpm` * **Git** — For cloning the repository ## Quick Start [Section titled “Quick Start”](#quick-start) 1. **Clone the repository** ```bash git clone https://github.com/yourusername/otso.git cd otso ``` 2. **Install Node dependencies** ```bash pnpm install ``` 3. **Build the Rust indexer** ```bash cd packages/otso-indexer cargo build --release ``` 4. **Run the indexer with demo data** The repository includes fixture data for testing: ```bash cargo run --release -- -c ../../data/config/sources.toml build ``` You should see output like: ```plaintext INFO Loaded 23 source configurations INFO Building 23 sources INFO pinboard events: 15 inserted, 0 skipped INFO spotify events: 25 inserted, 0 skipped ... INFO Updating projections... ``` 5. **Start the search UI** ```bash cd ../otso-search pnpm dev ``` 6. **Open the search interface** Navigate to `http://localhost:5173` in your browser. ## Using the Search UI [Section titled “Using the Search UI”](#using-the-search-ui) ### Basic Search [Section titled “Basic Search”](#basic-search) Type in the search box to find items across all sources. Results appear instantly as you type. * **Empty search** shows recent items * **Search terms** are matched against title and content * **Snippets** highlight where your query matched ### Timeline Histogram [Section titled “Timeline Histogram”](#timeline-histogram) Click the histogram icon to see your activity over time: * Each bar represents a month * Colors indicate different sources * Click a bar to filter by that time period * Click again to clear the filter ### Category Filters [Section titled “Category Filters”](#category-filters) Filter by content type using the category chips: | Category | What it includes | | -------- | ----------------------------- | | Reading | Bookmarks, highlights | | Music | Listens from Spotify, Last.fm | | Social | Tweets, toots, messages | | Notes | Apple Notes, Notion pages | | Browse | Browser history, searches | | Code | GitHub activity | ### Keyboard Navigation [Section titled “Keyboard Navigation”](#keyboard-navigation) * **↑/↓** — Move through results * **Enter** — Open selected item * **Escape** — Close document modal * **/** — Focus search box ## Adding Your Own Data [Section titled “Adding Your Own Data”](#adding-your-own-data) ### Data Format Requirements [Section titled “Data Format Requirements”](#data-format-requirements) Otso expects your data in SQLite databases. Each source database should have tables that can be queried for: | Field | Required | Description | | ------------- | -------- | ----------------------------------- | | `external_id` | Yes | Unique identifier within the source | | `title` | Yes | Display title for the item | | `content` | No | Full text content for search | | `occurred_at` | Yes | Timestamp (Unix epoch or ISO 8601) | | `url` | No | Link to original item | ### Adding a New Source [Section titled “Adding a New Source”](#adding-a-new-source) 1. **Place your database** Create a directory for your source and place the database inside: ```bash mkdir -p data/sources/my-source/chronicle cp ~/my-data.db data/sources/my-source/chronicle/chronicle.db ``` 2. **Add configuration** Edit `data/config/sources.toml`: ```toml [[sources]] name = "my-source" enabled = true db_path = "my-source/chronicle/chronicle.db" category = "notes" # Choose: reading, music, social, notes, etc. [[sources.queries]] table = "items" entry_type = "note" action_type = "CreateAction" object_type = "NoteDigitalDocument" sql = """ SELECT id as external_id, title, body as content, created_at as occurred_at, NULL as url FROM items """ ``` 3. **Switch to production data directory** In `sources.toml`, change the data directory: ```toml [defaults] data_dir = "data/sources" # Changed from "data/fixtures" ``` 4. **Rebuild the index** ```bash cd packages/otso-indexer cargo run --release -- build ``` 5. **Refresh the search UI** Your new source should appear in the category filters. ## Troubleshooting [Section titled “Troubleshooting”](#troubleshooting) ### ”Table not found” error [Section titled “”Table not found” error”](#table-not-found-error) Check that the table name in your SQL matches what’s in your database: ```bash sqlite3 data/sources/my-source/chronicle/chronicle.db ".tables" ``` ### No results appearing [Section titled “No results appearing”](#no-results-appearing) Verify events were indexed: ```bash cargo run --release -- stats ``` This shows event counts per source. ### Timestamp parsing issues [Section titled “Timestamp parsing issues”](#timestamp-parsing-issues) The indexer handles many formats automatically: * Unix timestamps (seconds or milliseconds) * ISO 8601 (`2024-01-15T10:30:00Z`) * Date strings (`2024-01-15`) * Various locale formats If parsing fails, events default to the current time with a warning. ## Next Steps [Section titled “Next Steps”](#next-steps) * [Configure more sources](/reference/sources-toml/) — Full reference for `sources.toml` * [Understand the event model](/reference/event-model/) — How data is stored internally * [See the roadmap](/overview/key-capabilities/#on-the-roadmap) — What’s coming next # Set Up POSSE to Mastodon > Connect and syndicate posts to Mastodon. ## Connect account [Section titled “Connect account”](#connect-account) Run: ```bash otso login mastodon https://mastodon.social ``` Follow the OAuth flow and store the token. ## Choose kinds [Section titled “Choose kinds”](#choose-kinds) Edit `otsorc.yml`: ```yaml posse: mastodon: kinds: [note, photo] ``` ## Test a post [Section titled “Test a post”](#test-a-post) Publish a note and confirm the Mastodon copy links back to your site.