Agentic Second Brain
A personal knowledge and workflow system: capture anything, and a background LLM pipeline turns it into structured notes, tasks, projects and searchable memory.
Overview
Agentic Second Brain is a personal knowledge system built around one idea: capture should be effortless, and structure should be the machine's job. You send in a page, a note, a voice memo or a file; a worker picks it up and decides what it actually is — a task with a deadline, a decision worth recording, a note belonging to an existing project — and files it accordingly.
It's a monorepo of four deployables: a FastAPI service, a job worker, a Next.js app, and a Chrome extension. The shared ai_core Python package holds everything interesting — the capture pipeline, the agents, hybrid search, the scheduler, review generation and export — so the API and the worker run the same logic rather than each having their own copy.
The design is written down before it is built, and the code says where it diverges. Modules carry docstrings that name the design-doc section they implement and state plainly what was deferred and why — detect_people needs a real entity-disambiguation strategy, so it was left out rather than shipped as string matching on first names.
Highlights
- A capture inbox fed from the web app, a scripted API, or a Manifest V3 Chrome extension that sends the current page or a text selection in one click.
- A background worker that claims jobs from a PostgreSQL queue and runs the capture through summarization, entity extraction, project matching, deadline and decision detection, tagging, and embedding.
- Hybrid search combining a pgvector similarity leg and a pg_trgm trigram leg, fused with reciprocal rank fusion so exact keyword hits and semantic matches both surface.
- Six agents — planner, memory, research, writer, review, workflow — exposed over FastAPI, with streaming responses for conversational memory queries.
- A deterministic task scheduler that scores tasks by urgency and project weight, builds an availability map, and greedily places them, flagging what it cannot fit as at-risk.
- Daily, weekly and monthly reviews plus an emailed digest, generated on their own cadences by the worker.
- Self-hosted Supabase for auth and storage, GitHub OAuth sign-in, personal access tokens for scripted clients, and CI with CodeQL scanning, Dependabot and GHCR image builds.
Design Notes
Reciprocal rank fusion instead of picking one search strategy
Vector similarity and full-text search fail in opposite directions: one misses exact terms, the other misses paraphrase. Both legs run, and each contributes 1/(60 + rank) to a document's score. An item found by only one leg still scores something — no cliff at the union boundary — and an item found by both outranks either, without having to normalize cosine similarity against trigram similarity, which are not comparable numbers.
pg_trgm over tsvector for the keyword leg
Trigram similarity tolerates typos and partial matches on short titles — searching 'pstgres' still finds 'Redis vs Postgres' — without adding a language-specific tsvector column to every searchable table. That mattered more here than raw ranking quality, since the corpus is short titles and snippets rather than long documents.
One structured LLM call instead of seven round trips
The capture pipeline's first eight stages — summarize, extract entities, identify project, detect deadlines and decisions, create a task if warranted, generate tags — share the same context and read better together, so they're one structured call against a single response schema. Embedding and related-note linking stay separate because they use a different API and need the note to exist first.
Agents propose; only the user commits
The hard rule across every agent is read and suggest freely, write only what is unambiguous. The workflow agent gathers stalled, overdue and at-risk signals deterministically and writes proposals into an agent_actions table with status 'proposed' — it never touches projects or tasks directly. A recheck window keeps the periodic sweep from re-proposing the same thing every pass.
Scheduling is deterministic first, AI-assisted second
Task placement is an urgency-decay score over a 14-day horizon, an availability map, and greedy placement — plain code you can reason about and test, not a prompt. The LLM is used where judgement genuinely helps; a calendar that silently rearranges itself for reasons you can't reconstruct is worse than no calendar.
A job queue that survives concurrent workers
The worker claims jobs with SELECT ... FOR UPDATE SKIP LOCKED, so several workers can drain the same queue without ever handing the same capture to two of them, and attempt counts are incremented at claim time so a crash mid-job is visible rather than silently retried forever.
The extension is a thin client, not a second backend
One-click capture posts to the same POST /captures endpoint already built for scripted sources, authenticated with a personal access token. The extension requests host permission only for the API origin the user types in, not for every site they visit.