You read a lot. Articles, documentation, blog posts, GitHub READMEs — all kinds of technical content that seems useful in the moment. But when you need that information later, it's gone. You remember reading something about rate limiting strategies or database connection pooling, but you can't find it. The URL is buried somewhere in your browser history, or you bookmarked it but forgot the title, or you saved it to a folder you never look at.
This guide shows you how to build a personal research pipeline that solves this problem. Not a read-it-later app — those just move the pile from your browser tabs to a different place where you still don't read it. Instead, you're building a system that captures full content, enriches it with AI-generated metadata, and makes everything semantically searchable so you can find articles by describing what you're looking for, not by remembering exact keywords.
The entire system runs locally. Python, SQLite, a few open-source libraries, and a local LLM for enrichment. No subscriptions, no data leaving your machine, no limits on how much you save.
Step 1: Capture — Getting Content Into the System
The capture problem has two parts. First, you need to save quickly — during a browsing session, you don't want to break flow to carefully curate a save. Second, you need to get full text, not just the URL. A URL alone is almost useless for search; the page might change, go offline, or hide behind a paywall later.
The simplest approach is a CLI tool that takes a URL and extracts the content immediately. You can call it from a terminal, wire it to a keyboard shortcut, or build a browser bookmarklet that sends URLs to a local endpoint.
For content extraction, trafilatura handles the messy work of pulling readable text out of HTML. It strips navigation, ads, comments, and other junk, leaving you with the main article content.
import httpx
import trafilatura
def extract_article(url: str) -> dict | None:
try:
response = httpx.get(url, timeout=15, follow_redirects=True,
headers={"User-Agent": "Mozilla/5.0"})
text = trafilatura.extract(response.text, include_comments=False,
include_tables=True, favor_recall=True)
if not text or len(text) < 200:
return None
return {"url": url, "text": text, "title": _extract_title(response.text)}
except Exception:
return None
def _extract_title(html: str) -> str:
import re
match = re.search(r']*>([^<]+)', html, re.IGNORECASE)
return match.group(1).strip() if match else "" The favor_recall=True setting tells trafilatura to be aggressive about including content — you'd rather have some extra noise than miss important paragraphs. The 200-character minimum filters out pages that failed to extract properly (login walls, JavaScript-only content, etc.).
For JavaScript-heavy pages, you'll eventually want a fallback that renders the page in a headless browser before extraction. But start simple — most technical articles work fine without JavaScript rendering.
Step 2: Store — Simple Schema for Research Notes
SQLite is enough for personal scale. You're not building a distributed system; you're building a single-user tool that might hold a few thousand articles. SQLite gives you ACID transactions, full-text search as a bonus, and a single file you can back up trivially.
The schema starts minimal. You can always add columns later.
import sqlite3
from datetime import datetime
def init_db(db_path: str = "research.db"):
conn = sqlite3.connect(db_path)
conn.execute("""
CREATE TABLE IF NOT EXISTS articles (
id INTEGER PRIMARY KEY AUTOINCREMENT,
url TEXT UNIQUE NOT NULL,
title TEXT,
text TEXT NOT NULL,
summary TEXT,
tags TEXT,
saved_at TEXT NOT NULL,
embedding BLOB
)
""")
conn.execute("CREATE INDEX IF NOT EXISTS idx_url ON articles(url)")
conn.commit()
return conn
def save_article(conn, url: str, title: str, text: str):
try:
conn.execute("""
INSERT INTO articles (url, title, text, saved_at)
VALUES (?, ?, ?, ?)
ON CONFLICT(url) DO UPDATE SET
title = excluded.title,
text = excluded.text,
saved_at = excluded.saved_at
""", (url, title, text, datetime.utcnow().isoformat()))
conn.commit()
return True
except Exception:
return FalseThe upsert pattern (ON CONFLICT ... DO UPDATE) means you can re-save an article and it just updates the existing record. Useful when you want to refresh content or when your capture tool accidentally runs twice.
Step 3: Enrich — Adding Metadata With a Local LLM
Raw article text is searchable, but structured metadata makes it better. A one-paragraph summary lets you scan results quickly. Tags give you faceted filtering. Key concepts help with cross-referencing.
Run articles through a local LLM to extract this metadata. Ollama makes it straightforward to run models locally, and structured output ensures you get JSON back instead of prose.
import httpx
import json
def enrich_article(text: str, ollama_url: str = "http://localhost:11434") -> dict:
prompt = f"""Analyze this article and return JSON with:
- summary: 2-3 sentence summary of the main points
- tags: list of 3-5 relevant technical tags (lowercase)
- key_concepts: list of 2-4 main concepts discussed
Article:
{text[:4000]}
Return only valid JSON, no other text."""
response = httpx.post(
f"{ollama_url}/api/generate",
json={
"model": "llama3.2",
"prompt": prompt,
"stream": False,
"format": "json"
},
timeout=60
)
try:
result = json.loads(response.json()["response"])
return {
"summary": result.get("summary", ""),
"tags": ",".join(result.get("tags", [])),
"key_concepts": result.get("key_concepts", [])
}
except (json.JSONDecodeError, KeyError):
return {"summary": "", "tags": "", "key_concepts": []}
def update_enrichment(conn, url: str, enrichment: dict):
conn.execute("""
UPDATE articles
SET summary = ?, tags = ?
WHERE url = ?
""", (enrichment["summary"], enrichment["tags"], url))
conn.commit()The enrichment step is separate from capture on purpose. Capture should be fast — save the content immediately. Enrichment can run in the background, batch processing articles you saved earlier. This separation means slow LLM inference doesn't block your workflow.
Step 4: Embed — Making Content Searchable Semantically
Keyword search finds articles containing specific words. Semantic search finds articles about specific concepts, even if they use different terminology. You want to search for "connection pooling best practices" and find articles that discuss database connection reuse, even if they never use the exact phrase "connection pooling."
Embeddings make this possible. You convert text into high-dimensional vectors where similar meanings cluster together. Then search becomes vector similarity.
from sentence_transformers import SentenceTransformer
import numpy as np
model = SentenceTransformer('all-MiniLM-L6-v2')
def embed_article(text: str) -> bytes:
# Combine title + summary + first chunk for a representative embedding
embedding = model.encode(text[:3000], normalize_embeddings=True)
return embedding.tobytes()
def cosine_similarity(a: bytes, b: bytes) -> float:
va = np.frombuffer(a, dtype=np.float32)
vb = np.frombuffer(b, dtype=np.float32)
return float(np.dot(va, vb))
def embed_and_store(conn, url: str, text: str, summary: str = ""):
# Use summary + text for richer embedding
combined = f"{summary}\n\n{text}" if summary else text
embedding = embed_article(combined)
conn.execute("UPDATE articles SET embedding = ? WHERE url = ?",
(embedding, url))
conn.commit()The model runs locally — no API calls, no per-query costs, no privacy concerns. The embeddings are stored directly in SQLite as binary blobs. For a few thousand articles, you can compute similarity scores by iterating through all embeddings in memory. It's fast enough.
If you scale beyond a few thousand articles, you'd move to a vector database, but premature optimization isn't worth it. Start with the simple version.
Step 5: Retrieve — Semantic Search Over Your Library
Now you can search by describing what you're looking for. Instead of remembering that you bookmarked an article with "PostgreSQL" and "timeout" in the title, you search for "handling slow database queries in production" and find relevant articles regardless of their exact wording.
def search(conn, query: str, limit: int = 10) -> list[dict]:
# Embed the query
query_embedding = embed_article(query)
# Get all articles with embeddings
cursor = conn.execute("""
SELECT url, title, summary, embedding
FROM articles
WHERE embedding IS NOT NULL
""")
results = []
for url, title, summary, embedding in cursor:
if embedding:
score = cosine_similarity(query_embedding, embedding)
results.append({
"url": url,
"title": title,
"summary": summary,
"score": score
})
# Sort by similarity score, return top results
results.sort(key=lambda x: x["score"], reverse=True)
return results[:limit]The difference between keyword search and semantic search shows up immediately in practice. A keyword search for "API rate limiting" only finds articles with those exact words. A semantic search finds articles about throttling, backpressure, request quotas, and exponential backoff — all conceptually related even if they use different vocabulary.
You can add hybrid search later (combining semantic similarity with keyword matching), but pure semantic search already works surprisingly well for personal research where you remember the concept but not the terminology.
Step 6: Surface — Getting Relevant Articles When You Need Them
Search is reactive — you have to remember to ask. The next level is proactive surfacing: when you're working on something, relevant articles from your library appear automatically.
The concept is straightforward. Take context from your current work — a file you're editing, a problem you're describing, code you're writing — and run it through the same semantic search. Articles that match your current context bubble up.
This is where manual systems consistently fail. People build elaborate folder structures and tagging systems, but they never actually surface old content when it's relevant. The folder is too specific ("PostgreSQL/Performance/Indexing") and you're working on something adjacent ("query optimization") so the connection doesn't happen.
Semantic matching handles adjacency naturally. An article about PostgreSQL index strategies surfaces when you're writing a MySQL query optimization function, because the concepts are similar even though the technology differs.
When to Build This vs. When to Use a Ready-Made Tool
Building your own research pipeline teaches you what the problem actually is. You discover edge cases: JavaScript-rendered pages that trafilatura can't handle, paywalled content, PDFs, video transcripts. You learn which metadata actually helps search quality and which is noise. You understand the tradeoffs between embedding models.
The DIY version also gives you full control. Your data stays local. You can modify the schema, change the enrichment prompts, swap embedding models. Nothing breaks because a service changes their API or shuts down.
Ready-made tools handle the edge cases you haven't hit yet. They've already solved JavaScript rendering, paywall bypassing, PDF extraction. They've optimized the enrichment pipeline. They surface content proactively without you wiring up integrations.
Both approaches are valid. If you want to understand the problem deeply and have time to iterate, build your own. If you want something working today that handles edge cases out of the box, use a tool designed for this.