PixelRAG — Search the Web by How It Looks, Not Just What It Says

Retrieval-Augmented Generation (RAG) has become the default architecture for grounding LLMs in external data. But the pipeline has a fundamental blind spot: it treats the web as text. HTML is parsed, tables are linearized, charts are discarded, and the visual structure that makes information navigable for humans is lost.

PixelRAG, from UC Berkeley's Sky Computing Lab (SkyLab) and BAIR, takes a radically different approach. Instead of parsing web pages to text, it renders them as screenshots and performs retrieval directly in pixel space. The result is a RAG system that sees documents the way a human does — tables stay tabular, charts stay graphical, and layout stays meaningful.

GitHub: StarTrail-org/PixelRAG | Paper: arXiv:2606.28344 | Demo: pixelrag.ai

Why PixelRAG Matters

Traditional text-based RAG pipelines have a well-known failure mode: they lose information at the parsing step. Consider a Wikipedia table of population statistics — a text parser might render it as a flat list of comma-separated values, destroying the column relationships. A chart showing quarterly revenue becomes unreadable. Complex layouts with sidebars, callouts, and multi-column text are mangled.

PixelRAG's paper shows that on standard benchmarks like Natural Questions (NQ) and SimpleQA, pixel-native retrieval outperforms text-based RAG — even on text-centric tasks. On multimodal benchmarks (MMSearch, LiveVQA, MoNaCo), it improves accuracy by up to 18.1% over text baselines. This challenges the assumption that web retrieval must go through a text abstraction layer.

Architecture Overview

The architecture follows a clean pipeline:

1. Document Sources — Web pages, PDFs, images, or local files are fed into the renderer
2. pixelshot Renderer — Uses Playwright's headless Chromium (via CDP) to capture each document as screenshot tiles. Each render runs in an isolated, throwaway Chrome profile
3. Embedding Pipeline — Three stages: chunk (slice tiles into overlapping shards) → embed (Qwen3-VL-Embedding 2B, LoRA fine-tuned on screenshot data) → build-index (FAISS vector index, GPU-accelerated)
4. FAISS Visual Search Index — Stores all screenshot embeddings. A pre-built index of 8.28 million Wikipedia articles (30M+ tiles) is available for immediate use
5. Search API — FastAPI-based server accepting text queries and image-based (visual) queries. Returns the top-k matching screenshot tiles
6. VLM Reader — A Vision Language Model reads the retrieved pixel tiles directly, without any intermediate text conversion. This is the key innovation: no parsing, no text extraction, just pixel-to-pixel reading
7. Answer — The VLM produces the final answer, having seen the document exactly as it appears visually

A separate training pipeline (train/ directory, a standalone uv project) provides LoRA fine-tuning of the Qwen3-VL-Embedding model on curated screenshot data with contrastive learning.

Prerequisites

• Python 3.10+
• A machine with at least 8 GB RAM (16 GB+ recommended for indexing)
• CUDA GPU recommended for embedding and indexing (macOS with Apple Silicon works via MPS)
• Playwright (auto-installed with pixelrag)
• For the pre-built Wikipedia index: ~220 GB free disk space (or skip and build a smaller custom index)

Installation

PixelRAG is modular — install only the stages you need:

# Core renderer (pixelshot CLI)
pip install pixelrag

# Full pipeline (chunk + embed + build-index)
pip install 'pixelrag[index]'

# Search server
pip install 'pixelrag[serve]'

# For PDF support
pip install 'pixelrag[pdf]'

Quick Start: Search the Live Wikipedia Index

The fastest way to try PixelRAG is to use the hosted API endpoint. No setup needed:

curl -X POST https://api.pixelrag.ai/search \
-H "Content-Type: application/json" \
-d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'

This searches a pre-built index of 8.28M Wikipedia articles. The response includes the top-k matching screenshot tiles as base64-encoded images.

Self-Hosted: Build Your Own Index

1. Render a Document to Screenshot Tiles

# Web page
pixelshot https://en.wikipedia.org/wiki/Python --output ./tiles

# PDF document
curl -sL -o paper.pdf https://arxiv.org/pdf/2503.09516
pixelshot paper.pdf --output ./tiles

# Multiple sources
pixelshot https://example.com my-document.pdf --output ./tiles

2. Create a Pipeline Configuration

# pixelrag.yaml
source:
  type: local
  path: ./my_docs

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: auto          # cuda on Linux, mps on macOS, cpu as fallback

output: ./my_index

3. Build and Serve the Index

# Build the full index (chunk → embed → build-index)
pixelrag index build

# Start the search API server
pixelrag serve --index-dir ./my_index --port 30001

4. Query Your Index

curl -X POST http://localhost:30001/search \
-H "Content-Type: application/json" \
-d '{"queries": [{"text": "What is this document about?"}], "n_docs": 3}'

Programmatic Usage

from pixelrag_render import render_url

# Single page to tiles
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")

# tiles is a list of PIL Image objects — feed them directly to a VLM
from transformers import Qwen2VLForConditionalGeneration, Qwen2VLProcessor
processor = Qwen2VLProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")
model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", device_map="auto"
)

Claude Code Integration

PixelRAG ships as a Claude Code plugin called pixelbrowse. Instead of fetching raw HTML, Claude screenshots a page with pixelshot and reads the image directly:

# Install the CLI globally
uv tool install pixelrag

# Install the Claude plugin
claude plugin marketplace add StarTrail-org/PixelRAG
claude plugin install pixelbrowse@pixelrag-plugins

# Use it
claude -p "screenshot https://news.ycombinator.com and summarize the top stories"

Comparison: PixelRAG vs Traditional RAG

• Parsing: Traditional RAG parses HTML → linearized text. PixelRAG renders to screenshots — no parsing
• Tables: Lost in text parsing. PixelRAG preserves tabular structure visually
• Charts/Diagrams: Inaccessible to text RAG. PixelRAG reads them as images
• Layout: Linearized and flattened. PixelRAG keeps the original layout
• Token Efficiency: PixelRAG can use image compression to reduce VLM token costs by up to 3× at lower resolutions while maintaining accuracy
• Pre-built Index: 8.28M Wikipedia pages (30M+ tiles) — ready to use at api.pixelrag.ai

Verification Checklist

• pixelshot successfully renders a web page to screenshot tiles
• The hosted API endpoint returns results: curl -s https://api.pixelrag.ai/search -d '...'
• Self-hosted index builds without errors: pixelrag index build
• Search API responds on the configured port: curl localhost:30001/search
• Claude Code pixelbrowse plugin renders and reads pages

Resources

• GitHub: github.com/StarTrail-org/PixelRAG
• Paper: arxiv.org/abs/2606.28344
• Live Demo: pixelrag.ai
• API Status: status.pixelrag.ai
• Hugging Face Indexes: StarTrail-org/pixelrag-faiss-indexes
• Training Data: Chrisyichuan/screenshot-training-natural-filtered-v2
• Berkeley SkyLab: sky.cs.berkeley.edu