RAG Architecture: FAISS vs Qdrant for Production AI

RAG Architecture: FAISS vs Qdrant for Production AI

When you’re building a Retrieval-Augmented Generation (RAG) pipeline, the vector database choice matters more than you think. We migrated our production Docebo consulting knowledge base from FAISS to Qdrant — here’s what we learned, what broke, and when you should (and shouldn’t) make the same move.

The Setup: What We Had

Our consulting platform, Apprendere Hub, serves Docebo expertise to clients through a RAG-powered chat and search interface. The knowledge base contains 280+ documents: Docebo configuration guides, connector integrations, SSO setup tutorials, migration playbooks, and API documentation.

The original stack (v1):

• FAISS (Facebook AI Similarity Search) for vector indexing
• all-MiniLM-L6-v2 embeddings (384-dimensional, sentence-transformers)
• BM25 lexical search for hybrid retrieval
• SQLite + pickle for chunk metadata storage
• 6,915 chunks indexed across 280 source documents

FAISS is fast. FAISS is simple. FAISS runs in-process. For a single-node prototype, it’s perfect.

But then we started scaling.

Why We Moved Away From FAISS

Three problems pushed us toward Qdrant:

1. In-Process Lock-In

FAISS runs inside your Python process. That means:

• Your embedding model, your API server, and your vector index all share the same RAM
• You can’t update the index without blocking queries
• No concurrent readers and writers
• Scaling means spinning up another full process with its own FAISS copy

On our DGX Spark (ARM64, 128 GB unified memory), this wasn’t a memory problem — but it was an architecture problem. Every restart meant reloading the index. Every index rebuild meant downtime.

2. No Native Filtering

FAISS gives you raw vector similarity. If you want to filter by category (“only search Docebo Connect docs”), you need to:

1. Retrieve N candidates from FAISS
2. Post-filter them in Python
3. Hope N was large enough that you didn’t miss good results

This is the “retrieve then filter” anti-pattern. It works, but it’s wasteful and unpredictable. Our clients search by domain (SSO, Connect, API) constantly, and the post-filtering was eating into relevance.

3. No Persistence Story

FAISS indices are binary files. You write_index() to disk and read_index() back. There’s no incremental update, no WAL, no replication. If your index gets corrupted, you rebuild from scratch.

We did this dance once after a power blip. Re-embedding 6,915 chunks took 15 minutes. Not catastrophic, but not confidence-inspiring either.

Why Qdrant

Qdrant is a purpose-built vector database with a REST API, native filtering, and persistence. It runs as a Docker container, speaks HTTP, and handles concurrent reads/writes natively.

Here’s what we gained:

Native Payload Filtering

results = qdrant.search(
    collection_name="apprendere-kb-bgem3",
    query_vector=query_embedding,
    query_filter=Filter(
        must=[
            FieldCondition(
                key="category",
                match=MatchValue(value="docebo-connect")
            )
        ]
    ),
    limit=10
)

Filtering happens inside the vector search, not after. This means more relevant results with smaller result sets and no wasted computation.

Built-in Persistence

Qdrant writes to disk continuously. Restart the container, your data is there. No rebuilds, no re-embedding, no anxiety.

Hybrid Search with BM25 Fusion

We kept our BM25 lexical index (pickle file, 10 MB) and fuse results from both Qdrant (semantic) and BM25 (lexical) using reciprocal rank fusion. The reranker (bge-reranker-v2-m3) then reorders the top candidates.

Query → Qdrant (semantic) → top-K
      → BM25 (lexical)   → top-K
      → Reciprocal Rank Fusion → top-N
      → bge-reranker-v2-m3 → final top-10

This three-stage pipeline catches cases where pure semantic search misses keyword matches (e.g., when a user searches for a specific Docebo setting name that doesn’t have strong semantic neighbors).

HTTP API = Language Agnostic

Our Python pipeline writes to Qdrant. Our future Node.js services can read from the same collection. Our n8n workflows can query it via HTTP nodes. No SDK lock-in.

The Migration: What Changed

We didn’t just swap the vector DB — we upgraded the entire retrieval stack:

Better Chunking = Fewer, Better Chunks

The v1 index had 6,915 chunks from 280 documents. The v2 index has 3,638 chunks from 284 documents. We chunked more aggressively (larger chunks, smarter boundaries), which reduced noise and improved retrieval precision.

Fewer chunks also means faster search and lower memory footprint — even with 1024-dim vectors (vs 384-dim in v1).

Embedding Model Upgrade

all-MiniLM-L6-v2 is a solid workhorse, but bge-m3 (from BAAI) brings:

• Multilingual support — important for our Spanish + English content
• Larger dimension (1024) — richer semantic representation
• Better domain adaptation — bge-m3 handles technical vocabulary (SSO, OIDC, SCORM) noticeably better

The trade-off: bge-m3 is a bigger model. We serve it through Ollama via our LiteLLM gateway, which keeps it loaded in GPU memory with 30-minute keep-alive. Query latency is ~40ms per embedding.

The Rollback Plan

Every migration needs an escape hatch. We kept the FAISS index as a fallback:

# Rollback to v1 stack
export QDRANT_COLLECTION=apprendere-kb
export RAG_EMBED_BACKEND=sentence-transformers
export RAG_EMBED_MODEL=all-MiniLM-L6-v2
export APPRENDERE_BM25_PATH=rag-index/bm25.pkl

The code checks the backend env var and routes accordingly. If Qdrant goes down, we can fall back to FAISS without re-embedding anything.

When You Should NOT Switch to Qdrant

FAISS is still the right choice when:

• You’re prototyping and need something running in 5 minutes
• Your index is small (<1,000 vectors) and fits in RAM
• You don’t need filtering — pure similarity search is all you do
• You’re on constrained hardware where running another Docker container isn’t feasible
• You need maximum query throughput on a single node — FAISS in-process is hard to beat for raw speed

Qdrant is worth the switch when:

• You need filtering (by category, source, date, etc.)
• Multiple services need to query the same index
• You want concurrent reads and writes
• Your index is large (>50K vectors) and growing
• You need persistence without rebuild downtime
• You’re running a multi-language stack (Python + Node.js + n8n, etc.)

Architecture Diagram

┌──────────────────────────────────────────────────┐
│                  Apprendere API (:8080)          │
│                                                   │
│  Query → Embed (bge-m3 via LiteLLM :4000)        │
│        → Qdrant Search (:6333) [semantic]         │
│        → BM25 Search (pickle) [lexical]           │
│        → Reciprocal Rank Fusion                  │
│        → Rerank (bge-reranker-v2-m3 :8201)       │
│        → Return top-K passages                    │
│        → LLM Context (via LiteLLM :4000)         │
└──────────────────────────────────────────────────┘
         │                          │
         ▼                          ▼
  ┌─────────────┐         ┌──────────────┐
  │  Qdrant     │         │   FAISS      │
  │  :6333      │         │  (fallback)  │
  │  3,638 pts  │         │  6,915 vecs  │
  │  1024-dim   │         │  384-dim     │
  │  bge-m3     │         │  MiniLM-L6   │
  └─────────────┘         └──────────────┘

Production Numbers

After 7+ days in production:

• Query latency p50: 120ms (embed + search + rerank)
• Query latency p95: 280ms
• Index build time: 105 seconds (vs 15 min for v1)
• Memory footprint: Qdrant container uses ~180MB RAM
• Disk usage: Qdrant collection data is ~45MB on disk
• Zero re-indexing events since deployment

Lessons Learned

1. Filter inside the database, not in your application code. This was the single biggest win. Native payload filtering in Qdrant eliminated an entire class of bugs around missing results and incorrect ranking.

2. Bigger embeddings aren’t always better — but they’re usually better. The jump from 384-dim to 1024-dim improved retrieval quality noticeably, especially for technical queries. The cost (slightly larger index, slightly slower embedding) is negligible.

3. Keep the fallback. Having FAISS as a rollback path gave us confidence to ship the migration without extensive staging testing. In production, the fallback has never been needed — but knowing it’s there is worth the 13MB of disk space.

4. Reranking is the secret sauce. The bge-reranker-v2-m3 model moved our retrieval from “pretty good” to “genuinely useful.” It catches cases where the semantic + BM25 fusion ranks a tangentially-related chunk above the actual answer.

5. Chunking strategy matters more than the database. Going from 6,915 chunks to 3,638 chunks (with smarter boundaries) improved results more than any other change. A vector DB can’t fix bad chunking.

What’s Next

• Langfuse tracing on every retrieval step to measure quality in production
• Argilla for curating relevance judgments and building an evaluation dataset
• DSPy for optimizing the fusion weights (currently using a simple reciprocal rank formula)
• Qdrant quantization to reduce memory further as the index grows

The stack works. The search is fast, the results are relevant, and we can finally filter by domain without a post-processing headache.

J4SGON builds sovereign AI infrastructure on ARM64. We run our own models, our own vector databases, and our own LLM gateways — no external API keys, no cloud lock-in. Read more about our approach or get in touch.