Vector databases are now foundational infrastructure for any team building RAG (retrieval-augmented generation) systems, semantic search, or embedding-based recommendations. The choices you make in vector database architecture determine retrieval latency, accuracy, scalability costs, and operational complexity.
This guide is an engineering-focused comparison for teams in production or approaching production with vector workloads.
What Vector Databases Actually Do
A vector database stores high-dimensional numerical vectors (embeddings) and performs approximate nearest neighbor (ANN) search — finding the N most similar vectors to a query vector efficiently. The core operation is:
- Embed your query text (e.g., user's question) using an embedding model (OpenAI text-embedding-3-large, BGE, E5)
- Search the vector index for the K most similar vectors to the query embedding
- Retrieve the associated metadata (the actual text chunks, document IDs, etc.)
- Pass the retrieved context to an LLM for response generation (RAG)
The key performance metrics are: - Recall@K: What fraction of the true top-K results does the ANN search return? (1.0 = perfect recall, 0.9 = 90% recall) - QPS (queries per second): Throughput for your query volume - Latency: P50 and P99 search latency - Index build time and cost: How long and how expensive to index N million vectors
Option 1: pgvector — When You Don't Need a Separate Database
pgvector is a PostgreSQL extension that adds vector similarity search to a standard Postgres database. If you're already running Postgres, pgvector is the simplest path to vector search.
When pgvector is the right choice: - You have an existing Postgres database with metadata you want to combine with vector search - Your vector dataset is under 5-10M vectors at 1536 dimensions - You want ACID transactions across vector and relational operations - You want to reduce operational complexity (one database instead of two)
pgvector HNSW (Hierarchical Navigable Small World) index: The HNSW index algorithm (added in pgvector 0.5+) dramatically improved performance over the original IVFFlat index. With HNSW, pgvector achieves competitive recall and latency at moderate scale.
Limitations: - Memory-bound: HNSW index is loaded into memory. At 1536 dimensions with 10M vectors, the index alone consumes ~60-80 GB of RAM — requiring a large Postgres instance. - Single-node: Postgres doesn't distribute vector indices across nodes. Scaling beyond a single large instance requires external sharding logic. - AWS RDS/Aurora pgvector is production-ready and removes operational overhead.
Recommended for: RAG systems with under 5M documents, teams with existing Postgres infrastructure, internal tools and knowledge bases.
Option 2: Pinecone — Managed Scale
Pinecone is a fully managed vector database that abstracts all infrastructure. You create an index, upsert vectors, and query — no cluster management, index tuning, or capacity planning.
When Pinecone is the right choice: - You need to scale to tens or hundreds of millions of vectors - Your team doesn't want to manage vector infrastructure - You're building a product where vector search is core and uptime matters - You want metadata filtering combined with ANN search
Pinecone architecture: Pinecone's managed indexes automatically scale, replicate for HA, and handle index organization internally. The API is simple:
from pinecone import Pinecone
pc = Pinecone(api_key="...")
index = pc.Index("your-index")
index.upsert(vectors=[{"id": "doc1", "values": embedding, "metadata": {"text": "..."}}])
results = index.query(vector=query_embedding, top_k=10, include_metadata=True)
Limitations: - Cost: Pinecone is significantly more expensive than self-hosted alternatives at scale. At 10M vectors, Pinecone runs $100-300+/month; equivalent self-hosted Weaviate on EC2 might run $200-500/month fully loaded but requires engineering overhead. - Vendor dependency: All your vectors are in Pinecone's infrastructure. Data portability requires re-embedding everything. - No SQL joins: You can't join Pinecone results against a relational database on the vector database side — application-layer joins required.
Recommended for: Startups and scale-ups that value velocity over operational control, teams where vector infrastructure management is not a core competency.
Option 3: Weaviate — Self-Hosted Control
Weaviate is an open-source vector database with both cloud-managed and self-hosted options. It supports multiple vector indexing algorithms, has a GraphQL API, and enables combining keyword and vector search (hybrid search).
When Weaviate is the right choice: - You want self-hosted control over your vector infrastructure - You need hybrid search (BM25 + vector) for better retrieval quality - You're dealing with multi-tenancy requirements - You want to run on your own AWS infrastructure
Weaviate key features: - HNSW index (primary) with quantization options for memory optimization - BM25 + vector hybrid search (often outperforms pure vector search for factual retrieval) - Multi-tenancy native: each tenant gets isolated vector space — important for SaaS RAG products - Modules for automatic embedding (Cohere, OpenAI, local models via transformers)
Deployment: Weaviate runs on Kubernetes (Helm chart) or Docker. For production on AWS, EKS with 3+ nodes and persistent storage (EBS) is standard. Plan for 60-80 GB memory per 10M vectors at 1536 dimensions.
Recommended for: Teams with Kubernetes infrastructure, SaaS products needing multi-tenancy, applications where hybrid search quality matters.
Option 4: Chroma — Development and Prototyping
Chroma is an open-source vector database optimized for developer experience. It runs in-process (no server required) for local development and can run as a server for production.
When Chroma is appropriate: - Early prototyping and development (RAG proof of concept) - Small-scale production (under 100K vectors) - Jupyter notebook and research workflows
When to move away from Chroma: - Production workloads over 100K vectors - Multi-user applications requiring concurrent access - Any workload requiring horizontal scaling
Chroma is the right tool for getting started fast. Build your RAG prototype with Chroma, then migrate to pgvector or Pinecone when the prototype is ready for production.
Architecture Patterns for RAG
Beyond the database choice, the architecture surrounding it matters for quality:
Chunking strategy: How you split source documents into chunks affects retrieval quality more than the vector database choice. Fixed-size chunks (256-512 tokens) are simple. Semantic chunking (splitting at natural boundaries) improves retrieval relevance. Recursive splitting (trying multiple sizes, keeping semantically coherent chunks) often produces the best results.
Hybrid search: Combining vector search (semantic similarity) with BM25 keyword search often outperforms pure vector search for factual queries. A document containing the exact phrase in the query should rank higher than one that's only semantically similar. Weaviate and Elasticsearch support hybrid search natively; pgvector + pg_trgm can approximate it.
Metadata filtering: Filter vectors by metadata before ANN search. Filter by document type, date, user permissions, or any structured attribute before running the vector similarity search. All major vector databases support this; implement it to reduce noise in retrieval results.
Reranking: After retrieving top-K candidates from vector search, apply a cross-encoder reranker (Cohere Rerank, BGE reranker, or ColBERT) to re-score and reorder candidates. Reranking consistently improves answer quality for RAG systems at the cost of ~50-100ms additional latency.
Production Checklist for Vector Database Deployment
- [ ] Defined chunking strategy and chunk sizes for your document types
- [ ] Embedding model selected and version locked (embedding model upgrades require re-indexing everything)
- [ ] Metadata schema defined (what filterable fields do you need?)
- [ ] Index backup strategy (periodic snapshots or export → S3)
- [ ] Latency SLAs defined (what's acceptable P99 for your search endpoint?)
- [ ] Monitoring: P50/P99 latency, recall benchmarks, index size growth
- [ ] Re-indexing strategy for when source documents update
Rutagon builds production RAG systems and AI search infrastructure for engineering teams. Contact us to discuss your vector database architecture.
Frequently Asked Questions
What embedding model should I use for RAG in 2026?
OpenAI text-embedding-3-large (3072 dimensions, can be reduced) performs well across most domains and has a simple API. For on-premises or cost-sensitive workloads, BAAI/bge-large-en-v1.5 and E5-large-v2 are strong open-source alternatives with competitive performance. Always benchmark on your specific domain's data — generic benchmarks don't predict domain-specific performance.
How do you handle embedding model version upgrades?
Changing embedding models requires re-embedding all your source documents with the new model, because embeddings from different models are not comparable in the same index space. Plan this as a migration: index the new embeddings in parallel, validate quality, then cut over. For large corpora (millions of documents), this is a significant engineering operation.
What's the cost difference between Pinecone and self-hosted Weaviate at scale?
At 50M vectors: Pinecone at ~$700+/month (pod-based pricing) vs self-hosted Weaviate on 3x r5.2xlarge ($1,100/month on-demand, ~$600/month 1-year reserved). At this scale, they're broadly comparable in cost. For 5M vectors or fewer, Pinecone's developer tier is significantly cheaper than provisioned self-hosted infrastructure. At 200M+ vectors, self-hosted becomes considerably cheaper.
Should I use a specialized vector database or stick with pgvector?
If you're already on Postgres and your scale is under 5M documents, start with pgvector — it's the lowest friction path. If you anticipate scaling beyond 10M vectors, have multi-tenancy requirements, or need hybrid search quality beyond what pgvector provides, plan for a dedicated vector database from the start. Migration from pgvector to Pinecone or Weaviate is straightforward (re-embed and re-index), but it's work.
How do you ensure retrieval quality in production?
Monitor retrieval quality with a labeled evaluation set: 100-200 question-answer pairs where you know the correct source document. Track "hit rate" (correct document in top-K results) over time. Regressions in hit rate (which can happen after embedding model changes, chunking changes, or data drift) are how you detect quality degradation before users notice.