--- name: vector-databases description: | Vector database integration for embeddings and similarity search. Pinecone, Weaviate, Qdrant, ChromaDB, pgvector. Index management, metadata filtering, hybrid search, and production optimization. USE WHEN: user mentions "vector database", "embeddings", "similarity search", "Pinecone", "Weaviate", "Qdrant", "ChromaDB", "pgvector", "HNSW", "ANN" DO NOT USE FOR: LangChain integration - use `langchain`; RAG architecture - use `rag-patterns`; traditional databases - use database skills allowed-tools: Read, Grep, Glob, Write, Edit --- # Vector Databases ## Pinecone ```typescript import { Pinecone } from '@pinecone-database/pinecone'; const pc = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! }); const index = pc.index('my-index'); // Upsert await index.namespace('docs').upsert([ { id: 'doc-1', values: embedding, metadata: { source: 'manual', topic: 'auth' } }, ]); // Query with metadata filter const results = await index.namespace('docs').query({ vector: queryEmbedding, topK: 5, filter: { topic: { $eq: 'auth' } }, includeMetadata: true, }); ``` ## ChromaDB (local/self-hosted) ```python import chromadb client = chromadb.PersistentClient(path="./chroma_db") collection = client.get_or_create_collection( name="documents", metadata={"hnsw:space": "cosine"}, ) # Add documents (auto-embeds with default model) collection.add( ids=["doc1", "doc2"], documents=["Auth guide content", "API reference content"], metadatas=[{"source": "manual"}, {"source": "api"}], ) # Query results = collection.query(query_texts=["how does login work?"], n_results=5) ``` ## pgvector (PostgreSQL extension) ```sql CREATE EXTENSION IF NOT EXISTS vector; CREATE TABLE documents ( id SERIAL PRIMARY KEY, content TEXT NOT NULL, embedding vector(1536), metadata JSONB DEFAULT '{}' ); CREATE INDEX ON documents USING hnsw (embedding vector_cosine_ops); -- Similarity search SELECT id, content, 1 - (embedding <=> $1::vector) AS similarity FROM documents WHERE metadata->>'source' = 'manual' ORDER BY embedding <=> $1::vector LIMIT 5; ``` ### Node.js with pgvector ```typescript import pgvector from 'pgvector'; await pgvector.registerTypes(pool); await pool.query( 'INSERT INTO documents (content, embedding) VALUES ($1, $2)', [text, pgvector.toSql(embedding)] ); const { rows } = await pool.query( 'SELECT *, 1 - (embedding <=> $1) AS similarity FROM documents ORDER BY embedding <=> $1 LIMIT $2', [pgvector.toSql(queryEmbedding), 5] ); ``` ## Qdrant ```typescript import { QdrantClient } from '@qdrant/js-client-rest'; const client = new QdrantClient({ url: 'http://localhost:6333' }); // Create collection await client.createCollection('documents', { vectors: { size: 1536, distance: 'Cosine' }, }); // Upsert await client.upsert('documents', { points: [{ id: 1, vector: embedding, payload: { source: 'manual' } }], }); // Search with filter const results = await client.search('documents', { vector: queryEmbedding, limit: 5, filter: { must: [{ key: 'source', match: { value: 'manual' } }] }, }); ``` ## Embedding Generation ```typescript import OpenAI from 'openai'; const openai = new OpenAI(); async function embed(texts: string[]): Promise { const response = await openai.embeddings.create({ model: 'text-embedding-3-small', // 1536 dims, cheapest input: texts, }); return response.data.map((d) => d.embedding); } ``` ## Anti-Patterns | Anti-Pattern | Fix | |--------------|-----| | No metadata filtering | Always store filterable metadata with vectors | | Wrong distance metric | Match metric to embedding model (cosine for OpenAI) | | Embedding model mismatch | Same model for indexing and querying | | No batching on upsert | Batch upserts (100-1000 vectors per call) | | Storing raw text in vector DB | Store text in primary DB, only IDs + vectors in vector DB | ## Production Checklist - [ ] Embedding model locked (changing requires full re-index) - [ ] Batch upserts with error handling - [ ] Metadata schema documented - [ ] Index type configured (HNSW for most cases) - [ ] Backup strategy for vector data - [ ] Monitoring: query latency, index size, recall metrics