Embedding Models Compared: Choosing the Right One for Your Use Case
Benchmarking OpenAI, Cohere, and open-source embedding models across retrieval accuracy, cost, and latency for real-world RAG pipelines.
EmbeddingsAIBenchmarks
Public leaderboards rank models on other people's data. The only ranking that matters is the one produced by your queries against your corpus, which takes an afternoon to build.
Build a small, honest eval set
Fifty real queries with the chunk that should win, labelled by hand. That is enough to separate a good model from a bad one for your domain.
python
def recall_at_k(model, queries, k=5) -> float:
index = model.encode([c.text for c in corpus], normalize=True)
hits = 0
for q in queries:
scores = index @ model.encode([q.text], normalize=True).T
top = scores.ravel().argsort()[::-1][:k]
hits += any(corpus[i].id == q.gold_chunk_id for i in top)
return hits / len(queries)
for name, model in candidates.items():
print(f"{name:28} recall@5={recall_at_k(model, eval_queries):.2f}")
Cost is a dimension parameter
Vector storage and search cost scale with dimensionality. Matryoshka-style truncation lets you trade a point of recall for a much smaller index — measure it rather than assuming.
python
import numpy as np
def truncate(vecs: np.ndarray, dims: int) -> np.ndarray:
out = vecs[:, :dims]
return out / np.linalg.norm(out, axis=1, keepdims=True)
for dims in (1536, 1024, 512, 256):
print(dims, round(recall_at_k(TruncatedModel(dims), eval_queries), 3))
How I choose
–Domain jargon heavy (legal, medical, internal tooling): test open-source models fine-tuned on your pairs — they often beat general hosted models.
–Multilingual corpus: pick a model trained multilingually rather than translating queries.
–Latency-sensitive on-device or edge: a small local model at 384 dims frequently loses under two points of recall.
–Whatever you pick, pin the model version. Re-embedding a corpus is a migration, not a config change.
A re-ranker on top of a mid-tier embedding model beat every embedding upgrade I tested. Spend the budget there first.