BERT
Bidirectional Encoder Representations from Transformers
Google's transformer-based language model that read text in both directions (left and right of each word) using a "fill-in-the-blank" training objective. It topped 11 NLP leaderboards on release and became the default model behind Google Search ranking within a year.
Cost
Free
Open weights — self-host
Context
512
Up to 512 tokens
How are Intelligence, Speed & Cost bucketed?
Intelligence and Speed buckets are percentile ranks on
Artificial Analysis. Cost buckets are fixed dollar
thresholds keyed off output-token price ($/M out).
Intelligence
- Top 1%≤ 1%
- Top 5%≤ 5%
- Top 10%≤ 10%
- Good≤ 25%
- Medium≤ 50%
- Below avg> 50%
Speed
- Top 1%≥ 345 tok/s
- Top 5%≥ 237 tok/s
- Top 10%≥ 196 tok/s
- Good≥ 146 tok/s
- Medium≥ 90 tok/s
- Slow< 90 tok/s
Cost
- Freeopen weights · self-host
- Low< $1 / M out
- Moderate$1–5 / M out
- High≥ $5 / M out
Why it matters
BERT, not GPT, was the model that converted the broader tech industry to "transformers everywhere." Its open release set the norm that NLP weights would be public — a norm OpenAI later broke with GPT-3 to build a business.
Core Capabilities
Long Documents
Handles entire codebases, books, and multi-doc RAG.
Research
Foundational paper or scientific contribution.
Context Window
512 tokens
short prompt
4k Chat 聊天
32k Long docs 长文档
128k Books 整本书
400k Multi-doc 多文档
1M Codebase 整个代码库
10M
512
Availability
API
Not available
Product / App
Not available
Open Source
Released
Enterprise
—
Pricing Model
Free / self-host
Open weights — pay only for compute.
Self-host Capability / Performance
Where this model sits relative to the middle 60% of models in the tree. All scores are 0–10 (higher is better).
Lower 20% Upper 80% This model
Context / memory
Context window size · log-scaled
6.0
9.0
0.0
Lower 20% 20th percentile — 20% of models score below this This model Where the current model lands Upper 80% 80th percentile — only 20% of models score above this
Percentile boundaries are computed across every model in the tree that reports the underlying benchmark for each capability.
What it feels like
- First system to surpass human performance on SQuAD v1.1 — 93.2 F1 vs human 91.2
- Bidirectional encoder unlocked deep contextual representations — every NLP benchmark moved overnight
- Pre-train + fine-tune paradigm became the dominant training recipe for the next 4 years
- Google Search adopted BERT in 2019 — first transformer in production at planet scale
- Spawned RoBERTa, ALBERT, DistilBERT, and the entire encoder-only lineage that powers today's embeddings
- Encoder-only design was eventually overshadowed by decoder-only GPT family for generation tasks
Best use cases
- Sentence-pair classification, NER, span extraction (the original BERT use cases)
- Building production search and embedding pipelines pre-2024
- Foundation for understanding every modern encoder model (mPNet, BGE, E5, etc.)
- Teaching transformer fundamentals via fine-tuning labs
Tools to try
Not ideal for
- Generative tasks (chat, code, long-form writing) — use GPT-style decoder models instead
- Frontier embedding work — newer models like BGE-M3, OpenAI text-embedding-3, Voyage outperform