How the ranking works

VectorBench ranks AI models by one thing: when people see two drawings of the same prompt and don’t know who made them, which do they prefer?

The matchup

Every model gets the same prompts and answers each with an SVG. A matchup is two of those drawings for one prompt, from two different models. Which drawing lands on the left is a coin toss, and the model names stay hidden until you vote.

You can say A is better, B is better, both are good, or both are bad. After you vote we show which model drew each one. Models with fewer votes come up more often so new arrivals settle quickly, and you’re never dealt a pair you’ve already judged.

The rating

Ratings come from a Bradley-Terry model, the same statistical method LM Arena uses, reported on the familiar Elo scale. It finds the set of ratings that best explains every vote at once, so the result doesn’t depend on the order votes arrived in and no vote counts more than another.

The scale is centred on 1000. A gap of 400 points means the stronger model is preferred ten times out of eleven; a gap of 100 means about 64% of the time. “Both good” and “both bad” each count as half a win for either side.

The interval and the rank

Each rating comes with a 95% interval, drawn as a beam on the leaderboard. We get it by refitting the ratings on 100 resampled copies of the votes and seeing how far each rating moves. A short beam means the votes agree; a long one means we need more of them.

A model’s rank is one more than the number of models that are confidently better than it, meaning their whole interval sits above its own. Two models whose beams overlap share a rank until the votes separate them. A model needs 50 votes before it’s ranked at all.

Submitted drawings

Anyone can submit a drawing for one of our prompts. We check each one before it joins the arena: it has to answer the prompt, come from the model named, and be the model’s unedited output. If you add your X or Discord, the drawing is credited to you whenever its model is revealed.

Keeping votes honest

Votes only count for a matchup we dealt, once, from the browser we dealt it to. We limit how fast one visitor can vote and discard votes cast faster than anyone could look. We keep a random ID in a cookie and a scrambled form of your network address to do this, and nothing else about you.

Spotted something wrong? Email us.