Creative Writing Bench V1

The human preferences used to train our reward model include judgments about originality. Its predictions may therefore favor stories that feel fresh and penalize writing that feels clichéd.

However, the benchmark does not independently verify whether a story’s wording or ideas are new relative to existing writing or a model’s training data. A high score indicates predicted reader preference. It does not establish that the story is novel or unique.

For two stories, we convert their reward-score difference into a preference probability:

Here, $i$ and $j$ identify an AI model or the human reference, $p$ identifies the prompt, and $r_{i,p}$ is the reward score for that response.

The comparison value $w_{i,j,p}$ is the estimated probability that readers would prefer response $i$ over response $j$. Equal reward scores give each response a 50% chance.

Refusals and empty responses are excluded from scoring and do not count as losses. For each prompt, the remaining contenders are scored against one another, without the excluded opponents.

For each prompt, we compare only the stories that were returned. We exclude refusals, empty responses, and self-comparisons.

$C_i$ is the set of eligible opponent-and-prompt comparisons for model or human reference $i$. A comparison is eligible only when both participants returned a story for that prompt.

We average these comparison values, giving each equal weight. Excluded comparisons do not contribute to the score or the number of comparisons used in the average.

Higher predicted win rates rank first:

In words: rank is one plus the number of higher predicted win rates. Exact ties share a rank, producing sequences such as 1, 2, 2, 4.

The professional and amateur views apply the same calculation to their respective prompt sets and human references.

A predicted win rate of 70% means an average comparison value of 0.70, across eligible comparisons. The percentage depends on the participating models, human reference, and prompts.