Artificial Analysis Decision Index Benchmarking Methodology
Artificial Analysis Decision Index
The Artificial Analysis Decision Index measures the capabilities and performance of decision models across document decisions and extraction, tool calling, knowledge, hallucination, reasoning and invariance. Version 2.1 combines six evaluations: AA-Paperclip, decisions and extraction on resumes, receipts and invoices, and research papers; EvalSafe-O*NET-AA, based on TypeSafe's O*NET benchmark of judgements about workplace documents; When2Call, NVIDIA's tool-calling evaluation; AA-Omniscience-Decision v1.1, a private dataset adapted from the knowledge and hallucination benchmark AA-Omniscience; HLE-MultiChoice v1.1, the text-only multiple-choice questions from Humanity's Last Exam with an option to decline; and AA-Symmetry v1.1, which checks that a decision holds when its options are moved or reworded, or its question is negated.
The index scores the probabilities a model gives, not only its picks. It is the weighted average of the Decision Score of AA-Paperclip, EvalSafe-O*NET-AA, When2Call, AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1, and of the Symmetry Score of AA-Symmetry, since AA-Symmetry checks that forms agree rather than that answers are right. For completeness, each evaluation's headline metrics, such as its all-pass rate, accuracy and non-hallucination rate, are still reported, in the Accuracy section of its page and of the index page. The index page's Accuracy section opens with their weighted average, using the index's weights, with AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 each splitting their 15% into accuracy at 10% and non-hallucination rate at 5%.
Decision Index evaluation suite
The Decision Index is a weighted average of its six evaluations: five Decision Scores and AA-Symmetry's Symmetry Score.
| Evaluation | Capability | Metric | Weight |
|---|---|---|---|
| AA-Paperclip | Decision-making and extraction | Decision Score over each document's 15 answers as one decision | 30% |
| EvalSafe-O*NET-AA | Decision-making and judgement | Decision Score | 15% |
| When2Call | Tool calling | Decision Score | 10% |
| AA-Omniscience-Decision v1.1 | Knowledge and hallucination | Decision Score | 15% |
| HLE-MultiChoice v1.1 | Reasoning | Decision Score | 15% |
| AA-Symmetry v1.1 | Decision invariance | Symmetry Score | 15% |
Tasks and Answers
Each task is posed as a state, the text to decide about, and one or more questions about it. A question is a choice among named options, a score on a scale, or a yes/no question. A model answers all of a task's questions in a single call.
Decision Models
We send the state and questions to the model's decision API. For a choice, the model returns the option it picks and a probability for every option. We score the option it picks, which is the option it gives the highest probability. For a yes/no question, the model returns the probability of yes, and we take its answer as yes when that probability is 0.5 or more.
Language Models
We send a fixed system prompt and the state as text, with each question and its options in the structured-output schema, and ask for one JSON object that gives a probability for every option of each choice and every level of each score, summing to 1, and the probability of yes for each yes/no question. We ask for this with structured outputs, rescale probabilities that don't sum to 1, and ask once more after a reply that doesn't fit. We then score a language model as we score a decision model: its answer is the option or level it gives the highest probability, and yes when the probability of yes is 0.5 or more. Because language models now give probabilities, we measure their calibration and invariance as well. In version 1.0, language models named one option for each question and gave no probabilities.
Decision Score
The Decision Score combines whether a model's chosen answer is right with how sure it was. For a question where the model gave its chosen answer probability , and is 1 if that answer is right and 0 if not, the question scores:
- Range: a right answer scores from 50 to 100 and a wrong one from 50 down to 0: 100 when the model is certain and right, 0 when it is certain and wrong. Confidence moves the score within each half, so 50 is the line between right and wrong.
- Evaluation score: the mean over its questions, which is , where accuracy is the share of chosen answers that are right and the Brier score is the mean of . This accuracy can differ from the one in the Accuracy section, which, for AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1, counts a declined question as wrong and, for EvalSafe-O*NET-AA, weights the three question types equally.
Why the Decision Score:
- Every threshold: a decision model's answer is typically acted on when its confidence clears a threshold, and passed to a person or a more capable model otherwise. Answering when , a right answer earns and a wrong one costs , and passing it on scores 0. These payoffs make the break-even point: an answer that is right with probability is worth on average, so answering pays exactly when the model is more than sure. Averaged over every threshold from 0 to 1, that payoff is , so the score is , the formula above. Each use case sets its own threshold; the score covers them all rather than choosing one.
- Proper: the Brier term rewards a model for reporting its true probability that its chosen answer is right. Overconfidence and underconfidence both cost points in expectation, and one confident miss costs at most one question's worth.
How we apply it:
- The chosen answer: the option the model gives the highest probability, and yes for a yes/no question when its probability is 0.5 or more. A tie scores the average over a random tie-break.
- Baselines: a uniform guess is expected to score on a question with answers.
- Declining: in AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1, the probability a model puts on the option that declines to answer, in any of its wordings, is kept. The chosen answer is the most probable other option, even when declining is more probable, and is that option's own probability. Probability on declining therefore lowers , softening the penalty when the chosen answer is wrong and capping the reward when it is right. Declining outright gives and scores 50.
- Missing answers: a question that errored or went unanswered scores a uniform guess's expected , where is its number of options, not counting the option that declines. An answer without probabilities counts as certain of its pick.
- AA-Paperclip: each document is scored as one decision: is 1 only if all 15 answers are right, and is the model's confidence that they all are, the product of its probabilities for its 15 chosen answers. Each API gives every answer its own probabilities, so this treats a document's answers as independent. Because a document's confidence is a product of 15 probabilities, small per-answer miscalibration compounds. A model that knows it will miss a document scores near 50 on it; one that misses a document it was sure of scores near 0.
- EvalSafe-O*NET-AA: the right answer is the reference models' consensus, its most likely answer, as for accuracy.
Symmetry Score
AA-Symmetry asks each question in several forms that mean the same thing, and checks that the decision and the confidence behind it hold. For a question with options, where in form the model gives its most probable option probability :
- Margin: is the margin of form : how far sits above chance, , rescaled to run from 0 for a uniform guess to 1 for certainty, so questions with 2, 4 and 5 options are scored alike.
- Range: a question scores 1 when its most probable option and that option's probability never move, less the more the probability moves, and 0 when the option changes or a form is unanswered. "I don't know" counts as an option here, since keeping to it is a decision too.
- Score: AA-Symmetry's Symmetry Score is the mean over its questions, from 0 to 100. Its all-pass permutation rate is shown in the Accuracy section.
Calibration
- Brier score: the mean of over an evaluation's questions, the term the Decision Score subtracts: the squared gap between the probability of the chosen answer and whether it is right, with declining and AA-Paperclip's documents handled as in the Decision Score. Lower is better, and 0 is certain and always right.
- Index Brier score: the five evaluations' Brier scores averaged with the index's weights, rescaled to sum to 1.
- Calibration curve: we group a model's answers into 10 bins of width 0.1 by the probability it gave the answer it picked, and plot each bin's accuracy against its mean probability. A perfectly calibrated model's bins lie on the diagonal. Unlike the Brier score, it uses the model's own pick, so declined, errored and unanswered questions are not taken into account, and it counts each of AA-Paperclip's 15 answers separately.
- Index calibration: we pool the bins of the evaluations scored against a correct answer: AA-Paperclip, EvalSafe-O*NET-AA, When2Call, AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1, weighting each evaluation equally, for the index's calibration curve.
- Calibration by benchmark: each evaluation's calibration curve uses its own bins and is shown on its page. AA-Symmetry has no calibration curve and is left out of the index's pool, because it measures whether answers hold across forms, not whether they're right.
Artificial Analysis Decision Index Evaluations
The six evaluations in the Artificial Analysis Decision Index v2.1.
AA-Paperclip
- Description: 1,500 documents, each with screening decisions to make and facts to extract, asked as 3,000 requests. Included in the Artificial Analysis Decision Index at 30% weighting.
- Dataset: our private set of three sets of 500 documents: resumes, receipts and invoices, and AI and machine learning research papers.
- Tasks: each document is asked two tasks, as two requests: a decision-making task involving five decisions and an extraction task involving ten facts from the document. That makes 15 answers per document. A document longer than a model's context window is cut to fit.
- Scoring: Decision Score, each document scored as one decision: its all-pass against the product of the probabilities of its 15 chosen answers. The headline metric, shown in the Accuracy section, is the all-pass rate: the share of documents with all 15 answers right, the band, the knock-out, each criterion and every extracted fact.
- Calibration: the calibration curve, unlike the Decision Score and Brier score, counts each of a document's 15 answers separately.
EvalSafe-O*NET-AA
- Description: Based on TypeSafe's EvalSafe O*NET benchmark: judgements about synthetic workplace documents written for O*NET occupations. TypeSafe scores all 7,500 questions against the reference models' probabilities, and we score the questions below against their shared answer. Included in the Artificial Analysis Decision Index at 15% weighting.
- Original dataset: typesafeAI/evalsafe-onet (Apache-2.0)
- Dataset: 150 documents with 50 questions each: 27 yes/no questions, 16 scores and 7 choices, 7,500 questions in all. We ask the 6,582 questions where both reference models, GPT-6 Astra and Claude Fable 5.1, give the same most likely answer, and leave out the other 918: where they disagree, one has a tie, or one gave no answer. Each question is asked on its own, with the whole document.
- Reference: the questions have no ground truth. The reference answer is the mean of the probabilities GPT-6 Astra and Claude Fable 5.1 gave each option, and its most likely answer is the correct answer.
- Scoring: Decision Score against the reference's answer, averaged over all questions. The headline metric, shown in the Accuracy section, is accuracy: the share of questions where the model's answer is the reference's, averaged over the three question types so each counts equally. For a score, the right answer is the level nearest the reference's expected score. Accuracy compares that with the level nearest the model's expected score; the Decision Score compares it with the model's most probable level.
When2Call
- Description: The multiple-choice test set of NVIDIA's When2Call (CC BY 4.0). Each task is a conversation with an assistant that has tools, posed as one choice among four candidate replies: answer directly, call a tool, ask for more information, or say it cannot answer. The order of the replies is shuffled per task. Included in the Artificial Analysis Decision Index at 10% weighting.
- Version: the Artificial Analysis Decision Index v2.0 asks the same When2Call tasks as v1.0, run again so that language models answer with probabilities.
- Original dataset: nvidia/When2Call
- Dataset: We use a filtered set of 3,423 tasks, down from 3,652 in the original test set. We filter out 229 “ask for more information” tasks where there is ambiguity about whether more information is required. The filtered-out tasks are listed below.
Task ID live_multiple_0-0-0 live_multiple_8-4-0 live_multiple_9-4-1 live_multiple_10-4-2 live_multiple_11-4-3 live_multiple_12-4-4 live_multiple_14-4-6 live_multiple_15-4-7 live_multiple_16-4-8 live_multiple_17-4-9 live_multiple_18-4-10 live_multiple_19-4-11 live_multiple_20-4-12 live_multiple_21-4-13 live_multiple_22-4-14 live_multiple_23-5-0 live_multiple_25-6-0 live_multiple_26-6-1 live_multiple_29-9-0 live_multiple_30-10-0 live_multiple_31-10-1 live_multiple_32-10-2 live_multiple_36-12-0 live_multiple_41-16-0 live_multiple_42-16-1 live_multiple_45-18-0 live_multiple_48-18-3 live_multiple_53-22-0 live_multiple_55-22-2 live_multiple_62-24-0 live_multiple_68-29-0 live_multiple_72-32-0 live_multiple_74-34-0 live_multiple_78-35-2 live_multiple_89-39-0 live_multiple_105-43-3 live_multiple_111-43-9 live_multiple_124-47-0 live_multiple_125-47-1 live_multiple_127-49-0 live_multiple_128-50-0 live_multiple_129-50-1 live_multiple_130-50-2 live_multiple_131-50-3 live_multiple_132-50-4 live_multiple_133-50-5 live_multiple_135-51-1 live_multiple_136-52-0 live_multiple_137-52-1 live_multiple_138-53-0 live_multiple_139-53-1 live_multiple_140-54-0 live_multiple_141-54-1 live_multiple_142-55-0 live_multiple_144-56-0 live_multiple_146-58-0 live_multiple_148-58-2 live_multiple_149-58-3 live_multiple_150-58-4 live_multiple_151-58-5 live_multiple_152-58-6 live_multiple_154-58-8 live_multiple_155-58-9 live_multiple_157-60-0 live_multiple_159-62-0 live_multiple_176-73-0 live_multiple_182-77-0 live_multiple_183-78-0 live_multiple_189-83-0 live_multiple_193-87-0 live_multiple_194-87-1 live_multiple_196-88-0 live_multiple_197-89-0 live_multiple_206-91-0 live_multiple_210-91-4 live_multiple_213-91-7 live_multiple_216-92-0 live_multiple_219-94-1 live_multiple_220-94-2 live_multiple_223-97-0 live_multiple_224-98-0 live_multiple_239-107-0 live_multiple_240-107-1 live_multiple_242-107-3 live_multiple_243-107-4 live_multiple_255-119-0 live_multiple_257-121-0 live_multiple_263-126-0 live_multiple_264-126-1 live_multiple_279-128-2 live_multiple_284-128-7 live_multiple_337-133-1 live_multiple_377-135-1 live_multiple_400-139-4 live_multiple_433-141-22 live_multiple_440-144-0 live_multiple_442-144-2 live_multiple_443-144-3 live_multiple_444-144-4 live_multiple_514-150-2 live_multiple_515-150-3 live_multiple_516-150-4 live_multiple_541-152-3 live_multiple_546-152-8 live_multiple_568-155-8 live_multiple_579-157-0 live_multiple_637-161-5 live_multiple_651-161-19 live_multiple_652-161-20 live_multiple_668-162-10 live_multiple_759-169-14 live_multiple_780-173-2 live_multiple_784-173-6 live_multiple_819-177-2 live_multiple_821-177-4 live_multiple_833-178-8 live_multiple_834-178-9 live_multiple_836-178-11 live_multiple_837-178-12 live_multiple_842-178-17 live_multiple_850-179-4 live_multiple_870-182-7 live_multiple_940-194-2 live_multiple_942-196-0 live_multiple_946-196-4 live_multiple_952-201-0 live_multiple_955-202-1 live_multiple_958-204-0 live_multiple_959-204-1 live_multiple_962-205-2 live_multiple_964-207-0 live_multiple_965-207-1 live_multiple_966-208-0 live_multiple_967-209-0 live_multiple_971-212-0 live_multiple_972-212-1 live_multiple_980-215-4 live_multiple_983-215-7 live_multiple_1012-241-0 live_multiple_1013-242-0 live_multiple_1015-244-0 live_multiple_1029-257-0 live_multiple_1031-259-0 live_simple_15-3-11 live_simple_22-5-0 live_simple_23-5-1 live_simple_24-5-2 live_simple_25-5-3 live_simple_28-7-1 live_simple_34-11-0 live_simple_38-15-0 live_simple_40-17-0 live_simple_41-17-1 live_simple_42-17-2 live_simple_43-17-3 live_simple_44-18-0 live_simple_45-18-1 live_simple_48-21-0 live_simple_51-23-0 live_simple_52-23-1 live_simple_53-24-0 live_simple_54-25-0 live_simple_56-26-0 live_simple_60-29-0 live_simple_62-29-2 live_simple_65-29-5 live_simple_71-35-0 live_simple_72-36-0 live_simple_85-46-0 live_simple_98-58-0 live_simple_101-60-0 live_simple_105-62-0 live_simple_106-63-0 live_simple_107-64-0 live_simple_108-65-0 live_simple_112-68-0 live_simple_118-74-0 live_simple_120-76-0 live_simple_123-79-0 live_simple_141-94-0 live_simple_142-94-1 live_simple_143-95-0 live_simple_145-95-2 live_simple_146-95-3 live_simple_147-95-4 live_simple_149-95-6 live_simple_150-95-7 live_simple_151-95-8 live_simple_152-95-9 live_simple_154-95-11 live_simple_155-95-12 live_simple_156-95-13 live_simple_157-95-14 live_simple_158-95-15 live_simple_159-95-16 live_simple_160-95-17 live_simple_162-96-1 live_simple_163-96-2 live_simple_164-97-0 live_simple_166-99-0 live_simple_167-99-1 live_simple_168-99-2 live_simple_170-99-4 live_simple_171-99-5 live_simple_172-99-6 live_simple_173-99-7 live_simple_174-100-0 live_simple_175-101-0 live_simple_176-102-0 live_simple_177-103-0 live_simple_178-103-1 live_simple_180-105-0 live_simple_181-106-0 live_simple_182-107-0 live_simple_183-108-0 live_simple_185-110-0 live_simple_186-111-0 live_simple_188-113-0 live_simple_190-115-0 live_simple_229-120-0 live_simple_230-121-0 live_simple_237-125-0 live_simple_240-125-3 live_simple_241-125-4 live_simple_242-126-0 live_simple_249-131-0 live_simple_250-132-0 live_simple_256-137-0 live_simple_257-137-1
- Scoring: Decision Score over the four replies. The headline metric, shown in the Accuracy section, is accuracy: the share of tasks answered with the reply of the correct category.
AA-Omniscience-Decision v1.1
- Description: The 500 private AA-Omniscience questions of AA-Omniscience-Decision, each a choice among five options: four answers, one of them correct, and an option that declines to answer. Included in the Artificial Analysis Decision Index at 15% weighting.
- Original dataset: AA-Omniscience
- Dataset: The questions, answers A to D and correct answers are those of AA-Omniscience-Decision v1.0. The declining option is worded in different ways: "I don't know" and variants such as "It would be best to consult a human expert".
- Scoring: Decision Score: the chosen answer is the most probable of the four answers, at its own probability, with the declining option's probability kept. The headline metrics, shown in the Accuracy section, are as in v1.0. Accuracy is the share of questions answered correctly. Non-Hallucination Rate is , where Hallucination Rate is the share of questions not answered correctly that were answered wrongly rather than declined or left unanswered.
HLE-MultiChoice v1.1
- Description: The text-only multiple-choice questions from Humanity's Last Exam, each with an option that declines to answer. Included in the Artificial Analysis Decision Index at 15% weighting.
- Dataset: cais/hle. We use 509 of the 513 questions in HLE-MultiChoice v1.0. The 4 listed below already use every letter from A to Z, so they have no letter left for the declining option. The declining option is the next letter after a question's last option, worded in the same ways as AA-Omniscience-Decision v1.1.
Task ID 66fc2689b72463921f6c40e8 6702bc770065c2f50235a34e 671aad23ce6e97a55a4d0d47 6722f2b2f3aeb8d4f9cf83ab
- Scoring: As AA-Omniscience-Decision v1.1: Decision Score, the chosen answer the most probable lettered answer, at its own probability, with the declining option's probability kept. The headline metrics, shown in the Accuracy section, are accuracy, an exact match on the answer letter, and non-hallucination rate, where declining or leaving a question unanswered is not a hallucination.
AA-Symmetry v1.1
- Description: Tests whether a decision survives changes to its question that should not change it. Included in the Artificial Analysis Decision Index at 15% weighting.
- Version: v1.1 asks the 120 questions of v1.0 in new forms. v1.0 asked them as 440 forms, 80 of the questions with their options in a different order or with the labels moved between options. Those forms become identical if an API sorts a question's options by their text before its model reads them, so v1.1 replaces them.
- Dataset: 120 questions from When2Call, WildGuardTest and AA-Omniscience-Decision v1.1, asked as 460 forms that mean the same thing.
- Options in the state (40 questions): the options are written at the end of the state, in a different order and list style in each form. The question gives only their labels.
- Options reworded (40 questions): the options are reworded whilst keeping the exact same meaning, so the answer and its confidence should not change.
- Question negated (40 questions): a yes/no question is also asked as its negation. These are unchanged from v1.0.
- Scoring: Symmetry Score: per question, 0 if the most probable option changes between forms, otherwise 1 less the range of its probability above chance across the forms, averaged from 0 to 100. The headline metric, shown in the Accuracy section, is the all-pass permutation rate: the share of questions where every form gives the same option the highest probability and, once each form's options are mapped back to the original, every option's probability is within 5 percentage points across the forms.
Previous Evaluations
WildGuardTest was in version 1.0 only. Its page still shows version 1.0 results, from language models that named one option and gave no probabilities.
WildGuardTest
- Description: The test set of Ai2's WildGuardMix. We ask up to three yes/no questions about each row: is the user's request harmful, is the assistant's reply a refusal, and is that reply harmful. Included in the Artificial Analysis Decision Index v1.0 at 25% weighting.
- Original dataset: allenai/wildguardmix (ODC-BY)
- Dataset: We use 835 of the original 1,725 test rows. We keep rows where all three human annotators agreed on all three labels (prompt harm, response refusal, and response harm), based on the annotator agreement fields on Hugging Face, and exclude questions that consistently trigger model refusals to better isolate model capability.
- Scoring: The average of three F1 scores: prompt harm, response harm and refusal.
Version History
Version 2.1
November 2026 to current
- Updated AA-Symmetry to v1.1. This takes the same original questions (before variation) and changes two variation forms to preserve benchmark integrity:
- Order variation (40 questions): We now put the option descriptions in the input
state, with the question giving only the labels. Question option ordering is still changed between variations - Options reworded (40 questions): Rather than moving the labels between options, we now change the wording of options slightly between variations. Option meaning stays the exact same and wording updates are small
- Question negated (40 questions): Unchanged from v1.0
- Order variation (40 questions): We now put the option descriptions in the input
Version 2.0
November 2026
- Scored each evaluation with a right answer by the Decision Score, and AA-Symmetry by its Symmetry Score, from the probabilities a model gives
- Added Brier scores, for the index and each evaluation scored by the Decision Score
- Added AA-Paperclip (30%), EvalSafe-O*NET-AA (15%) and AA-Symmetry v1.0 (15%)
- Updated AA-Omniscience-Decision to v1.1, with "I don't know" worded ten ways, and reduced its weighting from 25% to 15%
- Updated HLE-MultiChoice to v1.1, with an option to decline and without the 4 questions that use every letter from A to Z, and reduced its weighting from 25% to 15%
- Reduced When2Call's weighting from 25% to 10%
- Removed WildGuardTest
- Language models now answer with a probability for every option
Version 1.0
October 2026 to November 2026
- Launched the Artificial Analysis Decision Index with four evaluations: AA-Omniscience-Decision, HLE-MultiChoice, When2Call and WildGuardTest