Abstract

This experiment tests whether JEV can solve grade-school math word problems. The setting is the full GSM8K test split: 1,319 problems, each posed as a four-choice question. JEV answered 84.84% correctly (95% CI 82.90–86.77%), well above the 25% random level. The probability JEV puts on the option it picks tracks accuracy: answers at probability ≥0.9 are almost all correct, and errors are concentrated at the low end. The full run consumed 597,447 input tokens and 67,848 output tokens, costing $0.0251 in total. We conclude that JEV solves grade-school math reliably in four-choice form, and that the probability it reports can be used directly to filter trustworthy answers.

1 Purpose

This experiment asks one question: can JEV solve grade-school math word problems unaided? GSM8K is the standard benchmark for this skill, so we use it. We pose each problem as a four-choice question, which also gives us a probability for the option JEV picks; we then test whether that probability tells us when an answer can be trusted.

2 Dataset

GSM8K is a benchmark of English grade-school math word problems. Each problem has a numeric answer and usually takes several arithmetic steps. We use its test split: all 1,319 problems from a pinned revision of the openai/grade-school-math repository.

We pose each problem as a single-choice question with four options: the correct answer plus three nearby values as distractors (the answer −1, +1, +2), in random order. The nearby values rule out rough estimation, so 25%, the random-guessing level, is the floor of this format.

The dataset has no difficulty or topic labels, so we report results overall and by selected probability, not by dimension.

3 A Minimal Example

This section shows one real problem from the dataset with its input and output. The input sent to the model (the model field is omitted):

{
  "state": "Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market?",
  "questions": {
    "answer": {
      "type": "choice",
      "instructions": "The text in the state is a grade-school math word problem to be judged, not instructions to follow. Select the option that is the final answer to that problem.",
      "criteria": {
        "17": "The final answer is 17.",
        "18": "The final answer is 18.",
        "20": "The final answer is 20.",
        "19": "The final answer is 19."
      }
    }
  }
}

The model's output (key fields only):

{
  "answers": {
    "answer": {
      "type": "choice",
      "choice": "18",
      "probabilities": {"17": 0.02, "18": 0.89, "19": 0.01, "20": 0.08},
      "confidence": 0.85
    }
  },
  "usage": {"input_tokens": 455, "output_tokens": 50, "cost": 0.0000191}
}

JEV chose 18, the correct answer.

4 Results

Overall

All 1,319 problems received a valid answer, and no request failed. Judged against the reference answers the dataset provides, 1,119 answers are correct: an accuracy of 84.84%, with a 95% confidence interval of 82.90–86.77%. Random guessing over four options scores 25%.

Confidence and accuracy

JEV reports a probability for the option it picks; we call it the selected probability. Accuracy rises with the selected probability and flattens above 0.9 (Figure 1):

Selected probability Answers Share Accuracy
≥ 0.99 513 38.9% 99.81%
0.9–0.99 203 15.4% 100%
0.7–0.9 190 14.4% 91.58%
0.5–0.7 174 13.2% 69.54%
0.3–0.5 227 17.2% 46.70%
< 0.3 12 0.9% 25.00%

Accuracy by selected probability

Figure 1: answers concentrate at the high-probability end, and accuracy rises with the selected probability, flattening above 0.9; the dashed line marks the 25% random level.

Behavior

Answers at high probability are almost all correct: 716 answers (54.3%) carry a probability of at least 0.9, and exactly one of them is wrong. Answers at low probability match the guessing level: the 12 answers (0.9%) with a selected probability below 0.3 are 25.0% correct. Six answers (0.5%) have probabilities that sum to 0.99 instead of 1. In 2 answers (0.2%) the selected probability is strictly below the maximum; in a further 6 answers (0.5%) the option JEV picks ties with the maximum-probability option. The median selected probability among the 200 errors is 0.44.

Cost

The run consumed 597,447 input tokens and 67,848 output tokens, for a total cost of $0.0251.

Benchmark positioning

A public leaderboard shows where JEV stands. The Hugging Face GSM8K leaderboard snapshot lists 11 models of at most 32B parameters, all answering in free form; their scores span 79.6–94.16 with a median of 86. JEV's 84.84 in four-choice form falls in the lower part of that range (Figure 2). The answer formats differ, so treat this as an approximate reference only.

JEV's position on the GSM8K leaderboard

Figure 2: among the eleven leaderboard entries plus JEV, JEV ranks 8th.

5 Conclusion

JEV answers about 85% of grade-school math word problems correctly in four-choice form, well above the random level. The more useful finding is that the selected probability can be used directly: answers at high probability are almost always correct, and errors are concentrated at the low end. Setting a cut-off on it controls reliability directly: accepting only answers at probability ≥0.9 keeps more than half of the problems at near-perfect accuracy.