Asking a fast decision tool three times in shuffled order kept more of its right answers; plain reshuffling gets most of the gain

aWhat the model normally does

Jev gets a list of options, gives each a probability, and the top one wins

Jev is TypeSafe’s fast decision model. You send it a question with a list of options, and it returns a probability for each one. The highest probability is the pick. It is quick and very cheap, $0.042 per million input tokens, which makes it a natural engine for high-volume choices. Cruxes uses it the normal way, one call per decision card, to suggest a first pick.

The trouble is that the pick can change when the same options are listed in a different order. On 26 October, Song, Zheng, Ma and Zhou at Tencent (arXiv 2610.35826) showed this with a reorder test. Each question is shown eight times: in its original order, reversed, and in six random orders. The score is how many first-time right answers survive all eight. They report 73.2–74.5% for Jev and 77.2–90.2% for four frontier chat models.

We rebuilt that test on 800 held-out questions from SuperGPQA and LEXam. Jev, asked once J0, kept 64.1% [59.5, 69.0] of its right answers. Part of that is plain randomness. In a calibration run on 198 GPQA Diamond questions, reordering lost 20.7% [14.4, 27.3] of first right answers, and asking the same question in the same order eight times lost 9.6% [4.9, 14.6].

bExactly what Cruxes changes

Ask three times in shuffled order, average, and break a close tie head-to-head

Cruxes ReAsk J6 changes nothing inside Jev. It changes how many times Jev is asked and how the answers are combined. Every call still sends the full question through the ordinary choice API.

1. Ask once in the order given. This is call 1, the same call Jev normally gets. Here it gives the runner-up .49 and the right answer .32, so Jev alone picks wrong.
2. Ask twice more with the options shuffled. Each shuffle comes from a fixed seed. Jev never sees which order it got before.
3. Average by option. Probabilities are averaged by the option’s content, not its letter, so a moved option keeps its score. Here: right answer .43, runner-up .30.
4. Check if it is still close. If the top two are less than 0.5 apart, and they are the same pair call 1 was torn between, the decision is still close. The 0.5 threshold was fixed on the tune half before the test half was read.
5. Break the tie head-to-head. Jev is asked a two-option choice between just those two, once in each order. The two answers are averaged with the three-call average, rescaled to the pair. Here the final average gives the right answer .66.
6. Spend extra calls only where needed. If the three-call average is not close, it stands. Across the test half the re-ask averaged 3.2 calls per decision.

Across this question’s eight presentations, Jev asked once was right in 2. Cruxes ReAsk was right in all 8, with the head-to-head firing every time.

In the product: research only. Cruxes Rank asks Jev one choice question per card and shows the top option as the agent’s pick (apps/crux-chat/src/lib/jev-pick.ts, repository commit a8bab20a3). JV-1 recommends shuffle-averaged calls behind a flag, three plain shuffles or the full re-ask, chosen by latency budget.

cWhat we compared it with

The control spends the same calls without the Cruxes framing

• Jev, asked once. One call per presentation, the normal use.
• Shuffle-average ×k J1(k). Call 1 plus k−1 shuffled calls, averaged, for k from 2 to 6. This draws the price-for-steadiness curve every other arm is read against.
• Matched shuffles J6c. The same trigger as Cruxes ReAsk on the same questions, with the two head-to-head calls replaced by two more full-menu shuffles. Same calls, item by item. This isolates whether narrowing to the contested pair helps.
• Cruxes ReAsk · Duel J3 against its own matched shuffles J3c. The head-to-head straight after call 1, with no averaging first. This was the registered primary test of the Cruxes idea.
• Cruxes ReAsk · Crux-State J4 against a Neutral state J4c. One extra call whose instruction names the two contested options (“The decision hinges on two candidates: «X» and «Y». Compare them on what distinguishes them before choosing.”), against “Check the choice carefully before choosing.”

The paper’s frontier band comes from the paper’s own questions, which were not released, so it is a descriptive comparison, not a ranking.

dWhat we measured

Three shuffled asks plus a tie-break kept 83.4% of right answers; plain reshuffling reaches 81.2%

Cruxes ReAsk kept 83.4% [79.7, 86.9] of Jev’s right answers through eight reorderings, at 3.2 calls per decision. That sits inside the 77.2–90.2% the paper reports for four frontier chat models, measured on a different sample of questions. Against five plain shuffles, the difference in right answers lost was −2.7 points [−6.9, +1.4] at 0.64 times the calls, which meets the no-worse margin fixed in advance. It gave up 1.2 accuracy points [−2.1, −0.2] against five orders.

Why it works: randomness averages out

Jev is not deterministic. In our calibration run, about half (46%) of what looks like order bias appeared with no reordering at all. The paper’s own figures put that share at 25–32%, so treat our half as true of our run only. Random wobble shrinks when you ask more than once. Plain averaging over five orders cut the share of right answers lost to 19.2%. Re-asking only where the averaged answer was still close was no worse, at 3.2 calls instead of 5.

The Cruxes framing added nothing on Jev. Spending the same calls on more plain shuffles kept about as many right answers and was slightly more accurate. The gain comes from re-asking in shuffled orders, which the Cruxes harness does. The crux wording adds nothing here.

Two smaller question sets

The pattern held on GPQA Diamond (106 test questions, seen early during a scorer test), where the share of right answers lost fell from 18.5% to 10.7%, and on MMLU-Pro (147 questions), where it fell from 9.2% to 3.1%. Every Cruxes-against-control contrast reported there includes zero.

Cost and speed

A Jev call costs about $0.00002. The re-ask averages 3.2 calls, so a steadier decision costs well under a hundredth of a cent, and the calls can be sent in parallel. In a three-call probe, one Jev call took 0.24 to 1.61 seconds end to end, of which 44 to 69 milliseconds was Jev’s own server time. We have not timed the re-ask end to end.

Where it helps

Where steady, cheap choices pay off