Three Rolls, One Decision, and 14/3

Here is the game. I roll a fair die and pay you the number showing. You may accept it and walk away, or reject it and have me roll again. You may reject twice. The third roll, if you get that far, is whatever it is: no more choices, you are paid the face. Nothing is staked and you cannot lose anything, so the only question is how much the right to say no is worth.

The reflex answer is 3.5. It is the average of a die, it is off by a lot, and the size of the error is the interesting part. The game is worth $14/3$, about 4.67, and the gap of $7/6$ is the price of the option you were handed for free.

What 3.5 is actually the value of

A single roll of a fair die averages

That number is correct, and it answers a different question. It is the value of this game with the stopping right deleted, which you can check against the two degenerate policies: never stopping early pays you the third roll, and always stopping on the first roll pays you the first roll. Both are worth $7/2$. So 3.5 is not a rounding of the answer or a different convention. It is the value of the game a passive player is playing.

Read the game from the last roll

Decisions in the middle of the game depend on what happens later, so there is no way to start at roll one. You start at the end, where there is nothing left to decide, and walk back. This is backward induction, and the object it produces at each stage is a single number: what the rest of the game is worth to you before you see the next face.

Because the continuation value is one number rather than a function of what you are looking at, the optimal policy is automatically a threshold policy. There is no cleverer rule waiting to be found, no pattern in the previous rolls to exploit. That point is worth making explicitly, because a search over threshold policies only tells you the best threshold, not that thresholds are the right shape of answer.

The last roll is forced, so $V_1 = 7/2$. Now suppose two rolls remain and you have just seen a face $f$. Rejecting it is worth $7/2$, so you keep anything above that, which means 4, 5 and 6, and you reject 1, 2 and 3. Averaging over the six equally likely faces:

Why a four is worth keeping later and rejecting now

With three rolls available, the number a face has to beat is no longer $7/2$ but $17/4$. Only 5 and 6 clear it. A four, which you were happy to accept with two rolls left, is now worth rejecting, because you still hold two more attempts and those two attempts are worth 4.25 between them.

People sometimes read that as an inconsistency. It is the whole content of the problem. A four is not intrinsically good or bad; it is good or bad relative to what you are giving up by taking it. The thresholds rise as the number of remaining rolls rises, because the continuation value rises, and the continuation value rises because more attempts can only help a player who is allowed to stop whenever he likes.

Note what the rule does not say. It does not say stop on 5 or 6 throughout. Applied blindly at every stage, that policy hands you the third roll far too often, and it is measurably worse than the correct one. The threshold has to move.

The value of the game

One more step of the same computation, now with four faces continuing at $17/4$ and two stopping:

Consistency is not optimality, so it is worth knowing that this has been checked the brute way as well. There are 49 threshold pairs available if you allow each of the two thresholds to be any value from 1 to 7, and evaluating all of them over all 216 equally likely outcomes gives a maximum of exactly $14/3$, attained by one policy and no other. The recursion and the exhaustive sweep are answering slightly different questions, and both answers are needed.

The general recursion, and its ceiling

Nothing above used the number three. For any number of allowed rolls the same argument gives a recursion in one line:

The sequence it produces is $7/2,\ 17/4,\ 14/3,\ 89/18,\ 277/54,\ 1709/324$ and onward. It increases, since $\max(X, V_k) \ge \max(X, V_{k-1})$ pointwise, and it is bounded above by 6, since a payment can never exceed the largest face. So it converges, and the limit has to satisfy the fixed-point equation, whose only solution below 6 would require a face above 6. The limit is 6 itself.

Once $V_k$ passes 5 the recursion becomes linear, because only the six clears the threshold:

The gap to 6 shrinks by a factor of $5/6$ per extra roll, so it decays geometrically and the early rolls do nearly all the work. Ten rolls put the value at about 5.65 and twenty at about 5.94, and it takes more than forty rolls before the gap to 6 falls below a thousandth. The right to keep asking is valuable at first and then stops being worth much, which is the shape almost every optimal stopping problem has.

Where the answer changes

Three assumptions are doing real work, and each one moves the answer when you relax it.

The calculation is risk neutral. It maximises the expected payment and treats a certain 4.25 as interchangeable with a coin flip between 2 and 6.5. A player who prefers certainty stops lower, and there is nothing wrong with that; it is a different objective function, not a mistake. The thresholds derived above are optimal for the expected value and for nothing else.

There is no cost to rolling. Charge $c$ per roll and equation (4) becomes $V_{k+1} = \mathbb{E}[\max(X, V_k)] - c$, which lowers every continuation value and therefore every threshold. Discounting each future roll by a factor $\beta < 1$ does the same thing multiplicatively. Push either far enough and the thresholds fall to the point where you accept the first roll whatever it is.

And the payment is the face you stop on, not the best face you saw. If the rules paid you the largest of three rolls, with no decision to make at all, the game would be worth $119/24$, roughly 4.96, which is $7/24$ more than $14/3$. Being forced to commit at the moment you see a face costs something real. So 4.67 is the value of this game played perfectly, and it is not the most a die can give you in three rolls.

This problem is the smallest honest example of a family that includes deciding when to accept a job offer and when to sell an asset you can only sell once. The mathematics is the same: a value function computed from the end backwards, and a rule that says stop when what you are looking at beats what you would get by carrying on.

Sources and further reading

• The general framework, and why the value function is computed from the end — Optimal stopping
• The method used in equations (2) to (4) — Backward induction
• The recursion (4) is a one-line Bellman equation — Bellman equation
• The continuous-time and infinite-horizon cousins — Secretary problem

Every rational number in this article is exact rather than rounded, and each one was checked twice: once by symbolic backward induction, and once by evaluating all 216 outcomes under every one of the 49 threshold policies.