Saddle Point

Chapter 8 · begins on page 307

Many Agents

The chess automaton back to it on the cover
Try it yourself Learning to play Does the average of best replies become unexploitable, and what certifies it? Open the engine →

The models

Each model in the book is stated on one card: who decides, when, with what information and commitment, what they get, what they may do, and the question it answers.

Model A1 · p. 330

Does local bidding reach the centrally optimal assignment, and how would one know?

Model A1 · Chapter 8 · page 330

Does local bidding reach the centrally optimal assignment, and how would one know?

Assignment by auction; Model A1


1Decision-makers
four agents; the price board is a mechanism, not a player
2Time
bidding rounds until no agent is unassigned
3Information
each agent: its own values and the public prices; nothing else
4Commitment
the auction's rules are fixed; bids are binding
5Payoffs
team objective: the total value of the assignment
6Strategies
the myopic bid prescribed by the protocol

The analysis contract

(i) concept
the protocol's outcome, certified against the exact optimum
(ii) deviations
none inside the protocol; the certificate quantifies over all assignments
(iii) beliefs
none needed
(iv) selection
the unassigned agent of lowest index bids next; ties between tasks broken by lowest index
(v) attainment
exact; terminating; the duality gap is computed, not estimated
In the book: Chapter 8, Many Agents, page 330

Model A2 · p. 332

What do the agents agree on, how fast, and who controls the answer?

Model A2 · Chapter 8 · page 332

What do the agents agree on, how fast, and who controls the answer?

Synchronous averaging; Model A2


1Decision-makers
four agents on a graph; no centre
2Time
synchronous rounds, computed
3Information
each agent sees only its neighbours' current values
4Commitment
the update rule is fixed in advance
5Payoffs
team objective: agreement on the average (the stated intent)
6Strategies
the linear update; the stubborn agent's constant is a deviation from it

The analysis contract

(i) concept
the dynamical system's limit, identified by exact invariants
(ii) deviations
one: the stubborn agent (c)
(iii) beliefs
none
(iv) selection
none
(v) attainment
exact iterates; limits identified exactly; rates exact in (a), horizon-certified in (c)
In the book: Chapter 8, Many Agents, page 332

Model A3 · p. 333

Does the play's average become unexploitable, and what certifies the answer?

Model A3 · Chapter 8 · page 333

Does the play's average become unexploitable, and what certifies the answer?

Discrete fictitious play; Model A3


1Decision-makers
two players; the “learning algorithm” is their stated rule, not a third agent
2Time
steps computed, with checkpoints
3Information
each sees the full history of the opponent's actions
4Commitment
none; the rule is myopic by construction
5Payoffs
zero-sum in the first game; not in the second
6Strategies
fictitious play with the stated tie rule

The analysis contract

(i) concept
exploitability of the empirical mixtures (zero-sum); NashConv (general sum)
(ii) deviations
exact best responses; the measured gap is exact, not a lower bound
(iii) beliefs
the empirical counts, by definition of the dynamic
(iv) selection
lowest-index ties, stated
(v) attainment
exact at every checkpoint; limits cited (Robinson), non-convergence exhibited
In the book: Chapter 8, Many Agents, page 333Run this model →

Model A4 · p. 335

How much adaptation does a finite memory buy, alone and collectively?

Model A4 · Chapter 8 · page 335

How much adaptation does a finite memory buy, alone and collectively?

Tsetlin automaton and the Goore game; Model A4


1Decision-makers
one automaton against a stationary environment; three in the Goore game
2Time
stationary distributions of the induced Markov chains
3Information
own reward only; no communication
4Commitment
the transition rule is the design
5Payoffs
rewards as stated; the Goore curve is unimodal with an interior optimum
6Strategies
the automaton's rule; depth is the design parameter

The analysis contract

(i) concept
exact stationary distributions (birth–death chain; the joint -state chain)
(ii) deviations
none; the rule is fixed
(iii) beliefs
none, by design
(iv) selection
the chains are irreducible; the stationary distribution is unique
(v) attainment
exact (rational); the asymptotic-optimality limit cited, not computed
In the book: Chapter 8, Many Agents, page 335

Model A5 · p. 337

Is the plan optimal and provably so; what does losing one unit cost; whose assessment does the alliance end up holding?

Model A5 · Chapter 8 · page 337

Is the plan optimal and provably so; what does losing one unit cost; whose assessment does the alliance end up holding?

Patrol assignment and assessment pooling; Model A5


1Decision-makers
five members; the price board and the pooling rule are mechanisms, not players
2Time
(a) bidding rounds until every unit is assigned; (b) synchronous rounds of pooling
3Information
(a) each member: its own unit's values and the public prices; (b) each member: the current assessments of those it listens to
4Commitment
the protocols are fixed in advance; a stubborn member deviates from (b), a withholding member from (a)
5Payoffs
(a) team objective: total detection value; (b) none optimised: the object is what the protocol computes
6Strategies
the protocols' prescribed bids and updates; withholding and stubbornness as the stated deviations

The analysis contract

(i) concept
(a) the auction's outcome with its duality certificate; (b) the limit of the linear dynamics
(ii) deviations
withholding each unit in turn; one and two stubborn members
(iii) beliefs
none; the assessments are inputs, not posteriors
(iv) selection
lowest index at bidding ties; the optimum is unique
(v) attainment
exact; limits identified by invariants and proved by exact contraction certificates
In the book: Chapter 8, Many Agents, page 337

What is proved, in words

The book follows each result with one sentence that says what it means without symbols. The formal statement is a click away; the proofs are in the book.

Proposition 8.2 · The auction's outcome, with a certificate; Model A1 · p. 331

Agents bidding for tasks reach the best assignment, and the final prices, with the profits they imply, are the proof: with integer values, a duality gap below one leaves no room for a better assignment.

Proposition 8.2 · Chapter 8 · page 331

The auction's outcome, with a certificate; Model A1

In words Agents bidding for tasks reach the best assignment, and the final prices, with the profits they imply, are the proof: with integer values, a duality gap below one leaves no room for a better assignment.


At the auction terminates in rounds at the assignment of value , with final prices . The prices, together with the profits that each agent can compute from them, certify the outcome: the pair is dual-feasible ( for all , checked exactly), so every assignment's value is at most ; -complementary slackness bounds that dual objective by the achieved value plus ; here the gap is exactly , and with integer values a gap below makes the assignment exactly optimal. Enumeration of all assignments confirms the optimum and its uniqueness. The increment is not free: on the matrix with a single nonzero value , the auction at returns an assignment of value against the optimum : the guarantee degrades to , and an increment larger than the differences that matter loses them.

The proof is in the book: Chapter 8, Many Agents, page 331

Proposition 8.4 · Consensus values and rates; Model A2 · p. 332

Repeated local averaging converges on these three graphs, but whose average it reaches depends on the weights: the true mean under balanced weights, a hub-weighted mean otherwise, and a stubborn agent's own value if one never listens.

Proposition 8.4 · Chapter 8 · page 332

Consensus values and rates; Model A2

In words Repeated local averaging converges on these three graphs, but whose average it reaches depends on the weights: the true mean under balanced weights, a hub-weighted mean otherwise, and a stubborn agent's own value if one never listens.


(a) Under the Metropolis weights the sum is invariant (the matrix is doubly stochastic) and the disagreement contracts by the factor exactly at every step (, checked exactly for ): the matrix's eigenvalues are with the discrete Fourier eigenvectors, verified by exact multiplication. Every agent converges to the true average .

(b) Under the hub-and-leaves weights the matrix is row- but not column-stochastic; its exact left fixed vector is , the quantity is invariant (checked exactly at every computed step), and the agents agree on : the hub's number counts double, by design of the weights and of nothing else. (c) With agent 1 stubborn, the fixed-point space of the update is exactly the constants (, computed exactly), the maximum is non-increasing and the minimum non-decreasing (row-stochasticity), agent 1's value never moves, and every agent converges to the stubborn agent's initial value : with one has and , and since the first column of is , ; after steps the exact iterate is within of . The agent that refuses to average decides what everyone agrees on.

The proof is in the book: Chapter 8, Many Agents, page 332

Proposition 8.6 · Convergence and its failure, measured exactly; Model A3 · p. 334

Learning by best-responding to the opponent's history settles down in the zero-sum game (its measured exploitability falls at every checkpoint, and Robinson's theorem takes it to zero), and in Shapley's example stays far from equilibrium at every computed step, cycling for ever by Shapley's theorem.

Proposition 8.6 · Chapter 8 · page 334

Convergence and its failure, measured exactly; Model A3

In words Learning by best-responding to the opponent's history settles down in the zero-sum game (its measured exploitability falls at every checkpoint, and Robinson's theorem takes it to zero), and in Shapley's example stays far from equilibrium at every computed step, cycling for ever by Shapley's theorem.


On the zero-sum game the exploitability of the empirical mixtures — , computed with exact best responses — falls from at steps through at and at to at , decreasing at every checkpoint; Robinson's theorem (Robinson 1951) gives convergence to , which no finite run certifies. On Shapley's example the analogous quantity (the sum of the two players' exact best-response gains against the empirical profile, written in Appendix B (p. 547)) exceeds at every step (its minimum on that horizon is , at ): the empirical frequencies cycle in ever-longer blocks, and that they converge to nothing is Shapley's theorem (Shapley 1964, §5.3), not a conclusion of the finite run. With the same dynamic and the same tie rule, one changed field, the payoffs' structure, separates the two outcomes.

The proof is in the book: Chapter 8, Many Agents, page 334

Proposition 8.9 · Memory buys expediency; the collective concentrates on the optimum; Model A4 · p. 336

Simple automata that remember only their recent rewards prefer the better action more strongly as their memory grows, and three of them together spend most of their time at a collective optimum that none of them can see.

Proposition 8.9 · Chapter 8 · page 336

Memory buys expediency; the collective concentrates on the optimum; Model A4

In words Simple automata that remember only their recent rewards prefer the better action more strongly as their memory grows, and three of them together spend most of their time at a collective optimum that none of them can see.


(i) Against penalties the exact stationary probability of the better action is at depths : increasing at every step and approaching (here the closed form is , from detailed balance, which tends to ; Tsetlin's general asymptotic optimality for a best action with penalty probability below (Tsetlin 1961) is cited, not proved). (ii) In the Goore game the joint chain over all automaton states, solved exactly, puts stationary probability above on exactly two yes-votes, the maximum of the reward curve, although no participant knows the curve, sees the vote, or communicates: the mode of the vote count sits at the optimum (Figure 8.3 (p. 314)b shows the exact distribution). This is stationary concentration, not arrival: the chain keeps returning to the other vote counts, and nothing here proves that deeper memory or more players would drive the mass to one.

The proof is in the book: Chapter 8, Many Agents, page 336

Proposition 8.11 · Prices certify the plan and bound each withdrawal; the weights decide the assessment; Model A5 · p. 338

The final prices of a patrol auction prove the plan optimal and, without re-solving anything, bound what each member's withdrawal would cost; and an alliance that pools assessments by averaging ends up holding a weighted average whose weights belong to the protocol, or to a member that never updates.

Proposition 8.11 · Chapter 8 · page 338

Prices certify the plan and bound each withdrawal; the weights decide the assessment; Model A5

In words The final prices of a patrol auction prove the plan optimal and, without re-solving anything, bound what each member's withdrawal would cost; and an alliance that pools assessments by averaging ends up holding a weighted average whose weights belong to the protocol, or to a member that never updates.


(a) At the auction ends after bids with the frigate in Z3, the aircraft in Z2, the cutter in Z1, the drone group in Z4 and the minehunter in Z5, value , final prices and duality gap : the plan is exactly optimal, and enumeration shows it is the unique optimum. In the uncoordinated benchmark all five units go to Z1 and the value is . At the auction stops at . (b) For every unit and every increment, with the auction's assignment, its value, the unit's profit at the final prices and the duality gap,

where is the optimum without unit ; when the certificate proves optimal, and the bounds bracket the exact cost of withholding (at , where the auction stops at , the bracket brackets and not : for the aircraft the latter is ). Here the exact costs of withholding are for the frigate, aircraft, cutter, drone group and minehunter, each inside its bound (Table 8.2 (p. 324)a); the cutter, which holds the most valuable zone, is the cheapest to lose. (c) Let be row-stochastic and . If for some , then has a unique left fixed probability vector , and for every : member controls the share of the pooled assessment. Here: equal weights give and the limit (); the lead-nation weights give (in general ) and the limit (); with M5 stubborn, under either weighting, is the fifth unit vector and the limit is ( and ). Only the lead-nation protocol reaches the threshold . The change in one member's assessment that would put the pooled view exactly at the threshold is under equal weights, where the view is below the threshold and the change is an increase. The other four members can make it; M1 cannot, since its assessment of can rise by at most , and at the maximal report the pool reaches only . Under the lead-nation weights the view is above the threshold, and a reduction of by M1, or of by any other member, brings it down to it; each of these is feasible. (d) With M1 and M5 both stubborn there is no agreement: the limits are , each flexible member ending at a mixture of the two stubborn views with weights on M1 for M2, M3, M4; only M2 among the flexible members ends above the threshold.

The proof is in the book: Chapter 8, Many Agents, page 338

The worked case

A parallax in history

1874–1875 · Chapter 8 · p. 326

One postal territory, many administrations

Then

Berne, 1874: twenty-one states form one postal territory, each keeping its own postal law.

Now

Eight Baltic Sea Allies undertake to exchange information on incidents; units and laws stay national (Section 8.7 (p. 319)).

1874–1875 · Chapter 8 · page 326

One postal territory, many administrations

Then

Berne, 1874: twenty-one states form one postal territory, each keeping its own postal law.

Now

Eight Baltic Sea Allies undertake to exchange information on incidents; units and laws stay national (Section 8.7 (p. 319)).


What happened

The treaty forming a General Postal Union was signed at Berne on 9 October 1874 by twenty-one states, among them Germany, Austria-Hungary, France, Great Britain, Russia, Turkey and the United States; ratifications were exchanged on 3 May 1875, and it came into force on 1 July 1875 (Art. 19). Article 1 made the contracting countries, “under the title of General Postal Union, a single postal territory for the reciprocal exchange of correspondence between their post-offices”. Article 14 provided that the treaty's stipulations “do not involve any alteration in the interior postal legislation of any country”. Article 15 organised an International Office, “a central office, which shall be conducted under the surveillance of a Postal Administration to be chosen by the Congress”, its expenses borne by all the contracting administrations, charged with collecting and distributing information on the international service, giving opinions on disputes when asked, and facilitating the international accounts D [General Postal Union 1874, Arts. 1, 14, 15, 19, the treaty text].

The model

Fields 1, 4 and 5 of Models A1 and A2, as the worked case reads them. The administrations remain the decision-makers (Article 14 keeps each one's row of the matrix its own); the Union is the protocol agreed in advance, field 4; and the International Office is a mechanism, not a player (the price board of Model A1, the ring of Model A2), which collects and circulates but does not decide. The treaty fixes an interface and a tariff. It defines no common objective, and the auction's certificate proves optimality only for an objective that has been agreed (Exercise 8.1 (p. 328)) R.

What could be known

In 1875 the Office's information travelled by the mails it served, disputes were settled by opinions given on request, and accounts were reconciled after the fact. A data-sharing network of the kind in the WIS2 situation above makes an observation available to every subscribed node through a common protocol as soon as it is published, and an assignment of units to requests can be recomputed as the requests change. What the faster interface changes is the resolution at which distributed action can be coordinated. What it leaves unchanged is that a common protocol is not a common preference, and that an agreement reached by averaging across the interface is a weighted value whose weights someone set (Exercise 8.3 (p. 329)) R.

Where the parallel stops

Postal interoperability is not consensus and not an optimal allocation; the treaty of 1874 was replaced by a convention of 1 June 1878; and the models' four or five agents are not twenty-one administrations.

In the book: Chapter 8, page 326All sixteen parallaxes on the timeline →

The people

Chapter 8 · Many Agents

Lloyd S. Shapley

1923 – 2016

Mathematician and game theorist. He received his doctorate from Princeton in 1953 under Albert W. Tucker, worked at the RAND Corporation from 1954 to 1981 and then at the University of California, Los Angeles. He introduced stochastic games and the value iteration for their discounted and stopping versions, the value for cooperative games that bears his name, and, with David Gale, the stable-matching algorithm; his 1964 example showed that fictitious play need not converge outside the zero-sum world. He shared the 2012 Nobel Memorial Prize in Economic Sciences with Alvin Roth for the theory of stable allocations and the practice of market design — not for any of the results used in this chapter.

The life, in moments →

Main works

  • “Stochastic games” (PNAS 39, 1953)
  • “A value for -person games” (1953)
  • “College admissions and the stability of marriage”, with D. Gale (American Mathematical Monthly 69, 1962)
  • “Some topics in two-person games” (1964)

Chapter 8 · Many Agents

Julia Robinson

1919 – 1985

Mathematician. She received her doctorate at Berkeley in 1948 under Alfred Tarski, remained associated with Berkeley and became professor there in 1976. Her 1951 paper proved that in every finite zero-sum game the minimax bounds computed from the fictitious-play frequencies converge to the value — the convergence used in this chapter; the actions themselves need not settle — and her work on Hilbert's tenth problem, with Martin Davis and Hilary Putnam, laid the ground for its negative solution by Yuri Matiyasevich in 1970. She was the first woman elected president of the American Mathematical Society (1983–84).

The life, in moments →

Main works

  • “An iterative method of solving a game” (Annals of Mathematics 54, 1951)
  • “The decision problem for exponential Diophantine equations”, with M. Davis and H. Putnam (1961)

Chapter 8 · Many Agents

Mikhail L. Tsetlin

1924 – 1966

Mathematician and biophysicist, a Moscow State University graduate of 1952, who worked in Moscow, from 1958 in Israel Gelfand's seminar on the boundary of mathematics and physiology. His 1961 paper pioneered the study of finite automata in random media — how trivially simple devices can adapt to unknown environments — and the games of collectives of such automata, including the Goore game proposed in his circle in 1963 and published by others in 1965, showed that collectives can concentrate on an optimum of a function none of them observes; the line was continued by Varshavsky's work on the collective behaviour of automata. His papers were collected posthumously (Nauka, 1969; English edition 1973).

The life, in moments →

Main works

  • “On the behaviour of finite automata in random media” (Avtomatika i telemekhanika 22(10), 1961, pp. 1345–1354)
  • Automaton Theory and Modeling of Biological Systems (Academic Press, 1973; Russian original, Nauka, 1969)