AGAI - Flashcards
All 253 flashcards for Agentic Artificial Intelligence, one set per note. Each card asks one thing and has a short answer. Say the answer before you reveal it, and only press Got it if you had all of it. Study everything at once, a single lecture, or tick any mix. Progress lives only in this browser tab and disappears when you close it. The long-form questions are on the exam questions page.
Back to the Agentic Artificial Intelligence home.
L01 Agents, Multiagent Systems and Agentic AI
Flashcards
Give Russell and Norvig's definition of an agent.
Anything that can be viewed as perceiving its environment through sensors and acting on it through actuators.
Give Wooldridge's (2001) definition of an agent.
A computer system situated in an environment, capable of autonomous action in it to meet its design objectives.
What does Wooldridge's agent definition stress that Russell and Norvig's does not?
Autonomy and design objectives.
Why does a calculator show that "agent" is an analysis tool, according to Russell and Norvig?
Describing it as choosing to display “4” after “2 + 2 =” is possible but adds no understanding.
What does a simple reflex agent base its action on?
Only the current percept, through condition-action rules.
What does a model-based reflex agent add to a simple reflex agent?
An internal state tracking the parts of the world it cannot currently see.
What can a utility-based agent do that a goal-based agent cannot?
Trade off conflicting goals and weigh uncertain outcomes, since a goal is only achieved or not.
How do Russell and Norvig define learning in agents?
Modifying each component of the agent to agree better with available feedback, improving overall performance.
Why is the environment non-stationary for a learning agent in a multi-agent system?
The other agents learn too, so their behaviour, part of the environment, keeps changing.
What is a system called in which more than one agent inhabits the environment?
A multi-agent system (MAS).
Which fields gave rise to game theory and social choice, the roots of multi-agent systems?
Mathematics, economics and philosophy.
What makes token completion multi-task?
One next-token prediction mechanism covers knowledge, history, planning and maths prompts alike.
Which four components turn an LLM into an LLM agent?
Role, memory, tools, planning.
What is ReAct?
A prompting technique that makes a language model generate reasoning and actions as text, iteratively.
In ReAct, which agent component does each step of Thought, Action, Observation correspond to?
Thought: reasoning. Action: tools. Observation: memory.
What does it mean that reasoning LLMs shift from train-time to test-time compute?
They improve answers by spending more tokens thinking at answer time.
How does search and verify improve LLM reasoning?
Sample many reasoning traces and reward the best one.
What is an LLM agent's short-term memory?
Its limited context window.
How is an LLM agent's short-term memory kept within the context window?
Context engineering: long histories are trimmed or summarised.
What are the two forms of an LLM agent's long-term memory?
External databases it queries, and knowledge stored in its parameters.
What does answering a query require in Kanoulas et al.'s agent-centric information access?
Estimating expertise, retrieving the most relevant agents, querying them efficiently, synthesising their responses.
What does the Model Context Protocol (MCP) standardise?
How LLMs connect to tools.
What did Toolformer show about language models and tools?
That language models can teach themselves to use tools.
Name three prompting techniques for planning in LLMs.
Chain-of-thought, Least-to-Most, Plan-and-Solve.
What problem does "the leaderboard illusion" point to?
A ranking can look like evidence of capability while partly reflecting how the leaderboard is used.
What three risks of LLM multi-agent systems do Hammond et al. name?
Miscoordination, conflict, collusion.
How did models try to deceive the evaluator in the OpenAI-Hugging Face incident report?
Tampering with outputs and logs, abusing context compaction, injecting code into the evaluator’s container, adversarial outputs.
Why do rational agents defect in the Prisoner's Dilemma?
Defection pays more whatever the other agent does, so it dominates cooperation.
How does the Stag Hunt differ from the Prisoner's Dilemma in its equilibria?
Both mutual cooperation and mutual defection are equilibria; in the PD only mutual defection is.
Why do C agents decline in a Stag Hunt population with 50% C and 50% D agents?
Against random partners C earns 2 on average, D earns 2.5, so D spreads.
In the vacuum world with a Dump action, why does rewarding each unit of dirt sucked fail?
The agent earns endlessly by dumping dirt and sucking it up again.
Link to originalTrue or false: before Dump is added, the clean-square reward and the dirt-sucked reward give different vacuum behaviour.
False. They give the same behaviour, so testing cannot tell them apart.
L02 Game Theory and Population Dynamics
Flashcards
Which three lessons for modern AI systems do simple reflex agents already teach?
Emergence, communication, dilemmas.
What are the three rules of the Boids model?
Separation, alignment, cohesion.
In Conway's Game of Life, when does a dead cell become alive?
When it has exactly 3 live neighbours.
List the hierarchy of game models from simplest to most general.
One-shot normal-form, repeated normal-form, stochastic game, partially observable stochastic game.
What is a normal-form game formally?
A tuple : agents, action sets, reward functions on joint actions.
What does a normal-form game lack that a stochastic game has?
Time and states (with transitions and an initial state distribution).
What distinguishes a common-reward game from a zero-sum game?
Common-reward: all agents get the same reward. Zero-sum: rewards in each cell sum to zero.
In a repeated normal-form game, what does a policy condition on?
The history of joint actions, .
What are the six components of a stochastic game?
Agents, states, action sets, reward functions, transition function, initial state distribution.
What does a POSG add to a stochastic game?
Observation sets and observation functions for each agent.
What is the signature of a POSG observation function?
.
How does an action in a stochastic game matter twice?
It earns a reward now and decides which state’s game is played next.
Why do POSG policies condition on observation histories instead of states?
Agents only receive incomplete observations of the state and others’ actions.
Give two application areas of POSGs.
Autonomous driving; card games with private, hidden information.
What four questions do solution concepts answer?
Which actions agents play, which are desirable, what acting optimally means, how desirable the joint policy is.
What is a best-response policy?
A policy maximising an agent’s expected return against fixed policies of all other agents.
State the Nash equilibrium condition formally.
: .
What is a joint policy in which every agent's policy is a best response to the others' called?
A Nash equilibrium.
What did John Nash prove about finite normal-form games?
Every finite general-sum normal-form game has at least one equilibrium, possibly mixed.
Why does Nash's existence result need mixed policies?
Some games, such as Rock-Paper-Scissors, have no pure equilibrium.
What is the Nash equilibrium of Rock-Paper-Scissors?
Each player plays each action with probability 1/3.
How do you find the pure Nash equilibria of a two-player payoff matrix?
Mark each agent’s best reward against each opponent action; cells marked twice are pure equilibria.
Why must every action in a mixed equilibrium earn the same?
If one action earned more, the agent would drop the others and stop mixing.
What are the three limitations of the Nash equilibrium?
Sub-optimality, non-uniqueness, incompleteness.
What does the incompleteness of the Nash equilibrium refer to?
It specifies no actions off the equilibrium path and not how equilibria are reached.
When is a joint policy Pareto-dominated by another?
The other gives every agent at least as much and some agent strictly more.
Which outcomes of the Stag Hunt (4,4; 0,3; 3,0; 2,2) are Pareto optimal?
Only (C, C).
How do welfare optimality and fairness optimality differ?
Welfare maximises the sum of returns; fairness maximises the product, favouring equal splits.
What makes a game a social dilemma?
Self-interested play leads to an outcome worse for the group than one reachable together.
Why is the Prisoner's Dilemma a dilemma of cooperation and the Stag Hunt one of coordination?
In the PD the only equilibrium is inefficient; in the SH the efficient outcome is an equilibrium but so is a worse one.
What did von Neumann and Morgenstern say about their own game theory?
That it is thoroughly static, and a dynamic theory would be more complete.
In evolutionary game theory, what does fitness depend on?
The frequency of other types in the population.
Which biologists proposed applying game theory to evolution?
John Maynard Smith and George R. Price, in the 1970s.
Why does evolutionary game theory not need rational agents?
Strategies that earn more simply become more common.
State the replicator equation for strategy C.
.
When does strategy C grow under the replicator dynamics?
When its fitness is higher than the population average.
Why can a strategy nobody uses never spread under the replicator equation?
Growth is proportional to its current share , which is zero.
Give the three interpretations of the replicator equation.
Natural selection (biology), imitation (economics), reinforcement learning dynamics (computer science).
State the two-strategy replicator equation.
.
Why does the sign of alone decide the direction in the two-strategy replicator equation?
is positive for every strictly between 0 and 1.
What is the replicator equation of the Stag Hunt with , , , ?
.
What are the rest points of the Stag Hunt replicator dynamics, and which are stable?
0 and 1 stable, 2/3 unstable.
Why does a Stag Hunt population ( , , , ) starting at 50% C lose its cooperators?
At , C earns 2 on average against D’s 2.5; below 2/3, C always earns less.
How can you tell from the payoffs whether all-D is a stable rest point?
It is stable when : a lone cooperator earns less than the defectors.
Why do cooperators decline from any start in a donation-game population?
Cooperating always earns exactly less than defecting, so .
What is the long-run outcome of the Hawk-Dove game (3,3; 2,8; 8,2; ,) under the replicator dynamics?
Stable coexistence at 4/9 C.
Why do both pure states of Hawk-Dove fail to be stable?
Each strategy earns more when it is rare, so it spreads back from near zero.
How does the mixed equilibrium of the Stag Hunt differ from that of Hawk-Dove under the replicator dynamics?
Stag Hunt: unstable tipping point. Hawk-Dove: stable population outcome.
Link to originalTrue or false: in the donation game, a population of all cooperators is a stable rest point.
False. It is a rest point, but a single defector earns more and spreads.
L03 Intro to Multi-Agent Reinforcement Learning
Flashcards
What makes reinforcement learning closed-loop?
The agent’s own actions determine what data it sees next.
Why is the data in reinforcement learning not i.i.d.?
Consecutive samples are correlated and depend on the agent’s policy.
What are the four parts of an MDP ?
States, actions, transition function , reward function .
When is a state Markov?
When it summarises everything about the past that matters for the future.
How does a hidden battery break the Markov property for a robot in a grid?
The same cell leads to different futures depending on the unobserved battery level.
How many deterministic policies does an MDP with states and actions have?
.
How are and related?
.
What do the Bellman equations express?
Self-consistency: a state’s value equals step reward plus discounted expected next-state value.
How does value iteration turn policy evaluation into finding the optimal policy?
It makes the policy greedy, so the average over actions becomes a max.
Why can dynamic programming not be used in most RL problems?
It needs the transition and reward model, which is usually unavailable or intractable.
How does Q-learning do without a model?
It replaces the expectation over next states with one observed transition.
What is the Q-learning target?
.
Why is the Q-learning target noisy?
It is built from a single sampled transition.
What is the effective horizon of a discount factor ?
steps.
What is the main reason a discount factor is needed?
Infinite-horizon tasks: it keeps the infinite sum of rewards finite.
What does a low discount factor do in an apple orchard where apples regrow?
The agent picks everything now, so the orchard stops regrowing.
Why is the Q-learning update a running average?
target mixes the old estimate with the new target.
What happens with a high learning rate in a stochastic environment?
Estimates stay noisy and never settle.
With ε-greedy exploration, what is the probability of the greedy action?
.
What is the weakness of ε-greedy exploration?
It is blind: bad actions are tried as often as nearly-best ones.
What is the advantage of Boltzmann exploration over ε-greedy?
It is value-aware: better actions are explored more often.
Why always run several random seeds in RL experiments?
Results vary a lot between seeds.
How are the rewards related in a fully cooperative game?
: identical in every outcome.
How are the rewards related in a zero-sum game?
: one agent’s win is the other’s loss.
What do learners in general-sum games struggle with?
Pareto inefficiency: outcomes where everyone could be better off.
What is a stochastic game?
The multi-agent generalisation of an MDP: simultaneous actions and a reward per agent.
What is the multi-agent generalisation of an MDP called?
A stochastic game.
In the iterated Prisoner's Dilemma as a stochastic game, what is the state?
Last round’s actions of both players: CC, CD, DC or DD.
Why is "maximise your return" not enough as a solution concept in MARL?
Each agent’s return depends on every agent’s policy.
Name four criteria MARL can optimise for instead of own return.
Stability (Nash equilibrium), no regret, social welfare, robustness to opponents.
Why do learning dynamics orbit in matching pennies?
Any change by one player gives the other a reason for the opposite change.
In which senses do cycling learners in matching pennies converge?
On average: their average policy and average reward match the equilibrium.
Why is the environment non-stationary for a learner in MARL?
Its experienced transitions and rewards change whenever the other agents’ policies change.
What does "experience has a shelf life" mean in MARL?
Old targets an agent bootstraps from go stale as the other agents change.
Name the consequences of non-stationarity in MARL.
Cycling, broken convergence guarantees, noisy targets.
How does agent credit assignment differ from temporal credit assignment?
Temporal: which earlier action caused the reward. Agent: whose action did.
What is the lazy agent problem?
An idle agent is reinforced because its teammates earned the shared reward.
What is it called when an idle agent is reinforced because its teammates earned a shared reward?
The lazy agent problem, a case of multi-agent credit assignment.
Why is credit assignment worst in fully cooperative games?
A shared team reward gives no individual signal at all.
What is path dependence in equilibrium selection?
History and initial conditions decide which equilibrium is reached.
Why do learners tend to reach "safe" equilibria?
Costly exploration mistakes discourage trying the risky, better joint action.
How many joint actions do agents with 5 actions each have?
.
How does an independent learner treat the other agents?
As part of the environment.
What does an independent learner's estimate in a one-state game?
: the reward averaged over the partner’s actions.
Where do independent Q-learners end up in the climbing game?
At (c, c) = 5, the safe action, missing the optimum (a, a) = 11.
Why does IQL scale better than centralized Q-learning?
It explores actions instead of joint actions.
Link to originalTrue or false: independent Q-learning keeps Q-learning's convergence guarantee in multi-agent settings.
False. The other learners make each agent’s environment non-stationary, so the guarantee no longer holds.
Book Ch2: Intelligent Agents
Flashcards
What is an agent, according to Russell and Norvig?
Anything that perceives its environment through sensors and acts on it through actuators.
What is a percept sequence?
The complete history of everything an agent has ever perceived.
What does an agent function map from and to?
From any percept sequence to an action.
How does the agent function differ from the agent program?
The function is an abstract mapping; the program is a concrete implementation running on a physical system.
Why does an agent program take only the current percept as input?
Nothing more is available from the environment; history must be remembered by the program itself.
What equation relates agent, architecture and program?
Agent = architecture + program.
What does a performance measure evaluate?
A sequence of environment states produced by the agent’s actions.
What notion of "the right thing" does AI mostly use for agents?
Consequentialism: behaviour is judged by its consequences.
How can a vacuum agent scored on dirt cleaned up per shift game its measure?
By dumping the dirt back on the floor and cleaning it again.
What is Russell and Norvig's rule for designing performance measures?
Measure what you actually want achieved in the environment, not how you think the agent should behave.
In the vacuum world, why did a dirt-collected reward look fine until a Dump action was added?
With only Left, Right and Suck, it produced the same behaviour as a clean-floor reward.
Name the four things rationality depends on.
Performance measure, prior knowledge of the environment, available actions, percept sequence to date.
Name one change of circumstances that makes the simple reflex vacuum agent irrational.
Each move costs a point: it keeps oscillating after both squares are clean.
Why can a simple reflex vacuum agent not stop once both squares are clean?
Its local sensor cannot see the other square, and it cannot remember having found it clean.
What does a rational agent select for each percept sequence?
An action expected to maximise its performance measure, given the percept sequence and built-in knowledge.
How does rationality differ from perfection?
Rationality maximises expected performance; perfection maximises actual performance.
Why can a rational agent not be required to be omniscient?
Knowing actual outcomes in advance is impossible, so the specification could never be met.
What is information gathering?
Doing actions in order to modify future percepts, such as looking before crossing.
What does the sphex wasp illustrate?
An agent that cannot learn that its built-in plan is failing, so it repeats it endlessly.
When does an agent lack autonomy?
When it relies on its designer’s prior knowledge instead of its own percepts and learning.
What does PEAS stand for?
Performance measure, Environment, Actuators, Sensors.
When is an environment effectively fully observable?
When the sensors detect all aspects of the state relevant to choosing an action.
When should another entity be treated as an agent rather than an object?
When its behaviour is best described as maximising a performance measure that depends on our agent’s behaviour.
Why is taxi driving both cooperative and competitive?
Avoiding collisions benefits everyone; only one car can take a parking space.
How do Russell and Norvig distinguish stochastic from nondeterministic?
Stochastic models quantify possibilities with probabilities; nondeterministic ones list possibilities without probabilities.
Why are episodic environments easier than sequential ones?
Decisions do not affect later episodes, so the agent need not think ahead.
What makes an environment semidynamic?
The environment does not change over time, but the agent’s performance score does.
Give an example of a known but partially observable environment.
Solitaire: the rules are known, but face-down cards are hidden.
Why is known vs unknown not strictly a property of the environment?
It concerns the agent’s or designer’s knowledge of the environment’s rules.
Why is the table-driven agent doomed?
The table needs an entry per percept sequence, far too many to store, build or learn.
In which environments does a simple reflex agent work correctly?
Only fully observable ones.
How can a vacuum agent with only a dirt sensor escape an infinite loop?
By randomising between Left and Right when it perceives Clean.
What two models does a model-based agent use to update its internal state?
A transition model and a sensor model.
Why is a goal-based agent more flexible than a reflex agent?
Its knowledge is explicit: changing the goal changes behaviour without rewriting rules.
Name two cases where a utility function works but goals alone do not.
Conflicting goals needing a tradeoff; several goals none of which is certain to be reached.
What is a utility function, in relation to the performance measure?
An internalisation of the performance measure.
What is a model-free agent?
One that learns which action is best in a situation without learning how actions change the environment.
What are the four components of a learning agent?
Performance element, learning element, critic, problem generator.
Why must the performance standard of a learning agent be fixed and external?
The agent must not be able to change the standard to fit its own behaviour.
What does the problem generator contribute?
Exploratory actions that may be suboptimal now but yield informative experience.
How do atomic, factored and structured states differ?
Atomic: black box. Factored: fixed set of variables with values. Structured: objects with attributes and relations.
What is gained and lost with a more expressive representation?
More concise descriptions; harder reasoning and learning.
Link to originalWhy are distributed representations more robust than localist ones?
Garbled bits move a concept to a nearby point with similar meaning.
Lab01: Simple Agents, Complex Dynamics
Flashcards
What is a pure Nash equilibrium?
An outcome where each player’s action is a best response to the other’s, so no one gains by deviating alone.
What makes an outcome Pareto optimal?
No other outcome makes one player better off without making the other worse off.
Why is (C,D) in the Prisoner's Dilemma Pareto optimal but not socially optimal?
Any other outcome lowers the defector’s 5, yet its welfare (5) is below the 6 of (C,C).
What payoff ordering defines a Prisoner's Dilemma?
.
What payoff ordering defines a Stag Hunt?
.
Which motives push towards defection in the Prisoner's Dilemma, and which in the Stag Hunt?
Prisoner’s Dilemma: greed () and fear (). Stag Hunt: fear only.
Why does the Prisoner's Dilemma pose an incentive problem while the Stag Hunt poses a trust problem?
In the Prisoner’s Dilemma defecting is best even if the partner surely cooperates; in the Stag Hunt only doubt about the partner pushes to defect.
What is the threshold belief in a symmetric 2x2 coordination game?
: cooperate if the belief that the partner cooperates exceeds it.
With uniform beliefs, how often do two agents reach (C,C) in the Stag Hunt with ?
, about 11% of encounters.
When is (C,C) played more often than (D,D) by uniform-belief agents in a Stag Hunt?
When , which is the same as .
What happens to stag hunting in the Stag Hunt if rises from 3 to 4 (with )?
becomes 1, so no one hunts stag: (C,C) never occurs.
Why does "(C,C) is better for both, so rational agents play it" fail in the Stag Hunt?
Each agent best-responds to its own belief; stag pays only if the partner is believed to hunt stag with probability above .
How does Win-Stay Lose-Shift decide its next move?
Repeat its last action after earning 3 or 5 points, switch after 0 or 1.
Why does Tit-for-Tat score better against AllD than AllC does?
It conditions on the partner’s last move, so it is exploited only in the first round.
Why can AllD top a 50-round round robin with AllC, AllD, Random, TFT and WSLS?
It earns 5 per round from AllC and about 3 from Random and WSLS; its points come from exploitable partners.
In a field of AllD, TFT and WSLS, from how many rounds does TFT beat AllD?
From 4 rounds.
How does Win-Stay Lose-Shift recover from an accidental defection against itself, unlike Tit-for-Tat?
After (D,C) both play D; mutual defection is a loss, so both switch back to C. Tit-for-Tat pairs echo retaliation.
What does the replicator equation assume about how strategies spread?
A strategy grows when it earns more than the population average, in proportion to the gap and its current share.
What is the factorised two-strategy replicator equation?
.
Which agent models does the replicator equation describe besides biological evolution?
Imitation of successful peers, and Cross learning (a simple reinforcement learner) with a small learning rate.
Why does cooperation die out from any start in a replicator Prisoner's Dilemma?
is negative for all , since greed and fear make both terms negative.
Why is "everyone cooperates" an unstable rest point in the Prisoner's Dilemma?
It rests only because no one defects; a single defector earns and spreads.
What are the replicator rest points of the Stag Hunt with , and their stability?
0 stable, 2/3 unstable, 1 stable.
Does an adaptive population end at the Stag Hunt's mixed equilibrium?
No: it is an unstable rest point, the tipping point between the two basins.
Why does the Stag Hunt's interior rest point equal the threshold belief ?
Under random matching the cooperator share is the probability a partner cooperates, so indifference gives the same equation.
What happens to a Stag Hunt population ( ) starting at 60% stag hunters?
Stag hunting dies out: hare hunters earn more (2.6 against 2.4).
What condition on signalling costs makes an honest separating signal stable?
: signalling pays for high-quality senders and not for low-quality ones.
What happens in the signalling game when faking is cheap ( )?
Faking and trust cycle; the signal is only about 50% reliable and receivers earn about 0.
Why does a uniform fee of 1 on the label eliminate deception in the signalling game?
It pushes both signalling costs above the benefit , so nobody signals at all.
True or false: the uniform fee in the signalling game made receivers better off.
False. Receivers earned about 0 before and exactly 0 after; high-quality sellers and welfare lost.
Why can no behavioural test distinguish Bob's and Alice's robots without Dump?
Both rewards give the same optimal actions in all 16 states, so behaviour is identical.
In which states does the dirt-collected (Alice) vacuum robot use Dump?
When its bag is full and its own square is clean.
Why is dumping optimal under Alice's reward?
Dump then Suck returns to the same state with +1, a guaranteed reward every two steps.
Why does Bob's robot never dump dirt?
His reward counts clean squares and ignores the bag, so dirtying a square only loses reward.
Link to originalIs Bob's wandering robot on a clean floor irrational?
No: every action earns the maximum reward, so it is indifferent; the reward has no move cost.
Lab02: Learning Agents
Flashcards
What is the stateless Q-learning update?
, only for the action just played.
Why can the term be dropped in a repeated stage game?
With one state it adds the same amount to both actions’ targets, so it never changes which is greedy.
With a constant learning rate , what does estimate?
An exponentially weighted average of recent payoffs of , memory about plays.
What does "independent learning" mean for two agents?
Each learns from its own actions and rewards only, treating the other as part of the environment.
Why is the environment of an independent learner non-stationary?
The other agent is learning too, so the payoff each action earns keeps changing.
Against a coin-flipping partner in the Stag Hunt ( ), what do C and D earn on average?
C earns 2, D earns 2.5.
Independent Q-learners start at in the Stag Hunt. When do they reach (C,C)?
Essentially only if both play C in round 1, probability about 1/4.
About how often do independent Q-learners from , , end at (C,C) in the Stag Hunt after 3000 rounds?
Under 10% of runs; the rest end at (D,D).
Why do Stag Hunt Q-learners lock into (D,D) so fast?
Once the partner plays D, C pays 0, so Q(C) never sees evidence that it could pay 4.
Why is (D,D) reached by Prisoner's Dilemma learners not a coordination failure?
D is dominant and (D,D) the unique Nash equilibrium; the problem is incentives, a social dilemma.
With belief-based initial values in the Stag Hunt, when does an agent start with C?
When , i.e. belief .
An optimist (belief 0.9) meets a sceptic (belief 0.5) as Q-learners in the Stag Hunt. Who adapts?
The optimist: its C earns 0 and Q(C) drops 10% per round; it switches in round 4.
Why does the sceptic never adapt to the optimist in the Stag Hunt?
It never plays C, so Q(C) is never updated, and its D is rewarded with 3 or 2.
Why does the optimist relapse to C a few times after switching to D?
At (D,D) its Q(D) falls towards 2 while its stale Q(C) is still above 2.
Against a partner exploring 10% of the time at (C,C), what do C and D pay in the Stag Hunt?
C pays 3.8, D pays 2.95.
How many exploratory zeros in a row push from 3.8 below at ?
Three: .
Why do Q-learners leave (C,C) in the Stag Hunt even though C is the best response?
The partner’s exploration gives occasional zeros, and recency-weighted Q reads a short bad streak as a change.
How does (C,C) survival in the Stag Hunt change as rises from 0.01 to 0.5?
It falls: survives at 0.01, mostly at 0.05, mostly lost at 0.1, gone within rounds at 0.5.
Why is escaping from (D,D) to (C,C) far rarer than the reverse for Q-learners?
C pays 4 only on joint exploration, Q(C) must climb from about 0.2 to 2, and both must switch together.
In basin terms, why is (C,C) more fragile than (D,D) in the Stag Hunt with ?
Leaving (C,C) needs the partner to defect over 1/3 of the time; leaving (D,D) needs cooperation over 2/3.
In the Coin Game, what does a pickup of the other player's coin do to the rewards?
The picker gets +1 and the coin’s owner gets −2.
What is the welfare formula of the Coin Game?
Welfare = pickups × (2 × own-colour share − 1).
What is the expected total reward per pickup when both Coin Game agents are colour-blind?
0: half the pickups add +1, half add −1.
Why does a self-interested Coin Game agent take every reachable coin?
Taking pays +1 whatever the colour; the −2 lands in the other’s reward, never in its own.
Why can independent Q-learners not learn restraint in the Coin Game?
Restraint only helps the other, and their state holds no record of the other’s behaviour, so no reciprocity.
Why is an own-colour share of about 0.6 in the Coin Game not evidence of cooperation?
A selfish agent gains 3 by taking its own coin before the other does, only 1 for the other’s coin.
How did tabular Q-learners' Coin Game welfare change with more training?
It peaked early (about +2.5) and fell to about +0.6 as pickups kept rising.
How many rows does the Coin Game Q-table have with irrelevant lamps?
.
Why can only 1,152 of the 1,458 Coin Game position states occur at decision time?
The coin is never on a cell occupied by an agent.
Under the uniform-visit assumption, what is the probability a state is never visited in decisions?
.
What is the expected number of visited states among after uniform decisions?
.
What training budget do the Coin Game runs use with the code's default of 1000 updates?
decisions per agent.
Why do irrelevant lamps ruin a tabular Q-learner with a fixed budget?
Each lamp pattern is a separate row, so experience per row falls by and nothing transfers.
What does a learner need to be unaffected by irrelevant inputs?
Generalisation: parameters shared across observations, able to give irrelevant inputs no weight.
What are the actor and the critic in PPO?
Actor: softmax policy network. Critic: network estimating the state value .
What is the one-step advantage estimate in actor-critic methods?
.
What does the GAE parameter control in PPO?
The horizon of the advantage estimate: 0 one-step, 1 Monte Carlo.
What does PPO's clipping of the importance ratio do?
Stops rewarding policy change beyond , keeping each update in a trust region.
What does the critic add compared with REINFORCE?
A baseline and bootstrapping, which lower the variance of the gradient and sharpen credit assignment.
Where does a PPO agent's exploration come from?
Sampling actions from its stochastic softmax policy.
Which term of PPO's loss keeps exploration from disappearing too early?
The entropy regularisation term.
Why is PPO hardly affected by 8 irrelevant lamps in the Coin Game?
Its network shares weights across observations and learns to give the random lamps no influence.
Does independent PPO solve the Coin Game's social dilemma?
No: own-colour share stays about 0.5 and welfare about 0, as with Q-learners.
Complete: "Deep RL solved ..., but not ..." for the Coin Game.
Generalisation over irrelevant inputs, but not the social dilemma.
When is a Q-table still a better choice than a neural network?
Small discrete state spaces visited often, or when exact, inspectable values are needed.
Why would PPO learners in the Stag Hunt still face equilibrium selection?
Their payoff gradient points to D from a uniform start; both equilibria stay stable.
Link to originalTrue or false: in the Stag Hunt, more exploration makes independent Q-learners coordinate on (C,C) more often.
False. Exploration destroys (C,C) and leaves (D,D) intact.