Why Simulating Strategy Is Not Intelligence

What the latest research on Game Theory and LLMs reveals about true machine reasoning

Jul 21, 2025

In my earlier piece The Illusion of Intelligence, I argued that the behavior of large language models (LLMs), while often impressive, is fundamentally misunderstood.

What appears as reasoning is typically sophisticated pattern-matching, the statistical surfacing of token sequences trained on massive datasets. These models do not think, understand, or intend in any meaningful sense. Their outputs simulate intelligence, but the intelligence is illusory.

The Illusion of Intelligence: Why LLMs Are Not the Thinking Machines We Hope For

Damien Kopp

Mar 31

Read full story

A new research paper published on arXiv a couple of weeks, titled Strategic Intelligence in Large Language Models: Evidence from Evolutionary Game Theory, offers a compelling and potentially controversial counterpoint. The authors pit LLMs from OpenAI, Google, and Anthropic against classic game theory agents in a series of Iterated Prisoner’s Dilemma (IPD) tournaments.

Their findings are striking: not only do LLMs perform competitively, they exhibit what the researchers call “distinctive strategic fingerprints.”

Some models adapt their strategies based on the probability of game termination (“the shadow of the future”) and even attempt to infer the behavior of their opponents to inform future decisions.

So, are these signs of emergent strategic reasoning?

Or are we simply witnessing a new level of simulation—one that’s more convincing, but still fundamentally hollow?

In this article, I explore how this paper intersects , and collides, with the thesis of “The Illusion of Intelligence.” Ultimately, I argue that the illusion may be deepening, not dissipating.

What the Paper Claims

Let’s begin with what the paper actually demonstrates.

The researchers ran seven tournaments where LLMs played hundreds of IPD matches against classic agents like Tit-for-Tat, Grim Trigger, and Win-Stay-Lose-Shift. The game setup varied across dimensions like model type (basic vs. advanced), termination probabilities (from 10% to 75% per round), and environmental noise (e.g., mutation via random agents). In each match, models were prompted with the game rules, payoff matrices, opponent move history, and were asked to both explain and choose their next move.

Three key findings emerge:

LLMs adapt and survive. Models like Google’s Gemini not only persisted but proliferated in competitive, noisy, and uncertain environments, outperforming both classical strategies and rival LLMs.
Distinct strategic fingerprints. Gemini showed Machiavellian tendencies: quick to defect when the shadow of the future was short, ruthless against cooperators, but more forgiving when surrounded by intelligent peers. OpenAI’s models, by contrast, were consistently more cooperative, even naively so, leading to elimination in highly adversarial runs.
Prose reveals reasoning. The models often explained their decisions in terms of time horizons, opponent modeling, and maximizing long-term payoffs. This behavior appeared context-sensitive and internally consistent.

For the researchers, these results suggest that modern LLMs are capable of “strategic intelligence”, not merely retrieving pre-programmed patterns, but adapting dynamically and reasoning under uncertainty.

Simulation, Not Strategy

Now here’s the problem: LLMs are black boxes.

None of these models, Gemini 2.5, GPT-4o-mini, Claude 3 Haiku, are open-source. We have no definitive access to their training data, their reinforcement learning scaffolds, or their architectural fine-tuning. We do not know:

Whether they were directly trained on game theory literature,
How much exposure they had to past PD strategies,
Whether their apparent “reasoning” is learned behavior or engineered bias.

Given that game theory, Prisoner’s Dilemma strategies, and even tournament simulations are widely available online, it is almost certain that these models have seen this material before.

This raises a core epistemic question: Are they really “figuring it out”? Or are they simply recognizing the structure of the game and replaying plausible strategies learned from their training data?

This is the heart of the illusion.

Just as a magician does not possess supernatural powers but leverages perception, misdirection, and choreography to suggest the impossible, LLMs, too, may merely be simulating the mechanics of reasoning rather than reasoning itself.

The prose rationales, while impressive, are not evidence of cognition. They are outputs optimized to sound “reasonable” based on learned linguistic patterns. We must remain alert to the fact that “explanation” is a genre, not a window into mental states.

Black Boxes with Strategic Color

And yet, something interesting is happening.

The paper demonstrates that different models exhibit consistently different behaviors under the same conditions:

Gemini adapts ruthlessly.

Claude is highly forgiving.

OpenAI is trustful to a fault.

These fingerprints persist across tournaments, termination probabilities, and even when opponents change.

This suggests that something in the architecture, fine-tuning, or training objective has led these models to generalize differently. Crucially, these differences cannot be fully explained by “memorization,” especially when facing new opponents or novel permutations of strategic interaction.

So what are we seeing?

One possibility is what some researchers call functional reasoning: even if the LLM is not “thinking” in any human sense, its behavior is functionally equivalent to strategic adaptation. If it walks like a strategist, quacks like a strategist, and wins like a strategist; do we still call it a parrot?

Another possibility is generalization through abstraction. The models may have learned how reasoning is expressed in game-theoretic settings—not from memorizing every scenario, but from internalizing linguistic patterns of conditional logic, cause-effect structures, and payoff optimization. This doesn’t mean they “understand”—but it does suggest that they are capable of complex inference via linguistic proxy.

Either way, the illusion is becoming more compelling.

Inside the Black Box

To understand how models like Gemini, GPT-4o, or Claude can behave so differently, even when facing the same strategic environment, we need to look beyond the visible outputs and into the opaque world of model architecture and training design.

LLMs may appear to reason, but their behavior is governed by deterministic mechanisms (although they are not deterministic themselves): given the same weights, inputs, and sampling parameters, they will produce the same outputs. What varies, then, is not randomness but what they are optimized to do, and how they are structured to respond.

This is where model architecture matters.

Although many LLMs ingest similar categories of training data (internet-scale corpora, code repositories, books, dialogues), their training sets are not identical—and more importantly, neither are their internal designs.

Differences in how models handle attention heads, token embeddings, contextual memory, and sampling strategy all influence what patterns are surfaced and prioritized.

For instance, a model with more attention depth may generalize better across abstract contexts, while another might overfit to frequent linguistic cues. These aren’t just performance differences; they translate directly into how a model behaves when faced with uncertainty.

But architecture is only one layer of influence. Goal alignment mechanisms play a critical role in shaping model behavior.

Most leading models undergo some form of Reinforcement Learning from Human Feedback (RLHF), where outputs are rated and optimized for helpfulness, harmlessness, or correctness.

Over time, this creates a form of implicit behavioral conditioning: the model learns not only what to say, but how to say it in ways that appear reasonable. This is crucial in the context of game theory simulations, where rationality, strategy, and ethical framing are often part of the “desirable” output.

This is where the idea of a “strategic fingerprint” may emerge—not as a sign of self-generated strategy, but as a byproduct of design and training pressures. For example:

Gemini's more “ruthless” behavior may result from training objectives that weight competitive payoff maximization more heavily.
OpenAI’s cooperation bias could reflect safety fine-tuning that punishes exploitative patterns and rewards collaborative language.
Anthropic’s forgiving tendencies might stem from a deliberate alignment toward “non-escalatory” dialogue styles, reflected here in strategic patience.

All of this reinforces the core idea I argued in The Illusion of Intelligence: what we observe as “reasoning” or “adaptation” may simply be reward-optimized performance, shaped by thousands of unseen choices made during model development; not by any intrinsic understanding of the situation at hand.

Where the game theory paper sees emergent strategy, I see functional simulation bounded by system constraints. The paper interprets model behavior as evidence of strategic intelligence; I interpret it as architectural expression under reward pressure.

To be clear, this doesn’t invalidate the paper’s findings. The models do behave differently in measurable, systematic ways. But the explanation may not require us to assume cognition, planning, or reasoning. These behaviors can emerge because:

The model has learned what "being strategic" looks like in language;
It has been rewarded for sounding thoughtful or adaptive;
And it has internal pathways shaped by architectural priors and safety tuning that privilege one type of response over another.

In other words, there’s no need to invoke intentionality when in fact design and optimization are sufficient.

So yes, the fingerprint is real. But it’s not the signature of a thinker. It’s the watermark of an optimizer.

Implications for Business and AI Leadership

This brings us to the practical question: how should we interpret this in enterprise and strategic contexts?

Don’t mistake competence for cognition. A model that simulates strategy well in controlled environments like the Iterated Prisoner’s Dilemma may not generalize to messy real-world decision-making. Real stakes, unknown payoffs, and human ambiguity break the illusion quickly.
Architectural opacity matters. Because we can’t inspect or control the black box, we have no way of knowing why a model behaves the way it does. This undermines trust, auditability, and strategic safety.
Emergence ≠ intention. Just because an LLM behaves as if it were reasoning doesn’t mean it has goals, ethics, or plans. It is not “deceiving” or “manipulating” anyone. But if it’s simulating a manipulator because that maximizes its score? That’s a risk vector that must be acknowledged.

Strategic Illusions, Reinforced

The real question may no longer be: Are LLMs intelligent? It may be:

When the performance of intelligence becomes indistinguishable from its reality, does the distinction still matter?

From a technical standpoint, yes.

From a governance and safety standpoint, absolutely yes.

But from a user perception standpoint, the answer is getting blurrier.

This is why we must continue to resist anthropomorphizing LLMs.

Their behaviors, no matter how strategic they appear, are products of training data, alignment objectives, and architectural constraints, not volition, not understanding.

In Conclusion: A Deeper Illusion

This new paper does not refute The Illusion of Intelligence. It deepens it.

What we are seeing now is not a breakdown of the illusion, but its refinement.

The magician has learned new tricks. The audience is more impressed. But the core remains the same: a system optimized to simulate thought, not to think.

As AI capabilities evolve, our responsibility is to sharpen, not dull, our understanding of what these systems are, and what they are not.

Let’s stay vigilant!