A proposal for integrating theory of mind more deeply into frontier AI model development...by ChatGPT

27 Jun, 2026

I asked ChatGPT (5.5) to help me brainstorm some ideas of how theory of mind (my dissertation topic) could be more deeply integrated into frontier AI model development to improve model capabilities and usefulness.

This is the output, which is...good. Really good. A lot of ideas to follow-up on.

Integrating Theory of Mind into Frontier AI Model Development

Executive summary

Theory of mind (ToM) should be treated as a core capability layer for frontier AI, not as a niche social-feature add-on. In humans, ToM is the capacity to represent others’ beliefs, intentions, desires, knowledge, emotions, and perspective-dependent constraints; across psychology, philosophy, neuroscience, and anthropology, it is understood as a structured but non-unitary set of abilities for tracking hidden states under uncertainty, within interaction, and under cultural norms. Current large models already show partial ToM-like behavior, but the evidence is clear that these abilities are brittle, uneven, and often benchmark-specific: GPT-4-level systems can match or exceed human performance on some narrow tasks, yet still lag humans on broader batteries, higher-order recursion, psychological-state modeling, dynamic state tracking, and multimodal social understanding. [1]

The practical implication is straightforward: if a model cannot reliably infer what a user knows, wants, believes, misunderstood, intends next, or must not be told, then a large fraction of “agentic” failures are inevitable. The strongest performance gains from ToM integration will therefore appear in tasks where hidden human or agent state is the bottleneck: ambiguous instruction following, tutoring, collaborative planning, customer support, negotiation, coding agents, and multi-agent coordination. Early evidence already points in that direction. Perspective-taking prompting improves ToM benchmarks; symbolic belief tracking improves zero-shot performance and out-of-distribution robustness; dialogue agents that explicitly generate partner mental states do better at long-horizon social goals; coding agents with an explicit user-modeling partner substantially improve task success and user satisfaction; and repeated-game studies show that explicitly predicting an opponent’s behavior improves coordination with humans. [2]

The strongest proposal is therefore not “make the model more empathic.” It is: treat ToM as a first-class latent-state estimation problem that should shape data, objectives, curriculum, architecture, post-training, evaluation, deployment UX, and governance. Concretely, that means building models that can represent perspective-dependent world states, simulate alternative viewpoints, maintain bounded and auditable user models, learn from multi-agent interaction, and be evaluated on functional adaptation rather than only static false-belief question answering. At the same time, these gains create serious risks: a model that better models minds can also become better at manipulation, deception, privacy violation, and anthropomorphic overreach. Governance and product controls therefore have to be built in from the start, not added after capability work. [3]

Assumptions and scope

This proposal assumes an audience of technical research leadership, model architects, applied research teams, and product managers deciding how to allocate a 12–24 month frontier-model R&D portfolio. It also assumes that “performance” means more than benchmark accuracy in isolation: the target improvements are better capabilities in ambiguous human-AI collaboration, safer alignment to user intent, stronger long-horizon interaction quality, improved coordination with other agents, and fewer failures caused by misunderstanding users or other actors. Those are the domains where ToM is most likely to produce step-function gains rather than marginal ones. [4]

The architectural focus is transformer-based LLMs and VLMs, including tool-using and memory-augmented systems. “Integrating ToM” here does not mean attributing consciousness or personhood to models. It means operationally improving the model’s ability to infer and update latent mental-state variables that matter for prediction, explanation, coordination, pedagogy, privacy, and action selection. That framing is consistent with both the classic definition of ToM and recent computational accounts that treat social cognition as inference over abstract agent representations embedded in probabilistic causal models of action. [5]

I prioritize primary and official sources, especially peer-reviewed papers and official governance documents, with recent work from 2022–2026 foregrounded wherever possible. Older sources are used only where they remain foundational, such as false-belief development, simulation theory, and ToMnet. Several of the most practically relevant AI papers are still prepublication or workshop/submission-stage; where I rely on them, I treat them as directional evidence rather than settled consensus. [6]

Theory of mind across disciplines

Theory of mind is best understood as a family of capacities, not a single switch that is either “on” or “off.” Developmental psychology shows a staged progression from early joint attention, desire reasoning, and common ground toward false-belief understanding and more complex recursive mental-state reasoning, with classic false-belief performance changing sharply across the preschool years. More recent developmental work also emphasizes that children reason not only about isolated minds but about interacting minds, and that ToM can be strengthened by mental-state-rich conversation, storytelling, and role-play. [7]

Cognitive psychology and neuroscience frame ToM as structured mentalizing under computational constraint rather than unconstrained omniscient “mindreading.” Recent computational reviews argue that humans build abstract representations of agents, place them into probabilistic causal models of rational action, and then use restricted-scope models that trade off explanatory power against computational cost. Neuroscience has repeatedly implicated a mentalizing network centered on temporoparietal and medial prefrontal regions, while newer work suggests that perspective-taking, self–other discrimination, and recursive social inference rely on separable but interacting mechanisms rather than one monolithic module. [8]

Philosophy is valuable here because it keeps the AI framing honest. The central philosophical dispute is not whether people reason about minds, but how: “theory-theory” treats folk psychology as something like an internal explanatory theory, whereas simulation theory argues that we often understand others by placing ourselves in their position and reusing our own cognitive machinery under altered assumptions. Modern philosophical accounts also emphasize pluralism, narrative practice, and the possibility that different tasks recruit different mixtures of inference, simulation, and culturally learned interpretive routines. For AI design, that matters because it warns against betting on any single mechanism. A robust machine ToM stack should probably mix explicit models, simulation loops, and learned heuristics. [9]

Social psychology, sociology, and interaction research push ToM away from isolated puzzles and toward coordination in real communicative settings. Common ground is not decorative; it is what makes reference, implication, repair, and coordination possible. Turn-taking itself scaffolds perspective-taking by recursively making one speaker’s prior turn the object of the next turn. At the same time, the benefits of perspective-taking are context-sensitive rather than uniformly prosocial: it can improve coordination and reduce stereotyping in some settings, while producing mixed or fragile effects in others. For AI, the lesson is that ToM is not just inside-the-model reasoning about beliefs. It is a dynamic property of interaction, state alignment, and repair. [10]

Anthropology adds the most important corrective to naïve product thinking: cultures differ not only in what people believe, but in whether it is appropriate to presume access to another person’s inner states at all. The anthropology of mind documents “mental opacity” norms in which one should not casually assume that one knows what someone else thinks or intends. It also shows that ideas about mind–world boundaries, selfhood, and expressive norms vary sharply across social worlds. For AI systems, this means that stronger ToM should not automatically imply stronger explicit mind-reading behavior in the UI. In some markets, domains, and communities, the correct design is not “show the model’s inference about your inner state,” but “ask, verify, constrain, and keep inferences bounded.” [11]

A concise cross-disciplinary synthesis is below.

Why theory of mind is a performance lever

The strongest reason to invest in ToM is not that it makes models seem more human. It is that frontier-model failure modes are increasingly failures of partner modeling. When a model misreads what the user knows, what they are trying to optimize, whether they are joking, what they believe happened, what they are implicitly asking for, or what information is contextually inappropriate to reveal, the failure is often not a reasoning failure in the usual math-or-logic sense. It is a hidden-state inference failure. Recent computational work in psychology treats this precisely as the core problem of social cognition, and recent LLM papers show that functional performance depends heavily on whether the system explicitly represents such hidden states. [13]

The current benchmark landscape shows why this matters. Comprehensive evaluations find that frontier models can look strong on a few canonical tasks while remaining weak on breadth, recursion, dynamic updates, and real interaction. The problem is not just that models are “not yet human-level.” The deeper problem is that static benchmark success often overstates usable social intelligence. Both recent benchmark critiques and interaction studies argue that literal ToM — predicting beliefs or answers in a frozen scenario — is not the same thing as functional ToM, which is the ability to adapt to a partner across time and use inferred mental states to act well. [14]

The table below is the practical picture.

The most consequential takeaway is that generic “more reasoning” is not enough. A 2026 comparative study of reasoning models versus non-reasoning models on ToM benchmarks found that larger reasoning budgets often hurt performance, that slow thinking can collapse in social tasks, and that some models rely on option-matching shortcuts rather than genuine social inference. In other words, the path to better ToM is unlikely to be “just add more chain-of-thought.” It requires mechanisms and training that are specific to perspective, common ground, and latent-state tracking. [21]

Integrating theory of mind across the transformer model pipeline

The right way to build ToM into frontier models is to treat it as an end-to-end systems property. Data, objectives, curriculum, architecture, post-training, evaluation, and product affordances all have to align. If they do not, the model will either learn shallow benchmark tricks or acquire brittle social heuristics that do not survive deployment. That conclusion is reinforced by benchmark-robustness work, symbolic belief-tracking results, and new critiques of static ToM evaluation. [22]

A useful reference architecture is below. It is not a claim that one module already solves ToM. It is a practical blueprint synthesizing lessons from ToMnet, SymbolicToM, perspective-taking prompting, ToMAgent, ToM-SWE, multimodal ToM benchmarks, and mechanistic work on sparse ToM-sensitive subnetworks. [23]

flowchart LR
    A[User and environment signals] --> B[Perception stack text audio vision tools]
    B --> C[Shared world model]
    C --> D[Perspective selector]
    D --> E[Agent state store beliefs goals knowledge uncertainty]
    E --> F[Internal simulation loop]
    F --> G[Policy and decoder]
    G --> H[Response or action]

    I[Persistent memory opt-in and scoped] --> E
    J[Belief graph and common-ground tracker] --> E
    K[Safety and policy layer privacy deception high-stakes routing] --> G
    L[Post-action feedback corrections outcomes] --> I
    L --> J
    L --> C

The intervention portfolio below is the highest-leverage place to start.

Data and curation. The base data mix should move beyond generic internet text toward mental-state-dense interaction data. That means collecting and curating corpora where success depends on perspective and common ground: tutoring, clarification dialogues, enterprise support, task handoffs, planning conversations, collaborative coding, negotiation, fiction with explicit nested beliefs, and multi-party meetings. Annotation schemas should include at least: who knows what, who believes what, what each actor wants, what changed, what is private, what is merely inferred, and where the model should ask instead of infer. To reduce contamination and shortcut-learning, a meaningful fraction of training and evaluation data should be build-from-scratch rather than scraped from long-public false-belief inventories. Anthropology also suggests that the curation layer should encode culturally different norms for how explicitly people state or conceal mental states. [30]

Training objectives. Next-token prediction alone will not reliably induce stable ToM. The training stack should add auxiliary objectives that predict perspective-dependent world states and socially relevant latent variables. A good starting bundle is: belief-state prediction for each named actor; consistency losses on “who knows what” under perturbation; counterfactual next-action prediction conditioned on each actor’s beliefs rather than ground truth; dialogue lookahead objectives that reward mental-state representations useful for later task success; and contextual-privacy objectives that penalize disclosures which are inferable but inappropriate to reveal. Mechanistic work showing that tiny sparse parameter subsets strongly modulate ToM performance also suggests that explicit ToM objectives may not require wholesale architectural replacement to be effective. [31]

Curriculum. The developmental literature is an underused blueprint. The curriculum should begin with joint attention, shared reference, object use, desires, knowledge access, and common ground; then progress to first-order false belief, indirect requests, irony, and faux pas; then to higher-order recursive beliefs, deception, strategic interaction, and temporal mental-state evolution. This is not academic imitation for its own sake. It mirrors the increasing computational load of real human interaction and provides a more stable scaffold than throwing the model directly into high-order recursive tasks. [32]

Multi-agent simulation. If you want functional ToM, static text QA is insufficient. Models need repeated interaction with humans, synthetic agents, and policy-diverse partners under conditions where coordination, repair, trust, turn-taking, and strategic adaptation matter. Repeated-game results show that LLMs are relatively strong at self-interested games and weak at coordination-heavy ones, and that explicitly prompting the model to predict the other player’s behavior before acting improves coordination with human partners. That is a strong clue that social self-play and opponent modeling should become a standard post-pretraining stage. Dialogue-agent work points the same way: explicitly generating mental states and training them for dialogue utility improves long-horizon adaptation, and interlocutor-awareness work shows that partner-identity adaptation is simultaneously a capability gain and a safety issue. [27]

Architecture. I would not start by replacing the transformer. I would start by augmenting it with explicit state representations and bounded simulation. The most pragmatic options are below.

My recommendation is to combine four of these first: a symbolic/common-ground tracker, learned agent-state slots, an internal simulation loop, and a scoped user-memory layer. That bundle is much more likely to outperform “bigger base model + more RLHF” in real collaborative tasks. The reason is simple: ToM failures are often state-representation failures, and state representation is where plain transformers are weakest. SymbolicToM explicitly argues that scaling alone will not reliably induce ToM because the phenomenon is inherently symbolic and implicit, and newer reasoning-model work suggests that long generic reasoning traces can actively degrade ToM performance when they are not perspective-aware. [41]

Fine-tuning and RLHF. Post-training needs to stop rewarding only surface helpfulness. It should reward intent fidelity and socially appropriate adaptation. Preference data should compare not just “better answer vs worse answer,” but “better inferred goal,” “better uncertainty expression,” “better clarification choice,” “better common-ground repair,” and “better privacy decision.” For agentic systems, offline preference optimization should be followed by online or simulated interaction training where rewards reflect collaboration success, user correction rate, and policy compliance. Social reward modeling is promising, but it must contain explicit penalties for manipulative personalization, deceptive confidence, and privacy leakage; otherwise the same ToM machinery that improves collaboration will improve exploitation too. [42]

Evaluation. A serious ToM program should separate literal ToM from functional ToM. Literal ToM is static inference performance on benchmark tasks. Functional ToM is whether the model adapts well to real partners across time. Evaluation therefore has to combine static benchmark batteries with interactive studies, perturbation tests, and safety red teaming. It should include paraphrase perturbations, belief-likelihood manipulations, option-removal tests, recursive-depth escalation, dynamic state updates, multimodal narratives, and adversarial cases where the model is tempted to manipulate or over-infer private information. User-centered HCI critiques and benchmark critiques are right on this point: a model should not be called socially intelligent merely because it solves enough Sally-Anne variations. [43]

Deployment, product UX, and continuous learning. ToM-capable products should surface inference as inference. Good UX patterns include: “I may be inferring that you want X — is that right?”; editable user goals and constraints; transparent memory scopes; opt-in persistent state; per-fact deletion; request-to-confirm before acting on inferred intent in high-stakes contexts; and clear labels when the system is using profile or history signals. Continuous learning should come from explicit user corrections and consented outcome signals, not silent accumulation of increasingly intimate inferred mental profiles. Official risk frameworks are aligned with this direction: NIST emphasizes lifecycle risk management, oversight, tracking, and documentation for generative AI, while OECD and EU guidance emphasize transparency, meaningful human oversight, and proportionate guardrails. [44]

A practical evaluation stack is below.

Risks, governance, and product controls

The core risk is not abstract anthropomorphism. It is that better ToM is dual-use by default. A model that more accurately infers beliefs, desires, vulnerabilities, and likely reactions can be more persuasive, more manipulative, and more privacy-invasive. Recent studies show that GPT-4 with personalization was more persuasive than humans 64.4% of the time in conditions where persuasion differed, with an 81.2% relative increase in the odds of higher post-debate agreement. Privacy reasoning studies show that GPT-4 and ChatGPT revealed private information in inappropriate contexts 39% and 57% of the time, respectively. Separate work argues that deception abilities have already emerged in LLMs. Those three facts together are enough to reject any naive “just make it better at mindreading” program. [50]

A second risk is misinterpretation by the model. Human ToM itself is imperfect, culturally situated, and often repaired through interaction. Benchmarks that overfit static story puzzles can make teams think they have built real partner modeling when they have mainly built answer-pattern completion. Both classic developmental critiques and recent AI benchmark critiques warn that passing false-belief-style tests is neither necessary nor sufficient for broad social understanding. That means product teams should avoid building hard product logic on top of brittle inferred mental-state labels unless those labels are user-confirmable and uncertainty-aware. [51]

A third risk is multi-agent emergence. Repeated-game and interlocutor-awareness studies show that these systems already adapt to partner identity and can shift behavior substantially under social prompting. That is good for teamwork, but it also increases the risk of jailbreaking, collusive adaptation, strategic manipulation, and emergent coordination failures. The stronger the system becomes at modeling counterpart state, the more seriously evaluation must treat it as a strategic actor in some settings. [52]

A fourth risk is bias and fairness. If ToM layers are trained on noisy traces of preference or “user psychology,” they can easily become stereotype engines. Social-psychology evidence on perspective-taking is mixed, and anthropology shows that people differ in how they display intention, emotion, and privacy boundaries. Systems that confidently infer hidden states across demographic or cultural gaps will often be wrong in patterned ways. That argues for bounded inference, user correction, explicit uncertainty, and cross-cultural evaluation before launch. [53]

The right controls are not mysterious. They are below.

End-to-end proposal and roadmap

The most defensible compact proposal is a three-track program. The first track is core capability: ToM data, auxiliary objectives, explicit state modules, and interactive post-training. The second is evaluation and safety: functional ToM benchmarks, robustness tests, privacy/manipulation red teaming, and interpretability probes. The third is productization: transparent memory, user-editable goals, high-stakes routing, and launch only in products where mind-modeling creates value without making manipulation the business model. That means the first launch domains should be things like coding copilots, enterprise support, tutoring, collaborative planning, and agentic workflow orchestration — not consumer products optimized for engagement through personalized influence. [62]

The compact operating recommendation is below.

An 18‑month roadmap is realistic.

gantt
    title ToM R&D roadmap for a frontier model program
    dateFormat  YYYY-MM-DD
    axisFormat  %b %Y

    section Data and design
    Problem framing and taxonomy                    :a1, 2026-07-01, 2026-08-15
    Mental-state annotation schema                  :a2, 2026-07-15, 2026-09-15
    Curate interaction and fiction corpora          :a3, 2026-08-01, 2026-12-01
    Build procedural ToM data generator             :a4, 2026-08-15, 2026-11-15
    Multimodal ToM dataset pipeline                 :a5, 2026-10-01, 2027-02-01

    section Modeling
    Baseline breadth evaluation                     :b1, 2026-08-01, 2026-10-01
    Prototype belief tracker and common-ground module:b2, 2026-09-01, 2026-12-15
    Add agent-state tokens and simulation loop      :b3, 2026-11-01, 2027-03-15
    Continued pretraining with ToM objectives       :b4, 2026-12-01, 2027-04-15
    Interactive social fine-tuning and RL           :b5, 2027-02-01, 2027-06-01

    section Evaluation and safety
    Static benchmark suite                          :c1, 2026-09-15, 2027-01-15
    Dynamic and multimodal evaluation               :c2, 2027-01-01, 2027-05-15
    Interpretability probes and ablations           :c3, 2027-02-01, 2027-05-15
    Privacy and manipulation red-team               :c4, 2027-03-01, 2027-06-15
    External safety audit                           :c5, 2027-06-01, 2027-07-15

    section Product and deployment
    UX patterns for memory and correction           :d1, 2027-02-01, 2027-04-15
    Pilot in coding and enterprise support          :d2, 2027-04-15, 2027-07-15
    Governance review and launch decision           :d3, 2027-07-01, 2027-08-01
    Limited rollout with monitoring                 :d4, 2027-08-01, 2027-12-01

The go/no-go thresholds should be explicit. I would require, at minimum, broad gains on ToMBench/OpenToM/Hi-ToM or equivalent, measurable improvements in interactive partner-adaptation tasks, no statistically meaningful increase in privacy leakage or persuasive-manipulation capability under red teaming, and interpretable evidence that the system is actually maintaining perspective-dependent state rather than merely learning benchmark surface forms. That is stricter than current practice, but it is the correct bar if ToM is being sold internally as a foundational capability rather than a demo trick. [63]

Beyond transformers

The short version is blunt: I would not bet that plain larger transformers, even with bigger reasoning budgets, will produce robust machine ToM by themselves. The evidence is increasingly against that story. Symbolic belief-tracking work explicitly argues that scale alone will not induce ToM reliably, recent studies of reasoning models show that longer generic reasoning traces can hurt ToM performance, and benchmark critiques show that even strong literal-ToM performance may fail to transfer into functional adaptation. The near-term winning approach is much more likely to be hybrid. [64]

The strongest alternatives fall into five families. First, neuro-symbolic systems are already promising because mental-state reasoning is naturally relational and nested; SymbolicToM and COKE both point toward explicit graph- or chain-based representations of beliefs, intentions, and affective consequences that complement the generative fluency of transformers. Second, cognitive and Bayesian architectures matter because modern psychology increasingly models ToM as probabilistic inference over rational action under bounded computational budgets; if that framing is right, then explicit inverse-planning or Bayesian state-estimation layers may be necessary for hard social tasks even when the language backbone remains a transformer. Third, embodied and multimodal agents are likely essential for the next jump because many real mental states are expressed in gaze, timing, prosody, action, and egocentric context, and current multimodal ToM benchmarks show clear headroom. Fourth, modular or hierarchical transformers with dedicated agent-state memory and routing may outperform monolithic decoders because they can isolate perspective-dependent computation. Fifth, continual- and meta-learning systems should be taken seriously because ToM is partly the ability to rapidly build a model of a specific partner from very few interactions, which is exactly what ToMnet and recent user-modeling agents are designed to do. [65]

If I had to pick one strategic direction, it would be this: keep the transformer as the language-and-world-knowledge substrate, but build a hybrid social-cognition layer around it. That layer should include explicit partner-state representations, symbolic common-ground tracking, bounded internal simulation, and privacy-aware memory. That is more plausible than expecting next-token prediction plus generic reasoning scale to spontaneously become a stable model of minds. It is also more controllable, more auditable, and more compatible with product governance. [66]