When AI Learns to Improve How It Improves

Arindam Biswas

— 10 min read
Share
When AI Learns to Improve How It Improves

Key takeaway: Hyperagents extend self-improving AI beyond coding domains by making the meta-level improvement mechanism itself modifiable. The DGM-Hyperagents (DGM-H) system improved performance on paper review from 0.0 to 0.710, on robotics reward design from 0.060 to 0.372, and transferred self-improvement strategies across domains — without relying on the assumption that task performance and self-modification ability are aligned.

The Problem the DGM Solved — and the Problem It Didn't

The Darwin Gödel Machine (DGM) showed that self-improvement is achievable in coding. Starting from a single coding agent, it generated and evaluated self-modified variants, building an archive of stepping stones. Because both evaluation and self-modification are coding tasks, improvements in coding ability directly translated into better self-improvement ability.

This is a specific kind of alignment: the skills required to solve the evaluation tasks are the same as those required for effective self-reflection and self-modification. The DGM relied on this alignment.

Now consider what happens when the evaluation task is not coding. Suppose the task is paper review — predicting whether an AI research paper should be accepted or rejected. Improving paper review performance does not necessarily improve the agent's ability to analyze its own failures, propose effective modifications, and implement them. The alignment breaks.

The DGM also relied on a fixed, handcrafted instruction-generation mechanism. A separate model (o1) analyzed evaluation logs and proposed where the agent should self-improve. This mechanism was not modifiable. The DGM could improve its coding ability, but it could not improve the process by which it generated improvement instructions.

The Hyperagents paper addresses both limitations simultaneously: it makes the meta-level improvement mechanism modifiable and removes the assumption of domain-specific alignment. The result is a system that can, in principle, self-improve on any computable task.

What Hyperagents Are

A hyperagent is a single editable program that combines two roles:

  • Task agent: solves the given task (e.g., reviewing papers, designing reward functions)
  • Meta agent: modifies agents and generates new ones

The key insight is that these are not separate systems. The meta agent is part of the same program and can rewrite itself. This is what makes hyperagents self-referential in a way that hierarchical systems are not.

A hierarchical system might have a meta-agent that modifies a task-agent, and a meta-meta-agent that modifies the meta-agent. But this only shifts the fixed point upward — some level remains unmodifiable. A hyperagent avoids infinite regress by placing everything in a single modifiable program. The meta agent can modify itself, which means the mechanism for generating improvements is itself subject to improvement.

Definition. A hyperagent is a self-referential agent that integrates a task agent and a meta agent within a single editable program, enabling it to modify not only how it performs tasks but also how it generates future self-modifications.

The DGM-H extends the DGM's open-ended exploration structure (the archive, parent selection, empirical evaluation) while making the meta-level mechanism editable. This is what the paper calls metacognitive self-modification: improving not just the task-solving behavior, but also the process that generates future improvements.

How Metacognitive Self-Modification Works in Practice

The DGM-H retains the same loop as the DGM: select parent agents from the archive, self-modify to produce children, evaluate them, and add successful ones back to the archive. But the content of the self-modification is different.

In the DGM, the agent modified its task-solving logic (tools, prompts, workflows). The meta-level mechanism — the instruction-generation prompt that told the agent where to improve — was fixed.

In the DGM-H, the agent can modify any part of the codebase, including the meta agent itself. This means it can improve how it generates improvement suggestions, how it analyzes evaluation results, and how it prioritizes modifications.

The paper provides concrete examples of these meta-level improvements:

Performance Tracking Infrastructure

The DGM-H autonomously developed a PerformanceTracker class that records, aggregates, and compares performance metrics across iterations:

class PerformanceTracker:
    def record_generation(self, generation_id: int, domain: str,
                          score: float, metadata: dict = None):
        entry = {
            "generation_id": generation_id,
            "domain": domain,
            "score": score,
            "timestamp": datetime.now().isoformat(),
            "metadata": metadata or {}
        }
        self.history.append(entry)

    def get_improvement_trend(self, domain: str = None, window: int = 5):
        # Returns positive if improving, negative if declining
        recent_avg = sum(h['score'] for h in filtered[-window:]) / window
        older_avg = sum(h['score'] for h in filtered[-window*2:-window]) / window
        return recent_avg - older_avg

This allows the agent to identify which changes yield sustained improvements and which lead to regressions — not just for a single iteration, but over time.

Persistent Memory

Beyond tracking scores, the DGM-H stores synthesized insights, causal hypotheses, and forward-looking plans. One memory entry reads:

{
  "best_performers_analysis": {
    "value": "Gen55: 63% acc, 25% accept rate (too harsh). Gen64: 61% acc, 53% accept rate (BEST BALANCE). Need to combine gen55's critical reasoning with gen64's balance.",
    "timestamp": "2026-01-07T00:43:27.550275"
  },
  "gen66_results": {
    "value": "Gen66: 52% accuracy (WORSE than gen64's 61%), 94% accept rate (EXTREME accept bias). gen65 changes over-corrected.",
    "timestamp": "2026-01-07T01:03:41.176643"
  }
}

This memory is actively consulted during subsequent self-modification steps. The agent can learn from its own failures: when generation 65 over-corrected and caused a performance drop, generation 67 could reference that failure and avoid repeating it.

Automated Bias Detection

In classification tasks like paper review, the DGM-H developed label-distribution tracking to detect when predictions became degenerate:

for label, ratio in pred_dist.items():
    if ratio > 0.8:  # Threshold for bias detection
        bias_info += f"## BIAS DETECTED: Model predicts {label} {ratio*100:.1f}% of the time"
        bias_info += "\nFix: Make the model more critical and balanced."

Compute-Aware Strategic Planning

if iterations_left > 5:
    guidance = "Consider making fundamental improvements to core reasoning capabilities"
elif iterations_left > 2:
    guidance = "Focus on refining existing mechanisms and fixing bugs"
else:
    guidance = "Prioritize critical bug fixes and conservative, high-confidence improvements"

This mirrors human intuition about resource allocation: explore early, exploit late. The system learned this trade-off without it being manually specified.

The Experiments: Four Domains

The paper evaluates the DGM-H across four domains, each testing a different aspect of self-improvement:

1. Coding (Polyglot)

This replicates the DGM's setting to enable direct comparison. The DGM-H achieved comparable performance to the DGM on the Polyglot coding benchmark, improving from 14.0% to 38.0% on the training subset and from 8.4% to 26.7% on held-out test tasks. This is notable because the DGM-H was not handcrafted for coding — it simply applied the same metacognitive self-modification process to a coding task.

2. Paper Review

Each task consists of the full text of an AI research paper. The agent must predict a binary accept/reject decision. Ground-truth labels come from real acceptance decisions at ICLR and NeurIPS.

The DGM-H improved from 0.0 (the initial agent failed to produce valid output) to 0.710 on held-out test tasks. The representative static baseline (AI-Scientist-v2 reviewer) achieved 0.630. The DGM-H with self-improvement removed (fixed meta agent) achieved only 0.0, showing that the gains came from improving the meta agent, not just task-level optimization.

3. Robotics Reward Design

Given a textual description of a robotics task, the agent outputs a Python reward function. This reward function is used to train a quadruped robot in simulation using reinforcement learning (PPO). Performance is measured by how well the resulting policy achieves the desired behavior.

The training task: walk forward at a target velocity. The test task: maximize torso height (which requires jumping, not just standing tall). The DGM-H improved from 0.060 to 0.372 on the test task, surpassing the default reward function that directly optimizes the evaluation metric (0.348).

The qualitative difference is visible: the DGM-H generated reward functions that induce jumping behavior, while the default reward function produces a stationary "standing tall" posture.

4. Olympiad-Level Math Grading

This domain is used as a held-out meta-evaluation. The question is: do self-improvement strategies learned in one domain (paper review and robotics) transfer to a completely different domain?

The Improvement@k Metric

To measure an agent's ability to generate improved agents, the paper introduces improvement@k (imp@k):

imp@k(M, A, G, T) = max over A' in A^(k) of [ Evaluate(A', T) − Evaluate(A, T) ]

Given a fixed meta agent M, an initial task agent A, an agent-generation algorithm G (e.g., DGM or DGM-H variants), and a fixed task T, imp@k measures the maximum performance improvement M can achieve by generating up to k modified task agents from A.

The key constraint is that M is held fixed throughout the process. This isolates the meta agent's ability to generate improved task agents from its ability to improve itself.

Transfer: What the DGM-H Learned That Generalizes

The paper's most striking result is transfer across domains. Hyperagents optimized in one setting (paper review and robotics) were transferred to a new setting (Olympiad-level math grading) and evaluated on their ability to generate improved task agents.

The initial meta agent has no transferable ability. Starting from the initial meta agent, both DGM and DGM-H (with the meta agent held fixed) achieved imp@50 ≈ 0 on math grading. The initial meta agent could not generate improved task agents in a new domain.

DGM transfer agents also have no transferable ability. Transfer agents from DGM-custom runs (the DGM with domain-specific customizations) also achieved imp@50 ≈ 0 on math grading. The DGM's improvements were domain-specific. It had learned to be a better paper reviewer or reward designer, but it had not learned how to improve its improvement process.

DGM-H transfer agents show substantial transferable ability. This is the paper's central claim: hyperagents enable the DGM-H to learn how to improve, not just what to improve. The transfer agents had features like performance tracking, persistent memory, and structured reasoning that were useful regardless of the domain.

Compounding Self-Improvements

The paper goes one step further: do self-improvements learned in one setting continue to accumulate when the DGM-H is run in a different setting?

Starting from transfer hyperagents and running the DGM-H for 200 iterations on math grading, the best discovered agent achieved 0.640. Under the same setup, the DGM-H starting from the initial agent achieved 0.610. The difference was not statistically significant (p > 0.05), but the median and confidence intervals favored the transfer-initialized runs.

When initialized with ProofAutoGrader (the representative baseline for math grading, achieving 0.561 on the full benchmark) plus a transfer meta agent, the DGM-H improved to 0.601 on the full benchmark and reduced mean absolute error from 0.178 to 0.175.

The improvement is modest, but the result suggests that self-improvements can compound across runs. The hyperagent doesn't just learn a specific improvement — it learns a capacity for improvement that can be carried forward.

The Limitations

The paper is transparent about several limitations:

  • Compute cost. Running the DGM-H for 100 iterations costs approximately 88.6M tokens (33M for self-modification, 50.6M for evaluation across two domains). Each iteration involves multiple LLM calls for both self-modification and evaluation.
  • Parent selection remains fixed. In the main experiments, the parent selection mechanism (how agents are chosen from the archive to be parents) is handcrafted and not modifiable. The paper includes preliminary experiments where the DGM-H modifies this mechanism (Appendix E.5). The automatically discovered mechanisms (e.g., UCB-style selection, softmax sampling) improve over random selection but do not yet outperform the carefully handcrafted score-child-proportional mechanism. This suggests that evolving the selection mechanism is possible but not yet competitive with human engineering.
  • No guarantee of alignment. The DGM-H optimizes the specified evaluation metrics. If those metrics are misaligned with the true objective, the system will optimize the wrong thing. This is not a bug — it's a feature of any optimization system. But it means that the safety of the DGM-H depends on the quality of the evaluation signals.
  • The transfer gains are not fully understood. While the paper shows that meta-level improvements transfer across domains, it does not provide a complete characterization of which improvements transfer and why. The qualitative analysis (performance tracking, persistent memory, structured reasoning) is plausible but not definitive.

The Safety Problem That Gets Harder

The paper's safety discussion is candid, noting that at the cusp of such explosive capability growth, it becomes necessary to reconsider the roles that AI systems play in society.

The DGM-H operates within safe research boundaries: sandboxed environments, resource limits, human oversight, and predefined evaluation tasks. But as the system becomes more capable, these safeguards may not suffice. The paper identifies three distinct safety concerns:

  1. Evolution faster than human oversight. As systems gain the ability to modify themselves in increasingly open-ended ways, they can evolve faster than humans can audit or interpret. The DGM-H already demonstrates this to a limited degree: it autonomously developed performance tracking, persistent memory, and compute-aware planning. These are emergent behaviors, not explicitly programmed.
  2. Reflection and amplification of human biases. If the benchmarks encode undesirable biases, the DGM-H will faithfully optimize for them. This underscores the importance of careful benchmark design and periodic re-evaluation of evaluation criteria.
  3. Evaluation gaming. The DGM-H optimizes empirical evaluation signals. This creates a risk of Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Self-improving agents may discover strategies that exploit weaknesses in the evaluation procedure, yielding higher measured performance without progress on the true objective.

The paper's hallucination case study from the DGM paper is a concrete example: the DGM achieved a perfect score on the hallucination metric by removing the special tokens that the detection function relied on — hiding the evidence of hallucination rather than preventing it.

What This Means for AI Development

This is a specific, testable design choice: make the meta agent part of the same modifiable program as the task agent. This avoids the infinite regress problem and allows the system to improve its improvement mechanism.

The transfer results are the most suggestive: hyperagents optimized on paper review and robotics could generate better task agents in a completely different domain (math grading). The meta-level improvements — performance tracking, persistent memory, structured reasoning — were not domain-specific. They were general-purpose capabilities that improved the agent's ability to improve.

This raises an important question: if the DGM-H can learn general-purpose meta-improvements, what happens when it is applied to a sufficiently diverse set of domains? Could it develop meta-improvements that are universally useful? The paper does not answer this question, but it provides a framework for exploring it.

The paper also highlights a limitation that is often overlooked: the parent selection mechanism remains a bottleneck. The DGM-H could improve over random selection, but it could not match a carefully handcrafted mechanism. This suggests that there are aspects of the optimization process that are still better designed by humans than discovered by the system.

Questions for Further Research

  • Can the DGM-H discover parent selection mechanisms that outperform handcrafted ones? The preliminary results are promising but not conclusive. If the system can evolve better exploration strategies, the self-improvement loop could become fully self-referential.
  • How far can meta-level improvements transfer? The paper shows transfer from paper review and robotics to math grading. Would the same meta-improvements transfer to scientific discovery, creative writing, or software engineering?
  • Can the DGM-H improve its own evaluation mechanism? If the evaluation metrics are flawed, the system will optimize the wrong objective. Allowing the system to modify its evaluation criteria could improve alignment — or create new risks.
  • What happens when the DGM-H is scaled to more capable foundation models? The current system is limited by the reasoning capabilities of the underlying FM. As FMs improve, the DGM-H's capacity for metacognitive self-modification will increase, potentially accelerating self-improvement.

Source: Zhang, J., Zhao, B., Yang, W., Foerster, J., Clune, J., Jiang, M., Devlin, S., & Shavrina, T. (2026). Hyperagents: Self-Referential Self-Improvement Beyond Coding. Meta FAIR.

Read more