Tech Firms Pursue AI Systems That Self-Improve
Tech firms are racing to develop AI systems that can improve themselves, with companies like Google and startups exploring recursive self-improvement (RSI) as the next frontier in machine learning advancement. This process involves AI models designing and training new, more capable versions of themselves, automating key aspects of research and development that have traditionally required human intervention.
AI Automating Development Tasks
Google reports that AI agents now draft and refine 75% of new code internally, while Anthropic exceeds 80%, according to recent findings. These systems are being deployed to handle tasks like coding, mathematical reasoning, and testing, accelerating the pace of innovation. Tulsee Doshi, senior director at Google’s Gemini models, noted that the team has focused on enhancing Gemini’s ability to perform core developer functions, including building features and restructuring scripts. She emphasized that early AI-driven improvements have already yielded significant gains, though specific details remain undisclosed for competitive reasons.
The firm recently launched a voice-optimized Gemini model, developed with AI assistance to refine its system instructions—hidden guidelines it consults before responding to queries. Doshi highlighted that RSI tools helped quickly iterate on prompts to align with user experience goals. However, no company has yet achieved true RSI, which would require AI to independently execute the full scientific method: hypothesize, implement, analyze, and refine until breakthrough results emerge.
Startup Ambitions and Human Oversight Concerns
Among those betting heavily on autonomous AI research is French challenger Mistral, with Pierre Stock, its vice-president for science, calling RSI a “paradigm change” to outpace competitors. Similarly, startups like Discovery Loop, co-founded by former Google DeepMind executives, aim to automate scientific discovery across fields like drug development and cybersecurity. Jeff Clune, co-founder of Recursive and former OpenAI researcher, previously demonstrated an AI system that generated research ideas meeting academic conference standards.
Yet prominent AI researchers caution against relinquishing control. Hugo Larochelle of Montreal’s Mila institute distinguishes between “day science” tasks AI can handle, coding, tweaking experiments, and the more abstract “night science” of identifying unsolved problems. He argues humans must retain authority over resource allocation and strategic decisions, stating, “This is where we decide where we spend our resources.”
Concerns extend to existential risks, with a recent paper co-authored by Jeff Clune, Geoffrey Hinton, and Yoshua Bengio warning that AI-driven RSI could trigger an uncontrollable intelligence explosion. Bengio’s non-profit LawZero, backed by $300 million in government funding, is developing “Scientist AI” as a potential safeguard against such outcomes.
New Startups Pursuing Autonomous Research
Discovery Loop, launched by Jeff Dean and other ex-DeepMind leaders, aims to automate the scientific method across disciplines such as drug development and cybersecurity. Radical Ventures co-led its seed funding. Meanwhile, Recursive, founded by Jeff Clune and his students, demonstrated an AI system that generates and validates research proposals meeting academic standards, according to a 2024 paper.
Developing Safeguards Against Uncontrolled Growth
Risks accompanying AI-driven self-improvement have prompted urgent calls for oversight. A paper co-authored by Jeff Clune, Geoffrey Hinton, and Yoshua Bengio warned that unrestricted RSI could trigger an uncontrollable intelligence explosion, citing “extreme risks” from accelerating AI capabilities. Bengio emphasized the need for safeguards, stating that AI systems designing successors must not harbor hidden harmful objectives.
To address these concerns, LawZero, Bengio’s non-profit AI lab, secured $300 million in government funding to build “Scientist AI,” intended to act as a check on other advanced systems. Larochelle of Mila cautioned that human researchers must retain authority over strategic decisions, particularly in “night science” areas requiring creative problem identification. He argued humans uniquely determine which research directions to pursue, noting that AI can automate technical tasks but not replace human judgment in prioritizing scientific challenges.
