Embodied AI Scientists: Running Experiments in the Physical World

What happens when AI steps out of the purely computational realm and begins interacting with the physical world in a safe, structured, and scientific manner?

Imagine an autonomous laboratory of embodied AI scientists who design experiments, recruit participants, analyse data, and write papers all without human intervention, while simultaneously offering a collaborative framework where human and machine intelligence can combine for even more powerful scientific discovery. This isn't science fiction; it's what we've built at Explore Science.

We're excited to share a significant milestone in our journey: what we believe is the first AI system capable of autonomously conducting end-to-end scientific research with human participants. This breakthrough bridges a critical gap between in-silico AI and physical experimentation, creating valuable new inroads for scientific exploration.

Read on to learn:

• How we've created an embodied AI system capable of conducting real-world scientific experiments
• The innovative architecture that powers our autonomous research system
• A case study showcasing the results of our approach and system
• A discussion of the broader implications for science, research, and society

The full technical report will be released in the near future, so be sure to check back for the full details!​​

The link will be available here shortly.

From Computation to Physical Experimentation

Our vision is to contribute to humankind’s collective knowledge through the facilitation of autonomous scientific discovery, in part achieved by the extension of AI-driven scientific discovery from in-silico, to physical. While recent advances have showcased impressive capabilities in computational domains - such as Sakana AI's system which pioneered autonomous production of machine learning research papers - an extension of these systems in the context of our broader vision is to independently and iteratively test hypotheses through real-world experimentation. This boundary was particularly relevant in our first field of focus: psychology and cognitive science, where understanding human behaviour requires direct experimental interaction.

The embodied AI Scientist we have built enables AI to not just analyse existing literature or run simulations - it is an end-to-end pipeline for scientific discovery, capable of successfully formulating novel hypotheses, designing a study and constituent experiments, recruiting human participants for the experimental implementation, analysing collected data, producing supporting visualisations, synthesizing and interpreting findings, and constructing a final report on the findings.

Multi-Agent Architecture with Human-Inspired Cognition

Humans possess remarkable cognitive abilities, which allow us to navigate the complexities of scientific discovery. In pursuit of deep explanations for why our universe is the way it is, we can learn across domains, plan extended projects, and adapt our strategies based on feedback. Collaboration also often yields results that are vastly greater than any one researcher can attain working alone. At Explore, we asked the questions:

What would happen if we could empower AI agents with cognition-like principles?

What could be achieved if several of these agents, each capable of human-like higher-order executive functioning, worked together?

While Large Language Models (LLMs) excel at pattern recognition and knowledge retrieval, they typically lack the self-regulatory mechanisms that enable human scientists to manage highly complex multi-stage inquiries. We hypothesised that cognitively-enhanced agents may coordinate effectively over extended time horizons to complete complex scientific tasks and even the entire end-to-end scientific workflow… and we weren’t wrong. Our system autonomously produces scientific research through the coordinated and collaborative efforts of a task-force of specialised agents, like a master conductor leading a symphony orchestra composed of virtuosos and maestros.

The dynamic hierarchy of the system, which features layers of ‘Orchestrators’ (Agents with the ability to coordinate and spin up further sub-agents for themselves, such as the Method Agent and Data Analysis Agent) and ‘Specialists’ (Agents excelling in honed skillsets, such as the Coding Agent and Archivist Agent), results in an emergent cascade of expertise and directed attentional flows.

Cognitive Principles in Action

We identified early on in our development four key cognitive principles - abstraction, metacognition, decomposition, and autonomy - which serve as the architectural foundation for each of our agents. By implementing computational analogues of these executive functions into our ‘Cognitive Agents’, they demonstrated remarkable capacity to work effectively and smoothly in collaborative teams across the entire research project. Each principle plays a distinct role in enabling an agent to tackle the multifaceted challenges arising during tasks of the scientific workflow, and indeed many other complex tasks in the real-world.

Here is a brief overview of the core cognitive principles:

Abstraction: the cognitive process of focusing on general patterns rather than specific details, was operationalised by enabling agents to develop their own heuristics and instructions rather than constraining them with predetermined directives. For example, the Figure Creation Agent is not given specific rules for how a Figure should look, instead it is given broad instructions to come up with its own metrics for exceptional task completion given the specific research context.

Metacognition: the awareness and regulation of one's own thinking processes, was implemented through structured chains of self-reflective thought prior to reaching conclusions. For instance, when analysing experimental data, the Data Analysis Agent evaluates its own approach at every point. This allows the agent to recognise that the data type may violate assumptions for the tests it wants to run, prior to running statistical analyses. It can then evaluate whether it needs to switch to alternative options, while documenting its reasoning along the way.

Decomposition: the breaking down of complex problems into more manageable components, was critical to agents achieving outcomes of an exceptional quality. For example, when transforming a hypothesis into testable methodology, the Methods Agent systematically divides the process into discrete steps and validates each - identifying variables, designing manipulations, and selecting appropriate tasks. Critically, the transparency of this approach also makes the entire process accessible for objective evaluation by other agents and humans, resulting in robust research designs that effectively operationalise and test theoretical concepts.

Autonomy: self-directed goal pursuit. Whilst we have defined the overarching vision of the operational framework, each agent independently completes assigned tasks and creates new goals for itself in line with the overall vision. The agents continue working until they deem the required task has been completed to a satisfactory standard - confirmed by its supervising agent. Agents can also decide to create subagents, as needed, to help them complete the task, with each subagent adapting in real-time based on output quality.

For the multi-agentic system to maintain coherence over extended periods, we also found it crucial to integrate dynamic memory, which augments each agent with the prior knowledge and specific information necessary to carry out its task.

Cognitive Offloading and Dynamic Memory

Scientists don't memorise every paper they've read in full. They remember key concepts and will often refer back when needed to find specific details. In general, humans navigate complex tasks utilising sophisticated memory systems. We can i) hold and manipulate information in working memory, ii) selectively filter relevant details for the task at hand, and iii) offload information to external resources when our internal capacity is exceeded. This cognitive flexibility allows us to execute multi-step processes while maintaining focus on the most relevant information at each stage.

To emulate this cognitive capacity in our agentic framework, we developed a dynamic-Retrieval-Augmented Generation (d-RAG) system. AI often uses the same reference material regardless of context; however, our d-RAG system creates and evolves specialised knowledge repositories for each research direction traversed. The system progressively builds its knowledge base by processing retrieved academic papers in response to Agents’ queries, discarding irrelevant materials, and maintaining focussed representations tailored to the specific research question. This approach mimics how human researchers develop domain-specific expertise through targeted literature engagement - concentrating on the most relevant sources while setting aside less pertinent information. In addition, by allowing the system to offload complexity to specialised components, a concise and compact representation of the overall research state is able to be maintained at all points.

Mixture of Agents Approach

Due to the diversity of tasks within the scientific workflow - each posing unique cognitive challenges - we found it beneficial to assign different aspects to various frontier models. All models were trialled for each component, with particular models performing best given their unique capabilities and strengths. The primary model never operated alone, however, as a Mixture of Agents (MoA) approach (where additional models were employed within the same task) was found to also greatly increase the robustness of the system. The complementary mixing and matching of models enabled the system to collaboratively accomplish tasks that proved highly challenging for any individual model to complete alone.

While the original proof-of-concept of our system achieved end-to-end successful runs in December 2024 with earlier foundation models (e.g. Claude 3.5 Sonnet & GPT-4o), the latest frontier models facilitated meaningful improvements in accuracy, rigour, and performance reliability. With the newest models, we observed a large reduction in statistical reasoning errors and a notable decrease in the time required to generate valid analysis code (days to hours). The models employed in our system include various providers - Anthropic, OpenAI, xAI and Google - where the models shown below worked best for our purposes so far:

Implementation: Bridging AI and Human Participants

A novel innovation of our system is its ability to go out and run experiments in the real world. For our initial experiment, we interfaced the system directly with human participants via online platforms. To ensure our team’s ability to validate the findings, we provided the system with pre-validated cognitive tasks and questionnaires from published research; aligning with our ethical approval requirements. The Implementation Agent operationalised the research by:

• Using API interactions with Prolific, a web-based recruitment platform for human participants
• Defining participant recruitment parameters (sample size, age, and vision requirements) based on the ethics approval
• Creating a complete draft study on Prolific, compatible with the Pavlovia experimental hosting platform.

How did we ensure ethical compliance? In consideration of ethics and ethical compliance, we implemented a manual verification step prior to publication of the study online, carefully checking all parameters before recruiting participants. Though in reality, this system can run autonomously.

What about participant privacy? The system maintains rigorous privacy standards. All participant data is completely de-identified by the online recruitment platforms themselves. Our system never has access to any personal identifying information, working only with de-identified research data that conforms to ethical standards.

The Scientific Process, Automated

Our Embodied AI Scientist handles the entire research workflow (upward of 12 distinct phases) with key stages including:

Hypothesis Formulation

The system identifies promising research questions through extensive chain-of-thought reasoning and validates them through literature searches using the Semantic Scholar API to formulate a set of hypotheses and aims. Each idea is also scored against metrics including novelty, breakthrough potential, and feasibility; to inform the decision of which idea to pursue further.

Experimental Design

Good scientific practice demands meticulously designed methodologies. As such, we built the system to design rigorous, clear experimental protocols tailored to each research question. The process includes automated power analyses - writing and executing code scripts that extract parameters from extensive literature searches - to calculate precise sample sizes for statistical validity. The system also produces a detailed pre-registration, documenting hypotheses and complete data analysis plans - gold-standard practices used in clinical trials and robust psychological research to prevent p-hacking or HARKing (Hypothesizing After the Results are Known). This methodological rigour is built into the system's core functionality, with agents continuously checking their work against pre-registration documents throughout implementation and analysis, ensuring accordance with a priori intentions. When deviations are necessary, agents demonstrate flexible reasoning by transparently documenting the rationale for changes while maintaining scientific integrity, enabling rigorous critical peer review.

Data Analysis

Once data is collected, the system performs sophisticated, theory-driven analysis that goes far beyond simple data manipulation and rudimentary statistics. This includes:

• Developing theoretically justified data cleaning protocols tailored to each experimental paradigm
• Making reasoned decisions about outlier exclusion based on established research standards
• Calculating derived variables corresponding to theoretical constructs in the literature
• Implementing complex statistical procedures and modelling

Collaboration between numerous agents, each leveraging different frontier LLMs, allows the system to draw together varying opinions to dynamically adapt, address and rectify the errors that naturally occur when building an analysis pipeline from scratch. What is remarkable is the system's ability to handle data across hundreds of csv files, thousands of rows, hundreds of participants, and multiple data formats, requiring deep theoretical insights to construct appropriate variables and create meaningful measures.

Visualisation

Visually presenting complex data often plays a large role in enhancing the accessibility and impact of scientific findings. As such, our framework was developed with this ability in mind, ensuring experimental results could be translated into clear, effective, and informative visualisations. Following data analysis, the system autonomously generates visualisation strategies for both methodological procedures and experimental results. It creates custom SVGs for experimental task documentation and develops appropriate plots and figures to highlight key patterns in findings. The system determines the most suitable visualisation formats based on data characteristics and theoretical relevance, generating comprehensive captions that contextualise each visual element.

Report Generation

Once all experiments for the study are completed, the framework writes a comprehensive manuscript with embedded figures, validated references, and contextual discussion. It situates findings within broader theoretical frameworks and identifies implications for future research. To prevent reference hallucinations - a pervasive LLM problem - we built a rigorous verification framework into the system that extracts citations from generated text and systematically checks their authenticity against actual published papers in the literature, fixing any incorrect or fabricated references.

Simplified representation of key aspects in the embodied AI system.

Central to this entire process is thorough, uncompromising critical review. We've incorporated rigorous evaluation into every step of the pipeline. For a taste of the tough standards we apply, you can visit the manuscript review we built for researchers, Explore Science. The platform sets the bar high for scientific quality (and comes with a “tough-love” warning), aiming to help scientists improve their work prior to publication.

Cognitive Psychology Case Study

To illustrate the achievements of the system, here we present an interesting paper completed by the embodied AI scientist in cognitive psychology. The study investigated whether people who excel at remembering visual details also perform better at mentally rotating objects. While conventional wisdom might suggest yes, the AI scientist discovered something different. Results revealed only modest correlations between visual working memory sensitivity and mental rotation performance, which remained stable across varying task demands rather than strengthening under increased cognitive load. These processes seemed to maintain partial independence, which cautiously challenges theoretical models positing unitary resource models of visuospatial cognition. PDF

The entire process, from hypothesis formulation to polished manuscript document, required 12.5 hours of time-on-task and ~$98 (~27M input tokens/1.7M output tokens across models) in LLM token costs (excluding participant recruitment time of ~6 hours + participant payments). This represents a dramatic compression of the traditional research timeline, which typically spans multiple months or years. It literally takes a day. For those interested in taking a look at some of the outputs produced by the system to navigate project completion, all raw data, pre-registration, generated data analysis code, and methodological details of the paper are available in our GitHub repository.

A particularly impressive aspect of this AI-generated research is how it embraces methodological practices that even many human researchers forego, or may overlook. Section 3.5 of the paper stands out as a particularly noteworthy example, where the system conducted a thorough measurement reliability analysis - something rarely included in published studies despite being crucial for properly interpreting results. Furthermore, the system transparently and clearly acknowledged when hypotheses weren't supported, avoiding the publication bias that plagues human research. The paper's execution of a full multi-stage, complex, theoretically-driven data analysis - from multi-factor ANOVAs to attenuation-corrected correlations and reliability assessments - integrating theoretical predictions with empirical patterns and accounting for measurement limitations, also demonstrates sophisticated analytical prowess.

It is important to note that the paper presented here has had absolutely no human modification. As a result, there are some subtle issues to be noted, including minor presentation flaws relating to the spatial configuration of Figures, and methodological issues such as decisions to maintain quite conservative data cleaning criteria thresholds, which resulted in relatively high exclusion rates of participants. It is worthwhile noting, however, that this is not necessarily a problem, considering that online testing environments are inherently prone to lower data quality than laboratory settings. The paper could also benefit from additional references to support some claims, and further justification for choices, such as using different sample sizes for different analyses which is defensible for maximizing available data. As we mentioned earlier, our manuscript review, Explore Science, is extremely comprehensive, and flagged several points of improvement for this paper. In collaboration with human input and amendments, however, the papers produced by our system could be polished to the standard required for a top-tier publication with minimal effort.

Our scientific paper is forthcoming, which outlines the technical aspects of our system, its performance characteristics, the full experimental results, detailed evaluation, and further highlights from other completed research papers. Stay tuned! @ExScienceAI

Implications and Future Directions

The development of embodied autonomous scientific systems marks a significant shift in how research can be conducted, introducing both promising opportunities and substantial challenges that deserve careful consideration.

Ethical Dimensions of AI-Driven Science

In our implementation with cognitive psychology experiments, human researchers-maintained verification and approval of all experimental components prior to participant interaction. As these systems evolve toward greater autonomy, robust oversight mechanisms become increasingly essential to uphold ethical standards. With every new technological innovation, there are ethical considerations that must be carefully considered. Although potential risks exist with autonomous AI systems, with sufficient consideration of design, AI scientific exploration can remain safe and aligned with human values. The transparency of such systems represents another critical ethical dimension that we believe is essential for externally validating the contributions of AI frameworks to scientific discovery. To this end, open-sourcing the pre-registration plan and code scripts produced during the autonomously conducted investigation enables independent evaluation of the quality and accuracy of the work, and helps establish trust in the rigour of the scientific workflow, while acknowledging the evolving role of AI in knowledge creation.

Human-AI Collaboration in Scientific Discovery

Scientific discovery stands upon a new precipice - one where AI systems are not just assisting research but actively participating in generating new knowledge. While our system can do so autonomously, we view this system not as a replacement for human scientists, but rather as a tool that will amplify human creativity and insight. Applications involving collaborative research, where AI handles aspects of the scientific workflow that lie outside of the researchers’ interests or training, will likely dramatically accelerate discovery by expanding and expediting research endeavours. Particularly in the information explosion of the modern research landscape, where exponentially increasing research volume exceeds human time and cognitive resources, we believe that AI systems may prove particularly valuable in processing vast cross-disciplinary information to identify connections that may otherwise be missed. A complementary approach to scientific discovery that leverages the unique strengths of both human and artificial intelligence, will likely advance scientific knowledge beyond what could be accomplished independently.

Limitations

The current system incorporates extensive error-handling and verification throughout many stages of the end-to-end process, but it’s not infallible. To illustrate, the most computationally expensive aspect, data analysis, involves upward of 12 distinct steps, where the system fixes hundreds of errors independently. Despite this impressive capacity to detect and self-correct problems, issues still occasionally occur, such as non-convergence of models. As such, for each phase of the pipeline, we carefully verified outputs, and on rare occasions, needed to restart an analysis step. As the intelligence and capabilities of frontier LLMs continues to improve, we foresee the frequency of such issues decreasing, similar to the reduction in errors we have seen between initial versions of the framework with Claude 3.5 Sonnet and GPT-4o, through to the current implementation with Claude 3.7 Sonnet and o3-mini. Indeed, the quality of the systems’ research outputs has received preliminary validation from four external senior scientists (in the fields of cognitive neuroscience and psychology), with further details provided in the paper being released.

Future Work The implementation bridge we have achieved with our embodied AI scientist framework represents a crucial step toward autonomous scientific discovery, enabling AI systems to test hypotheses through direct interaction with human participants. Our current implementation - focussed on online experimentation - provides the foundation for future work to expand beyond cognitive psychology and enhance the system's capacity for autonomous theory refinement based on unexpected results.

Today, our system can design and run psychology experiments. Tomorrow, it might discover patterns in climate or astronomical data that humans have overlooked.

Today, it requires a little guidance on occasion. Tomorrow, it might propose entirely new theoretical frameworks.

The boundary between AI and scientific discovery is dissolving. The question isn't whether AI will transform science, but how we will harness this transformation to solve humanity's greatest challenges. We believe that by creating adaptable, collaborative systems where human ingenuity and AI capabilities can complement each other, we will accelerate breakthroughs across scientific domains to solve some of the biggest problems of our time.

Join the Exploration

At Explore Science, we're both excited by what we've achieved and humbled by how much remains to be investigated. We'd love to hear your thoughts on potential applications, partnerships, or technical extensions. Whether you're interested in collaborations, have questions, or simply want to share your perspective, you can email us or join the conversation on X @ExScienceAI. Bridging into the unknown