A collection of open source projects that I've set up, analyzed, and studied extensively. These cutting-edge AI research frameworks have significantly influenced my work and thinking in AI, agent systems, and technical development.
The implementation insights gained from working with these projects have directly informed architectural decisions in our education platform development and other professional initiatives.
Disclaimer: These are projects I have explored, studied, or otherwise found valuable in my professional development.
Below are the projects I've highlighted on my resume, along with dozens more that have been influential in my professional development. The first three projects represent cutting-edge AI research frameworks I've particularly focused on.
Description: The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery (Lu, Chris and Lu, Cong and Lange, Robert Tjarko and Foerster, Jakob and Clune, Jeff and Ha, David).
A groundbreaking system for fully autonomous scientific discovery using frontier LLMs—by running end-to-end experiments across generative AI, diffusion models, and grokking. My own thinking about novel agent design and multi-modal data generation was refined through exploring this project.
Citation: arXiv:2408.06292
Description: Automated Design of Agentic Systems (Hu, Shengran and Lu, Cong and Clune, Jeff).
An eye-opening project that explores recursive AI design through a "meta-agent" capable of inventing and coding novel agents—pushing the frontier of emergent agency, and LLM-based invention. Working with ADAS gave me firsthand insight into the pace and potential of AI-driven agentic system design, and deepened my understanding of emergent agency, self-improvement, and the broader implications for AI development. It sharpened my perspective on both the possibilities and the responsibilities that come with increasingly autonomous systems.
Citation: arXiv:2408.08435
Description: A Universal and Generative Physics Engine for Robotics and Beyond.
A unified physics and generative agent framework for automating multimodal data generation—spanning physical video realism, camera and character motion, robotic control, interactive 3D scenes, articulated object generation, and expressive speech animation.
Repository: GitHub Repository
Description: A lightweight library for building multimodal AI agents that can generate text, image, audio, and video. It provides a unified API for LLMs with enhanced capabilities like memory, knowledge integration, tools, and reasoning.
Key Features:
Technology Stack: Python, Various LLM providers, Vector databases
Description: Educational materials for deep learning based on the book "Deep Learning for Coders with Fastai and PyTorch" by Jeremy Howard and Sylvain Gugger. Provides comprehensive Jupyter notebooks covering deep learning concepts and implementation.
Key Components:
Technology Stack: Python, fastai, PyTorch, Jupyter Notebooks
Description: A tool that transforms GitHub repositories into interactive system design/architecture diagrams for visualization. It allows users to quickly understand repository structure and navigate to relevant files.
Key Features:
Technology Stack: Frontend: Next.js, TypeScript, Tailwind CSS, ShadCN; Backend: FastAPI, Python, Server Actions; Database: PostgreSQL with Drizzle ORM; Deployment: Vercel (Frontend), EC2 (Backend)
Description: A comprehensive framework for building autonomous AI agents with their own cognitive architecture, memory systems, and ability to pursue objectives independently.
Key Applications:
Technology Stack: Python with various LLM integrations
Description: A framework enabling multimodal AI models to operate a computer by viewing the screen and performing mouse and keyboard actions to achieve objectives, mimicking human computer interaction.
Key Features:
Technology Stack: Python, OpenAI API, Screen capture and input simulation
Description: A groundbreaking initiative that leverages OpenAI's agent-based APIs to create a self-organizing, ethically governed ecosystem of AI agents. Inspired by the ACE Framework.
Key Features:
Technology Stack: OpenAI APIs, Python-based implementation, Agent communication frameworks
Description: A multi-agent AI framework focused on sequential prompt methodology, allowing multiple specialized AI agents to collaborate on tasks.
Key Features:
Technology Stack: Python, OpenAI API
Description: Research implementation focused on methodologies for automatically designing complex AI agent systems with appropriate architectures and behaviors.
Features:
Technology Stack: Python, Various agent frameworks, evaluation metrics
Description: A framework for building AI-powered applications that interact with users across multiple modalities, containing reasoning applications built on top of the xRx framework.
Features:
Technology Stack: Docker, Web technologies, Various AI integrations
Description: An implemented AI-powered system for managing phone calls with voice assistant features, transcription, and call routing capabilities.
Features:
Technology Stack: Python, Telephony APIs, Speech recognition libraries
Description: Automated presentation slide generation system using AI to create educational or business content from textual inputs.
Features:
Technology Stack: Python, Presentation generation libraries, LLM integration
Description: A project for tracking or summarizing new research papers from arXiv, focused on AI and machine learning publications.
Features:
Technology Stack: Python, arXiv API, NLP for summarization
Description: A tool that turns any GitHub pull request into a text diff for LLM ingestion, making it easier to feed PR changes into large language models.
Key Features:
Technology Stack: Frontend: Jinja, Tailwind CSS; Backend: FastAPI, Python; Deployment: Railway
View Resume View Software & AI Portfolio View Undergraduate Experience