HUMAN.EXE Intention & AI Commander Explian

RATIONALE: Intention/Inquiry - why are you doing what you are doing?

My research focus has shifted from general game development research to a more specific question: how to enhance the capabilities of solo game developers through an AI Commander system, particularly in the field of enemy AI. Traditional behavior trees and state machine designs are both time-consuming and limited by the resources of independent developers, while directly using LLMs to control each enemy unit faces challenges of real-time performance and network latency.

My research proposes an alternative: retaining traditional behavior trees to handle basic enemy behaviors (such as shooting when seeing a player), while introducing LLM as an "AI Commander" to make high-level strategic decisions (such as enemy positions, patrol routes, and tactical arrangements). This approach maintains real-time responsiveness of the game while enhancing the intelligence and unpredictability of enemy behavior.

I hope that through this project, I can explore a method that allows independent developers to create intelligent enemies similar to those in AAA games at a lower cost, thereby making the game experience richer and more challenging.

Context: Field -who else is doing what you are doing? What do you learn from these other practitioners?

My project exists at the intersection of game AI research and LLM application. After extensive literature research, I found that most research on the combination of LLM and games focuses on dialogue systems, with little in-depth research on tactical decision-making. Here are some related works that are of significant reference value to my project:

Rainbow Six：This game's enemy AI system adopts an omniscient defensive strategy, where enemies can grasp the entire map information and make tactical deployments. This mode inspired my design of the "omniscient mode" of AI Commander, allowing AI to formulate strategic defense plans by understanding the entire game environment.

Edge of Tomorrow：The concept in this movie/game, where the protagonist constantly respawns and learns from failures, inspired my "learning mode". AI Commander can remember player behavior patterns and continuously learn and adjust strategies in each interaction.

Large Language Models and Games: A Survey and Roadmap by Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N. Yannakakis：This paper, which can be found on Google Scholar, is the most relevant paper to my research. It discusses four key aspects:

1.Multimodal Technology Integration: Modern LLMs can process and generate multimodal content beyond traditional text, such as images.

2.Technical Application Role Classification: LLMs can play multiple roles in games, including game players, NPCs, player assistants, and game hosts.

3.Open Source Technical Solutions: The paper focuses on open source models like Mistral and Llama.

4.Technical Boundary Discussion: The paper discusses the potential and limitations of LLMs in a balanced way.

NVIDIA's AI BOSS Research: This proves the feasibility of AI-driven BOSS in games, but their method requires substantial computational resources, which is not suitable for independent developers.

Method: Practice: How will you do what you are doing? This is not about software but about capturing the way you work.

I will adopt a practice-oriented research method, combining theoretical analysis and prototype development. Specific methods include:

1.Conceptual Design and Theoretical Analysis: Clarify the logic, requirements, and minimum viable standards of the AI Commander system, and define information flows and interaction methods through sketches and pseudocode.

2.Unity Prototype Development: Based on the HUMAN.EXE project, create a test environment with basic cover and enemy NPCs to validate the AI Commander system. This will include:

Designing LLM API interfaces

Creating data structures for game state and command conversion

Implementing four different operation modes: omniscient mode, regular defense mode, learning mode, and limited information mode

3.Testing and Evaluation: Evaluate the technical feasibility (latency, stability, API cost) and quality effects (perceived intelligence, strategic consistency, tactical diversity, player engagement) of the AI Commander system through systematic observation.

4.Iterative Optimization: Continuously improve system design, implementation, and LLM prompts based on test results.

5.Multi-model Comparison: If time permits, I will test multiple large language models, such as Chat-GPT, Claude, and Google Gemini. Currently, based on my experience and specific experimental processes, DeepSeek offers the lowest cost.

Production:

Tools:

Unity Engine: Main development platform, used to create test environments and AI Commander system prototypes

Deep Seek API: Choose DeepSeek as the main LLM due to its cost-effectiveness, suitable for independent developers

Cursor, Claude, C#: Write game logic and API integration code

Adobe Audition and ElevenLabs: Sound design and synthesis

GitHub: Version control and code management

Google Scholar: Academic research and paper search

TXYZ: Supplementary paper research tool

Development Process:

Game State Processing: Develop a system that extracts relevant information from the game environment and formats it into a structure understandable by LLM

API Communication Layer: Implement stable and efficient communication between Unity and DeepSeek API

Local Testing Foundation: First conduct basic tests locally, and gradually open more permissions to DeepSeek

Command Parsing System: Parse LLM responses into executable instructions for enemy NPCs in the game

Visualization: Create visualization systems for debugging and demonstrating AI decision processes

Time Schedule

Weeks 8-9: Complete the concept design and architecture planning of the AI Commander system

Weeks 10-11: Implement basic functions, conduct preliminary tests and adjustments

Weeks 12-13: Optimize the system, prepare final demonstrations and documentation