AIRI: Self-hosted AI virtual companion with support for real-time voice and game interaction

AIRI is an open source, self-hosted AI virtual companion project designed to provide users with customizable digital character experiences. It is centered on the Large Language Model (LLM) and supports real-time voice chat, game interactions (e.g. Minecraft and Factorio), and virtual character (VRM/Live2D) animation. Users can run AIRI in a browser or on the desktop, without complex dependencies, using web technologies or local GPU acceleration. Inspired by Neuro-sama, the goal of the project is to create a virtual companion that plays games, chats and performs multiple tasks with the user.AIRI emphasizes user autonomy and supports localized deployment to ensure privacy and flexibility. The project is maintained by the Moeru AI community and is continuously updated, and developers are welcome to contribute.

Function List

• Real-time Voice Interaction: Supports voice input and output from a browser or Discord, using ElevenLabs Speech Synthesis Technology.
• Gameplay Interaction: able to execute natural language commands in Minecraft to complete tasks such as digging and building; assist the player in Factorio (in development).
• Virtual character support: Integrated VRM and Live2D models with animation effects such as automatic blinking and line-of-sight tracking.
• Local Inference: Model inference with WebGPU or local NVIDIA CUDA/Apple Metal, no cloud dependencies.
• In-memory systems: support for local databases (e.g. DuckDB WASM) to record user interaction history.
• Cross-platform support: runs on web browsers, Windows and macOS.
• Developer Friendly: Provides a plug-in system to support user extensions.

Using Help

Installation process

To use AIRI, users need to clone the GitHub repository and perform a simple configuration. Below are the detailed installation steps:

1. clone warehouse::
   Open a terminal and enter the following command to clone the AIRI repository:

   git clone https://github.com/moeru-ai/airi.git
   cd airi
   

2. Installation of dependencies::
   AIRI use pnpm as a package manager. Ensure that Node.js 22+ is installed and run it:

   pnpm install
   

3. Configuring Environment Variables::
   Copy the sample configuration file and fill in the necessary information:

   cp packages/agent/.env.example packages/agent/.env.local
   

   compiler packages/agent/.env.local, add the following:

   OPENAI_API_KEY=your_openai_api_key
   OPENAI_API_BASEURL=your_openai_api_baseurl
   

   If using the Factorio module, configuration is also required:

   cp packages/factorio-wrapper/.env.example packages/factorio-wrapper/.env.local
   

   compiler packages/factorio-wrapper/.env.localThe WebSocket and RCON addresses are set, for example:

   WS_SERVER_HOST=localhost
   FACTORIO_WS_HOST=localhost
   RCON_API_SERVER_HOST=localhost
   

4. Linking Factorio modules (optional)::
   If you use the Factorio function, you need to create symbolic links:

   ln -s /path/to/airi-factorio/packages/autorio/dist /path/to/factorio/data/autorio
   

5. Run AIRI::
   Select the mode of operation according to the platform:
   • browser (software): To start the web version, go to http://localhost:5173(Specific ports are subject to actual configuration).
   • desktop: Runs the desktop version, supports NVIDIA CUDA or Apple Metal acceleration, and executes:

     pnpm run start
     

Functional operation flow

1. Real-time voice interaction

AIRI supports voice input and output via browser or Discord. Users need to configure ElevenLabs speech synthesis API:

• exist .env.local Add the ElevenLabs API key to the list.
• After launching AIRI, open the browser interface or the Discord client and click on the Voice Input button.
• Say a command (e.g., "Help me build a house"), and AIRI will respond by voice and perform the task.
• Voice interaction supports automatic detection of the user's speaking status, suitable for real-time communication scenarios.

2. Minecraft game interaction

AIRI integrates Minecraft robotics, which can be controlled by natural language commands:

• Make sure the Minecraft server (1.20+) is running and configure the .env.local The server information in the

  BOT_USERNAME=your_bot_username
  BOT_HOSTNAME=localhost
  BOT_PORT=25565
  BOT_VERSION=1.20
  

• Start AIRI and connect to the server.
• Enter a command into the interface, such as "Dig 10 rocks" or "Follow me", and AIRI will execute it automatically.
• Features include path planning, item management, combat support, and more. For example, type in "Build a 3×3 cabin" and AIRI will automatically collect wood and complete the construction.

3. Virtual character animation

AIRI supports VRM and Live2D models to enhance the interactive experience:

• Upload a VRM or Live2D model file on the Setup page.
• The model automatically performs Juno, which includes animation effects such as automatic blinking and line-of-sight tracking.
• The user can adjust the model's actions through the interface, such as "look to the mouse" or "move eyes randomly".

4. Local reasoning and privacy

AIRI supports WebGPU inference without the need for cloud servers:

• Make sure the device supports WebGPU (modern browsers like Chrome/Edge).
• Select a local model (e.g. LLaMA) in the settings to run inference in the browser.
• Desktop support for NVIDIA CUDA or Apple Metal for better performance.

5. Extended functionality

Developers can add functionality through the plug-in system:

• interviews services catalog, refer to the documentation to create custom skills.
• Example: Add a new Game Interaction module, need to implement the skill logic and register it to AIRI.

caveat

• Ensure that the network is stable and that voice and gaming functions require a low-latency connection.
• Local inference requires a high-performance device; at least 16GB of RAM is recommended.
• Regularly check your GitHub repository for updates to get the latest features and fixes.

application scenario

1. Gamemate
   AIRI can interact with players in Minecraft and Factorio to execute commands or provide assistance, for players who want an intelligent companion.
2. Virtual Assistant
   Communicate with AIRI via voice and text to accomplish daily tasks such as looking up information and taking notes, suitable for personal entertainment or work scenarios.
3. Developer Testing
   Developers can utilize AIRI's plug-in system to test AI models or develop new features for AI research and application development.
4. Virtual Character Fans
   Users can customize VRM/Live2D characters to create personalized digital companions for virtual character enthusiasts.

QA

1. Does AIRI need high performance equipment?
   The browser version runs on regular devices, but native inference requires a WebGPU-enabled browser or GPU device (e.g., NVIDIA/Apple device).
2. How do you ensure data privacy?
   AIRI supports local deployment and inference, data does not need to be uploaded to the cloud to guarantee privacy.
3. Does it support multiple languages?
   Yes, AIRI's language model supports multiple languages, and speech synthesis supports multilingual output.
4. How can I participate in the development?
   Visit the GitHub repository, submit a Pull Request or join the Telegram/Discord discussion in the Moeru AI community.