JoyAgent-JDGenie: an open source multi-intelligence framework to support automated processing of complex tasks

JoyAgent-JDGenie is a multi-intelligence body framework open-sourced by Jingdong, focusing on automating complex tasks. It has performed well in GAIA benchmarks with an accuracy of 75.15%, surpassing several similar products. This framework works out-of-the-box, does not rely on cloud platforms, and is suitable for local deployment. Users can enter queries or tasks to let the system automate operations such as code generation, document processing and report generation. It supports mounting sub-intelligence or tools to flexibly adapt to different needs. The project code is hosted on GitHub and based on Apache-2.0 license, allowing developers to use and modify it freely.

Function List

• Supports multi-intelligence collaboration to handle complex tasks such as code, documentation, report generation, and more.
• Out-of-the-box framework that runs without complex configuration.
• Supports local deployment and does not rely on cloud platforms, suitable for enterprise-level applications.
• Provides modular tool mounts that can be customized with sub-intelligences to meet specific needs.
• Integrated multi-tasking capabilities such as code debugging, data analysis and automated reporting.
• Supports multimodal input and handles multiple data types such as text and images.
• Provide sample data and test cases for developers to get started quickly.

Using Help

JoyAgent-JDGenie has a clear usage process for developers to quickly integrate into their projects. The following is a detailed installation and usage guide to help users get started from scratch.

Installation process

1. environmental preparation
   Make sure you have Python 3.8 or higher and Git installed on your system; Linux or macOS is recommended, Windows users should install WSL or use a compatible environment.

   sudo apt update
   sudo apt install python3 python3-pip git
   

2. Cloning Project Code
   Download the source code for JoyAgent-JDGenie from the GitHub repository. Open a terminal and enter the following command:

   git clone https://github.com/jd-opensource/joyagent-jdgenie.git
   cd joyagent-jdgenie
   

3. Installation of dependencies
   The project relies on a number of Python libraries, so make sure you install them beforehand. pip Updated. Run the following command to install the required dependencies:

   pip3 install --upgrade pip
   pip3 install -r requirements.txt
   

4. Configuring Environment Variables
   JoyAgent-JDGenie Support for AWS Claude and OpenAI's API as language model drivers. You need to configure at least one API key. Edit .env file (or create a new one), add the following:

   export OPENAI_API_KEY="你的OpenAI密钥"
   # 或者
   export AWS_CLAUDE_API_KEY="你的Claude密钥"
   

   After saving, run source .env Make the environment variable effective.

5. Install ChromeDriver (optional)
   If you need to handle web tasks, you need to install ChromeDriver, here are the installation commands for Linux:

   wget https://storage.googleapis.com/chrome-for-testing-public/131.0.6778.85/linux64/chromedriver-linux64.zip
   unzip chromedriver-linux64.zip
   sudo mv chromedriver /usr/bin/chromedriver
   sudo chmod +x /usr/bin/chromedriver
   

   By default, Chrome runs in headless mode (no GUI). To visualize the browser, edit the configuration file and set the headful=TrueThe

6. Verify Installation
   Run the following command to test if the framework is working properly:

   python3 -m simulated_web_agent.main --help
   

   If a help message is output, the installation was successful.

Functional operation flow

The core function of JoyAgent-JDGenie is to accomplish complex tasks through multi-intelligence collaboration. Below are the detailed operation steps of the main functions.

1. Running the mission

The project provides 1000 sample character data (located in the example_data/personas/json/) that can be used for testing. The command to run the task is as follows:

python3 -m simulated_web_agent.main --persona "example_data/personas/json/virtual customer 0.json" --output "output" --llm-provider openai

• --persona: Specifies a persona data file containing a description of the task, such as purchasing an item or generating a report.
• --output: Specify the output directory to save the task results.
• --llm-provider: Selection of language models (openai maybe claude).

The sample character data format is as follows:

{
"persona": "Persona: Michael ...",
"intent": "buy a large, inflatable spider decoration for halloween",
"age": 42,
"gender": "male",
"income": [30001, 94000]
}

After running, the system will invoke the intelligent body according to the task goal to complete operations such as web browsing, data extraction or report generation.

2. Customizing sub-intelligences

JoyAgent-JDGenie supports the mounting of sub-intelligences or tools to fulfill specific needs. For example, adding a sub-intelligence for data analysis:

• exist tools/ directory to create a new tool script that follows the interface specifications in the project documentation.
• modifications config.yaml, add the tool path:

  tools:
  - path: "tools/custom_analyzer.py"
  name: "DataAnalyzer"
  

• Rerun the task and the system will automatically load the new tool.

3. Handling multimodal tasks

The framework supports processing multiple inputs such as text, images, etc. For example, upload an image and ask for a description to be generated:

• Place the image file into the input/ Catalog.
• Modify the mission profile to specify the image path and mission target:

  {
  "input": "input/image.jpg",
  "task": "generate image description"
  }
  

• Running the task, the system calls the multimodal intelligences to generate the results.

4. Viewing the output

After the task is completed, the results are saved in the --output The specified directory. The output file includes the task log, generated content, and debugging information. Check output/ directory to ensure that the task is performed correctly.

caveat

• Make sure the network connection is stable and the API key is valid.
• If the task takes too long to execute, adjust the configuration file's timeout Parameters.
• Regularly update the project code for the latest features:

  git pull origin main
  

application scenario

1. Automated Code Debugging
   Developers can use JoyAgent-JDGenie to automatically analyze code errors and generate repair suggestions. Upload a code file, specify the task as "Debug Code", and the system will call the intelligent body to analyze and output a solution.
2. Generate business reports
   Business users can enter sales data or market research tasks and the framework automatically generates structured reports. Ideal for quickly generating financial analysis or market trend reports.
3. Web Data Crawling
   The system supports automated web browsing and data extraction. For example, if you input "capture product price of e-commerce platform", the intelligent body will visit the target website and return structured data.
4. Personalized Task Processing
   With customizable sub-intelligence, users can implement specific functions such as auto-responding to emails or batch processing documents, suitable for office automation scenarios.

QA

1. Does JoyAgent-JDGenie require cloud service support?
   Not required. The framework supports local deployment and is suitable for enterprise applications or environments with data privacy requirements.
2. How to choose the right language model?
   Supports both OpenAI and AWS Claude; OpenAI for fast responses and Claude for complex tasks. Depending on your needs, you can use the .env file to configure the corresponding API key.
3. What types of tasks can be handled?
   Code generation, document processing, report generation, web crawling and multimodal tasks are supported. Specific tasks are defined through character data or configuration files.
4. How do I extend the functionality?
   exist tools/ directory to add custom tool scripts and update the config.yaml Documentation. Refer to the official documentation to ensure interface compatibility.