WebThinker: An Intelligent Reasoning Tool that Supports Autonomous Web Search and Report Writing

WebThinker is an open source project designed to enhance the autonomous research capabilities of large-scale reasoning models (LRMs). It allows models to independently perform web search, navigation and information extraction during the reasoning process, and ultimately generate detailed reports. The project is developed by the Information Retrieval and Natural Language Processing Laboratory of Renmin University of China (RUC-NLPIR), based on the QwQ-32B reasoning model and Qwen-32B-Instruct auxiliary model.WebThinker breaks through the limitations of traditional retrieval-enhanced generation (RAG) with a model that performs search and navigation operations directly, enabling end-to-end task processing. It excels in knowledge-intensive complex reasoning tasks (e.g., GPQA, GAIA, WebWalkerQA, HLE) and open-ended report generation tasks, dramatically reducing the time and cost of information gathering for researchers. The project is under the MIT license, and the code and documentation are publicly available on GitHub for researchers and developers.

Function List

• Autonomous Web Search: The model can automatically initiate a search and extract relevant information based on task requirements.
• Deep web navigation: supports clicking on links or buttons to explore web content in depth.
• Automatic report generation: Write reports as you search, generating structured, customized content.
• Report editing and checking: Provides tools to check and revise reports to ensure accuracy and consistency of content.
• Support multiple inference modes: run different task modes through scripts to adapt to diversified needs.
• integrated (as in integrated circuit) Crawl4AI Ans: Optimize the extraction of JavaScript rendered web pages.
• Open source and extensible: the code is open and allows users to customize models and features.

Using Help

Installation process

To use WebThinker, users need to follow the steps below to configure the environment and run the project. Below is a detailed installation and usage guide to ensure that users can get started quickly.

1. Create and activate the Conda environment

WebThinker relies on a Python 3.9 environment, which is recommended to be managed with Conda. Run the following command to create and activate the environment:

conda create -n webthinker python=3.9
conda activate webthinker

This creates a file named webthinker of the virtual environment to ensure dependency isolation.

2. Clone the project and install dependencies

Clone the WebThinker repository from GitHub and install the necessary Python packages:

git clone https://github.com/RUC-NLPIR/WebThinker.git
cd WebThinker-main
pip install -r requirements.txt

requirements.txt Contains all the dependency packages required by the project, such as the vLLM and Crawl4AI. ensure that the network connection is stable to complete the installation.

3. Configuration inference models and auxiliary models

WebThinker requires an inference model and an auxiliary model. The official recommendation is to use QwQ-32B as the inference model and Qwen-32B-Instruct as the auxiliary model for core inference and web page reading and report writing tasks respectively. Users need to deploy these models via vLLM. Before running, make sure the models have been downloaded and started locally or on the server. The deployment steps can be found in the official documentation (linked in the README of the GitHub repository).

4. Setting up a web parsing client

To improve web crawling, it is recommended that the <WebThinker-main>/scripts/search/bing_search.py The Crawl4AI client is configured in Crawl4AI. Crawl4AI can process dynamic web content rendered in JavaScript to improve the reliability of information extraction. Users need to register for the Crawl4AI service and obtain an API key, then follow the documentation to configure it.

5. Run WebThinker

After completing the above configuration, the user can run different inference modes with the provided scripts. Example:

python scripts/run_inference.py --mode report_generation

This command starts the report generation mode. Other modes (such as complex reasoning tasks) can be specified in the script. Before running, make sure the modeling service is started.

Main Functions

The core of WebThinker is autonomous search and report generation. Below is a detailed flow of the main functions:

Autonomous web search

After the user enters a research question, WebThinker automatically invokes a search tool (e.g., Bing search) to obtain relevant web pages. The model analyzes the initial search results, extracts keywords, and decides whether further search is needed. For example, if a user asks for "latest AI research trends", WebThinker will search for relevant academic articles and news to filter for high-quality content.

In-depth web navigation

WebThinker's Deep Web Explorer module allows models to click on links or buttons on web pages to drill down into information. For example, when visiting an academic website, the model can click on the "Next Page" or "Download PDF" button to access more detailed data. No manual intervention is required, as the model automatically determines which links are most relevant.

Report generation and editing

WebThinker offers three reporting-related tools:

• Chapter content writing: The model automatically generates specific sections of the report, such as "Background" or "Data Analysis", based on the search results. Users can specify the structure of the report and the model will populate the content as needed.
• Reporting inspections: The model checks the logic and completeness of the report, flagging missing information or inconsistent sections.
• Report Editor: Users can adjust the content of the report by invoking the editing function through a script. For example, run:

  python scripts/edit_report.py --report_path ./outputs/report.txt
  

  This command allows the model to optimize the wording and structure of the specified report.

Optimizing Crawl with Crawl4AI

Crawl4AI is able to parse complex content on dynamic web pages, such as news pages containing JavaScript. Users are asked to bing_search.py Configure the API key in Crawl4AI and ensure network connectivity. Once configured, the model automatically calls Crawl4AI to extract web text, images, or other elements.

caveat

• hardware requirement: High-performance GPUs (e.g. NVIDIA A100) are required to run large models such as QwQ-32B. Ensure hardware supports vLLM inference.
• network environment: The search and crawl functions require a stable network. It is recommended to use a proxy or VPN to deal with possible geo-restrictions.
• Model Selection: Users may try other command fine-tuning models (e.g. LLaMA) as auxiliary models, but need to ensure compatibility.
• Output View: The generated report is saved in the <WebThinker-main>/outputs/ directory, which contains a list of all the features associated with Grok3 DeepSearch and the Gemini 2.0 Deep Research's Comparative Report.

With these steps, you can easily deploy WebThinker and use its search and report generation features. The project documentation is detailed and suitable for users with some technical background.

application scenario

1. academic research
   For researchers who need to quickly gather the latest advances in a particular field, WebThinker automatically searches academic papers, news, and blogs to generate a report with citations and analysis. For example, if you type in "recent advances in quantum computing", the model will search arXiv and Google Scholar to produce a structured report.
2. market analysis
   Business analysts need to know what their competitors are up to. WebThinker searches industry news, company websites, and social media to generate reports on market trends, saving time on manual collection.
3. technology development
   WebThinker navigates technical blogs and GitHub repositories, extracting key information and generating technical summaries to assist in project development.

QA

1. What models does WebThinker support?
   The official recommendations are QwQ-32B (inference model) and Qwen-32B-Instruct (auxiliary model). Users may try other command fine-tuning models, but need to ensure vLLM compatibility.
2. How to handle dynamic web content?
   Configure the Crawl4AI client, edit the <WebThinker-main>/scripts/search/bing_search.pyCrawl4AI parses JavaScript rendered content.
3. How long does it take to generate a report?
   Depends on task complexity and network speed. Simple reports take about 5-10 minutes and complex tasks can take more than 30 minutes.
4. Is programming experience required?
   Deploying WebThinker requires basic Python and Conda knowledge. The use of pre-configured scripts is simple and suitable for beginners.