Idlefish Intelligent Monitoring Robot: an AI tool to monitor and analyze Idlefish goods in real time

ai-goofish-monitor is an idle fish commodity monitoring tool based on Playwright and AI technology. It helps users track the product information on the Idlefish platform in real time, analyze the product details and seller profiles through AI, and filter out high-quality products that meet the demand. The tool is equipped with an intuitive web management interface that simplifies the task creation and monitoring process. Users can describe their needs in natural language, and AI automatically generates screening rules. It supports multi-task concurrency, real-time notification and a high degree of customization, suitable for users who need to quickly find the target product. The project is open source, hosted on GitHub and easy to install and use.

Function List

• AI-driven task creation: The user describes the requirements in natural language, and the AI automatically generates complex product screening rules.
• Multi-task concurrent monitoring: Supports monitoring multiple keywords at the same time, with tasks running independently without interfering with each other.
• Real-time commodity analysis: Immediately after discovering a new product, AI analyzes the product image, description, and seller information.
• Visual Web Interface: Provides task management, log viewing and filtering rule editing functionality without the need for operational command lines.
• Instant notification push: Push eligible product information in real time via ntfy.sh or enterprise WeChat bots.
• Highly customizable: Support for setting up separate keywords, price ranges, and AI analysis instructions for each task.
• Headless mode operation: Crawlers can choose to run in headless mode, reducing resource consumption and suitable for long-term monitoring.
• PC Link to Mobile Link: Automatically converts PC product links to cell phone links for improved ease of operation.

Using Help

Installation process

1. Cloning Project Code
   Run the following command in the terminal to clone the project locally:

   git clone https://github.com/dingyufei615/ai-goofish-monitor
   cd ai-goofish-monitor
   

2. Installing Python dependencies
   Make sure Python 3.8 or above is installed, and then install the dependencies required by the project:

   pip install -r requirements.txt
   

3. Configuring Environment Variables
   In the project root directory, create the .env file, fill in the following configuration:

   # OpenAI API 配置
   OPENAI_API_KEY="sk-..."
   OPENAI_BASE_URL="https://api.openai.com/v1"
   OPENAI_MODEL_NAME="gpt-4o"
   # ntfy 通知服务配置
   NTFY_TOPIC_URL="https://ntfy.sh/your-topic-name"
   # 企业微信机器人通知（可选）
   WX_BOT_URL="https://qyapi.weixin.qq.com/cgi-bin/webhook/send?key=xxxxx"
   # 浏览器配置
   LOGIN_IS_EDGE=false
   # 是否将 PC 链接转为手机链接
   PCURL_TO_MOBILE=true
   # 是否以无头模式运行爬虫
   RUN_HEADLESS=true
   

   • OPENAI_API_KEY: Get the API key from the official OpenAI website.
   • NTFY_TOPIC_URL: Get the topic URL for receiving notifications after registering for the ntfy.sh service.
   • WX_BOT_URL: If you use Enterprise WeChat Notification, you need to configure the Enterprise WeChat Robot key.
   • To use the Edge browser, set LOGIN_IS_EDGE set up as true; if you need to turn off headless mode (e.g., if you encounter a sliding CAPTCHA), set the RUN_HEADLESS set up as falseThe
4. Get Idlefish login status
   Run the login script to generate the session state file:

   python login.py
   

   The script will open a browser window showing the Idle Fish login QR code. Scan the QR code with the Idlefish App to complete the login. After successful login, the script closes automatically and generates the following script in the project root directory xianyu_state.json File. This file records the login status for use by the crawler.

5. Configuring Monitoring Tasks
   compiler config.json file to add monitoring tasks. Example:

   [
   {
   "keyword": "二手手机",
   "price_range": [500, 2000],
   "enable": true,
   "ai_prompt": "筛选成色好、卖家信誉高的手机"
   },
   {
   "keyword": "笔记本电脑",
   "price_range": [1000, 5000],
   "enable": true,
   "ai_prompt": "优先选择轻薄本，排除翻新机"
   }
   ]
   

   • keyword: Search Keywords.
   • price_range: Price range.
   • ai_prompt: AI screening commands with support for natural language descriptions.

Using the Web Interface

1. Launching the Web Interface
   Run the following command to start the Web service:

   python web.py
   

   Open your browser and visit http://localhost:5000(port numbers may be different, refer to the terminal prompt).

2. Creating Tasks
   • Click "Create Task" in the web interface.
   • Enter a natural language description, such as "Looking for a $500-2000 used iPhone, 90% new in color, from a reputable seller".
   • Click "Generate" and AI automatically creates the filter rule and saves it to the config.jsonThe
3. Start monitoring
   • Return to the main interface and click the "Start All" button.
   • The crawler starts monitoring all enabled tasks and real-time logs are displayed on the interface.
4. Viewing and editing
   • Log View: The interface displays the crawler's running log in real time for easy debugging and monitoring.
   • Rules Editor: Click on the "Edit" button of the task to directly modify the AI filtering instructions or keywords.
   • Results Screening: The interface displays the eligible products, click to view the product details or jump to the idle fish page.
5. Receive notification
   • well configured NTFY_TOPIC_URL After that, eligible items will be pushed to the phone or desktop via ntfy.sh.
   • If you use Enterprise WeChat, the notification will be sent to the designated group chat.

Command line run (optional)

If you are not using the web interface, you can run the main crawler script directly:

python main.py

Script loading config.json The monitoring starts automatically for all tasks in the Logs are output to the terminal.

Featured Function Operation

• AI Analytics: The tool uses the GPT-4o model to analyze product images, descriptions, and seller information. For example, if you type in "filter reputable sellers", the AI will synthesize the seller's transaction history and ratings to eliminate risky sellers.
• multitasking: Each task runs independently of the other. Users can enable or disable tasks at any time from the web interface.
• real time notificationAI analyzes new products and sends push notifications as soon as they go live, with a response time that is usually within a few seconds.

caveat

• Ensure network stability to avoid login status failure.
• regular inspection .env The API key and notification configurations in the file are valid.
• If you encounter a sliding CAPTCHA, it is recommended that the RUN_HEADLESS set up as false, complete the validation manually.
• The project depends on Playwright, if the installation fails, you can refer to the official Playwright documentation to solve the problem.

application scenario

1. Second-hand goods trading
   Users want to buy used cell phones or computers in Idle Fish, but need to quickly screen out reasonably priced items in good condition. ai-goofish-monitor automatically monitors keywords, AI analyzes the details of the items, and pushes out quality items that meet the criteria, saving time for manual screening.
2. Collection Search
   Collecting enthusiasts need to monitor specific categories of rare items, such as limited edition figures or antiques. Users can set up precise keywords and AI filtering rules to get real-time information on new collectibles hitting the shelves.
3. bulk buying
   Merchants need to purchase used equipment in bulk. The tool supports concurrent multi-tasking, monitors multiple categories, and AI screens for items that meet quality standards and quickly informs them via notifications to make it easier to grab them.

QA

1. How do I make sure my login status is valid?
   (of a computer) run python login.py generating xianyu_state.json After the file, the crawler will automatically use the file to access Idlefish. If the login fails, re-run the login script to update the status.
2. Is the AI analysis accurate?
   The AI is based on GPT-4o models, which are analyzed in conjunction with product images, descriptions and seller profiles. Accuracy depends on the filtering instructions provided by the user. It is recommended to describe the requirements in clear natural language.
3. What notification methods are supported?
   Supports ntfy.sh push (mobile, desktop) and enterprise WeChat bot notifications. Required in .env file to configure the corresponding URL.
4. How many tasks can be monitored?
   Theoretically there is no limit to the number of tasks, but it is affected by the performance of the hardware. It is recommended to run 5-10 tasks at the same time on an average computer to ensure performance.
5. What should I do if I encounter a CAPTCHA?
   commander-in-chief (military) .env in the file RUN_HEADLESS set up as false, the CAPTCHA validation is completed manually at runtime, after which the headless mode can be restored.