GPT-Crawler is an open source web crawler tool developed by the BuilderIO team specifically designed to automatically extract content from websites and generate structured knowledge files (output.json). Its core features include:
- Automated Content Crawling: Crawls the text of a web page by specifying a URL and CSS selector.
- Dynamic Web Support: Capture dynamic content rendered by the client using headless browser technology.
- Highly configurable: Allows to set the maximum number of pages to crawl (maxPagesToCrawl), file size limit (maxFileSize) and exclude specific resource types (e.g. images/videos).
- Multi-scene Adaptation: It supports local running, Docker container deployment, and API calls.
The generated files can be used directly to create custom GPTs or AI assistants, e.g. uploaded to the OpenAI platform as a knowledge base source.
This answer comes from the articleGPT-Crawler: Automatically Crawling Website Content to Generate Knowledge Base DocumentsThe
May not be reproduced without permission:AI productivity tools " What is GPT-Crawler and what are its core functions?
