Peekaboo: macOS screen capture and visual quiz tool

Peekaboo is a tool designed for macOS to provide AI assistants with fast, non-intrusive screen capture and visual quizzing (VQA) capabilities. It uses Apple's ScreenCaptureKit technology to enable efficient screen capture that supports capturing the entire screen, specific apps, or windows, and analyzing image content in conjunction with local or cloud-based AI models. MCP Server two ways of use, suitable for developers and need to automate the visual processing of the scene. It is designed with privacy in mind and supports local AI modeling without interfering with user operations during the capture process. Tools are installed via npm packages or Homebrew for easy integration into the development process.

Function List

• Quickly capture macOS screens with support for the entire screen, specific apps, or windows.
• Supports Visual Question and Answer (VQA) for AI analysis of captured images.
• Provide fuzzy matching function to precisely locate the target window.
• Multiple output formats are supported, including PNG, JPG, and Base64.
• Automatically removes window shadows and borders to improve image clarity.
• Integrate local AI models (e.g. Ollama) or cloud-based models (e.g. GPT-4 Vision, Claude).
• Non-intrusive snapping without disturbing the current window focus.
• Provides TypeScript support and JSON API for developer integration.
• Supports window and application management to get information about running applications and windows.

Using Help

Installation process

Peekaboo offers a variety of installation methods to suit different user needs. Below are the detailed steps:

Way 1: Installation via Homebrew (recommended)

Homebrew is a package management tool for macOS that makes installing Peekaboo quick and easy.

1. Open a terminal and add Peekaboo's Homebrew repository:

   brew tap steipete/tap
   

2. Install Peekaboo:

   brew install peekaboo
   

3. Verify the installation and run the following command to view the version:

   peekaboo --version
   

Way 2: Installation via npm (includes MCP server)

Ideal for developers who need MCP server functionality.

1. Make sure Node.js is installed (recommended version 16 or higher).
2. Runs in the terminal:

   npm install -g @steipete/peekaboo-mcp
   

3. Verify the installation:

   peekaboo --version
   

Approach 3: Build from Source

Ideal for developers who need to customize functionality.

1. Clone a GitHub repository:

   git clone https://github.com/steipete/peekaboo.git
   cd peekaboo
   

2. Install the dependencies:

   npm install
   

3. Build the CLI and MCP servers:

   npm run build:all
   

4. (Optional) Install the CLI to the system path:

   ./scripts/build-cli-standalone.sh --install
   

Way 4: Directly download the binary file

1. Download the latest version of the binary:

   curl -L https://github.com/steipete/peekaboo/releases/latest/download/peekaboo-macos-universal.tar.gz | tar xz
   

2. Move to the system path:

   sudo mv peekaboo-macos-universal/peekaboo /usr/local/bin/
   

Configuring AI Models

Peekaboo supports both local and cloud-based AI models for visual quiz functions. Below is a list of local Ollama model as an example:

1. Install Ollama:

   brew install ollama
   ollama serve
   

2. Download the visual model (e.g. llava or qwen2-vl):

   ollama pull llava:latest
   ollama pull qwen2-vl:7b
   

3. Configure Peekaboo:

   peekaboo config edit
   

   Set the AI provider in the configuration file to <ollama/llava:latest> or other models.

Main Functions

screen capture

Peekaboo supports multiple capture modes:

• Full screen: Capture full-screen content.

  peekaboo capture screen --output screen.png
  

• application-specific: Captured by application name (fuzzy matching supported).

  peekaboo capture app "Visual Studio Code" --output code.png
  

• specific window: Captured by window title or index.

  peekaboo capture window --title "Document" --output doc.png
  

• output format: Supports PNG, JPG or Base64.

  peekaboo capture screen --format base64
  

Visual Question and Answer (VQA)

Users can ask questions about the captured image and the AI model will analyze the image and answer.

1. Capture images and ask questions:

   peekaboo capture screen --question "What is the main text on the screen?" --output result.json
   

2. The output is in JSON format and contains images and AI responses.

window management

Lists running applications and windows:

peekaboo list apps
peekaboo list windows --app "Safari"

caveat

• Ensure that macOS grants Peekaboo screen recording permissions (System Settings > Privacy & Security > Screen Recording).
• Local AI models require sufficient hardware resources, and it is recommended to use more than 16GB of RAM.
• The network connection is stable to ensure that the cloud model is functioning properly.
• Detailed documentation is available at GitHub:

  open https://github.com/steipete/peekaboo
  

application scenario

1. Developer Debugging
   Developers can use Peekaboo to capture application windows and analyze UI elements or error messages. For example, capture a code editor window and ask "Are there any syntax errors in the code" to quickly pinpoint the problem.
2. Automated workflows
   With Peekaboo, AI assistants can monitor screen content and automate tasks. For example, capturing a browser window and extracting web page text for data analysis.
3. Education and training
   Teachers can capture the interface of the software and use the VQA functionality to generate teaching materials. For example, ask "What are the data points in the chart?" to quickly organize lesson content.
4. Remote technical support
   Technical support staff can capture a user's screen, analyze interface problems and provide guidance without the user having to manually send screenshots.

QA

1. What AI models does Peekaboo support?
   Local models (e.g., Ollama's llava and qwen2-vl) as well as cloud-based models (e.g., GPT-4 Vision, Claude) are supported.
2. Is there a need for continuous networking?
   No internet connection is required to use the local model, and a stable internet connection is required to use the cloud model.
3. How do I ensure that capturing does not interfere with operations?
   Peekaboo uses ScreenCaptureKit, which does not change the focus of the window when capturing, ensuring non-intrusive operation.
4. How are permissions handled?
   The first time you run it, you need to grant screen recording permission in the system settings, otherwise you can't capture the screen.