ScreenCoder: A tool to convert UI screenshots into editable HTML/CSS code

ScreenCoder is an open source tool designed to transform user interface (UI) screenshots or design sketches into production-ready HTML and CSS code. It uses a modular multi-intelligence framework , through visual understanding , layout planning and code generation of three steps , the complex design into a clear structure , editable code . Developers or designers can quickly turn design prototypes into actual web pages, saving time on manual coding.ScreenCoder supports multiple generation models, such as Doubao, Qwen, GPT and Gemini, which are suitable for different development needs. The code is publicly available on GitHub, allowing users to freely download, modify and deploy.

Function List

• Convert UI screenshots or design sketches into HTML/CSS code.
• Supports a modular multi-intelligence body framework, divided into three phases: visual recognition, layout planning and code generation.
• Multiple generation models (Doubao, Qwen, GPT, Gemini) are supported, and users can choose according to their needs.
• Provides editable code output for easy customization and adjustment by developers.
• Automatic generation of large-scale image-code pairs for optimizing model performance.
• Supports rapid prototyping and pixel-accurate interface implementation.

Using Help

Installation process

To use ScreenCoder, users need to first download the code from the GitHub repository and configure the environment. Below are the detailed installation steps:

1. clone warehouse
   Run the following command in the terminal to clone the ScreenCoder project locally:
   "`bash
   git clone https://github.com/leigest519/ScreenCoder.git
   cd screencoder

2. Creating a Virtual Environment
   To avoid dependency conflicts, it is recommended to create a Python virtual environment:
   "`bash
   python3 -m venv .venv
   source .venv/bin/activate

For Windows users:source .venv\Scripts\activateThe

3. Installation of dependencies
   Install the Python libraries required for the project:
   "`bash
   pip install -r requirements.txt

4. Configuration Models and API Keys
   ScreenCoder supports multiple generation models (e.g. Doubao, Qwen, GPT, Gemini). Users need to create the corresponding API key file in the project root directory (e.g.doubao_api.txt,qwen_api.txtetc.) and paste the API key into the file.
   Then, in theblock_parsor.pycap (a poem)html_generator.pyin the configuration file to set the desired model. For example, modify the model parameter in the configuration file to:
   "`python
   model = "Doubao" # or Qwen, GPT. Gemini

Usage Process

The operation of ScreenCoder is divided into the following steps, and users can complete the conversion of UI to code through the command line:

1. Block Detection
   (of a computer) runblock_parsor.pyScript that analyzes the input UI screenshot and identifies key components (e.g., navigation bar, sidebar, content area, etc.). The command is as follows:
   "`bash
   python block_parsor.py

This step generates labels and location information for the components in preparation for subsequent layout planning.

2. Generate code with placeholders
   (of a computer) runhtml_generator.pyscript that generates the initial HTML code with placeholders:
   "`bash
   python html_generator.py

The output code contains the page structure and placeholders (e.g. gray image blocks) for further editing.

3. Content Mapping and Final Code Generation
   After generating the placeholder code, ScreenCoder maps the recognized UI components to the final HTML/CSS code. Users can manually adjust the code as needed to ensure that the style and layout meet expectations.

Featured Function Operation

• Visual Recognition (Grounding Agent)
  ScreenCoder's Visual Recognition Module analyzes UI screenshots using Visual-Linguistic Modeling (VLM) to automatically detect and mark key elements in the interface, such as buttons, navigation bars, text boxes, etc. This feature ensures that the generated code structure is highly consistent with the design. Users simply upload screenshots in PNG or JPG format, and the system automatically processes and outputs component tags.
• Layout planning (Planning Agent)
  The Layout Planning module organizes identified components to form a hierarchical page structure based on front-end engineering principles. For example, it will place the navigation bar at the top, the sidebar on the left, and the main content area in the center. Users do not need to specify the layout manually, the system will automatically generate a reasonable HTML structure.
• Code generation (Generation Agent)
  The code generation module generates HTML and CSS code with adaptive prompts. The generated code is clean, semantically clear and conforms to modern web development standards. Users can copy the code directly into the project after generation or adjust the style and functionality as needed.

caveat

• Ensure that input screenshots are clear and UI elements are well defined, avoiding blurry or overly complex images.
• Check the validity of the API key to ensure that the selected model is properly served by the API.
• If you need to adjust the code, it is recommended that you use a code editor (e.g. VS Code) to check and optimize the CSS styles after generation.

With the above steps, users can quickly turn a design into usable web code suitable for rapid prototyping or production environment deployment.

application scenario

1. Rapid Prototyping
   Designers or developers need to quickly turn UI designs into interactive web prototypes. ScreenCoder automatically generates HTML/CSS code to shorten the time from design to development.
2. Front-end automation
   Front-end development teams want to automate code generation and reduce repetitive tasks.ScreenCoder's modular framework delivers high-precision code for large-scale projects.
3. Learning and teaching
   Programming beginners or front-end development courses can use ScreenCoder to turn actual UI design into code and learn the structure and writing style of HTML/CSS.
4. Cross-platform interface development
   Developers need to use the same design for multiple platforms (e.g., web and mobile), and the code generated by ScreenCoder can be used as a basis for minor adjustments to adapt to different devices.

QA

1. What file formats does ScreenCoder support?
   ScreenCoder supports UI screenshots or design sketches in PNG and JPG formats. Make sure the images are clear and the UI elements are well defined for the best conversion results.
2. Is programming experience required?
   No in-depth programming experience is required, but users familiar with HTML/CSS can better tune the generated code. Beginners can use the generated code directly.
3. How do I choose to generate a model?
   existblock_parsor.pycap (a poem)html_generator.pySet models (e.g. Doubao, Qwen, etc.) in it and provide corresponding API keys. Different models may affect the generation speed and code quality, so it is recommended to test according to the requirements.
4. Does the generated code need to be adjusted manually?
   The generated code can usually be used directly, but complex designs may require manual adjustments to CSS styling or HTML structure to meet specific needs.
5. Is ScreenCoder free?
   ScreenCoder is an open source project, and the code is publicly available for free on GitHub. However, the use of certain generative models (e.g., GPT, Gemini) requires obtaining API keys on your own, which may involve costs.