llmware: an open source framework for rapidly building enterprise-class RAG applications

llmware is an open source framework focused on helping developers quickly build enterprise-class Retrieval Augmentation Generation (RAG) applications. It provides over 50 small, specialized Large Language Models (LLMs) that support running in local or private cloud environments, and is particularly suited for data security-sensitive industries such as finance, legal and compliance. llmware integrates a complete tool chain from data processing to model inference, allowing developers to connect to enterprise knowledge bases and perform tasks such as quizzing, classification, extraction, etc., without complex configuration. It is designed with the goal of lowering the barriers to AI development so that both beginners and professional developers can get started quickly. The framework supports a wide range of databases and vector stores, runs efficiently, and can even be used on regular laptops without the need for expensive GPU servers. llmware is developed by AI Bloks, follows the Apache 2.0 license, has an active community, and encourages users to participate in contributing.

Function List

• RAG assembly line: Supports complete lifecycle management from knowledge source access to generative AI models.
• 50+ Specialized Models: Includes SLIM, DRAGON, and BLING series of models optimized for tasks such as question and answer, classification, and extraction, respectively.
• Multi-database support: Compatible with text databases such as MongoDB, PostgreSQL, SQLite, and 10 vector databases such as Milvus and PGVector.
• Efficient local operation: Small models can be run on a regular PC without a GPU, reducing costs.
• Integration of knowledge sources: Supports multiple data sources such as PDF, web pages, audio, etc., which are automatically processed and embedded in the model.
• Fact-checking and auditing: Provide evidence validation, fact-checking, and cue history tracking to reduce modeling illusions.
• UI Example: Provides interactive interface examples to help developers build applications quickly.
• automation script: Provides one-click installation scripts to simplify environment configuration and sample runs.

Using Help

Installation process

The installation process for llmware is simple and supports installation via PyPI or GitHub source. Below are the detailed steps:

1. Installation via PyPI::
   • Open a terminal and run the following command to install the base version:

     pip3 install llmware
     

   • If full functionality is required (including OCR, vector databases, etc.), run:

     pip3 install 'llmware[full]'
     

   • Note: A full installation will include optional dependencies such aspymilvus,pytesseractetc. for users who need advanced features.
2. Installation via GitHub source code::
   • Clone the llmware repository:

     git clone https://github.com/llmware-ai/llmware.git
     

   • Go to the project catalog:

     cd llmware
     

   • Run the automated installation script to install the dependencies and configure the examples:

     python3 welcome_to_llmware.py
     

   • The script automatically installs therequirements.txtdependencies, copy the example files, and run thewelcome_example.pyfor a quick experience.
3. Dependent on environmental requirements::
   • Make sure Python 3.8+ is installed on your system.
   • For OCR functionality, Tesseract (v5.3.3) and Poppler (v23.10.0) need to be installed manually.
   • If you are using MongoDB, you need to make sure that themongo-c-driverversion lower than 1.24.4 to avoid segmentation errors.
   • Resize the system stack (under Linux environment):

     ulimit -s 160000
     

   • For Docker users, the stack size is set at runtime:

     docker run --ulimit stack=32768000:32768000
     

Main Functions

1. Creation of RAG applications

At the heart of llmware is building RAG applications, and the following is a typical operational flow:

• Creating Libraries: UseLibraryClasses create knowledge bases to store documents.

  from llmware.library import Library
  lib = Library().create_new_library("my_library")
  

• Add file: Import PDF, text and other documents into the library.

  lib.add_files("/path/to/documents")
  

• Generate vector embedding: Using an embedding model (e.g.industry-bert-contracts) generating vectors.

  from llmware.retrieval import Query
  q = Query(lib)
  q.generate_embeddings("industry-bert-contracts")
  

• perform a search: Retrieve information through textual or semantic queries.

  results = q.text_search_by_page("contract termination", page_num=1)
  

• invocation model: Load the model and combine the query results to generate answers.

  from llmware.prompts import Prompt
  prompter = Prompt().load_model("llmware/dragon-yi-6b-gguf")
  prompter.add_source_query_results(results)
  response = prompter.prompt_with_source("What is the notice period for termination?")
  

2. Use of SLIM models

SLIM models are designed for specific tasks and output structured data (e.g. JSON, SQL). Example of operation:

• Load SLIM model::

  from llmware.prompts import Prompt
  prompter = Prompt().load_model("llmware/slim-sentiment")
  

• executive reasoning::

  response = prompter.prompt_main("This contract is favorable to us.")
  print(response)  # 输出：{"sentiment": "positive"}
  

3. Fact-checking and auditing

llmware provides evidence validation features to ensure that answers are accurate:

• Verify Answer::

  stats = prompter.evidence_comparison_stats(response)
  print(stats)  # 输出证据匹配度
  

• Save Interaction History::

  prompter.save_state()
  

4. UI examples

llmware provides interactive interface examples located in theexamples/UI/Catalog:

• Run the example:

  python3 examples/UI/invoice_ui.py
  

• The example supports uploading PDF invoices and running the BLING model for Q&A.

Featured Function Operation

1. Multi-model Agent workflow

The SLIM model supports multi-step Agent workflows. For example, combiningsentimentcap (a poem)intentModel:

• Loading multiple models::

  prompter1 = Prompt().load_model("llmware/slim-sentiment")
  prompter2 = Prompt().load_model("llmware/slim-intent")
  

• Perform multi-step reasoning::

  text = "I want to cancel my subscription."
  sentiment = prompter1.prompt_main(text)  # 输出：{"sentiment": "negative"}
  intent = prompter2.prompt_main(text)     # 输出：{"intent": "cancellation"}
  

2. Contract analysis

llmware's DRAGON model optimizes contract analysis:

• Analyzing contracts::

  from llmware.library import Library
  from llmware.retrieval import Query
  lib = Library().create_new_library("contracts")
  lib.add_files("/path/to/contracts")
  q = Query(lib)
  results = q.text_query_with_document_filter("termination", {"file_source": "contract.pdf"})
  prompter = Prompt().load_model("llmware/dragon-yi-6b-gguf")
  prompter.add_source_query_results(results)
  response = prompter.prompt_with_source("Summarize termination provisions")
  

3. Vector database integration

A variety of vector databases are supported, such as Milvus:

• Configuring Milvus::

  curl -o docker-compose.yaml https://raw.githubusercontent.com/llmware-ai/llmware/main/docker-compose-redis-stack.yaml
  docker-compose up
  

• Generate Embedding::

  q.generate_embeddings("industry-bert-contracts", vector_db="milvus")
  

application scenario

1. Enterprise contract management
   llmware can be used to analyze large numbers of contracts and extract key terms (e.g., termination clauses, payment terms). Organizations can import contracts into the library and use the DRAGON model for semantic queries to quickly generate a summary of terms, suitable for legal teams to improve efficiency.
2. Financial data processing
   Finance teams can use SLIM models to analyze invoices or financial reports, extracting key fields (e.g. amounts, dates). Batch processing is supported to generate structured JSON output for easy integration into ERP systems.
3. Knowledge Base Q&A
   Enterprises can build an internal knowledge base and combine it with the BLING model to realize the Q&A function. Employees can query company policies, procedures, etc. through natural language, and the model returns accurate answers with evidence sources.
4. Local AI Deployment
   For data-sensitive industries, llmware supports running AI models on local PC or private cloud to avoid data leakage. It is suitable for banks, medical organizations and other scenarios that require high security.

QA

1. Does llmware require a GPU to run?
   Not required. llmware's BLING and SLIM models are optimized to run on the CPU and are suitable for regular laptops. the DRAGON model may require higher-performance hardware, but is still supported for GPU-less environments.
2. How do you deal with modeling illusions?
   llmware provides fact-checking and evidence-verification capabilities through theevidence_comparison_statscap (a poem)evidence_check_sourcesMethods to verify responses to ensure consistency with knowledge sources.
3. What file formats are supported?
   Supports PDF, text, CSV, JSON, audio, web pages, etc. OCR function handles text in images and is suitable for scanned documents.
4. How do I get involved in the llmware community?
   Issues or pull requests can be submitted via GitHub, join the discussion on the Discord channel (https://discord.com/invite/fCztJQeV7J), or check out the contribution guidelines.