Bifrost: A High Performance Gateway for Connecting Multiple Large Language Models

Bifrost is a high-performance Large Language Model (LLM) gateway built using the Go language. Its core role is to provide developers with a unified interface for connecting to and managing multiple different big model vendors, such as OpenAI, Anthropic, Amazon Bedrock, etc. Bifrost is designed to simplify the development process by eliminating the need for separate integration code for each model. With this gateway, applications simplify the development process by eliminating the need to write separate integration code for each model.Designed with a focus on performance and reliability, Bifrost adds only microseconds of latency when processing large numbers of requests. It has built-in automatic failover and load balancing, so when a model or vendor goes down, the system can automatically forward requests to alternate options, ensuring continuity and stability of service. In addition, Bifrost provides a visual web interface that allows users to monitor requests, manage model configurations and view analytics in real-time, greatly simplifying operations and maintenance.

Function List

• Unified API Interface: Connects to over 10 major big model vendors including OpenAI, Anthropic, Amazon Bedrock, Mistral, Ollama and more through a single API endpoint.
• high performance processing: Built using the Go language, it adds only about 11 microseconds of latency overhead on average under a load of 5,000 requests processed per second.
• Built-in Web User Interface: Provides a visual configuration interface and real-time monitoring dashboards that allow users to manage vendors, monitor logs, and analyze metrics directly in the browser, eliminating the need to manually edit configuration files.
• Automatic Failover: When a request from a model or provider fails, it can automatically retry or switch to a preset alternate model, guaranteeing the stability of the service.
• Load Balancing and Key Management: Supports dynamic and weighted management of API keys, efficiently distributing requests across multiple keys or providers.
• Out-of-the-box observability: Native support for Prometheus indicators allows for easy integration into existing monitoring systems with no additional configuration required.
• Multiple integration methodsThree usage models are supported: running as a standalone HTTP service, as a Go language package integrated directly into an application, or as a direct replacement for the existing OpenAI/Anthropic SDK (just change the base URL of the API).
• Plug-in Architecture: Designed with a plugin-first architecture and support for Model Context Protocol (MCP) for easy extension and integration with external tools.

Using Help

Bifrost offers a variety of flexible ways to use it, the fastest way being through the npx command to start a local HTTP service, this way you can have a fully functional AI gateway in less than 30 seconds without installing any dependencies.

1. Quick start (HTTP service)

This is the easiest and fastest way for all developers. It starts a local server and a companion web administration interface.

Environmental requirements:

• Node.js (version 18+) is installed.
• Have an API key from at least one big model vendor (e.g. OpenAI).

Operational Steps:

Step 1: Start the Bifrost service
Run the following command in your terminal (command line tool):

npx @maximhq/bifrost

When executed, this command automatically downloads and runs Bifrost. the service listens by default to the local 8080 Ports.

Step 2: Configure the vendor
Once the service has started, open the following address in your browser to access the Bifrost web administration interface:
http://localhost:8080
In this interface, you can visually add and manage different big model vendors. For example, to add OpenAI, all you need to do is click the "Add Vendor" button and fill in your OpenAI API key. You can also set advanced options such as model weights and priorities for load balancing and failover.

Step 3: Test API Calls
Once the configuration is complete, your application will be able to access Bifrost's 8080 port to call the big model now. and Bifrost will forward your request to the target provider. You can use the curl command to test whether the service is working properly:

curl -X POST http://localhost:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "openai/gpt-4o-mini",
"messages": [
{"role": "user", "content": "你好，Bifrost！🌈"}
]
}'

If you receive a reply from the model, your gateway is running successfully.

2. As a direct replacement for existing code

If you have an SDK for OpenAI or Anthropic integrated into your code, you can use Bifrost with minimal code changes.

Operational Steps:

1. Follow the Quick Start method above to run the Bifrost service and complete the vendor configuration.
2. In your application code, locate the base_url maybe baseURL Parameters.
3. Change this URL from the official interface address to the address of the Bifrost service.

Code Example:
Suppose your original OpenAI Python code looked like this:

from openai import OpenAI
client = OpenAI(
api_key="YOUR_OPENAI_API_KEY",
# base_url="https://api.openai.com/v1"  # 这是原来的地址
)

You just need to change it to:

from openai import OpenAI
client = OpenAI(
api_key="YOUR_OPENAI_API_KEY", # 这里的 key 依然需要，但 Bifrost 会使用自己管理的 key
base_url="http://localhost:8080/v1" # 指向 Bifrost 网关
)

Once the modifications are complete, your application requests will be forwarded through Bifrost and you will automatically get all the gateway functionality such as failover, load balancing, etc. without having to change any other business logic code.

3. Integration as a Go package

For Go language developers, the core features of Bifrost can be integrated directly into their applications as libraries for maximum performance and control.

Operational Steps:

1. In your Go project, use the following command to get the Bifrost core package:

   go get github.com/maximhq/bifrost/core
   

2. Import and use Bifrost in your code. you can configure vendors, routing rules, and plugins from code. This approach avoids additional HTTP communication overhead and provides the best performance. You can refer to the official documentation of the project for the detailed API and usage.

application scenario

1. Improving the stability of AI applications
   For applications in production environments that require continuous service, model stability is critical. With Bifrost's automatic failover feature, when the primary model (e.g., GPT-4) becomes inaccessible for any reason, the system automatically switches to an alternate model (e.g., Claude 3 or another model) to ensure that user requests are always handled, thus avoiding service interruptions.
2. Reduce and manage the cost of multi-model usage
   Different models have different pricing strategies. Developers can configure multiple models in Bifrost and set up routing rules, e.g., giving computationally intensive and complex tasks to powerful but expensive models, while assigning simple and routine tasks to lower-cost models. In this way, operational costs can be significantly optimized with guaranteed results.
3. Simplify multi-cloud or hybrid cloud deployments
   Organizations may be using models from different cloud providers (e.g. AWS Bedrock, Azure OpenAI) at the same time.Bifrost provides a unified API entry point that shields against underlying vendor differences. This makes application deployment and migration easier and avoids platform lock-in.
4. Rapid experimentation and switching to new models
   Models in the AI space are updated and iterated very quickly. As new and better models emerge, developers can quickly add new models and test them through Bifrost's web interface, even enabling A/B testing on a traffic-ratio basis. The entire process eliminates the need to modify and redeploy application code, dramatically accelerating the pace of innovation and iteration.

QA

1. What are the advantages of Bifrost over other similar tools such as LiteLLM?
   Bifrost's primary strength is performance. It was built from the ground up using the Go language and is designed for high concurrency, low latency production environments. According to official performance tests, Bifrost's latency overhead is much lower than Python-based tools such as LiteLLM on the same hardware, allowing it to handle higher concurrency of requests. In addition, it comes with a visual web interface that makes configuration and monitoring easier and more intuitive.
2. Does using Bifrost affect data privacy?
   Bifrost is a completely open source and self-deployable gateway. This means you can deploy it on your own servers or in a private cloud environment. All requests and data flow only within the infrastructure you control before being sent directly to the final big model provider, and Bifrost itself does not store or send your data to any third-party servers, ensuring data privacy and security.
3. Does Bifrost support Streaming responses?
   Yes, Bifrost fully supports streaming responses for large models. When you make a request to a model that supports streaming output (such as OpenAI's chat model), Bifrost sends chunks of data generated by the model back to the client in real-time, which is critical for building applications such as real-time chatbots or code generation.
4. What is the exact process for configuring failover?
   In Bifrost's web administration interface, you can set up a list of Fallback models for one or more primary models. For example, you can set the openai/gpt-4o-mini set as the primary model, and then set the anthropic/claude-3-haiku cap (a poem) google/gemini-1.5-flash added to its standby list. When a pair of gpt-4o-mini If the request fails, Bifrost automatically tries alternate models in the order of the list until the request succeeds.