Supercharging Open-source LLMs: Your Gateway to 250+ Models

In the ever-evolving world of AI, open-source Large Language Models (LLMs) are becoming increasingly popular. As a developer building AI-powered applications, you're likely aware of their advantages:

• Enhanced privacy and security
• Greater model ownership
• Improved model steerability
• Customization possibilities

Open-source models offer unparalleled flexibility and control. Models like Llama 3.1, Mistral, Qwen 3, and Deepseek Coder are pushing the boundaries of what's possible.

But transitioning from closed-source models like OpenAI or Anthropic to open-source alternatives can be challenging. That's where Portkey comes in, making your open-source LLM apps production-ready with just a few lines of code.

Portkey: Your Open-source LLM Superpower

Portkey works with ALL major open source inference provider: Fireworks, Anyscale, Together AI, Perplexiy, Mistral, Google Vertex, AWS Bedrock, Azure Inference, Github Models, Deepinfra, Ollama, Openrouter, Cerebras, Predibase, Workers AI, and many more!

Here's how Portkey makes your app built on OSS LLMs prod-ready:

1. Universal API: Seamlessly route requests to any open-source LLM.
2. Intelligent Fallbacks: Set up automatic fallbacks to closed-source models like OpenAI for uninterrupted service.
3. Canary Testing: Easily route a portion of your traffic to open-source LLMs for testing.
4. Comprehensive Metrics: Get deep insights into cost, latency, and performance for all your LLM requests.
5. Debug with Ease: Use Portkey's dashboard as your single source of truth for quick issue resolution.
6. Production-Ready Suite: Leverage Portkey's Gateway, Observability, Guardrails, and Prompt Management for robust deployments.

List of Open Source Models on Portkey

In this blog, we've written up a short guide for whenever you want to integrate a particular model in your app - how you can use the model, switch from OpenAI, and caveats!

Getting Started is a Breeze

Integrating Portkey with your existing code is surprisingly simple.

If this is your OpenAI code:

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ.get("OPENAI_API_KEY"),
)

chat_completion = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": "Say this is a test",
        }
    ],
    model="gpt-3.5-turbo",
)

Switching to Any Open-source Model is a 3-line Change:

Simply modify the provider in createHeaders, add Authorization, and change the model name - that's it!

from openai import OpenAI
from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders

client = OpenAI(
    api_key=os.environ.get("FIREWORKS_API_KEY"),
    base_url=PORTKEY_GATEWAY_URL,
    default_headers=createHeaders(
        provider="fireworks",
        api_key="PORTKEY_API_KEY"
    )
)

chat_completion = client.chat.completions.create(
    model="llama-3.1",
    messages=[{"role": "user", "content": "Say this is a test"}],
)

With these simple modifications, you're now using an open-source model through Portkey's powerful Gateway.

Local Models? We've Got You Covered

For those looking to keep things close to home, Portkey supports local model deployments too. Whether you're using Ollama, LocalAI, Nvidia Triton, vLLM server, or any locally hosted LLM, Portkey seamlessly integrates with your setup.

Portkey: Your Open-source  Co-pilot

1. Effortless Switching: Transition between models with a single line of code change.
2. Robust Fallbacks: Ensure high availability by falling back to closed-source models when needed.
3. Granular Control: Implement canary testing to gradually shift traffic to open-source models.
4. Deep Insights: Leverage Portkey's dashboard for comprehensive performance metrics and quick debugging.
5. Production-Ready: From Gateway to Guardrails, Portkey provides everything you need for confident deployment.

Key Considerations When Transitioning from Closed Source to Open-Source Models

Assuming you've verified that the open-source model's outputs meet your use case requirements and can effectively replace GPT-4o / Claude-3.5-Sonneet / Gemini, here are crucial factors to consider for production reliability:

1. Performance Metrics: Analyze 90-day latency and API uptime statistics.
2. Output Consistency: Verify that the model doesn't frequently produce loops or garbage text.
3. Provider Variations: Be aware that each inference provider may add their "flavor" to the open-source model (e.g., quantization techniques). Don't expect identical outputs from the same model served by different providers.
4. Parameter Responsiveness: Test how model behavior changes with different parameters (e.g., temperature settings between 0 and 2) to ensure expected responses.
5. Fallback Strategies: Implement fallbacks to more stable models in case of failures.
6. Rate Limits: Understand the rate limits for your chosen inference providers, especially for newly added or high-parameter models which may have severe initial restrictions.
7. Error Handling: Prepare for unexpected 500 errors by setting up failovers and robust error handling mechanisms.

Start Your Open-source LLM Journey Today

Open-source LLMs offer exciting possibilities, and Portkey ensures you can harness their power without the headaches. Whether you're looking to reduce costs, increase flexibility, or maintain data sovereignty, Portkey is your trusted companion in the world of open-source AI.

Ready to supercharge your AI applications with open-source LLMs? Sign up for Portkey today and join our community of AI innovators to share experiences, get support, and stay at the cutting edge of AI development

By embracing open-source models and leveraging Portkey's Gateway, you're not just adopting cutting-edge AI technology – you're joining a movement that's shaping the future of AI development and deployment. Start exploring the vast world of open-source LLMs today and unlock new possibilities for your AI applications!