LiteLLM

Connect LiteLLM to AI Studio using its SDK or proxy.

This quickstart guide walks you through connecting Hyperstack AI Studio with LiteLLM. You’ll learn how to authenticate, configure environments, and make chat/completion requests to Hyperstack-hosted models using either the LiteLLM SDK or Proxy.

Full Guide

For a full walkthrough, including production-ready proxy workflows and advanced configurations, see How to Integrate Hyperstack AI Studio as a Provider in LiteLLM.

Why Use LiteLLM with Hyperstack

LiteLLM simplifies the developer experience when working with OpenAI-compatible models by abstracting away provider-specific logic. Hyperstack AI Studio complements this by offering scalable model hosting, fine-tuning, and management behind an OpenAI-compatible API.

When used together, LiteLLM provides a streamlined interface to Hyperstack’s powerful backend infrastructure. Hyperstack hosts and serves the models, while LiteLLM handles routing, configuration, and API abstraction - ideal for everything from experimentation to production deployment.

Hyperstack AI Studio	LiteLLM
AI backend runs inference, fine-tuning, evaluation, and hosting	Interface layer routes requests and provides SDK/proxy access
OpenAI-compatible API	Supports OpenAI schema out of the box
Scalable, secure model infrastructure	Unified function calls and proxy endpoints
Fine-tuned custom model hosting	Easy local or cloud integration

To integrate LiteLLM with Hyperstack, you can choose between two supported workflows depending on your environment:

• Use the Python SDK Integration for local development, scripts, or lightweight projects.
• Use the Proxy Server Integration for production workloads, API management, and team-based workflows.

Each method is compatible with any Hyperstack-hosted model via OpenAI-compatible endpoints.

Option 1: Python SDK Integration

Use this approach for local dev, testing, or scripts.

1. Install LiteLLM

   Install the core LiteLLM SDK via pip:

   pip install litellm

2. Get Your API Key and Model Details

   Generate the necessary credentials to authenticate with Hyperstack AI Studio.

   a. Log in to the Hyperstack Console
   b. Navigate to the AI Studio Playground and select your desired model (e.g., meta-llama/Llama-3.1-8B-Instruct)
   c. Click on the API tab to retrieve your Base URL and Model ID
   d. Visit the API Keys page and generate a new key
   e. Copy and securely store the generated API key

3. Set Up Environment Variables

   Configure your authentication and base endpoint as environment variables:

   import os
   os.environ["OPENAI_API_KEY"] = "<your-hyperstack-api-key>"
   os.environ["OPENAI_API_BASE"] = "https://console.hyperstack.cloud/ai/api/v1"

4. Make a Completion Call

   Run a basic test query to verify the setup:

   from litellm import completion
   
   response = completion(
     model="openai/meta-llama/Llama-3.1-8B-Instruct",
     messages=[{"role": "user", "content": "Who won the World Cup in 2022?"}]
   )
   
   print(response["choices"][0]["message"]["content"])

Option 2: Proxy Server Integration

For team-based or production workloads, use the LiteLLM proxy server. It allows centralized management of models, API keys, usage tracking, and routing.

1. Download Config Files

   You will need two files: one for Docker Compose and another for Prometheus monitoring.

   curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/docker-compose.yml
   curl -O https://raw.githubusercontent.com/BerriAI/litellm/main/prometheus.yml

2. Create a .env File

   Define your admin credentials and encryption salt in a .env file:

   echo 'LITELLM_MASTER_KEY="sk-admin"' > .env
   echo 'LITELLM_SALT_KEY="sk-salt"' >> .env
   source .env

3. Start the Proxy Server

   Run the following to launch the proxy:

   docker compose up

   Once started, access the admin dashboard at:

   http://0.0.0.0:4000

4. Add Your Hyperstack Model

   After logging into the admin dashboard with your master key:

   a. Go to the Models section and click Add Model
   b. Select the OpenAI-Compatible Provider option
   c. Enter the following details:

   • Model Name: meta-llama/Llama-3.1-8B-Instruct
   • Base URL: https://console.hyperstack.cloud/ai/api/v1
   • API Key: The key you generated in the Hyperstack Console

   d. Click Add Model to register it

5. Test Your Model in the Proxy Playground

   Navigate to the Playground tab in the LiteLLM dashboard:

   a. Select the model you just added
   b. Enter a prompt like:

   What's the capital of Nepal?

   c. Confirm the model response:

   The capital of Nepal is Kathmandu.

6. Make cURL or SDK Requests

   You can now access the Hyperstack model via REST API requests to the proxy endpoint:

   curl --location 'http://0.0.0.0:4000/chat/completions' \
   --header 'Content-Type: application/json' \
   --data '{
     "model": "meta-llama/Llama-3.1-8B-Instruct",
     "messages": [
       {"role": "user", "content": "What LLM are you?"}
     ]
   }'

   This will return a valid OpenAI-compatible response routed to your Hyperstack-hosted model.

Next Steps

For details on how to move from prototype to production - including caching, load balancing, and managing multiple models - see How to Integrate Hyperstack AI Studio as a Provider in LiteLLM.

Back to top