LiteLLM

LiteLLM is an open-source library that gives you one consistent, OpenAI-compatible interface across many model providers. You can use it in two ways with CompactifAI:

• Python SDK: call litellm.completion(...) directly in your own code without running a separate server.
• Proxy server: run litellm --config ... locally (or in your infra) and point any OpenAI-compatible client — Claude Code, IDE plugins, internal tools — at it.

You need a CompactifAI API key and the model id for each model you plan to use (for example hypernova-60b or glm-5-2). LiteLLM ships a native compactifai/ provider that already knows CompactifAI’s API endpoint — see the official LiteLLM CompactifAI provider docs for the canonical reference.

How it works

Your app / CLI  -->  LiteLLM (proxy or SDK)  -->  CompactifAI API

LiteLLM translates requests into CompactifAI’s OpenAI-compatible /v1/chat/completions format, forwards them, and streams the response back in the same format your client already expects.

1. Install LiteLLM

For SDK-only usage:

pip install litellm

For the proxy server:

pip install 'litellm[proxy]'

2. Option A — Use the Python SDK directly

No proxy needed — call CompactifAI straight from the SDK:

import os
from litellm import completion

os.environ["COMPACTIFAI_API_KEY"] = "your-compactifai-api-key"

response = completion(
    model="compactifai/hypernova-60b",
    messages=[{"role": "user", "content": "Hello!"}],
)

print(response.choices[0].message.content)

Swap compactifai/hypernova-60b for any other model id from the models catalog. Tool calling, streaming (stream=True), and the other standard OpenAI-style parameters (temperature, tools, tool_choice, response_format, etc.) are all supported the same way — see Tool Calling for examples.

3. Option B — Run the LiteLLM proxy

Step 1: Set your API key

export COMPACTIFAI_API_KEY="your-compactifai-api-key"

Step 2: Create a litellm_config.yaml:

model_list:
  - model_name: hypernova-60b
    litellm_params:
      model: compactifai/hypernova-60b
      api_key: os.environ/COMPACTIFAI_API_KEY
  - model_name: glm-5-2
    litellm_params:
      model: compactifai/glm-5-2
      api_key: os.environ/COMPACTIFAI_API_KEY
litellm_settings:
  drop_params: true

Replace the model_name entries with any models from the models catalog you have access to. You can discover the full list of available model IDs programmatically via GET /v1/models (see API reference). The litellm_params.model value must be prefixed with compactifai/ so LiteLLM routes through its built-in CompactifAI provider — no api_base override needed unless you’re explicitly targeting a non-default environment.

Step 3: Start the proxy:

litellm --config litellm_config.yaml --port 4000

Call it exactly like the OpenAI API:

curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer anything" \
  -d '{
    "model": "hypernova-60b",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Note

The proxy’s own Authorization header value isn’t checked against CompactifAI — your real CompactifAI key is the one set in litellm_config.yaml via os.environ/COMPACTIFAI_API_KEY. If you expose the proxy beyond your machine, put it behind your own auth/network controls.

4. Point other tools at the proxy

Once the proxy is running on http://localhost:4000, any tool that supports a custom OpenAI-compatible base URL can use CompactifAI through it — for example Claude Code sets ANTHROPIC_BASE_URL to the LiteLLM proxy address.

Troubleshooting

Issue	Solution
Connection refused on localhost:4000	Make sure the LiteLLM proxy is running (litellm --config litellm_config.yaml --port 4000)
401 Unauthorized from CompactifAI	Verify COMPACTIFAI_API_KEY is set and matches the key from your MultiverseIAM dashboard
Model not found error	Ensure the model_name in litellm_config.yaml (or the model argument in the SDK) matches an id from the models catalog, and is prefixed with compactifai/
drop_params errors about unsupported fields	Keep litellm_settings.drop_params: true in the config so LiteLLM strips fields CompactifAI doesn’t accept

For request and response fields, see Chat Completion and the API reference.