LangWatch supports connecting to any model that exposes an OpenAI-compatible API, including local inference servers (Ollama, vLLM, TGI), cloud deployments (Databricks, Azure ML), and custom APIs.
Adding a Custom Model
- Navigate to Settings in your project dashboard
- Select Model Provider from the settings menu
- Enable Custom model
- Configure your model:
| Field | Description |
|---|---|
| Model Name | A descriptive name for your model (e.g., llama-3.1-70b) |
| Base URL | The endpoint URL for your model's API |
| API Key | Authentication key (if required) |
For local models that don't require authentication, enter any non-empty string as the API key.
Example Configurations
Ollama
| Field | Value |
|---|---|
| Base URL | http://localhost:11434/v1 |
| API Key | ollama |
vLLM
| Field | Value |
|---|---|
| Base URL | http://localhost:8000/v1 |
| API Key | Your configured token |
Databricks
| Field | Value |
|---|---|
| Base URL | https://<workspace>.cloud.databricks.com/serving-endpoints |
| API Key | Your Databricks personal access token |
Using Custom Models
Once configured, your custom models appear in the model selector throughout LangWatch, including the Prompt Playground and when configuring scenarios.
When referencing your custom model in code or API calls, use the format:
custom/<your-model-name>For example, if you configured a model named llama-3.1-70b, reference it as custom/llama-3.1-70b.
To check it worked, open the Prompt Playground and confirm your model is listed in the model selector, then send it a message to see that the endpoint answers.
Related
- LiteLLM Integration - Unified interface for multiple providers
- Tracking LLM Costs - Configure cost tracking
- Prompt Playground - Test prompts with custom models