> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenTelemetry Migration

> Integrate LangWatch with existing OpenTelemetry setups to enhance tracing, analysis, and agent evaluation workflows.

The LangWatch Python SDK is built entirely on top of the robust [OpenTelemetry (OTel)](https://opentelemetry.io/) standard. This means seamless integration with existing OTel setups and interoperability with the wider OTel ecosystem.

## LangWatch Spans are OpenTelemetry Spans

It's important to understand that LangWatch traces and spans **are** standard OpenTelemetry traces and spans. LangWatch adds specific semantic attributes (like `langwatch.span.type`, `langwatch.inputs`, `langwatch.outputs`, `langwatch.metadata`) to these standard spans to power its observability features.

This foundation provides several benefits:

* **Interoperability:** Traces generated with LangWatch can be sent to any OTel-compatible backend (Jaeger, Tempo, Datadog, etc.) alongside your other application traces.
* **Familiar API:** If you're already familiar with OpenTelemetry concepts and APIs, working with LangWatch's manual instrumentation will feel natural.
* **Leverage Existing Setup:** LangWatch integrates smoothly with your existing OTel `TracerProvider` and instrumentation.

Perhaps the most significant advantage is that **LangWatch seamlessly integrates with the vast ecosystem of standard OpenTelemetry auto-instrumentation libraries.** This means you can easily combine LangWatch's LLM-specific observability with insights from other parts of your application stack. For example, if you use `opentelemetry-instrumentation-celery`, traces initiated by LangWatch for an LLM task can automatically include spans generated within your Celery workers, giving you a complete end-to-end view of the request, including background processing, without any extra configuration.

## Leverage the OpenTelemetry Ecosystem: Auto-Instrumentation

One of the most powerful benefits of LangWatch's OpenTelemetry foundation is its **automatic compatibility with the extensive ecosystem of OpenTelemetry auto-instrumentation libraries.**

When you use standard OTel auto-instrumentation for libraries like web frameworks, databases, or task queues alongside LangWatch, you gain **complete end-to-end visibility** into your LLM application's requests. Because LangWatch and these auto-instrumentors use the same underlying OpenTelemetry tracing system and context propagation mechanisms, spans generated across different parts of your application are automatically linked together into a single, unified trace.

This means you don't need to manually stitch together observability data from your LLM interactions and the surrounding infrastructure. If LangWatch instruments an LLM call, and that call involves fetching data via an instrumented database client or triggering a background task via an instrumented queue, all those operations will appear as connected spans within the same trace view in LangWatch (and any other OTel backend you use).

### Examples of Auto-Instrumentation Integration

Here are common scenarios where combining LangWatch with OTel auto-instrumentation provides significant value:

* **Web Frameworks (FastAPI, Flask, Django):** Using libraries like `opentelemetry-instrumentation-fastapi`, an incoming HTTP request automatically starts a trace. When your request handler calls a function instrumented with `@langwatch.trace` or `@langwatch.span`, those LangWatch spans become children of the incoming request span. You see the full request lifecycle, from web server entry to LLM processing and response generation.

* **HTTP Clients (Requests, httpx, aiohttp):** If your LLM application makes outbound API calls (e.g., to fetch external data, call a vector database API, or use a non-instrumented LLM provider via REST) using libraries instrumented by `opentelemetry-instrumentation-requests` or similar, these HTTP request spans will automatically appear within your LangWatch trace, showing the latency and success/failure of these external dependencies.

* **Task Queues (Celery, RQ):** When a request handled by your web server (and traced by LangWatch) enqueues a background job using `opentelemetry-instrumentation-celery`, the trace context is automatically propagated. The spans generated by the Celery worker processing that job will be linked to the original LangWatch trace, giving you visibility into asynchronous operations triggered by your LLM pipeline.

* **Databases & ORMs (SQLAlchemy, Psycopg2, Django ORM):** Using libraries like `opentelemetry-instrumentation-sqlalchemy`, any database queries executed during your LLM processing (e.g., for RAG retrieval, user data lookup, logging results) will appear as spans within the relevant LangWatch trace, pinpointing database interaction time and specific queries.

To enable this, simply ensure you have installed and configured the relevant OpenTelemetry auto-instrumentation libraries according to their documentation, typically involving an installation (`pip install opentelemetry-instrumentation-<library>`) and sometimes an initialization step (like `CeleryInstrumentor().instrument()`). As long as they use the same (or the global) `TracerProvider` that LangWatch is configured with, the integration is automatic.

#### Example: Combining LangWatch, RAG, OpenAI, and Celery

Let's illustrate this with a simplified example involving a web request that performs RAG, calls OpenAI, and triggers a background Celery task.

<CodeGroup>
  ```txt requirements.txt theme={null}
  langwatch
  openai
  celery
  opentelemetry-instrumentation-celery
  ```

  ```python example.py theme={null}
  import langwatch
  import os
  import asyncio
  from celery import Celery
  from openai import OpenAI
  from langwatch.types import RAGChunk

  # 1. Configure Celery App
  celery_app = Celery('tasks', broker=os.getenv('CELERY_BROKER_URL', 'redis://localhost:6379/0'))

  # 2. Setup LangWatch and OpenTelemetry Instrumentation
  from opentelemetry_instrumentation.celery import CeleryInstrumentor
  CeleryInstrumentor().instrument()

  # Now setup LangWatch (it will likely pick up the global provider configured by Celery)
  langwatch.setup(
      # If you have other OTel exporters, configure your TracerProvider manually
      # and pass it via tracer_provider=..., setting ignore_warning=True
      ignore_global_tracer_provider_override_warning=True
  )

  client = OpenAI()

  # 3. Define the Celery Task
  @celery_app.task
  def process_result_background(result_id: str, llm_output: str):
      # This task execution will be automatically linked to the trace
      # that enqueued it, thanks to CeleryInstrumentor.
      # Spans created here (e.g., database writes) would be part of the same trace.
      print(f"[Celery Worker] Processing result {result_id}...")
      # Simulate work
      import time
      time.sleep(1)
      print(f"[Celery Worker] Finished processing {result_id}")
      return f"Processed: {llm_output[:10]}..."

  # 4. Define RAG and Main Processing Logic
  @langwatch.span(type="rag")
  def retrieve_documents(query: str) -> list:
      # Simulate RAG retrieval
      print(f"Retrieving documents for: {query}")
      chunks = [
          RAGChunk(document_id="doc-abc", content="LangWatch uses OpenTelemetry."),
          RAGChunk(document_id="doc-def", content="Celery integrates with OpenTelemetry."),
      ]
      langwatch.get_current_span().update(contexts=chunks)
      time.sleep(0.1)
      return [c.content for c in chunks]

  @langwatch.trace(name="Handle User Query with Celery")
  def handle_request(user_query: str):
      # This is the root span for the request
      langwatch.get_current_trace().autotrack_openai_calls(client)
      langwatch.get_current_trace().update(metadata={"user_query": user_query})

      context_docs = retrieve_documents(user_query)

      try:
          completion = client.chat.completions.create(
              model="gpt-5",
              messages=[
                  {"role": "system", "content": f"Use this context: {context_docs}"},
                  {"role": "user", "content": user_query}
              ],
              temperature=0.5,
          )
          llm_result = completion.choices[0].message.content
      except Exception as e:
          langwatch.get_current_trace().record_exception(e)
          llm_result = "Error calling OpenAI"

      result_id = f"res_{int(time.time())}"
      # The current trace context is automatically propagated
      process_result_background.delay(result_id, llm_result)
      print(f"Enqueued background processing task {result_id}")

      return llm_result

  # 5. Simulate Triggering the Request
  if __name__ == "__main__":
      print("Simulating web request...")
      final_answer = handle_request("How does LangWatch work with Celery?")
      print(f"\nFinal Answer returned to user: {final_answer}")
      # Allow time for task to be processed if running worker locally
      time.sleep(3) # Add a small delay to see Celery output

      # To run this example:
      # 1. Start a Celery worker: celery -A your_module_name worker --loglevel=info
      # 2. Run this Python script.
      # 3. Observe the logs and the trace in LangWatch/OTel backend.
  ```
</CodeGroup>

In this example:

* The `handle_request` function is the main trace.
* `retrieve_documents` is a child span created by LangWatch.
* The OpenAI call creates child spans (due to `autotrack_openai_calls`).
* The call to `process_result_background.delay` creates a span indicating the task was enqueued.
* Critically, `CeleryInstrumentor` automatically propagates the trace context, so when the Celery worker picks up the `process_result_background` task, its execution is linked as a child span (or spans, if the task itself creates more) under the original `handle_request` trace.

This gives you a unified view of the entire operation, from the initial request through LLM processing, RAG, and background task execution.

## Integrating with `langwatch.setup()`

When you call `langwatch.setup()`, it intelligently interacts with your existing OpenTelemetry environment:

1. **Checks for Existing `TracerProvider`:**
   * If you provide a `TracerProvider` instance via the `tracer_provider` argument in `langwatch.setup()`, LangWatch will use that specific provider.
   * If you *don't* provide one, LangWatch checks if a global `TracerProvider` has already been set (e.g., by another library or your own OTel setup code).
   * If neither is found, LangWatch creates a new `TracerProvider`.

2. **Adding the LangWatch Exporter:**
   * If LangWatch uses an *existing* `TracerProvider` (either provided via the argument or detected globally), it will **add its own OTLP Span Exporter** to that provider's list of Span Processors. It does *not* remove existing processors or exporters.
   * If LangWatch creates a *new* `TracerProvider`, it configures it with the LangWatch OTLP Span Exporter.

## Default Behavior: All Spans Go to LangWatch

A crucial point is that once `langwatch.setup()` runs and attaches its exporter to a `TracerProvider`, **all spans** managed by that provider will be exported to the LangWatch backend by default. This includes:

* Spans created using `@langwatch.trace` and `@langwatch.span`.
* Spans created manually using `langwatch.trace()` or `langwatch.span()` as context managers or via `span.end()`.
* Spans generated by standard OpenTelemetry auto-instrumentation libraries (e.g., `opentelemetry-instrumentation-requests`, `opentelemetry-instrumentation-fastapi`) if they are configured to use the same `TracerProvider`.
* Spans you create directly using the OpenTelemetry API (`tracer.start_as_current_span(...)`).

While seeing all application traces can be useful, you might not want *every single span* sent to LangWatch, especially high-volume or low-value ones (like health checks or database pings).

## Selectively Exporting Spans with `span_exclude_rules`

To control which spans are sent to LangWatch, use the `span_exclude_rules` argument during `langwatch.setup()`. This allows you to define rules to filter spans *before* they are exported to LangWatch, without affecting other exporters attached to the same `TracerProvider`.

Rules are defined using `SpanProcessingExcludeRule` objects.

```python  theme={null}
import langwatch
import os
from langwatch.domain import SpanProcessingExcludeRule
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

# Example: You already have an OTel setup exporting to console
existing_provider = TracerProvider()
existing_provider.add_span_processor(
    SimpleSpanProcessor(ConsoleSpanExporter())
)

# Define rules to prevent specific spans from going to LangWatch
# (They will still go to the Console exporter)
exclude_rules = [
    # Exclude spans exactly named "GET /health_check"
    SpanProcessingExcludeRule(
        field_name="span_name",
        match_value="GET /health_check",
        match_operation="exact_match"
    ),
    # Exclude spans where 'http.method' attribute is 'OPTIONS'
    SpanProcessingExcludeRule(
        field_name="attribute",
        attribute_name="http.method",
        match_value="OPTIONS",
        match_operation="exact_match"
    ),
    # Exclude spans whose names start with "Internal."
    SpanProcessingExcludeRule(
        field_name="span_name",
        match_value="Internal.",
        match_operation="starts_with"
    ),
]

# Setup LangWatch to use the existing provider and apply exclude rules
langwatch.setup(
    api_key=os.getenv("LANGWATCH_API_KEY"),
    tracer_provider=existing_provider, # Use our existing provider
    span_exclude_rules=exclude_rules,
    # Important: Set this if you intend for LangWatch to use the existing provider
    # and want to silence the warning about not overriding it.
    ignore_global_tracer_provider_override_warning=True
)

# Now, create some spans using OTel API directly
tracer = existing_provider.get_tracer("my.app.tracer")

with tracer.start_as_current_span("GET /health_check") as span:
    span.set_attribute("http.method", "GET")
    # This span WILL go to Console Exporter
    # This span WILL NOT go to LangWatch Exporter

with tracer.start_as_current_span("Process User Request") as span:
    span.set_attribute("http.method", "POST")
    span.set_attribute("user.id", "user-123")
    # This span WILL go to Console Exporter
    # This span WILL ALSO go to LangWatch Exporter
```

Refer to the `SpanProcessingExcludeRule` definition for all available fields (`span_name`, `attribute`, `library_name`) and operations (`exact_match`, `contains`, `starts_with`, `ends_with`, `regex`).

## Debugging with Console Exporter

When developing or troubleshooting your OpenTelemetry integration, it's often helpful to see the spans being generated locally without sending them to a backend. The OpenTelemetry SDK provides a `ConsoleSpanExporter` for this purpose.

You can add it to your `TracerProvider` like this:

<CodeGroup>
  ```python Scenario 1: Managed Provider (Recommended) theme={null}
  import langwatch
  import os
  from opentelemetry import trace
  from opentelemetry.sdk.trace import TracerProvider
  from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

  # Create your own TracerProvider
  my_tracer_provider = TracerProvider()

  # Add the ConsoleSpanExporter for debugging
  my_tracer_provider.add_span_processor(
      SimpleSpanProcessor(ConsoleSpanExporter())
  )

  # Now, setup LangWatch with your pre-configured provider
  langwatch.setup(
      tracer_provider=my_tracer_provider,
      # If you are providing your own tracer_provider that might be global,
      # you might want to set this to True if you see warnings.
      # ignore_global_tracer_provider_override_warning=True
  )

  # Spans created via LangWatch or directly via OTel API using this provider
  # will now also be printed to the console.

  # Example of creating a span to test
  tracer = my_tracer_provider.get_tracer("my.debug.tracer")
  with tracer.start_as_current_span("My Test Span"):
      print("This span should appear in the console.")
  ```

  ```python Scenario 2: Global Provider (Illustrative) theme={null}
  # Ensure necessary imports if running this snippet standalone
  import os
  import langwatch
  from opentelemetry import trace
  from opentelemetry.sdk.trace import TracerProvider # Needed for isinstance check
  from opentelemetry.sdk.trace.export import SimpleSpanProcessor, ConsoleSpanExporter

  # In this case, you might try to get the global provider and add the exporter.
  # Note: This can be less predictable if other libraries also manipulate the global provider.

  langwatch.setup(
      ignore_global_tracer_provider_override_warning=True # If a global provider exists
  )

  # Try to get the globally configured TracerProvider
  global_provider = trace.get_tracer_provider()

  # Check if it's an SDK TracerProvider instance that we can add a processor to
  if isinstance(global_provider, TracerProvider):
      global_provider.add_span_processor(SimpleSpanProcessor(ConsoleSpanExporter()))

  # Example span after attempting to modify global provider
  # Note: get_tracer from the global trace module
  global_otel_tracer = trace.get_tracer("my.app.tracer.global")

  with global_otel_tracer.start_as_current_span("Test Span with Global Provider"):
      print("This span should appear in console if global provider was successfully modified.")
  ```
</CodeGroup>

This will print all created spans to your console

## Accessing the OpenTelemetry Span API

Since LangWatch spans wrap standard OTel spans, the `LangWatchSpan` object (returned by `langwatch.span()` or accessed via `langwatch.get_current_span()`) directly exposes the standard OpenTelemetry `trace.Span` API methods. This allows you to interact with the span using familiar OTel functions when needed for advanced use cases or compatibility.

You don't need to access a separate underlying object; just call the standard OTel methods directly on the `LangWatchSpan` instance:

```python  theme={null}
import langwatch
from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

langwatch.setup() # Assume setup is done

with langwatch.span(name="MyInitialSpanName") as span:

    # Use standard OpenTelemetry Span API methods directly on span:
    span.set_attribute("my.custom.otel.attribute", "value")
    span.add_event("Specific OTel Event", {"detail": "more info"})
    span.set_status(Status(StatusCode.ERROR, description="Something went wrong"))
    span.update_name("MyUpdatedSpanName") # Renaming the span

    print(f"Is Recording? {span.is_recording()}")
    print(f"OTel Span Context: {span.get_span_context()}")

    # You can still use LangWatch-specific methods like update()
    span.update(langwatch_info="extra data")
```

This allows full flexibility, letting you use both LangWatch's structured data methods (`update`, etc.) and the standard OpenTelemetry span manipulation methods on the same object.

## Understanding `ignore_global_tracer_provider_override_warning`

If `langwatch.setup()` detects an existing *global* `TracerProvider` (one set via `opentelemetry.trace.set_tracer_provider()`) and you haven't explicitly passed a `tracer_provider` argument, LangWatch will log a warning by default. The warning states that it found a global provider and will attach its exporter to it rather than replacing it.

This warning exists because replacing a globally configured provider can sometimes break assumptions made by other parts of your application or libraries. However, in many cases, **attaching** the LangWatch exporter to the existing global provider is exactly the desired behavior.

If you are intentionally running LangWatch alongside an existing global OpenTelemetry setup and want LangWatch to simply add its exporter to that setup, you can silence this warning by setting:

```python  theme={null}
langwatch.setup(
    # ... other options
    ignore_global_tracer_provider_override_warning=True
)
```