> ## Documentation Index
> Fetch the complete documentation index at: https://langwatch.ai/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Security

> Security model, encryption, secrets management, and hardening for LangWatch

This page covers the security features and best practices for self-hosted LangWatch deployments.

## Authentication & Authorization

LangWatch handles user authentication with support for:

* Email/password (default)
* SSO providers: Azure AD, Okta, Auth0, Google, GitHub, GitLab

See [SSO Configuration](/docs/self-hosting/configuration/sso) for setup guides.

**Role-Based Access Control (RBAC)** controls what users can do within a project:

* Organization-level roles (owner, admin, member)
* Project-level permissions

**SCIM provisioning** (Enterprise) enables automated user lifecycle management from your identity provider.

**API tokens** are signed with JWT (`API_TOKEN_JWT_SECRET`) for SDK authentication.

## Encryption

### At Rest

| Data Store         | Encryption Method                                            |
| ------------------ | ------------------------------------------------------------ |
| PostgreSQL         | Provider-level encryption (RDS: AES-256, Cloud SQL: AES-256) |
| ClickHouse         | Encrypted volumes (EBS encryption, PD encryption)            |
| S3                 | Server-side encryption (SSE-S3 or SSE-KMS)                   |
| Stored credentials | Application-level encryption via `CREDENTIALS_SECRET`        |

The `CREDENTIALS_SECRET` environment variable is used to encrypt API keys and credentials stored in PostgreSQL (e.g., LLM provider keys configured in the UI). This is application-level encryption on top of database-level encryption.

### In Transit

| Path                                         | Encryption                                                |
| -------------------------------------------- | --------------------------------------------------------- |
| Client to App                                | TLS at Ingress, Load Balancer                             |
| App to PostgreSQL                            | TLS (configure via connection string: `?sslmode=require`) |
| App to ClickHouse                            | HTTPS (configure ClickHouse with TLS certificates)        |
| App to Redis                                 | TLS (configure via connection string: `rediss://...`)     |
| Inter-service (App, Workers, NLP, LangEvals) | Plain HTTP within cluster (use a service mesh for mTLS)   |

<Tip>
  For inter-service encryption, deploy a service mesh like Istio or Linkerd. This adds mTLS between all pods without application changes.
</Tip>

## Secrets Management

### Development (Auto-Generated)

For development, enable `autogen.enabled: true` in the Helm chart. This generates random secrets automatically. Not suitable for production, secrets change on reinstall.

### Production (Kubernetes Secrets)

Create secrets manually and reference them in the Helm chart:

```bash theme={null}
kubectl create secret generic langwatch-secrets \
  --namespace langwatch \
  --from-literal=credentialsEncryptionKey=$(openssl rand -hex 32) \
  --from-literal=nextAuthSecret=$(openssl rand -hex 32) \
  --from-literal=cronApiKey=$(openssl rand -hex 32)
```

Reference in `values.yaml`:

```yaml theme={null}
secrets:
  existingSecret: langwatch-secrets
```

### Production (External Secret Managers)

For tighter security, use an external secrets operator to sync secrets from your cloud provider:

* **AWS Secrets Manager**: via [External Secrets Operator](https://external-secrets.io/)
* **HashiCorp Vault**: via [Vault Secrets Operator](https://developer.hashicorp.com/vault/docs/platform/k8s/vso)
* **Azure Key Vault**: via [Azure Key Vault Provider](https://azure.github.io/secrets-store-csi-driver-provider-azure/)

The Helm chart's `secretKeyRef` pattern works with any Kubernetes Secret, regardless of how it was created.

## Network Security

### Recommended Network Architecture

* **Only the LangWatch App should be exposed externally** via Ingress or Load Balancer
* All other components (Workers, NLP, LangEvals, PostgreSQL, ClickHouse, Redis) should be on internal networks only (ClusterIP services)
* Place databases in private subnets with no internet access
* Use VPC endpoints, PrivateLink for S3 access

### Private control-plane paths are blocked at the ingress

Everything under `/api/internal/*` is the app's private control plane — the Langy
agent's callbacks and the AI gateway's usage callbacks. Those routes authenticate
with a shared secret, but they are never meant to be reachable from the internet.

The chart blocks them at the ingress by default, via `ingress.blockedPaths`
(default `["/api/internal"]`). Each blocked prefix is routed to a Service with no
endpoints, so the controller has nowhere to forward the request:

```yaml theme={null}
ingress:
  enabled: true
  # Defaults to ["/api/internal"]. Add prefixes your install keeps in-cluster.
  blockedPaths:
    - /api/internal
    - /api/cron   # only if you do NOT drive cron from an external scheduler
```

Things worth knowing before you change it:

* **Blocked requests answer 503** on ingress-nginx. The exact code is
  controller-dependent (the Ingress API does not specify what a backend with no
  endpoints returns), and this is verified against ingress-nginx only — if you
  run a different controller, confirm it answers 5xx rather than falling through
  to another rule before relying on the block. Expect a low background rate in
  your ingress 5xx metrics from scanners — that is the block working, not the
  app faulting.
* **In-cluster callers are unaffected** — the agent, gateway and CronJobs use the
  internal Service.
* **It covers this ingress only.** If you publish the app another way (NodePort,
  LoadBalancer, your own Ingress), restrict these paths where that route ends.
* **The chart refuses configurations that would defeat the block.** Each of
  these renders a manifest that looks protected and is not, so the chart fails
  at render time rather than shipping it:
  * an `ingress.hosts` path nested under a blocked prefix (it out-matches the
    blackhole)
  * a blocked prefix with a trailing slash, a repeated slash, or surrounding
    whitespace (none of them match any real request path)
  * a regex `pathType: ImplementationSpecific` path, which ingress-nginx ranks
    *above* prefix rules
  * the `nginx.ingress.kubernetes.io/default-backend` annotation, which is
    defined as the handler for a backend with no endpoints — exactly what the
    blackhole is, so it converts the block into a proxy
  * a host whose `http.paths` list is empty, which would leave an Ingress whose
    only rules are blackholes
* **The block depends on nobody being able to create an EndpointSlice named for
  the blackhole Service.** It is selector-less, so endpoints can only arrive by
  hand — restrict `discovery.k8s.io/endpointslices: create` in this namespace if
  that is not already the case.
* **AWS LBC (default `target-type: instance`) and GKE without NEG** reject a
  ClusterIP backend and stop reconciling the Ingress. Set
  `alb.ingress.kubernetes.io/target-type: ip` (or the NEG equivalent), or empty
  `blockedPaths` and block at the load balancer instead.
* **To disable**, set `blockedPaths: []` in a values file or use `--set-json`;
  plain `--set` assigns a string and is rejected.
* **Gateway or Langy agent running outside the cluster?** Their callbacks arrive
  over the public ingress and will be blocked — narrow the list rather than
  emptying it.

### Kubernetes Network Policies

Restrict traffic between pods:

```yaml theme={null}
# Example: only allow app and workers to reach ClickHouse
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: clickhouse-access
  namespace: langwatch
spec:
  podSelector:
    matchLabels:
      app: clickhouse
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: app
        - podSelector:
            matchLabels:
              app.kubernetes.io/component: workers
      ports:
        - port: 8123
```

### Firewall Rules

| Source         | Destination   | Port | Protocol |
| -------------- | ------------- | ---- | -------- |
| Internet, VPN  | App (Ingress) | 443  | HTTPS    |
| App            | PostgreSQL    | 5432 | TCP      |
| App, Workers   | ClickHouse    | 8123 | HTTP     |
| App, Workers   | Redis         | 6379 | TCP      |
| Workers        | NLP           | 5561 | HTTP     |
| Workers        | LangEvals     | 5562 | HTTP     |
| NLP, LangEvals | External LLMs | 443  | HTTPS    |
| CronJobs       | App           | 5560 | HTTP     |

## Pod Security

Every LangWatch pod — app, workers, NLP, LangEvals, gateway, and the cron pods — and every bundled datastore — PostgreSQL, Redis, ClickHouse, Keeper — runs hardened. The defaults the first-party services inherit:

```yaml theme={null}
# Pod-level (global.podSecurityContext)
runAsNonRoot: true
runAsUser: 1000
fsGroup: 1000
seccompProfile:
  type: RuntimeDefault

# Container-level (global.containerSecurityContext)
runAsNonRoot: true
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
  drop: [ALL]
```

The bundled datastores carry the same posture but keep their image's own uid — PostgreSQL and Redis run 999, ClickHouse and Keeper 101, the gateway 65532, and Keeper's init container 65534. If you write a Gatekeeper `MustRunAs` constraint, allow all of those, not just 1000: a constraint that misses the init container's 65534 denies the whole Keeper pod. (The bundled Prometheus also runs 65534, though `strict-admission` removes it.)

`runAsNonRoot` is deliberately set at both pod and container level. Kubernetes inherits the pod-level value, but some Gatekeeper constraints read the container-level field directly and deny pods that only carry it on the pod.

No LangWatch pod mounts a ServiceAccount token (`global.automountServiceAccountToken: false`); none of these services talk to the Kubernetes API. The bundled Prometheus does mount one — it is an upstream subchart, and `strict-admission` removes it.

The root filesystem is read-only on every one of those pods. Anything a process writes at runtime (the app's `/tmp`, ClickHouse's server logs and `/tmp`, Postgres' socket directory) lands on an `emptyDir` mounted over that path; persistent data stays on its PVC. The image layer is never writable.

### Strict admission control

On a cluster enforcing Pod Security Admission [`restricted`](https://kubernetes.io/docs/concepts/security/pod-security-standards/) or an equivalent Gatekeeper / Kyverno bundle, apply the [`strict-admission` overlay](https://github.com/langwatch/langwatch/blob/main/charts/langwatch/examples/overlays/strict-admission.yaml):

```bash theme={null}
helm install lw . \
  -f examples/overlays/size-prod.yaml \
  -f examples/overlays/access-ingress.yaml \
  -f examples/overlays/strict-admission.yaml
```

A **default** install does not pass on its own. Everything LangWatch authors already complies — read-only root, non-root at both levels, dropped capabilities, `RuntimeDefault` seccomp, no privilege escalation, no automounted token, CPU + memory requests and limits — but three bundled components cannot, and the overlay turns all three off:

* **Prometheus** (`prometheus.chartManaged`) is an upstream subchart LangWatch doesn't control. Its pods carry no `readOnlyRootFilesystem`, no seccomp profile, and no resource limits on the config-reload sidecar. Bring your own, or disable metrics.
* **The Langy assistant** (`langyagent.chartManaged`) runs its manager as root with `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETUID` and `SETGID`. That is by design: the manager gives every assistant worker a distinct UID, and without those capabilities sibling workers share a UID and can read each other's credentials. Don't force it non-root — deploy it on a cluster that allows it, or run without the assistant.
* **The ClickHouse preflight Job** (`clickhouse.preflight.enabled`) shells out to `kubectl` to check your Secret's keys, so it needs both an automounted token and a writable root for kubectl's discovery cache. It only renders when you supply the ClickHouse Secret yourself rather than using autogen — which is the production path, so the overlay pins it off.

One more toggle in the overlay isn't an admission failure but an unmet dependency:

* **Gateway HPA** (`gateway.autoscaling.enabled`) scales on a custom metric via `prometheus-adapter`. Set it to `false` if you don't run one; the overlay pins `gateway.replicaCount: 2` instead.

With Prometheus off, app metrics reporting (`app.telemetry.metrics.enabled`) is off by default too, so nothing exposes or scrapes a `/metrics` endpoint; the overlay pins it off to keep the two coupled. Anonymous usage analytics (`app.telemetry.usage.enabled`) is a separate setting, on by default; set it to `false` for an air-gapped or no-egress install.

In-cluster datastores satisfy read-only-root policies, but managed databases (the `postgres-external`, `redis-external`, and `clickhouse-external` overlays) are still the better production choice: you get backups, HA, and patching, and there are no datastore pods to admit.

## PII Redaction

LangWatch includes a built-in PII redaction pipeline step that automatically detects and masks personally identifiable information in traces before storage.

* **Enabled by default** in the Helm chart
* Disable with `app.features.disablePiiRedaction: true` (not recommended)
* Runs as part of the event sourcing pipeline in workers

## Multitenancy

LangWatch enforces tenant isolation at the application level:

* Every ClickHouse query includes `WHERE TenantId = ...` as the first predicate
* PostgreSQL queries include `projectId` in WHERE clauses
* API tokens are scoped to a specific project
* Cross-tenant data access is prevented at the query layer

## Supply Chain

LangWatch container images and CLI packages are published with verifiable supply-chain attestations so operators can confirm an artifact was built by LangWatch CI from a specific source commit.

### Container images (Docker Hub)

Every release of `langwatch/langwatch`, `langwatch/langwatch_nlp`, `langwatch/langevals`, and `langwatch/ai-gateway` is signed with [Sigstore](https://www.sigstore.dev/) cosign using keyless OIDC. Both the multi-arch index manifest and each per-platform manifest (linux/amd64, linux/arm64) are signed by digest. A CycloneDX SBOM is generated per platform and attached as a cosign attestation against the matching platform manifest digest, so the SBOM you verify always corresponds to the architecture you actually pulled.

Verify a signature with [cosign](https://github.com/sigstore/cosign):

```bash theme={null}
cosign verify langwatch/langwatch:<tag> \
  --certificate-identity-regexp '^https://github\.com/langwatch/langwatch/' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com
```

Inspect the attached SBOM for the platform you pulled (cosign resolves the right per-platform manifest digest automatically when you pass a tag):

```bash theme={null}
cosign download attestation \
  --predicate-type https://cyclonedx.org/bom \
  langwatch/langwatch:<tag> \
  | jq -r '.payload | @base64d | fromjson | .predicate' \
  > langwatch.cdx.json
```

The per-platform `*.cdx.json` files (e.g. `langwatch-linux-amd64.cdx.json`, `langwatch-linux-arm64.cdx.json`) are also attached to each `langwatch@vX.Y.Z` [GitHub release](https://github.com/langwatch/langwatch/releases).

### npm CLI

The `langwatch` npm package is published with [npm provenance attestations](https://docs.npmjs.com/generating-provenance-statements) via GitHub Actions OIDC, also backed by Sigstore. The provenance link is visible on the [package page](https://www.npmjs.com/package/langwatch) and can be verified with `npm audit signatures`.

## Production Hardening Checklist

* [ ] `autogen.enabled: false`, use manually created secrets
* [ ] All secrets stored in a secrets manager (not inline in values.yaml)
* [ ] TLS enabled on Ingress (HTTPS only)
* [ ] Database connections use TLS (`?sslmode=require`)
* [ ] PostgreSQL, ClickHouse, Redis in private subnets (no public access)
* [ ] Network policies restrict pod-to-pod traffic
* [ ] S3 buckets have public access blocked
* [ ] ClickHouse backups enabled and tested
* [ ] Monitoring and alerting configured
* [ ] Secret rotation procedure documented
* [ ] Pod security contexts verified (non-root, read-only filesystem, `RuntimeDefault` seccomp, no automounted token)
* [ ] On strict admission clusters: apply `examples/overlays/strict-admission.yaml` (`prometheus.chartManaged: false`, `langyagent.chartManaged: false`, and — without a custom metrics API — `gateway.autoscaling.enabled: false`)
* [ ] Ingress rate limiting configured
* [ ] Audit logs enabled (Enterprise)
* [ ] Image signatures verified at pull time (admission controller or `cosign verify` in CI)