DNS and TLS

​

SaaS: dedicated NLB + ACM cert + Cloudflare DNS-only

(customer) ─DNS─▶ gateway.langwatch.ai
                       ▼
             Cloudflare CNAME (DNS-only / gray cloud)
                       ▼
       a9837743b…elb.eu-central-1.amazonaws.com  ← AWS NLB hostname
                       │
                       ├─ TLS terminates here (ACM cert: gateway.langwatch.ai)
                       ▼
            Service `langwatch-gateway-service` (LoadBalancer, port 443)
                       ▼
                    Gateway Pod :5563

• NLB instead of nginx-ingress + ELB. The gateway has different traffic shape than the LangWatch app: long-lived SSE streams, layer-4 passthrough, sub-ms latency budget. NLB suits this; the app’s ELB+nginx posture would buffer SSE without the unbuffered annotations and add a layer-7 hop on every request.
• ACM cert instead of cert-manager. The cert is provisioned via AWS Certificate Manager and validated by DNS-01 (CNAME records published in Cloudflare). cert-manager runs on the app cluster but the gateway uses the AWS-native path because TLS terminates on the NLB itself.
• Cloudflare in DNS-only mode (the gray cloud, NOT proxied). Critical, see Cloudflare proxy mode is wrong for the gateway below.

​

Self-hosted: point your own hostname at the gateway

• AWS NLB + ACM + Cloudflare: recommended for AWS, matches SaaS.
• nginx-ingress + cert-manager: what the chart renders by default; works on any cluster with an nginx-ingress controller.
• AWS ALB: alternative for shops standardised on AWS Load Balancer Controller.
• GCP, Azure, on-prem: same shape, different controller.

1. An external DNS record: gateway.acme.com must resolve to the load balancer in front of the gateway.
2. A valid TLS certificate: either issued by ACM (NLB/ALB termination), cert-manager (nginx-ingress termination), or your own PEM.
3. A Service or Ingress resource that exposes the gateway pod on :443, the chart renders this from ingress.enabled (nginx posture) or from a Service-level override (NLB+ACM posture).

​

nginx-ingress + cert-manager (default chart)

ingress:
  enabled: true
  className: nginx
  host: gateway.acme.com
  path: /v1
  pathType: Prefix
  tls:
    enabled: true
    secretName: gateway-tls
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-buffering: "off"        # SSE needs unbuffered
    nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"    # long-lived streams
    nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"

# clusterissuer-letsencrypt.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - dns01:
          cloudflare:
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

​

AWS NLB + ACM + Cloudflare (recommended for AWS self-hosts)

(customer) ─DNS─▶ gateway.acme.com
                       ▼
             Cloudflare CNAME (DNS-only / gray cloud)
                       ▼
         abc123.elb.eu-central-1.amazonaws.com  ← AWS NLB hostname
                       │
                       ├─ TLS terminates here (ACM cert: gateway.acme.com)
                       ▼
            Service type=LoadBalancer (port 443 → 5563)
                       ▼
                    Gateway Pod :5563

1. Provision the ACM certificate (request before the LoadBalancer Service so the cert ARN is ready when the controller provisions the NLB):
   aws acm request-certificate \
     --domain-name gateway.acme.com \
     --validation-method DNS \
     --region <your-region>
   
   Then describe the cert to find the validation CNAME records you need to publish in Cloudflare:
   aws acm describe-certificate --certificate-arn <arn> \
     --query 'Certificate.DomainValidationOptions[].ResourceRecord' \
     --region <your-region>
   
   Returns a Name and Value per domain, add both as CNAME records in Cloudflare (proxy: DNS-only). ACM polls these continuously; status flips from PENDING_VALIDATION → ISSUED once DNS propagates. Typical: 1-5 min, worst-case ~30 min. ⚠️ No manual force-revalidate API. If you wait 30+ min and ACM is still pending, verify the CNAME with dig CNAME _<token>.gateway.acme.com +short, the answer must match the value in aws acm describe-certificate. If dig resolves correctly, ACM will pick it up shortly; if not, fix the Cloudflare record. To watch status:
   aws acm describe-certificate --certificate-arn <arn> \
     --query 'Certificate.Status' --region <your-region>
   
2. Configure the gateway Service as LoadBalancer with NLB + ACM annotations. This is a Service-level config (not the Ingress); set it via the chart’s Service block or apply directly:
   # values.yaml override (or templates/service.yaml customisation)
   gateway:
     service:
       type: LoadBalancer
       port: 443
       targetPort: 5563
       annotations:
         service.beta.kubernetes.io/aws-load-balancer-type: nlb
         service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
         service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:aws:acm:<region>:<account>:certificate/<id>
         service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "443"
         service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
   
   The AWS Load Balancer Controller sees these annotations and provisions an NLB with the ACM cert attached. Wait for the NLB hostname to appear:
   # Service name follows the chart's `gateway.fullname` template, by
   # default `<release>-gateway`. Replace `langwatch-gateway` below if
   # you installed under a different release name.
   kubectl get svc langwatch-gateway -n langwatch \
     -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
   
   Provisioning is typically 2-5 min after the cert flips ISSUED.
3. Add a CNAME in Cloudflare pointing gateway.acme.com → the NLB hostname. Proxy status: DNS-only (gray cloud, NOT orange). See the next section for why.
   Type:  CNAME
   Name:  gateway
   Target: abc123.elb.eu-central-1.amazonaws.com
   Proxy: DNS only (gray cloud)
   TTL:   Auto
   
   Do not use an A record, NLB underlying IPs rotate and any pinned IP will fail within days.
4. Verify end-to-end:
   dig +short gateway.acme.com           # → elb.<region>.amazonaws.com hostname
   openssl s_client -connect gateway.acme.com:443 -servername gateway.acme.com </dev/null 2>&1 | grep -E 'subject=|issuer='
   curl -sSf https://gateway.acme.com/healthz
   
   /healthz returns {"status":"ok","version":"..."} when the pod is live.

​

Cloudflare proxy mode is wrong for the gateway

1. Origin auth complexity. Cloudflare Full (Strict) does accept any publicly-trusted CA at the origin, so an ACM cert on the NLB will validate without a 526 by default, but turning the proxy on opts you into Cloudflare’s origin-auth surface. The day someone enables Authenticated Origin Pulls or expects mTLS between Cloudflare and the NLB, you’ll need to install Cloudflare-issued origin certs (or the Cloudflare CA bundle) on the NLB just to keep the gateway reachable. That’s operational fragility for zero benefit on a data plane Cloudflare shouldn’t be inspecting in the first place.
2. Latency. Cloudflare proxy adds ~10-30 ms per request even for cached zero-config setups, on top of the ~50 ms Cloudflare edge → AWS region path. The AI Gateway aims for sub-ms overhead beyond bifrost; halving that budget on every request defeats the design.
3. SSE, streaming buffering. Cloudflare’s proxy buffers small response chunks for compression and bot-detection scoring. SSE responses with single-token deltas (every OpenAI-shape stream) can stall by hundreds of milliseconds at the edge before being released to the client. The result: streaming feels broken even when it’s working.
4. Privacy. With Cloudflare proxying, the edge sees every customer’s plaintext LLM prompt and completion (TLS terminates at Cloudflare). For enterprise customers under data-residency or PII constraints, that’s a non-starter, the whole reason to run a gateway is to keep prompt content within your own boundary.

​

AWS ALB (alternative)

ingress:
  className: alb
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}]'
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/backend-protocol: HTTP
    # Use ACM instead of cert-manager when the ALB terminates TLS:
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:<region>:<account>:certificate/<id>

​

GCP, Azure, on-prem

• GCP GKE: ingress.className: gce + a ManagedCertificate resource. No per-request idle timeout limit, but response streaming is capped at 30 s unless you use a BackendConfig with timeoutSec: 3600.
• Azure AKS: ingress.className: webapprouting.kubernetes.azure.com, similar pattern, cert-manager compatible.
• On-prem bare metal: any ingress controller that supports Kubernetes’ standard Ingress resource works. nginx-ingress, Traefik, HAProxy all tested.

​

Why a separate hostname instead of a path prefix?

1. No rewrite middleware. A path prefix would require nginx.ingress.kubernetes.io/rewrite-target to strip /gateway before forwarding, doable, but it breaks OpenAI SDK compatibility because SDKs build request URLs by joining base_url + "/v1/chat/completions" and base_url = "https://app.acme.com/gateway" makes SDKs double-path.
2. Observability and rate limits isolate cleanly. You can pin Grafana dashboards and WAF rules to gateway.* without chasing path prefixes.
3. TLS cert lifetimes don’t cross-contaminate. A bad deploy on the app side doesn’t invalidate the gateway’s cert.

​

Cloudflare-specific notes

• Proxy mode (orange cloud) is wrong for the AI Gateway. See Cloudflare proxy mode is wrong for the gateway. Always use DNS-only (gray cloud).
• Bot Fight Mode can false-positive on Claude Code and OpenAI SDK traffic patterns (steady POST cadence from a single IP). Even in DNS-only mode, BFM can fire on the zone level, disable on the gateway hostname or the whole zone if the gateway lives in a dedicated zone.
• Cache rules: gateway responses should not be cached. The default “respect origin headers” is correct, the gateway sets Cache-Control: no-store on every response. Don’t add a Page Rule that forces caching, and don’t enable Cloudflare’s “Cache Everything” preset.
• Universal SSL stays on automatically when the record is in the zone, but is unused in DNS-only mode (TLS terminates at the origin’s cert, not Cloudflare’s). Leave it on, it has no effect when the record is gray-cloud.

​

Verifying end-to-end

# DNS resolves to the right load balancer
dig +short gateway.acme.com

# TLS is the cert you expect (ACM ARN matches `Subject:` line)
openssl s_client -connect gateway.acme.com:443 -servername gateway.acme.com </dev/null 2>&1 \
  | grep -E 'subject=|issuer='

# Health
curl -sSf https://gateway.acme.com/healthz

# Auth echo (requires a VK from your langwatch control plane)
curl -sSf https://gateway.acme.com/v1/models \
  -H "Authorization: Bearer vk-lw-01JXXX..."

# Full completion (tests auth + routing + upstream + observability pipeline)
curl -sSf https://gateway.acme.com/v1/chat/completions \
  -H "Authorization: Bearer vk-lw-01JXXX..." \
  -H "Content-Type: application/json" \
  -d '{"model":"openai/gpt-5-mini","messages":[{"role":"user","content":"ping"}]}'