Tools

Tools: Skip the Cortex tenant headers - Deploy multi-tenant Prometheus in 5 minutes (2026)

2026-04-16 0 views admin
Tools: Skip the Cortex tenant headers - Deploy multi-tenant Prometheus in 5 minutes (2026)

The tenant header problem

Prerequisites

Setup: Complete multi-tenant stack

Configuration: OIDC integration

Step 1: Configure OIDC provider

Step 2: Configure OpenLBAC OIDC

Testing: Query rewriting in action

Step 1: Verify policy enforcement

Step 2: Observe automatic query rewriting

Step 3: Test with existing tools

Optimization: Production deployment

Horizontal scaling

Monitoring and alerting

Audit and compliance

Troubleshooting common issues

Issue: JWT token validation fails

Issue: Queries return empty results

Issue: High query latency

Why this approach works better If you've tried Cortex or Mimir for multi-tenant Prometheus, you've hit the same wall: every client needs tenant headers. Your existing Grafana dashboards break. CLI tools need updates. API integrations require modification. Your proof-of-concept becomes a migration project. There's a better approach: query-level access control with zero client changes. Current multi-tenant solutions require the X-Scope-OrgID header on every request: This breaks existing infrastructure and creates security vulnerabilities - headers can be spoofed, misconfigured, or simply forgotten. Before deploying OpenLBAC, ensure you have: Clone and deploy the full OpenLBAC stack with example Prometheus and Keycloak: This starts four OpenLBAC components: For Keycloak (included in docker-compose): Update the OIDC configuration in config/oidc.yml: Test different user contexts to see query rewriting: Rewritten for backend-team: Rewritten for frontend-team: Point your existing tools to OpenLBAC instead of Prometheus directly: Your existing dashboards work unchanged. CLI tools work unchanged. The only change is the endpoint URL. Scale proxy instances for high throughput: Nginx configuration for load balancing: Monitor OpenLBAC components: Key metrics to monitor: Enable audit logging for compliance frameworks: Each query execution logs: Symptoms: 401 Unauthorized responses from lbac-proxy Solution: Verify OIDC configuration and token format: Symptoms: Valid queries return no data after policy application Solution: Check policy rule logic and label presence: Symptoms: Increased response times after OpenLBAC deployment Solution: Optimize policy complexity and caching: OpenLBAC adds <5ms latency in most configurations. Higher latencies indicate policy complexity issues or network bottlenecks. Zero client modification: Existing tools continue working unchanged. No headers to manage or spoof. Real-time enforcement: Policies apply at query execution, not just dashboard level. API access and CLI tools automatically controlled. Enterprise IdP integration: Leverage existing Keycloak, Okta, or Auth0 groups. No custom authentication development required. Comprehensive audit: Every query decision logged for PCI DSS, GDPR, and SOC 2 compliance requirements. Five-minute deployment: Docker Compose to production-ready in minutes, not weeks of client migration. The tenant header approach assumes you can modify every client. The query rewriting approach assumes you can't - and works anyway. What's your experience with multi-tenant observability? Have you hit the tenant header wall, or found other approaches that preserve existing integrations? Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse

Command

Copy

# Every client needs modification -weight: 500;">curl -H "X-Scope-OrgID: team-alpha" http://prometheus:9090/api/v1/query?query=up # Grafana datasources need tenant configuration # CLI tools need wrapper scripts # API clients need header injection # Every client needs modification -weight: 500;">curl -H "X-Scope-OrgID: team-alpha" http://prometheus:9090/api/v1/query?query=up # Grafana datasources need tenant configuration # CLI tools need wrapper scripts # API clients need header injection # Every client needs modification -weight: 500;">curl -H "X-Scope-OrgID: team-alpha" http://prometheus:9090/api/v1/query?query=up # Grafana datasources need tenant configuration # CLI tools need wrapper scripts # API clients need header injection -weight: 500;">git clone https://github.com/openlbac/openlbac cd openlbac # Deploy complete stack: OpenLBAC + Prometheus + Keycloak -weight: 500;">docker-compose up -d -weight: 500;">git clone https://github.com/openlbac/openlbac cd openlbac # Deploy complete stack: OpenLBAC + Prometheus + Keycloak -weight: 500;">docker-compose up -d -weight: 500;">git clone https://github.com/openlbac/openlbac cd openlbac # Deploy complete stack: OpenLBAC + Prometheus + Keycloak -weight: 500;">docker-compose up -d # -weight: 500;">docker-compose.yml (relevant sections) services: lbac-server: image: openlbac/lbac-server:latest ports: - "8090:8090" environment: - DATABASE_URL=postgresql://postgres:password@postgres:5432/openlbac lbac-proxy: image: openlbac/lbac-proxy:latest ports: - "8080:8080" environment: - LBAC_CORE_URL=http://lbac-core:9090 - UPSTREAM_PROMETHEUS_URL=http://prometheus:9090 lbac-core: image: openlbac/lbac-core:latest ports: - "9090:9090" environment: - LBAC_SERVER_URL=http://lbac-server:8090 prometheus: image: prom/prometheus:latest ports: - "9091:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml # -weight: 500;">docker-compose.yml (relevant sections) services: lbac-server: image: openlbac/lbac-server:latest ports: - "8090:8090" environment: - DATABASE_URL=postgresql://postgres:password@postgres:5432/openlbac lbac-proxy: image: openlbac/lbac-proxy:latest ports: - "8080:8080" environment: - LBAC_CORE_URL=http://lbac-core:9090 - UPSTREAM_PROMETHEUS_URL=http://prometheus:9090 lbac-core: image: openlbac/lbac-core:latest ports: - "9090:9090" environment: - LBAC_SERVER_URL=http://lbac-server:8090 prometheus: image: prom/prometheus:latest ports: - "9091:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml # -weight: 500;">docker-compose.yml (relevant sections) services: lbac-server: image: openlbac/lbac-server:latest ports: - "8090:8090" environment: - DATABASE_URL=postgresql://postgres:password@postgres:5432/openlbac lbac-proxy: image: openlbac/lbac-proxy:latest ports: - "8080:8080" environment: - LBAC_CORE_URL=http://lbac-core:9090 - UPSTREAM_PROMETHEUS_URL=http://prometheus:9090 lbac-core: image: openlbac/lbac-core:latest ports: - "9090:9090" environment: - LBAC_SERVER_URL=http://lbac-server:8090 prometheus: image: prom/prometheus:latest ports: - "9091:9090" volumes: - ./prometheus.yml:/etc/prometheus/prometheus.yml # Access Keycloak admin console open http://localhost:8081 # Default credentials: admin/admin # Create realm: "observability" # Create groups: "platform-team", "backend-team", "frontend-team" # Create users and assign group memberships # Access Keycloak admin console open http://localhost:8081 # Default credentials: admin/admin # Create realm: "observability" # Create groups: "platform-team", "backend-team", "frontend-team" # Create users and assign group memberships # Access Keycloak admin console open http://localhost:8081 # Default credentials: admin/admin # Create realm: "observability" # Create groups: "platform-team", "backend-team", "frontend-team" # Create users and assign group memberships oidc: provider_url: "http://localhost:8081/realms/observability" client_id: "openlbac-client" client_secret: "your-client-secret" scopes: ["openid", "profile", "groups"] groups_claim: "groups" policies: - name: "platform-team" groups: ["platform-team"] rules: - label: "namespace" operator: "=~" values: ["production|staging|development"] - name: "backend-team" groups: ["backend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=~" values: ["api|database|cache"] - name: "frontend-team" groups: ["frontend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=" values: ["web"] oidc: provider_url: "http://localhost:8081/realms/observability" client_id: "openlbac-client" client_secret: "your-client-secret" scopes: ["openid", "profile", "groups"] groups_claim: "groups" policies: - name: "platform-team" groups: ["platform-team"] rules: - label: "namespace" operator: "=~" values: ["production|staging|development"] - name: "backend-team" groups: ["backend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=~" values: ["api|database|cache"] - name: "frontend-team" groups: ["frontend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=" values: ["web"] oidc: provider_url: "http://localhost:8081/realms/observability" client_id: "openlbac-client" client_secret: "your-client-secret" scopes: ["openid", "profile", "groups"] groups_claim: "groups" policies: - name: "platform-team" groups: ["platform-team"] rules: - label: "namespace" operator: "=~" values: ["production|staging|development"] - name: "backend-team" groups: ["backend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=~" values: ["api|database|cache"] - name: "frontend-team" groups: ["frontend-team"] rules: - label: "namespace" operator: "=" values: ["production"] - label: "-weight: 500;">service" operator: "=" values: ["web"] # Platform team member (full access) -weight: 500;">curl -H "Authorization: Bearer $PLATFORM_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Backend team member (-weight: 500;">service-filtered) -weight: 500;">curl -H "Authorization: Bearer $BACKEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Frontend team member (web -weight: 500;">service only) -weight: 500;">curl -H "Authorization: Bearer $FRONTEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Platform team member (full access) -weight: 500;">curl -H "Authorization: Bearer $PLATFORM_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Backend team member (-weight: 500;">service-filtered) -weight: 500;">curl -H "Authorization: Bearer $BACKEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Frontend team member (web -weight: 500;">service only) -weight: 500;">curl -H "Authorization: Bearer $FRONTEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Platform team member (full access) -weight: 500;">curl -H "Authorization: Bearer $PLATFORM_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Backend team member (-weight: 500;">service-filtered) -weight: 500;">curl -H "Authorization: Bearer $BACKEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" # Frontend team member (web -weight: 500;">service only) -weight: 500;">curl -H "Authorization: Bearer $FRONTEND_TOKEN" \ "http://localhost:8080/api/v1/query?query=up" rate(http_requests_total{-weight: 500;">status="200"}[5m]) rate(http_requests_total{-weight: 500;">status="200"}[5m]) rate(http_requests_total{-weight: 500;">status="200"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service=~"api|database|cache"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service=~"api|database|cache"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service=~"api|database|cache"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service="web"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service="web"}[5m]) rate(http_requests_total{-weight: 500;">status="200", namespace="production", -weight: 500;">service="web"}[5m]) # Grafana datasource configuration apiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources data: prometheus.yml: | apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://lbac-proxy:8080 # Changed from prometheus:9090 access: proxy jsonData: httpHeaderName1: "Authorization" secureJsonData: httpHeaderValue1: "Bearer $GRAFANA_OIDC_TOKEN" # Grafana datasource configuration apiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources data: prometheus.yml: | apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://lbac-proxy:8080 # Changed from prometheus:9090 access: proxy jsonData: httpHeaderName1: "Authorization" secureJsonData: httpHeaderValue1: "Bearer $GRAFANA_OIDC_TOKEN" # Grafana datasource configuration apiVersion: v1 kind: ConfigMap metadata: name: grafana-datasources data: prometheus.yml: | apiVersion: 1 datasources: - name: Prometheus type: prometheus url: http://lbac-proxy:8080 # Changed from prometheus:9090 access: proxy jsonData: httpHeaderName1: "Authorization" secureJsonData: httpHeaderValue1: "Bearer $GRAFANA_OIDC_TOKEN" # -weight: 500;">docker-compose.override.yml services: lbac-proxy: deploy: replicas: 3 nginx: image: nginx:alpine ports: - "8080:80" volumes: - ./nginx.conf:/etc/nginx/nginx.conf # -weight: 500;">docker-compose.override.yml services: lbac-proxy: deploy: replicas: 3 nginx: image: nginx:alpine ports: - "8080:80" volumes: - ./nginx.conf:/etc/nginx/nginx.conf # -weight: 500;">docker-compose.override.yml services: lbac-proxy: deploy: replicas: 3 nginx: image: nginx:alpine ports: - "8080:80" volumes: - ./nginx.conf:/etc/nginx/nginx.conf upstream lbac_proxies { server lbac-proxy-1:8080; server lbac-proxy-2:8080; server lbac-proxy-3:8080; } server { listen 80; location / { proxy_pass http://lbac_proxies; proxy_set_header Authorization $http_authorization; } } upstream lbac_proxies { server lbac-proxy-1:8080; server lbac-proxy-2:8080; server lbac-proxy-3:8080; } server { listen 80; location / { proxy_pass http://lbac_proxies; proxy_set_header Authorization $http_authorization; } } upstream lbac_proxies { server lbac-proxy-1:8080; server lbac-proxy-2:8080; server lbac-proxy-3:8080; } server { listen 80; location / { proxy_pass http://lbac_proxies; proxy_set_header Authorization $http_authorization; } } # prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: 'lbac-proxy' static_configs: - targets: ['lbac-proxy:8080'] metrics_path: '/metrics' - job_name: 'lbac-core' static_configs: - targets: ['lbac-core:9090'] metrics_path: '/metrics' # prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: 'lbac-proxy' static_configs: - targets: ['lbac-proxy:8080'] metrics_path: '/metrics' - job_name: 'lbac-core' static_configs: - targets: ['lbac-core:9090'] metrics_path: '/metrics' # prometheus.yml global: scrape_interval: 15s scrape_configs: - job_name: 'lbac-proxy' static_configs: - targets: ['lbac-proxy:8080'] metrics_path: '/metrics' - job_name: 'lbac-core' static_configs: - targets: ['lbac-core:9090'] metrics_path: '/metrics' # config/audit.yml audit: enabled: true sink_type: "elasticsearch" elasticsearch: endpoint: "http://elasticsearch:9200" index: "openlbac-audit" events: - "query_execution" - "policy_violation" - "authentication_failure" - "authorization_decision" # config/audit.yml audit: enabled: true sink_type: "elasticsearch" elasticsearch: endpoint: "http://elasticsearch:9200" index: "openlbac-audit" events: - "query_execution" - "policy_violation" - "authentication_failure" - "authorization_decision" # config/audit.yml audit: enabled: true sink_type: "elasticsearch" elasticsearch: endpoint: "http://elasticsearch:9200" index: "openlbac-audit" events: - "query_execution" - "policy_violation" - "authentication_failure" - "authorization_decision" # Check token structure echo $TOKEN | base64 -d | jq . # Verify groups claim exists # Ensure issuer matches provider_url # Check client_id in token audience # Check token structure echo $TOKEN | base64 -d | jq . # Verify groups claim exists # Ensure issuer matches provider_url # Check client_id in token audience # Check token structure echo $TOKEN | base64 -d | jq . # Verify groups claim exists # Ensure issuer matches provider_url # Check client_id in token audience # Verify labels exist in your metrics -weight: 500;">curl http://prometheus:9090/api/v1/labels # Test policy rules manually -weight: 500;">curl "http://localhost:8090/api/v1/policies/test" \ -d '{"query": "up", "user_groups": ["backend-team"]}' # Verify labels exist in your metrics -weight: 500;">curl http://prometheus:9090/api/v1/labels # Test policy rules manually -weight: 500;">curl "http://localhost:8090/api/v1/policies/test" \ -d '{"query": "up", "user_groups": ["backend-team"]}' # Verify labels exist in your metrics -weight: 500;">curl http://prometheus:9090/api/v1/labels # Test policy rules manually -weight: 500;">curl "http://localhost:8090/api/v1/policies/test" \ -d '{"query": "up", "user_groups": ["backend-team"]}' # config/performance.yml cache: policy_cache_ttl: 300s query_cache_ttl: 60s optimization: max_rule_complexity: 10 parallel_policy_evaluation: true # config/performance.yml cache: policy_cache_ttl: 300s query_cache_ttl: 60s optimization: max_rule_complexity: 10 parallel_policy_evaluation: true # config/performance.yml cache: policy_cache_ttl: 300s query_cache_ttl: 60s optimization: max_rule_complexity: 10 parallel_policy_evaluation: true - Docker and Docker Compose installed - An existing Prometheus instance (or willingness to deploy one) - An OIDC provider with group claims (Keycloak, Okta, Auth0) - Basic understanding of PromQL and observability concepts - lbac_proxy_query_duration_seconds - Query rewriting latency - lbac_proxy_queries_total - Request rate and error rate - lbac_core_policy_updates_total - Policy propagation health - lbac_core_connected_proxies - Proxy connectivity -weight: 500;">status - User identity and group membership - Original query and rewritten query - Data sources accessed - Timestamp and request metadata

🏷️ Tags

toolsutilitiessecurity toolscortextenantheadersdeploymultiprometheusminutes

More from Tools

Tools: Infrastructure as Code with AWS CloudFormation From Fundamentals to Production Patterns (2026)

Tools: Infrastructure as Code with AWS CloudFormation From Fundamentals to Production Patterns (2026)

2026-04-16 0
Tools: Load Balancing with Nginx: Practical Examples

Tools: Load Balancing with Nginx: Practical Examples

2026-04-16 0
Tools: Building a Self-Hosted Bot Management Platform with Docker, FastAPI and React (2026)

Tools: Building a Self-Hosted Bot Management Platform with Docker, FastAPI and React (2026)

2026-04-16 0
Tools: Linux 7.0 Kernel Deep Dive + Step-by-Step Installation Guide

Tools: Linux 7.0 Kernel Deep Dive + Step-by-Step Installation Guide

2026-04-16 0

Trending

1

CVE-2025-61481: Critical Remote Code Execution Vulnerability in MikroTik RouterOS & SwitchOS

2025-10-27 • 189 views
2

CVE-2025-43939: Dell Unity OS Command Injection (High)

2025-10-30 • 148 views
3

Google disputes false claims of massive Gmail data breach

2025-10-30 • 130 views
4

Microsoft: DNS outage impacts Azure and Microsoft 365 services

2025-10-30 • 88 views
5

3.5B Accounts, 1 Critical Flaw: Meta Closes WhatsApp Data-Harvesting

2025-11-25 • 81 views