The setup we inherited

What we actually needed

What changed

The numbers

Trade-offs and Limitations

What I'd tell a peer team

Further Reading TL;DR: We had 11,247 lines of Python middleware handling per-tenant LLM cost attribution, rate limiting, and provider failover. Replaced about 60% of it with Bifrost's virtual keys and governance features. Some honest gaps remain, which is why this is a writeup and not a sales pitch. Nexus Labs runs enterprise agent automation. Each customer gets isolated workloads. Each workload makes between 200 and 50,000 LLM calls per day across OpenAI, Anthropic, Bedrock, and Vertex. When I joined, we had a Python middleware doing four things at once: API key rotation per provider, per-tenant rate limits in Redis, cost attribution via request tagging, and fallback logic when a provider returned 429s. 11,247 lines of Python. Three engineers had touched it. Two had left. One of them had encoded their team-internal pricing assumptions inline. Every model deprecation became a sprint. Three things, in priority order: I evaluated three gateways before picking one. Here is the comparison after running each through a 2-week eval against our actual traffic shape. LiteLLM was the real contender. Larger community, more battle-tested in production for some workload shapes. Where it lost for us: setting up hierarchical budgets across customer to team to workload tiers required more YAML wrangling than we wanted, and the failover behavior on streaming requests was less predictable under our tests. Portkey was strong on dashboards. We didn't want a hosted dependency for our cost control path. The piece that surprised me most was the virtual keys model. From the docs (governance/virtual-keys), every tenant gets a virtual key. The key carries the budget cap, rate limit, allowed providers, and allowed models. Our orchestrator stopped caring about provider routing entirely. Config that replaced 4,200 lines of Python: Our orchestrator now does one thing: pick a virtual key based on tenant. Send the request. Done. The latency number was the biggest surprise. Bifrost is Go. Our middleware was Python doing synchronous Redis calls. We knew that was a problem. Solving it wasn't on the roadmap. Migration was harder than the docs suggest. Our cost attribution data didn't map cleanly. We had legacy fields like team_internal_billing_code baked into every log. Mapping these to virtual key metadata took a full sprint, and the team still grumbles about it. Semantic caching is risky for our workload. Our agents call LLMs with tool results embedded in prompts. Two prompts that look 92% similar can require very different responses. We disabled semantic caching for the agent path. Enabled it only for our content generation path, where we saw a 31% hit rate. MCP gateway integration is newer than the rest. We use it for filesystem access from a customer-facing automation agent. Works fine. But debugging when a tool call fails requires more log digging than the rest of the platform. No native cost-anomaly alerting yet. Budget caps work. But "this customer's usage spiked 3x in 2 hours" is still wired up via Prometheus alerts and PagerDuty by hand. Portkey has this in their hosted product. If real-time anomaly alerts are your top requirement, weight that. If you have one provider and one customer, you don't need this. Use the provider's SDK. If you have 3+ providers, multiple customer tiers, and someone on your team has written class CostTrackingMiddleware more than once, evaluate. Spin up the Docker container (quickstart). Point staging traffic at it for a week. Look at the metrics. Decide. The model is the easy part. Cost attribution is the part that wakes you up at 2am when a customer's bill is wrong. Templates let you quickly answer FAQs or store snippets for re-use. Hide child comments as well For further actions, you may consider blocking this person and/or reporting abuse