Case Study

Command Intelligence for Cloud Infrastructure

A unified engineering command center that gives a cloud AI agency real-time visibility across every production service — from auth to AI inference — with sub-minute incident detection and zero framework overhead.

15+

Services Monitored

<1 min

Incident Detection

Palette Commands

Framework Overhead

Project

Internal Ops Platform

Type

Engineering Command Center

Stack

Vanilla JS · Keycloak 26 · FastAPI · Gemini

Deploy

Coolify · Docker · Traefik SSL

Built by

Del Rey Valmoria

// 01 — The Problem

Flying Blind Across 15+ Production Services

A cloud AI agency running 15+ production services had no unified visibility. Keycloak, NocoDB, n8n, MinIO, and a FastAPI bridge each had their own dashboards, logs, and alert paths — or none at all. The ops team was checking services individually, on rotation, by memory.

The mean time to detect an incident was over 10 minutes. In most cases, a client reported the issue first. There was no single pane of glass that could answer the most basic question: is everything running right now?

The team also had no quick-access layer for operational tasks. Routine actions — checking deployment status, pulling a service log, triggering a workflow — required navigating four or five separate admin UIs. There was no AI assistant wired into internal context, no command launcher, no alert routing to Discord.

Every escalation path was manual. Every status check was a tab-switch. The tooling was not keeping pace with the complexity of what it was supposed to operate.

Three root causes: no single visibility layer, no sub-minute detection, and no operational shortcut system. The fix had to unify all three without adding a React bundle to an internal tool.

// 02 — The Architecture

One Hub. Every Service.

The hub is a Vanilla JS SPA served by nginx behind Traefik SSL. Auth flows through Keycloak 26 PKCE. All AI and data calls route through a FastAPI bridge that enforces JWT isolation, circuit breaking, and rate limiting before touching backend services.

Vanilla JS Keycloak 26 PKCE S256 PostgreSQL 16 FastAPI Gemini 2.5 Flash Redis AES-256-GCM NocoDB n8n v2.10 MinIO nginx Traefik SSL Coolify v4 Prometheus

// 03 — How It Works

Five Layers, One Pane of Glass

Each layer of the hub was designed to solve a distinct operational problem. Together they form an always-on command surface that replaces manual tab-switching with a single authenticated dashboard.

Auth Layer — Keycloak 26 PKCE

// PKCE S256 · ag-internal realm · TOTP MFA · 8h session timeout

Every session begins with a full Keycloak PKCE authorization code flow. The hub exchanges the code for a JWT signed with RS256. All subsequent API calls to the FastAPI bridge carry this token in Authorization headers. TOTP multi-factor is enforced at the realm level — no bypass path. Postmark delivers OTP and password-reset emails. Sessions time out at 8 hours; the JS client silently refreshes via the iframe check-sso mechanism before expiry.

keycloak-js 26.2.3 · silent-check-sso.html · Postmark SMTP

Health Ribbon — 5-Pillar Status

// 30s refresh · Keycloak · NocoDB · n8n · MinIO · Bridge

A 28px status bar pinned to grid row 3 polls five critical services every 30 seconds. Each pillar shows a green/amber/red pill derived from the bridge health endpoint. A compound status normalizer (e.g. RUNNING:HEALTHY → running) prevents false P0 alerts from Coolify's compound status strings. Any red pill fires a Discord alert through the AG Observability error handler. Before this ribbon, mean detection time was 10+ minutes. After: under one minute.

normSt() · 30s poll · Discord webhook routing

Ctrl+K Command Palette — 18 Commands

// COMMAND · BUSINESS · ENGINEERING · INTELLIGENCE · TOOLS groups

A keyboard-first command launcher (Ctrl+K) exposes 18 operational commands across five nav groups: COMMAND, BUSINESS, ENGINEERING, INTELLIGENCE, and TOOLS. Commands include: open Coolify, trigger a workflow, view service logs, pull deployment status, open the AI rail, and navigate to any hub section. Fuzzy search filters results in real time. No mouse required for any common operation.

18 commands · fuzzy match · keyboard-first

AI Rail — Gemini 2.5 Flash

// 320px slide-in · JWT-isolated · bridgeHeaders() async

A 320px slide-in panel wires a Gemini 2.5 Flash assistant directly into the hub's operational context. The AI rail calls the FastAPI bridge, not Gemini directly — the bridge enforces JWT validation, rate limiting, and a circuit breaker before forwarding. Redis caches session context with AES-256-GCM encryption. The assistant can answer questions about service state, help draft runbook entries, and summarize recent incidents.

FastAPI bridge · Redis session cache · circuit breaker

Alert Routing — Dynamic Recipients

// NocoDB ag_notify_recipients · 5-min Redis cache · zero hardcoded emails

Notification recipients are stored in NocoDB, not in environment variables. The bridge fetches the active recipient list at startup and caches it in Redis with a 5-minute TTL. Updates take effect within one cache window — no redeploy required. A Ops Log card in the hub provides a CRUD interface for managing recipients without touching infrastructure. All critical and warning alerts route to tenant-specific Discord channels via the AG Observability error handler.

NocoDB CRUD · 5-min Redis TTL · Discord webhook routing

// 04 — Engineering Decisions

Why These Choices. Not Other Choices.

Every architectural decision was made against a concrete alternative. Here are the three that most defined the system's character.

Vanilla JS over React / Vue / Svelte

Verdict: zero bundle overhead, instant load, no build step

An internal ops tool is loaded dozens of times per day by a small team. Every second of load time is engineer-hours lost. A React SPA with a Vite build pipeline adds ~150-300 KB of JS before a single component renders. Vanilla JS with DOM APIs loads the entire hub in under 200ms on the internal network. No build step means instant iteration — edit a file, refresh, done. The trade-off (no component model, manual DOM management) is acceptable for a single-team internal tool with well-defined, stable UI sections.

Rejected: React + Vite

+150-300 KB bundle · build step required · npm dependency surface · hydration overhead

Chosen: Vanilla JS

Zero bundle · instant load · no build pipeline · direct DOM · zero dependencies

Self-hosted Keycloak 26 over Auth0 / Clerk

Verdict: full PKCE control, custom FreeMarker theme, zero per-MAU cost

Auth0 and Clerk both charge per monthly active user (MAU) — costs that scale with team size regardless of usage patterns. More critically, neither allows the level of token control, realm isolation, or custom login theme that a multi-tenant agency platform requires. Keycloak 26 runs on the same self-hosted infrastructure as all other services. It supports PKCE S256, TOTP MFA, per-realm role sets, and a fully custom FreeMarker login theme (ag-tenant) that matches brand identity. The operational cost is one PostgreSQL database and one Docker container.

Rejected: Auth0

Per-MAU pricing · limited realm isolation · no custom SMTP control · SaaS lock-in

Chosen: Keycloak 26

Zero MAU cost · full PKCE · TOTP MFA · custom theme · multi-realm · self-hosted

FastAPI Bridge over Direct Backend Calls

Verdict: JWT isolation, circuit breaking, and Prometheus metrics in one layer

The alternative was calling Gemini, NocoDB, and Coolify directly from the browser JS — exposing API keys in client-side code and bypassing any rate-limiting or circuit-breaking logic. The FastAPI bridge creates a single JWT-authenticated entry point. Every outbound call to a backend service is made server-side with service credentials that never reach the browser. The bridge also implements a circuit breaker for Gemini (preventing runaway inference spend during outages) and exposes a /metrics Prometheus endpoint for latency and error rate tracking.

Rejected: Direct browser calls

API keys exposed client-side · no rate limiting · no circuit breaker · no metrics

Chosen: FastAPI Bridge

JWT-isolated · circuit breaker · Redis session cache · Prometheus /metrics endpoint

// 05 — Proof

Results That Closed the Gap

The hub went from zero to production in a single sprint. The metrics below represent the delta between the pre-hub baseline and steady-state operation after the health ribbon, command palette, and AI rail were fully wired.

<1 min

Incident Detection

15+

Services Monitored

Palette Commands

0 KB

Framework Bundle

Dashboard Interface

Health Ribbon + Service Grid

The hub's primary view: a 28px health ribbon at the top with live service status pills, followed by a service card grid showing deployment state, version, and uptime per service.

CTO Intelligence Hub — live operations dashboard with health ribbon and service cards

LIVE — app.arellanoglobal.org

Command Palette

Ctrl+K — 18-Command Launcher

The command palette overlays the dashboard on Ctrl+K. Fuzzy search filters 18 commands across five groups in real time.

CTO Hub — Ctrl+K command palette overlay with 18 commands

CTRL+K COMMAND PALETTE — 18 COMMANDS

Operational Impact

Detection Time: 10 minutes → under 1 minute

The health ribbon polls five critical service endpoints every 30 seconds and routes any degraded status directly to Discord. Before the hub, incident detection relied on manual checks or client reports — averaging 10+ minutes. The ribbon cut that to under 60 seconds for any observable failure in the monitored stack.

Zero Tab-Switching for Common Operations

Coolify, n8n, NocoDB, service logs, and deployment status are all reachable via the Ctrl+K palette in under two keystrokes. The 18-command set covers 100% of the team's routine operational actions.

AI Context In the Same Window

The Gemini AI rail is wired into service state. The team asks questions in natural language and gets answers grounded in live infrastructure context, without context-switching to a separate chat tool.

No Framework. No Bundle. Instant Load.

The Vanilla JS architecture means zero build pipeline and under 200ms load time on the internal network. Any engineer can edit the dashboard and see changes on refresh — no npm install, no build step, no hot-reload config.

Dynamic Alerts. Zero Config Redeploys.

Notification recipients are managed through the hub UI itself, stored in NocoDB, and cached with a 5-minute Redis TTL. Adding or removing an alert recipient takes 10 seconds and requires no infrastructure change.