Monitoring AI Infrastructure with Grafana

Running local AI infrastructure without observability is like flying a drone with a blindfold and a vibe check. You know the GPU is somewhere under the hood, you know the models are loading, but you can’t see latency drift, memory pressure, or token throughput degradation until something catches fire. We run a fully self-hosted AI stack on ARM64 — Ollama, vLLM, LiteLLM, Qdrant, Langfuse, and a dozen other services — and the only reason it stays upright is because we instrumented it from day one. This post walks through the actual monitoring stack we use in production: Prometheus for metrics collection, Grafana for visualization, and the specific exporters and dashboards that give us visibility into model performance, container health, and hardware utilization. If you’re building a sovereign AI stack or self-hosting on ARM64, this is the observability layer you need.

Why Observability Matters for Local AI

Cloud providers give you dashboards for free. When you self-host, you build your own. That’s not a downside — it’s the point. Your monitoring stack runs on your hardware, stores metrics in your database, and never ships telemetry to a third party. For organizations operating under EU AI Act compliance frameworks or data residency requirements, this isn’t optional.

But the practical reason is simpler: AI infrastructure is noisy. Models consume GPU memory unpredictably. Inference latency drifts under load. Container health fluctuates. Disk I/O from vector databases spikes during reindexing. Without real-time metrics, you’re debugging blind — and in a stack with 50+ containers, “blind” means spending hours correlating logs across services to find the one that’s OOM-killing.

A proper observability stack answers three questions:

1. Is the system healthy right now? (Alerting)
2. What changed? (Historical metrics)
3. Where is the bottleneck? (Cross-correlation between layers)

The Stack: Prometheus + Grafana + Exporters

Our monitoring architecture follows the standard Prometheus model, adapted for AI workloads:

┌─────────────────────────────────────────────────┐
│                    Grafana                       │
│         (Dashboards + Alerting UI)               │
└──────────────────┬──────────────────────────────┘
                   │ queries
┌──────────────────▼──────────────────────────────┐
│                 Prometheus                       │
│        (Time-series DB + Scraping)               │
└──┬──────┬──────┬──────┬──────┬─────────────────┘
   │      │      │      │      │
   ▼      ▼      ▼      ▼      ▼
 Node   cAd-   GPU  LiteLLM  Langfuse
Export  visor  Exp.  /metrics  (traces)

Each exporter exposes metrics in Prometheus format at a known endpoint. Prometheus scrapes them every 15 seconds, stores them as time-series data, and Grafana queries that data to render dashboards. The whole stack runs in Docker, managed via docker-compose, and takes about 1.5 GB of RAM total.

Node Exporter — System-Level Metrics

Node Exporter is the baseline. It collects CPU usage, memory consumption, disk I/O, network traffic, and filesystem statistics from the host. Our node-exporter-full Grafana dashboard gives us the system view: are we out of RAM? Is the SSD wearing out? Is network bandwidth saturated?

On ARM64, Node Exporter runs natively as an aarch64 binary — no emulation overhead. It’s a single container that exposes :9100/metrics.

cAdvisor — Container-Level Metrics

cAdvisor provides per-container resource utilization: CPU, memory, network, and filesystem usage for every Docker container. When you’re running 50+ containers, this is how you find the one eating 144% CPU (yes, that happened to us with cAdvisor itself — more on that later).

One lesson from production: cAdvisor’s default configuration is aggressive. On a system with many containers, it can consume more CPU than the workloads it’s monitoring. We fixed this by disabling disk metrics (--disable_metrics=disk.*) and increasing the housekeeping interval to 30 seconds:

# docker-compose.yml (monitoring service)
cadvisor:
  image: gcr.io/cadvisor/cadvisor:latest
  command:
    - --housekeeping_interval=30s
    - --disable_metrics=disk.*,tcp
  ports:
    - "8080:8080"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
    - /sys:/sys:ro
    - /var/lib/docker/:/var/lib/docker:ro

After this change, cAdvisor dropped from 144% CPU to 0.3%. That’s not a typo. The default config was spending nearly two full cores scraping disk stats for 50+ containers every second. Tune your scrapers, folks.

NVIDIA GPU Exporter — Hardware Acceleration Metrics

For ARM64 systems with NVIDIA GPUs (Jetson Orin, DGX Spark), the NVIDIA GPU Exporter exposes GPU temperature, memory usage, utilization, power draw, and fan speed via nvidia-smi. On our system, it runs on port 9835 and feeds into the nvidia-gpu Grafana dashboard.

This is critical for AI workloads. When vLLM loads a model into GPU memory, you want to see the VRAM allocation spike and stabilize. When Ollama evicts a model to free memory, you want to see the drop. Without GPU metrics, you’re guessing at whether your models are actually loaded or paged out to system RAM.

LiteLLM — AI Model Performance Metrics

This is where standard monitoring stacks stop and AI-specific observability begins. LiteLLM is our model router — every LLM call in the stack goes through it. LiteLLM natively supports Prometheus metrics by adding a single line to its config:

litellm_settings:
  callbacks: ["prometheus"]
  require_auth_for_metrics_endpoint: false  # let Prometheus scrape without auth

LiteLLM exposes metrics at /metrics on port 4000. Prometheus scrapes it every 15 seconds. The metrics include:

• litellm_deployment_total_requests — requests per model
• litellm_deployment_failed_requests_total — failures by model
• litellm_request_latency_seconds_seconds{quantile="0.95"} — p95 latency
• litellm_total_tokens — token consumption rate
• litithm_deployment_in_flight_requests — concurrent in-flight requests

We add a scrape target in prometheus.yml:

scrape_configs:
  - job_name: litellm
    static_configs:
      - targets: ['litellm:4000']
    metrics_path: /metrics

That’s it. LiteLLM handles the rest. Every model call — whether it goes to Ollama, vLLM, or a cloud fallback — is automatically instrumented with model name, token count, latency, and status code.

The Dashboards

We run four Grafana dashboards. Here’s what each one shows and why it matters.

1. Node Exporter Full — System Health

The standard Node Exporter Full dashboard covers:

• CPU utilization per core (16 panels)
• Memory usage (used, buffered, cached, free)
• Disk I/O per device (read/write rates, queue depth)
• Network traffic per interface
• Filesystem usage and inodes

Use case: “Is the host healthy?” This catches RAM exhaustion, disk failures, and network saturation before they cascade into service outages.

2. cAdvisor Dashboard — Container Health

The cAdvisor dashboard shows per-container:

• CPU usage (percentage and cores)
• Memory usage (RSS, cache, total)
• Network I/O (rx/tx bytes per second)
• Filesystem reads/writes

Use case: “Which container is misbehaving?” When something starts eating resources, this dashboard pinpoints it in seconds. We caught the cAdvisor CPU issue here first — the panel showed a single container at 144% while everything else was under 5%.

3. NVIDIA GPU Metrics — Accelerator Health

Custom dashboard tracking:

• GPU temperature and thermal throttling events
• VRAM usage (allocated vs. total)
• GPU utilization (compute, memory bandwidth)
• Power draw (watts) against TDP limit
• Process-level GPU memory breakdown

Use case: “Is the GPU actually being used?” Idle GPUs waste power. Saturated GPUs cause latency spikes. This dashboard tells you which state you’re in and whether you need to rebalance model loading.

4. AI Model Performance — LLM Observability

This is the dashboard we built ourselves. 14 panels covering the full LLM request lifecycle:

Use case: “Are the models performing?” This is the dashboard that justifies the infrastructure spend. When latency p95 drifts from 200ms to 2s, you see it here first. When a model starts returning 500s, the failure panel turns red before users complain.

Alerting

Dashboards are reactive — you have to look at them. Alerts are proactive — they ping you. Grafana supports alert rules natively, and we configure critical alerts for:

• GPU temperature > 85°C → Thermal throttling imminent
• Container memory > 90% of limit → OOM-kill incoming
• LLM failure rate > 5% → Models are breaking
• p95 latency > 3s → Users are waiting too long
• Disk usage > 80% → Time to clean up

Alerts route through Grafana’s notification channels. We use Discord webhooks for non-critical alerts and direct pings for critical ones. The key principle: alert on symptoms, not causes. “GPU at 90°C” is a symptom. “vLLM process using 28GB VRAM” is a cause. Alert on the former, investigate the latter.

Langfuse — The Missing Layer

Prometheus and Grafana cover metrics. But they don’t tell you what the models said or why a request failed. That’s where Langfuse comes in. It’s an open-source LLM observability platform that traces every model call at the application layer.

Langfuse runs alongside Grafana in our stack (port 3001) and integrates with LiteLLM via the success_callback: ["langfuse"] config setting. Every LLM call gets:

• Full prompt and completion text
• Token counts (prompt + completion)
• Model name and parameters
• Latency breakdown (TTFT, total)
• Cost estimate
• User/session tracking

When a user reports “the model gave a bad answer,” Langfuse lets you find the exact request, see the prompt, see the response, and trace the entire conversation chain. Prometheus tells you that latency spiked. Langfuse tells you why.

Practical Setup Guide

If you’re starting from scratch, here’s the minimum viable monitoring stack for a local AI deployment:

# docker-compose.monitoring.yml
version: '3.8'

services:
  prometheus:
    image: prom/prometheus:latest
    ports: ['9090:9090']
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana:latest
    ports: ['3001:3000']
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/dashboards:/etc/grafana/provisioning/dashboards
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=${GRAFANA_ADMIN_PASSWORD}

  node-exporter:
    image: prom/node-exporter:latest
    ports: ['9100:9100']

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    command: ['--housekeeping_interval=30s', '--disable_metrics=disk.*,tcp']
    ports: ['8080:8080']
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro

  nvidia-gpu-exporter:
    image: utkuozdemir/nvidia-gpu-exporter:latest
    ports: ['9835:9835']
    runtime: nvidia
    environment:
      - NVIDIA_VISIBLE_DEVICES=all

volumes:
  prometheus-data:
  grafana-data:

Add LiteLLM as a scrape target (as shown above), import the dashboards from Grafana’s dashboard library or build your own, and you have production-grade observability in about 30 minutes of setup time.

Lessons from Production

After running this stack for two months on our DGX Spark (ARM64, 51+ containers), here’s what we learned:

1. Tune cAdvisor immediately. The default config is a CPU hog on container-dense systems. Disable disk metrics and increase the housekeeping interval. Your CPU graph will thank you.

2. LiteLLM metrics are the highest-signal instrument in the stack. System metrics tell you the machine is healthy. LiteLLM metrics tell you the AI is healthy. Both matter, but only one tells you whether users are getting good responses.

3. Alert on p95 latency, not average. Averages hide tail latency. If 95% of requests complete in 50ms but 5% take 5 seconds, your average looks fine and your users are furious.

4. Track token consumption over time. Tokens are the currency of LLM infrastructure. If a model suddenly starts consuming 3x tokens for the same workload, something changed — maybe a prompt template, maybe a model update. Token rate panels catch this early.

5. Use Langfuse for debugging, Grafana for operations. They’re complementary, not redundant. Grafana answers “is the system fast?” Langfuse answers “is the model smart?”

6. Disk space monitoring is not optional. Prometheus data accumulates. Langfuse’s ClickHouse backend accumulates. Container logs accumulate. Without disk alerts, you’ll hit 100% and everything breaks at once.

Conclusion

Observability for self-hosted AI isn’t a luxury — it’s the difference between a stack you trust and a stack you fear. The tools are all open source, they run natively on ARM64, and the setup time is measured in hours, not weeks. Start with Node Exporter and Grafana. Add cAdvisor when your container count exceeds ten. Add LiteLLM metrics when you route your first LLM call. Add Langfuse when you need to debug model behavior. Layer by layer, you build a picture of the system that no cloud provider’s dashboard can match — because it’s yours, running on your hardware, observing your workloads.

For the full architecture behind this monitoring stack, check out our guides on building a local AI stack, self-hosting on ARM64, and what sovereign AI means for European businesses.