HYPHAby S33D
Documentation › Integrations

AI and GPU: Ollama, vLLM, LiteLLM, TGI, llama.cpp and NVIDIA GPUs

Inference queues, latencies, tokens, KV cache, loaded models and GPU health, read-only by the agent, which never starts a generation.

At a glance

SoftwareCollectionActivationNamespace
Ollamalocal API (/api/version, /api/tags, /api/ps)auto on 127.0.0.1:11434ollama.*
vLLMOpenMetrics reader, vllm profile (port 8000)declare itvllm.*
LiteLLMOpenMetrics reader, litellm profile (port 4000)declare itlitellm.*
Text Generation InferenceOpenMetrics reader, tgi profile (port 8080)declare ittgi.*
llama.cpp (llama-server)OpenMetrics reader, llamacpp profile (port 8080)declare itllamacpp.*
NVIDIA GPUsnvidia-smioff by defaultgpu.*
NVIDIA GPUs (DCGM)OpenMetrics reader, dcgm profile (port 9400)declare itgpu.*

None of these integrations sends a generation request: the agent reads lists and metrics, it never loads a model nor incurs a cost.

Ollama

Once a minute the agent reads three API routes: the version, the installed models (the duration of this list is the response time) and the models loaded in memory.

integrations:
  ollama:
    url: http://127.0.0.1:11434     # default; any other address turns the integration on
    instances:                      # other Ollama servers, read from this agent
      - {name: gpu-2, url: http://192.0.2.12:11434}
MetricMeaning
ollama.models.installed, ollama.models.sizeinstalled models, total disk space
ollama.model.sizesize of each model (tags ollama_model, ollama_family, ollama_parameters, ollama_quantization)
ollama.models.loaded, ollama.loaded.size, ollama.loaded.vramloaded models and memory in use
ollama.model.loaded.size, .vram, .vram_pctper loaded model; below 100 %, part of the model runs on the CPU
ollama.model.loaded.expires_inseconds before unloading (absent with keep_alive: -1)
ollama.model.loaded.context_lengthcontext of the loaded model
ollama.response_timeduration of the model list, in seconds

Check ollama.can_connect: version, installed and loaded models.

Inference servers through the OpenMetrics reader

vLLM, LiteLLM, TGI and llama.cpp publish their metrics in the Prometheus format. A built-in profile sets the port, the families kept, their HYPHA names and the labels kept:

integrations:
  openmetrics:
    - profile: vllm                          # http://127.0.0.1:8000/metrics
    - profile: litellm
      bearer_token_file: /etc/hypha/litellm.key
    - profile: tgi
      url: http://127.0.0.1:3000/metrics
    - profile: llamacpp                      # llama-server started with --metrics

Each target publishes the openmetrics.can_connect check and carries the instance_name tag (the profile name, unless name is set). How the reader works (rates, histograms, series cap) is described in the configuration.

vLLM

Labels kept: model_name, engine, finished_reason. Names from earlier vLLM versions (gpu_cache_usage_perc, time_per_output_token_seconds) are read under the same HYPHA names.

LiteLLM

Prometheus metrics are part of LiteLLM's open-source edition. Turn the callback on in its config:

litellm_settings:
  callbacks: ["prometheus"]

Since version 1.85, /metrics requires a key: give it to the agent through bearer_token_file (a file only the agent can read), or serve the metrics on a separate port.

Labels kept: requested model, model, model group, provider, response code, error class. The user, their e-mail address, their IP address and the hash of their key are never kept.

Text Generation Inference

The profile follows the names of the TGI documentation. The Docker image listens on port 80 inside the container: set the url of the published port.

llama.cpp

Start llama-server with --metrics.

NVIDIA GPUs

Through nvidia-smi

Without an exporter, the agent reads nvidia-smi every 30 seconds. The integration is off by default:

integrations:
  nvidia_gpu:
    enabled: auto          # on when nvidia-smi is present; true to force it
    # path: /usr/bin/nvidia-smi
    # processes: false     # without the list of compute processes
MetricMeaning
gpu.countnumber of GPUs
gpu.utilization, gpu.memory.utilizationGPU and memory controller usage (%)
gpu.memory.total, .used, .free (bytes), gpu.memory.usage (%)GPU memory
gpu.temperature, gpu.memory.temperaturetemperatures (°C)
gpu.power.draw, gpu.power.limit (W), gpu.power.usage (%)power
gpu.clock.sm, gpu.clock.mem, gpu.clock.sm.maxclocks (MHz)
gpu.fan.speed, gpu.pstatefan (%), performance state (0 = P0)
gpu.ecc.corrected, gpu.ecc.uncorrectedECC errors since the driver was loaded
gpu.throttle.thermal1 when clocks are reduced because of heat
gpu.processes, gpu.process.memorycompute processes and memory per program (process_name)

Tags: gpu (index), gpu_uuid, gpu_name. Checks: gpu.can_query (nvidia-smi answers, driver loaded and matching its library) and gpu.health per GPU (critical: uncorrected ECC errors; warning: thermal slowdown).

Values the GPU does not provide ([N/A], unified memory of Grace and DGX Spark machines, no fan) are not sent. A field unknown to the driver is removed from the query and reading resumes.

Through the DCGM exporter

With dcgm-exporter (Kubernetes, compute clusters), declare the dcgm profile:

integrations:
  openmetrics:
    - profile: dcgm        # http://127.0.0.1:9400/metrics

Metrics take the same names as the nvidia-smi ones when the unit is the same (gpu.utilization, gpu.temperature, gpu.power.draw, gpu.clock.sm, gpu.memory.used in bytes since dcgm-exporter 4.9); the others keep their own name: gpu.fb.used_mib, gpu.fb.free_mib (MiB, up to 4.8), gpu.utilization_ratio (fraction, 4.9), gpu.xid_error, gpu.pcie.replays, gpu.remapped_rows.*, gpu.engine.active, gpu.tensor.active, gpu.dram.active. Tags: gpu, uuid, modelname, and pod, namespace, container on Kubernetes. Memory temperature is not kept: DCGM reports a sentinel value on GPUs without a sensor.

Pick nvidia-smi or DCGM on a given machine, not both.

Monitor templates

IA et GPU (AI and GPU) family of the catalog: Ollama unreachable or slow, inference server unreachable (by instance_name), vLLM or TGI inference queue, vLLM time to first token, vLLM KV cache saturated, requests deferred by llama.cpp, LiteLLM deployment down, GPU too hot, GPU saturated, GPU memory full, GPU faulty, nvidia-smi failing, GPU XID error.

Ready-made dashboards

Dashboards › Catalog, IA et GPU (AI and GPU) family: NVIDIA GPUs, Ollama, vLLM, LiteLLM, Text Generation Inference and llama.cpp.

Updated: 20 November 2026