zmlai/llmd repository overview

  ███████╗███╗   ███╗██╗             ██╗    ██╗     ██╗     ███╗   ███╗██████╗
  ╚══███╔╝████╗ ████║██║            ██╔╝    ██║     ██║     ████╗ ████║██╔══██╗
    ███╔╝ ██╔████╔██║██║           ██╔╝     ██║     ██║     ██╔████╔██║██║  ██║
   ███╔╝  ██║╚██╔╝██║██║  .ai     ██╔╝      ██║     ██║     ██║╚██╔╝██║██║  ██║
  ███████╗██║ ╚═╝ ██║███████╗    ██╔╝       ███████╗███████╗██║ ╚═╝ ██║██████╔╝
  ╚══════╝╚═╝     ╚═╝╚══════╝    ╚═╝        ╚══════╝╚══════╝╚═╝     ╚═╝╚═════╝

⁠ZML/LLMD - High Performance LLM Inference Server

This is an alpha of ZML/LLMD, the ZML.ai⁠ homegrown high-performance LLM inference server (read more in the release blog post⁠). Please report any issues to the ZML team.

ZML/LLMD is a work in progress and may change significantly in the future.

⁠Features

• Easy Setup - Just mount your model and run
• Super fast cold boot - Loads models at SSD speed (~1-2s for 8B BF16)
• Cross-Platform Support - Transparent supports NVIDIA CUDA, AMD ROCm, Intel OneAPI and Google TPU
• Lightweight - CUDA: 1.7GB, ROCm: 3.8GB, TPU: 278mb, Intel: 356mb
• OpenAI API Compatible - Drop-in replacement for OpenAI endpoints
• Modern features - Continuous batching, paged attention, sharding
• High Performance - Optimized for fast inference
• Native HuggingFace, S3 and GCS support - built-in userland VFS supports zero copy loading from network storage to GPU

⁠Quick Start

⁠NVIDIA GPUs

Requires NVIDIA Container Toolkit⁠

docker run -p 8000:8000 --shm-size=256GB --gpus=all -e HF_TOKEN -it zmlai/llmd \
    --model=hf://Qwen/Qwen3-8B

⁠AMD GPUs

docker run -p 8000:8000 --device=/dev/kfd --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:rocm \
    --model=hf://Qwen/Qwen3-8B

⁠Intel GPUs

docker run -p 8000:8000 --device=/dev/dri -e HF_TOKEN -it zmlai/llmd:oneapi \
    --model=hf://Qwen/Qwen3-8B

⁠Google TPU

docker run --net=host --privileged -e HF_TOKEN -it zmlai/llmd:tpu \
    --model=hf://Qwen/Qwen3-8B

⁠Supported model types

Here are the models we support:

• gemma3
• gemma3_text
• gemma4
• gemma4_unified
• gemma4_text
• lfm2
• llama
• ministral3
• mistral3
• mistral
• qwen2
• qwen3
• qwen3_5
• qwen3_5_moe

⁠OpenAI Compatible API

Once running, the server provides OpenAI-compatible endpoints at http://localhost:8000, example:

curl -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3",
    "messages": [
      {"role": "user", "content": "What is the capital of France?"}
    ]
  }'

⁠Support

• Website: zml.ai⁠
• Discord: Join our community⁠

⁠Copyright (c) 2026 ZML