aistaging/minicpm-v-4_5-safetensors repository overview

⁠MiniCPM-V 4.5

MiniCPM-V 4.5 is a state-of-the-art multimodal large language model (MLLM) that achieves GPT-4o-level performance on vision-language tasks with only 8B parameters. Built on Qwen3-8B and SigLIP2-400M, this model excels at single image understanding, multi-image reasoning, and high-FPS video processing - all optimized to run efficiently on edge devices including smartphones and tablets.

The model introduces groundbreaking features including a unified 3D-Resampler architecture that compresses video tokens by 96x, enabling efficient high-FPS video understanding. It achieves an average score of 77.0 on OpenCompass, surpassing widely used proprietary models like GPT-4o-latest and Gemini-2.0 Pro. With support for controllable hybrid fast/deep thinking modes, state-of-the-art OCR capabilities, and multilingual support for 30+ languages, MiniCPM-V 4.5 delivers exceptional performance across diverse vision-language tasks.

MiniCPM-V 4.5 is designed for easy deployment across multiple platforms including llama.cpp, Ollama, vLLM, and SGLang, with quantized models available in 16 different formats (int4, GGUF, AWQ). The model can be fine-tuned on new domains using Transformers and LLaMA-Factory, and includes optimized implementations for local iOS devices.

⁠Characteristics

⁠Using this model with Docker Model Runner

docker model run minicpm-v-4_5-safetensors

For more information, check out the Docker Model Runner docs⁠.

⁠Architecture

MiniCPM-V 4.5 introduces a unified 3D-Resampler that enables efficient video understanding by grouping and jointly compressing up to 6 consecutive video frames into just 64 tokens. This achieves a 96× compression rate for video tokens, allowing the model to process significantly more video frames without additional LLM computational cost, enabling high-FPS (up to 10 FPS) video and long video understanding.

⁠Benchmarks

⁠Overall Performance

MiniCPM-V 4.5 achieves an average score of 77.0 on OpenCompass, outperforming models significantly larger in size:

⁠OpenCompass Efficiency

⁠Video-MME Performance

Both benchmarks were evaluated using 8×A100 GPUs for inference.

⁠Key Capabilities

State-of-the-art Vision-Language Performance: Surpasses GPT-4o-latest, Gemini-2.0 Pro, and Qwen2.5-VL 72B, making it the most performant MLLM under 30B parameters.

Efficient Video Understanding: Processes high-FPS video (up to 10 FPS) and long videos with state-of-the-art performance on Video-MME, LVBench, MLVU, MotionBench, and FavorBench.

Advanced OCR and Document Parsing: Achieves leading performance on OCRBench, surpassing proprietary models like GPT-4o-latest and Gemini 2.5. Processes high-resolution images up to 1.8 million pixels (1344x1344) with 4x fewer visual tokens than most MLLMs.

Controllable Hybrid Thinking: Supports both fast thinking mode for efficient daily use and deep thinking mode for complex problem-solving tasks, switchable based on user needs.

Multilingual Support: Trained with RLAIF-V and VisCPM techniques, supporting 30+ languages with trustworthy behaviors and reduced hallucinations.

⁠Example Use Cases

⁠Deployment Options

MiniCPM-V 4.5 supports multiple deployment frameworks:

• Edge Devices: llama.cpp, Ollama
• Cloud Serving: vLLM (v0.10.2+), SGLang
• Quantization: GGUF, BNB, AWQ formats in 16 sizes, int4 quantization
• Fine-tuning: Transformers, LLaMA-Factory
• Mobile: Optimized iOS app for iPhone and iPad

⁠Key Techniques

Unified 3D-Resampler: Jointly compresses multiple consecutive video frames with temporal awareness, enabling efficient high-density video understanding while maintaining unified encoding for images, multi-image inputs, and videos.

Unified Learning for OCR and Knowledge: Dynamically corrupts text regions in documents with varying noise levels, teaching the model to adaptively switch between accurate text recognition and multimodal context-based knowledge reasoning.

Hybrid Fast/Deep Thinking with Multimodal RL: Uses a new hybrid reinforcement learning method incorporating RLPR and RLAIF-V to jointly optimize both fast and deep thinking modes, enhancing performance while reducing hallucinations.

⁠Links

• Hugging Face Repository⁠
• GitHub Repository⁠
• Cookbook⁠
• Technical Report⁠
• iOS Demo⁠
• Online Demo⁠

⁠Considerations

• The model requires custom code and trust_remote_code=True for usage with Transformers
• Video processing requires organizing video data into frames and temporal_ids sequences
• Maximum video frames supported: 180 frames × 3 packing = 540 effective frames
• For optimal performance, use sdpa or flash_attention_2 attention implementation (not eager)
• High-resolution image processing supports up to 1.8 million pixels (e.g., 1344x1344)
• As an LMM trained on multimodal corpora, generated content does not represent the views of model developers
• Users should be aware of potential data security issues, public opinion risks, and misuse scenarios

⁠Generated by

This model card was automatically generated using cagent-action⁠. Want to learn more about Docker Model Runner? Check out the project repository: https://github.com/docker/model-runner⁠.