ai/nomic-embed-text-v2-moe-safetensors repository overview

⁠nomic-embed-text-v2-moe-safetensors

Nomic Embed Text v2 MoE is a state-of-the-art multilingual Mixture of Experts (MoE) text embedding model designed for high-performance retrieval tasks across approximately 100 languages. This model achieves competitive performance with models twice its size while maintaining efficiency through its sparse MoE architecture, which activates only 305M of its 475M total parameters during inference. The model excels at multilingual retrieval tasks and was trained on over 1.6 billion high-quality text pairs.

The model is the first general-purpose MoE text embedding model and was introduced in the paper "Training Sparse Mixture Of Experts Text Embedding Models". It features Matryoshka representation learning, allowing flexible embedding dimensions from 768 down to 256 with minimal performance degradation, enabling up to 3x reductions in storage costs. This makes it particularly well-suited for retrieval-augmented generation (RAG) applications where both performance and efficiency are critical.

Nomic Embed v2 MoE delivers state-of-the-art multilingual performance compared to models in the ~300M parameter class and remains competitive with larger models up to 568M parameters. The model is fully open-source, with released weights, training code, and training data, ensuring complete reproducibility of the training pipeline.

⁠Characteristics

⁠Using this model with Docker Model Runner

docker model run nomic-embed-text-v2-moe-safetensors

For more information, check out the Docker Model Runner docs⁠.

⁠Benchmarks

⁠Performance Comparison on BEIR and MIRACL

Performance comparison with other open-weights embedding models in the ~300M parameter class:

Performance comparison with larger models (~560M parameters):

⁠Matryoshka Embedding Performance

Performance on BEIR at different embedding dimensions:

The model maintains strong performance even when truncated to 256 dimensions, enabling significant storage and compute savings.

⁠Training Architecture

The model was trained using:

• 1.6 billion high-quality text pairs across multiple languages
• Consistency filtering to ensure high-quality training data
• Matryoshka representation learning for dimension flexibility
• Both weakly-supervised contrastive pretraining and supervised finetuning

⁠Links

• Blog Post⁠
• Hugging Face Model⁠
• Technical Paper⁠
• Training Code⁠
• Atlas Platform⁠

⁠Considerations

• Task Prefixes Required: The model requires task-specific prefixes for optimal performance. Use search_query: for queries/questions and search_document: for documents
• Custom Code: Must use trust_remote_code=True when loading the model to use the custom MoE architecture implementation
• Language Variation: Performance may vary across different languages, with best results on high-resource languages
• Resource Requirements: Despite efficiency improvements, MoE architecture may require more resources than traditional dense models during training and deployment
• Sequence Length: Maximum input length is 512 tokens; longer sequences will be truncated
• Matryoshka Trade-offs: While dimension truncation (e.g., to 256) provides significant storage savings, there may be some performance degradation depending on the task
• Optimal Use Cases: Best suited for multilingual retrieval, semantic search, and retrieval-augmented generation (RAG) applications

⁠Generated by

This model card was automatically generated using cagent-action⁠. Want to learn more about Docker Model Runner? Check out the project repository: https://github.com/docker/model-runner⁠.