- Published on
Mistral AI has released Mistral 7B, a 7.3 billion parameter language model, which it claims outperforms Llama 2 13B on all benchmarks and Llama 1 34B on many.
The model uses Grouped-query attention for faster inference and Sliding Window Attention to handle longer sequences.
It is being released under the Apache 2.0 licence, with the company’s reference implementation and deployment options on various clouds.
Mistral 7B has been fine-tuned for chat, achieving better performance than Llama 2 13B on MT-Bench, a metric for evaluating multilingual instruction models.
The company said it was looking forward to working with the community on developing moderation for the model to allow it to be used in environments that require guardrails for outputs.