$AMD released a state-of-the-art, fully open model trained entirely on MI300X and MI325X GPUs using ROCm

Instella-MoE is a 16-billion-parameter mixture-of-experts model that activates only 2.8 billion parameters per token, reducing inference costs while remaining competitive with dense and MoE models with similar or larger active parameter counts
It supports a 64K context window and was trained through six stages, from pre-training to reinforcement learning
AMD also introduced two new efficiency techniques:
- Gated Multi-head Latent Attention, which filters low-value attention outputs.
- FarSkip-Collective, which overlaps communication and computation, improving pre-training speed by 12.7% and reducing time-to-first-token by up to 39.2%.