Hardware Requirements and Technical Tradeoffs
The high hardware threshold of Grok-2 stems from three major technical characteristics: 1) the 128-expert MoE architecture needs to maintain 286 billion active parameters; 2) 8-way tensor parallelism requires ultra-fast NVLink interconnections; and 3) FP8 quantization requires support from next-generation compute cards such as the H100.
For developers with limited resources, the model can be experienced in these ways:
- Cloud Service Solutions: Lambda Labs offers hourly rental instances of pre-installed environments (~$12.5/hour) to support rapid release of resources
- Quantitative Lite: The grok-2-mini 4bit version from the community runs on a single 24GB GPU and retains the capacity of 85%.
- API Access: xAI expects to launch an official API in 2024Q4, and the pricing strategy may be based on 1/3 of GPT-4's pricing.
Performance trade-offs: 1) Turning off some experts (-expert-dropout 0.3) can reduce the memory usage of 40%; 2) Using an optimized inference framework such as vLLM can improve the throughput of 20%; 3) For batch size=1 scenarios you can try to --quantization fp4 Mode.
This answer comes from the articleGrok-2: xAI's Open Source Hybrid Expert Large Language ModelThe
English 
简体中文 
日本語 
Deutsch 
Português do Brasil