| Model | HellaSwag (0-shot) | MMLU (5-shot) | GSM8K (8-shot) | Avg. Latency (ms/token) | |---------------------------|--------------------|---------------|----------------|--------------------------| | Lite 1.4B (Base) | 62.3 | 41.7 | 24.5 | 8.2 | | Phi-2 (2.7B) | 71.9 | 57.8 | 48.1 | 18.4 | | StableLM-2-1.6B | 67.2 | 49.3 | 35.6 | 11.7 | | | 71.4 | 56.9 | 46.8 | 9.4 |
BigBooster closes the gap with 2.7B models while running at nearly the same speed as the base 1.4B model. 5. Ablation Study | Component | Avg. Accuracy | Δ Latency | |-------------------------------|---------------|------------| | Full booster (no routing) | 73.1% | +112% | | Random routing (15% tokens) | 61.5% | +10% | | No auxiliary load-balancing | 67.4% | +14% | | Proposed (learned routing) | 71.4% | +15% |
Model weights and inference code will be released under Apache 2.0. This paper is a conceptual proposal; actual implementation results may vary.
| Model | HellaSwag (0-shot) | MMLU (5-shot) | GSM8K (8-shot) | Avg. Latency (ms/token) | |---------------------------|--------------------|---------------|----------------|--------------------------| | Lite 1.4B (Base) | 62.3 | 41.7 | 24.5 | 8.2 | | Phi-2 (2.7B) | 71.9 | 57.8 | 48.1 | 18.4 | | StableLM-2-1.6B | 67.2 | 49.3 | 35.6 | 11.7 | | | 71.4 | 56.9 | 46.8 | 9.4 |
BigBooster closes the gap with 2.7B models while running at nearly the same speed as the base 1.4B model. 5. Ablation Study | Component | Avg. Accuracy | Δ Latency | |-------------------------------|---------------|------------| | Full booster (no routing) | 73.1% | +112% | | Random routing (15% tokens) | 61.5% | +10% | | No auxiliary load-balancing | 67.4% | +14% | | Proposed (learned routing) | 71.4% | +15% |
Model weights and inference code will be released under Apache 2.0. This paper is a conceptual proposal; actual implementation results may vary.