InternLM2 Technical Report
About
The evolution of Large Language Models (LLMs) like ChatGPT and GPT-4 has sparked discussions on the advent of Artificial General Intelligence (AGI). However, replicating such advancements in open-source models has been challenging. This paper introduces InternLM2, an open-source LLM that outperforms its predecessors in comprehensive evaluations across 6 dimensions and 30 benchmarks, long-context modeling, and open-ended subjective evaluations through innovative pre-training and optimization techniques. The pre-training process of InternLM2 is meticulously detailed, highlighting the preparation of diverse data types including text, code, and long-context data. InternLM2 efficiently captures long-term dependencies, initially trained on 4k tokens before advancing to 32k tokens in pre-training and fine-tuning stages, exhibiting remarkable performance on the 200k ``Needle-in-a-Haystack" test. InternLM2 is further aligned using Supervised Fine-Tuning (SFT) and a novel Conditional Online Reinforcement Learning from Human Feedback (COOL RLHF) strategy that addresses conflicting human preferences and reward hacking. By releasing InternLM2 models in different training stages and model sizes, we provide the community with insights into the model's evolution.
Key insights
What problem does InternLM2 address?
InternLM2 is an open-source LLM family (1.8B, 7B, 20B) trained on 2T+ tokens of text, code, and long-context data. It uses Grouped-Query Attention and a novel Conditional Online RLHF to reconcile conflicting human preferences and reduce reward hacking.
How does it differ from prior open-source LLMs?
Unlike prior reports that seldom detail data processing, InternLM2 documents full pipelines for text, code, and long-context data. On MMLU 5-shot, InternLM2-7B scores 65.8 versus Llama2-7B's 46.8. InternLM2-Chat-20B reaches 79.6 on GSM8K 4-shot, surpassing GPT-3.5's 78.2.
What are the practical uses of these findings?
InternLM2 supports long-context applications like retrieval-augmented generation and agents, with strong tool-use and code-interpreter capabilities. Models and checkpoints across training stages are released for community research; no production deployment is documented.
What limitations do the authors note?
The authors note that different parameter sizes do not yield significant differences in long-context performance, requiring further investigation. They also flag potential data contamination, presenting LM-loss metrics to detect test-set leakage and overfitting.