Long Context

Base of RoPE Bounds Context Length
This work contributes to the investigation of the lower bounds of the Base in RoPE, providing a theoretical foundation for the long-context extrapolation of models.
Xin Men*, Mingyu Xu*, Bingning Wang†, Qingyu Zhang, Hongyu Lin, Xianpei Han, Weipeng Chen
PDF 引用
Base of RoPE Bounds Context Length