Reproducing Chinchilla Scaling on a Budget

Training a 70B parameter model costs millions of dollars. Scaling laws exist so you don't have to guess how to spend that budget. Here's what I learned reproducing them on a free GPU. Scaling laws are basically rules that tell us how model performance improves as you increase quantities such as mode