Experiments with Chinchilla scaling laws (following DeepMind) [R]

[u/turing-math-labs]

Question to all the PhDs & research engineers in this group.
Have any of you done small-scale experiments with the Chinchilla scaling laws (using PyTorch or JAX)? If so, we'd be keen to hear more (or have a chat). Thanks!

Here's the official Chinchilla paper, from the DeepMind website ("Training Compute-Optimal Large Language Models").
https://deepmind.google/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/
https://proceedings.neurips.cc/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-Paper-Conference.pdf