- scaling
- ml
- reflections
•
•
-
What Scaling Laws For Hybrid LLMs Can Tell Us About Pretraining Mixtures
Some thoughts on how scaling laws and data efficiency can guide data mixtures and choices.
-
What Hybrid Models Mean for Scaling
Understanding how hybrid models lie in various spectra between dense and linear attention.
-
From Muon to Spectra
Deriving dynamics for ESDs afforded by Muon
-
A Simple Toy Model Bridging HTSR & $\alpha$-REQ
Bounding conditioning of features with that of weights
-
Muon and Manifold Versions
A dive into Muon, manifold constraints, and optax.
-
Celebration is the secret
Inspiration from someone I look up to
-
New POVs on hypernetworks
Hypernetworks from the POV of INRs and functional similarity