a collection of my paper implementations, thoughts, experiments, and extensions.
- predicting the order of upcoming tokens is better than MTP for language modeling
- feedforward network latent structure does not have linear gains while scaling language models
- optimizing snake1d activation kernel in triton : a worklog
- on pre-norm vs post-norm in transformers
- optimizing multilingual compression via biased tokenization
- deriving and coding up the naive bayes algorithm
- SCRIPT-bpe
- euclidean vs cosine distance: an experiment with t-SNE from scratch