881 B
Frankenstein
Embedding hackathon repo
Notebook embedding.ipynb: Try to swap components between 2 LMM models. Recommended to start. Has more comments and helper functions.
Notebooks rotation, scaling and reflection.ipynb: Only attempt an operation on the input and output embeddings of one model. Does it break the model? Or is it invariant?
Notebook rotation_all.ipynb: This notebook then attempts to rotate the entire model. So all weights in all transformer layers, etc. This is not as easy as it sounds, and highly model specific: different models have very different internal layers and representations. Layers may have different shapes, or are concatenated (such as the kvq matrices).
Notebook rotation_fixed.ipynb: What happens if you try to rotate the input embedding, and then rotate back just before the first activation function in the first neural network?