dotlabs
🚀
2
Dot Labs
company
AI & ML interests
byte-level research • the dots company
Recent Activity
appvoid
updated a
Space about 9 hours ago
appvoid
updated a
model 1 day ago
appvoid
updated a
dataset 3 days ago
appvoid
published a
dataset 3 days ago
Join Us!
4
#1 opened 3 days ago
by 
appvoid
Post
117
appvoid
published a
model 3 days ago
Post
179
byte-level, deep networks, flow-matching, matryoshka...
Post
177
Much of the misunderstanding on loops and byte-level models comes from fear on inventing the wheel, not reinventing it
Post
147
How much should OpenAI offer for a name like Dot Labs?
- 1 reply
Post
171
Reality has not limits if you know how to shape it, one step at a time.
- 1 reply
appvoid
updated a
dataset 17 days ago
Post
86
How old were you when you discovered good AI agents favor the Strategy Design Pattern over any other?
- 1 reply
appvoid
published a
dataset 26 days ago
Post
133
LLMS for edge devices?
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?
That's the idea for the next model.
Keep in tune.
What if we go the reliable route instead of the speed route?
What if we make it run on sbcs with few megabytes available?
That's the idea for the next model.
Keep in tune.
Post
143
We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
appvoid
published a
Space about 1 month ago
Post
133
Any thoughts on Nvidia acquiring this website?
I don't know what to feel about it. But would be great if huggingface gets something similar to Kaggle with free GPU hours (or even days) for training.
I don't know what to feel about it. But would be great if huggingface gets something similar to Kaggle with free GPU hours (or even days) for training.
- 3 replies
Post
3309
Love how the small lm community is getting identity over time:
- Channel-Mixing
- XSA
- Three-tower
- Digit aware
- Loops
No one is doing the same! That's so cool.
- Channel-Mixing
- XSA
- Three-tower
- Digit aware
- Loops
No one is doing the same! That's so cool.
- 36 replies