Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
NILKNARFGonzo 
posted an update 13 days ago
Post
103
just recieved my stack of 10 floppy disks - you know what that means

floppyx4 is canceled, floppyx10 is next

here's the intended specs:
- official tokenizer (the actual tokenizer for gpt-2)
- actual gpu training (barely)
- sharegpt (if i can afford it computationally)
- full thing fitting on 10 floppy disks (not just the safetensors file)
- and if needed different arch (like llama)

also unsloth on a gpu from 2015 is insane

Should i make CDx1? i have a CD and DVD's btw

·

I'd recommend using a very small head_dim, and using the freed parameters for feed-forward width or something.