Day 14

baby-llm: Minimal GPT Training Script

baby-llm: Minimal GPT Training Script

I really dont understand technically how an LLM is built - so today I tried to get OpenAI's Codex to build one for me.  Together we created a simple starter script that builds an GPT-style LLM using a corpus and running on my local machine's GPU.  The build is still going on, but I think it's working.

Of course the big question I had was what material will the LLM learn from.  There are plenty of books available on line so I chose to go with the classic 1866 novel by Fyodor Dostoevsky Crime & Punishment.  Reading this several years ago was amazing, and I'm curious now what it means to interact with an LLM that only understands Dostoevsky.

The Github repository is filled with descriptions of the process, which I intend to go back through when I have time.  The current run should take about 10hrs...

← All Projects