Can I Train My Own LLM From Scratch?

Short answer: Yes, you can train your own LLM from scratch if you think small. A tiny LLM (large language model, the kind of AI behind chatbots) can be trained from zero on one rented gaming GPU. A 64 million parameter model takes a little over two hours and costs less than a dollar. It will chat, but it will not replace ChatGPT. What you get is the whole recipe, understood line by line.

You type a question into ChatGPT and a paragraph comes back. It feels like a mind. Most people assume building one takes a data center and a billion dollars.

For the giant ones, it does. But the recipe is the same at every size, and at the small end it fits on one graphics card. This post walks that recipe from the very first idea, then shows what happened when I trained a tiny one myself on a plain 2-core cloud machine with no GPU.

The thread to hold onto: an LLM is a next-word guesser that got very good with practice. Everything else is detail.

What does training an LLM from scratch actually mean? §

It means starting with a model that knows nothing and teaching it, by practice, to guess the next piece of text.

Start with something you already do. Ever finished a song lyric before the singer? When you read “Twinkle, twinkle, little…”, your brain fills in “star” before you see it. You learned that from hearing the song many times.

An LLM does the same thing with numbers. First the text is cut into tokens (small chunks of text, often a word or part of a word). Each token gets an ID number. The model reads a run of tokens and outputs a score for every possible next token.

Inside, the model is a huge pile of numbers called parameters (the adjustable knobs that store what the model learned). At the start they are random, so its guesses are random too.

Training is a loop. Show the model real text, let it guess the next token, measure how wrong it was, then nudge every knob a tiny bit so the right answer becomes more likely. That “how wrong” score is called loss (one number that says how surprised the model was by the real answer). Repeat millions of times and the knobs slowly fill with patterns of language.

“From scratch” means the knobs really do start random. You are not downloading someone else’s trained model and tweaking it.

What is inside a small LLM? §

It is a stack of identical layers that pass a running summary of the text upward, one step at a time.

That stack is called a transformer (the standard design behind almost every modern LLM). Chat models use the decoder-only kind (a version that only reads left to right and predicts what comes next).

Each layer does two jobs. Both matter. Attention (a step where every token looks back at earlier tokens and picks which ones matter) lets “it” in a sentence find the thing “it” refers to. Then a small feed-forward network (a simple block of math applied to each token on its own) mixes that information. Stack 8 of these layers and the model can track grammar, facts and style.

The original transformer design had two halves, an encoder and a decoder. A chat LLM keeps only the decoder half and stacks it many times. A small open source model called MiniMind does exactly that: its dense version is a decoder-only stack of 8 layers.

What are the steps from raw text to a chatbot? §

There are two must-do stages, pretraining and fine-tuning, plus optional polish on top.

Pretraining comes first. The model reads a big pile of ordinary text and learns to continue it. After this stage it can finish sentences, but it does not know it is supposed to answer you. Ask it a question and it may simply write more questions.

Next is SFT (supervised fine-tuning, extra training on example conversations). This time the model sees thousands of “user asks, assistant answers” pairs and learns the shape of a helpful reply. This is the step that turns a text-continuer into a chatbot.

After that come optional extras. LoRA (a cheap way to fine-tune by training a small add-on instead of the whole model) teaches a narrow skill, like medical answers. DPO (training on pairs of good and bad answers so the model prefers the good one) and other RL (reinforcement learning, learning from scores instead of examples) methods polish behavior.

MiniMind ships code for every one of these stages, written in plain PyTorch (the most common Python library for building neural networks) instead of hiding them behind big helper libraries. That is why it is a good teaching example: you can read every line.

What happened when I trained a tiny one myself? §

On a 2-core machine with no GPU, a 1.5 million parameter model went from random letters to play-script lines in 25 minutes.

The question was simple: does the loop above really work, without any special hardware? So I wrote a short PyTorch script with the same basic parts as MiniMind: attention, feed-forward blocks, the same kind of normalization and shared input and output weights. Then I trained it on about one million characters of Shakespeare.

To keep it small, each token was one character. That gives a vocabulary (the full list of tokens the model knows) of only 65 symbols.

Before training, the loss was 4.177. That number has a plain meaning: it matches guessing evenly among all 65 characters. The model knew nothing, and its writing proved it:

3UhzFSN$G.rMk$ZFvQM&lXfXVv;RMpPbid'YvlUmX&IXXfqaXgSfq;GVcqJD&.enS'hpK

3UhzFSN$G.rMk$ZFvQM&lXfXVv;RMpPbid'YvlUmX&IXXfqaXgSfq;GVcqJD&.enS'hpK

Then the loop ran. Would it learn anything at all? Each line below is the loss on text the model never trained on:

A loss of 2.296 means the model has narrowed each guess down to about 10 likely characters instead of 65. Here is what it wrote after 8 minutes:

And s thou meen ot men sirs vevears ndoue
They hacings and mall t festinct torery walllle
Bis lith sencot, somes ponch not pewens in thedr ne pom tar I do

And s thou meen ot men sirs vevears ndoue
They hacings and mall t festinct torery walllle
Bis lith sencot, somes ponch not pewens in thedr ne pom tar I do

It is not English yet. But look at what it learned on its own: capital letters at line starts, short lines, and real words like “And”, “thou”, “They” and “I do”. Nobody told it any of that.

So I let it run to 25 minutes. The loss fell to 1.797, about 6 likely characters per guess. A few of its lines:

AMPHOMPSOPEY:
I have that they not mother, most; thou servation
Is the uto protied that he cortain'd their
And now the feaven a hand in wereturn.
There's to made a hours, but bl

AMPHOMPSOPEY:
I have that they not mother, most; thou servation
Is the uto protied that he cortain'd their
And now the feaven a hand in wereturn.
There's to made a hours, but bl

It invented a speaker name in capitals followed by a colon, exactly like a play script. It uses “There’s” and “cortain’d” with apostrophes. Every bit of that came from guessing the next character and being corrected about three thousand times.

That is the whole trick of an LLM, visible in under half an hour. Bigger models do exactly this, with more knobs, more text and far more steps.

Can I train a real LLM without a GPU? §

You can learn on a CPU, but a useful model needs a GPU, because the math is thousands of times too slow otherwise.

How big is the gap? I built a model with the same width, depth and vocabulary as MiniMind’s 64M model. My simpler version came out at about 54M parameters. Then I timed one training step on the same 2-core machine. Each step processed 8 chunks of 340 tokens.

One step took 9.3 seconds. That is about 292 tokens per second. At that speed, reading one billion tokens of training text would take roughly 40 days, nonstop.

A GPU (graphics card, a chip built to do thousands of multiplications at once) exists for exactly this. Training is almost entirely matrix multiplication (multiplying big grids of numbers), and a GPU does that in parallel. So a job that takes weeks on a laptop processor finishes in hours on one gaming card. Running a finished model is a far lighter job than training one; see whether a 70B model can run on a 4GB GPU.

How much does it cost to train a small LLM? §

For a 64M parameter chatbot on one rented RTX 3090, the full pretrain plus fine-tune run costs about 3 yuan, which is under half a US dollar.

MiniMind’s own measurements on a single NVIDIA RTX 3090 (a consumer gaming GPU with 24 GB of memory) break it down like this:

Seven yuan is roughly one US dollar, so the whole run lands well under a dollar of rented GPU time. Rental prices move around, so treat it as a ballpark.

For scale, the smallest MiniMind model is about 1/2700 the size of GPT-3. That ratio is the honest summary of what you are building: a real LLM, made the real way, but thousands of times smaller than the famous ones.

How do you train MiniMind step by step? §

Clone it, install the requirements, download two datasets, then run the pretrain script and the SFT script in order.

You need a machine with an NVIDIA GPU, Python, and about 3 GB of free space for the two small datasets. Renting a cloud GPU by the hour works fine.

1. Get the code: git clone --depth 1 https://github.com/jingyaogong/minimind
2. Install: cd minimind && pip install -r requirements.txt
3. Download pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl from the MiniMind dataset page into the dataset folder.
4. Pretrain: cd trainer && python train_pretrain.py
5. Fine-tune into a chatbot: python train_full_sft.py
6. Chat with it: go back up one folder and run python eval_llm.py

The scripts save weights into an out folder as they go, and the chat script loads the fine-tuned weights from there by default. If you only want to see the result first, the project also offers ready-trained weights you can download and chat with.

One catch before you start. MiniMind’s training data is mostly Chinese, with some English mixed in. Its tokenizer (the tool that cuts text into tokens) has only 6,400 tokens, which keeps the model small. So expect it to answer best in Chinese, and treat English replies as a bonus. If you want an English-only model, the same scripts work on your own English text.

What will your own tiny LLM be good at? §

It is great for learning how LLMs work and poor at facts, so build it to understand, not to replace a real assistant.

There is a fair case against doing this at all. If you only want a private chatbot, downloading an open model and running it locally gets you a far smarter assistant in minutes. Strata vs Ollama compares two ways to do that on a 12 GB GPU. Fine-tuning an existing model gets you a custom one for less effort.

Still, those routes keep the inside of the model a black box. Training from scratch is the only way to see every stage happen: the loss falling, grammar appearing, then the switch from “continues text” to “answers questions”. After one run, words like pretraining, SFT and LoRA stop being jargon and become things you did.

Expect clear limits. A 64M model makes up facts often, forgets long conversations, and struggles with math. Even MiniMind’s early chat versions reach basic dialogue while knowledge and generalization stay limited.

So should you try it? §

Yes, once, with a plan: tiny model on your CPU first, then MiniMind on a rented GPU.

Run the 25-minute CPU experiment to watch the loss drop with your own eyes. Then rent a GPU for an afternoon and train MiniMind end to end. Two short steps, and you will know more about LLMs than most people who use them every day.

It is still a next-word guesser. The difference is that you know how it got good.

Common questions about training your own LLM from scratch §

Can I train an LLM from scratch on my laptop? §

You can train a tiny one, a few million parameters, on a laptop processor in minutes and watch it learn. A model big enough to chat usefully needs a GPU, or it would take weeks.

How long does it take to train a small LLM from scratch? §

MiniMind’s 64M model takes about 2.3 hours on one RTX 3090 for pretraining plus fine-tuning. My own 1.5M parameter test showed clear learning in 8 minutes on 2 CPU cores, and play-script lines by 25 minutes.

How much data do you need to train an LLM? §

A tiny learning model can train on a few books’ worth of text, about one million characters. MiniMind’s small pretraining file is about 1.2 GB, and big commercial models use vastly more.

Is it better to fine-tune or train an LLM from scratch? §

For a useful assistant, fine-tuning an existing model is cheaper and gives better results. Training from scratch is the better choice when your goal is to understand how LLMs work.