What is the training loop? It is the four-step ritual repeated thousands of times: predict, score the error, compute gradients, nudge weights. Every deep learning framework hides this behind model.fit or a training step.
Why do learning rate and epochs matter? They are the two knobs beginners feel first. Learning rate sets step size; epochs set how many times the model sees the full dataset. Get either wrong and training either crawls, explodes, or memorizes instead of learning.
Picture the hiker in fog descending a valley (the loss surface). Each step of the loop:
weight = weight - learning_rate * gradient.The learning rate (often written as eta or "eta") is step length:
An epoch is one full pass through the training set. Real datasets are processed in mini-batches so you get many updates per epoch without loading everything at once.
Underfitting vs overfitting — the studying trap:
| Problem | Plain-English idea | Symptom |
|---|---|---|
| Underfitting (high bias) | Model is too simple — like a student who only learns "all four-legged animals are dogs" | Poor on both training and test data |
| Overfitting (high variance) | Model memorizes training noise word-for-word instead of learning patterns | 100% on practice, fails on new data |
Healthy training: both train and validation loss drop, then flatten. Overfitting: train keeps falling while validation rises. Underfitting: both stay high.
Batch vs epoch. Suppose 10,000 examples and batch size 100. Each epoch has 100 update steps. After 20 epochs you have taken 2,000 gradient steps — but each example has been seen about 20 times.
Why shuffle. Randomizing order each epoch reduces the chance that a weird contiguous slice of data biases every update the same way.
Learning rate schedules (awareness). Many runs start with a constant learning rate, then decay it (step decay, cosine). Warmup starts small and ramps up for large-batch transformer training. For this lesson, a carefully chosen constant learning rate is enough.
| Learning rate | Typical symptom |
|---|---|
| Much too high | Loss → NaN or oscillates upward |
| Slightly high | Fast early drop, then unstable plateaus |
| Good | Smooth decrease of train (and val) loss |
| Too low | Tiny slope; needs huge epoch count |
Monitoring. Log train loss every N steps and validation loss every epoch. Plot both. Early stopping reads this plot automatically. Watch for exploding loss after a learning rate change — that is your first debugging signal.
Toy linear regression with mini-batch gradient descent. We fit y ~= w * x + b on noisy data and watch loss fall.
import numpy as np
rng = np.random.default_rng(42)
n = 200
x = rng.normal(size=n)
y = 3.0 * x - 1.0 + rng.normal(scale=0.3, size=n)
idx = rng.permutation(n)
train_idx, val_idx = idx[:160], idx[160:]
x_train, y_train = x[train_idx], y[train_idx]
x_val, y_val = x[val_idx], y[val_idx]
def mse(y_true, y_pred):
return np.mean((y_true - y_pred) ** 2)
def forward(x, w, b):
return w * x + b
w, b = 0.0, 0.0
lr = 0.05
epochs = 40
batch_size = 32
history = []
for epoch in range(epochs):
perm = rng.permutation(len(x_train))
x_epoch, y_epoch = x_train[perm], y_train[perm]
for start in range(0, len(x_train), batch_size):
xb = x_epoch[start:start + batch_size]
yb = y_epoch[start:start + batch_size]
pred = forward(xb, w, b)
err = pred - yb
dw = 2.0 * np.mean(err * xb)
db = 2.0 * np.mean(err)
w -= lr * dw
b -= lr * db
train_loss = mse(y_train, forward(x_train, w, b))
val_loss = mse(y_val, forward(x_val, w, b))
history.append((epoch, train_loss, val_loss))
print(f"learned w={w:.3f}, b={b:.3f} (true ~ 3, -1)")
print("epoch | train_mse | val_mse")
for epoch, tr, va in history[::8]:
print(f"{epoch:5d} | {tr:9.4f} | {va:7.4f}")Try lr = 1.0 and watch loss explode. Try lr = 1×10⁻⁵ and notice almost no movement after many epochs.
optimizer.zero_grad() and you train on summed ghosts of past batches.The training loop is forward → loss → backward → update, repeated over batches and epochs — with learning rate setting step size and train/val curves telling you whether you are learning or memorizing.