Mini-batch gradient descent

One Epoch, Step by Step

A 2-3-1 network with tanh hidden units and squared-error loss trains on 8 samples split into 2 batches of 4. Every click advances one sub-step: a sample goes forward, its loss is added to the batch total, and after the last sample the batch loss is averaged, differentiated, and applied as one weight update. After both batches the epoch is finished. All numbers are computed live.

Position in the epoch

Network

forward: activations, positive weightsloss, gradients, negative weights (dashed)

Current batch

samplex₁x₂yŷℓ = (ŷ−y)²

Loss over the epoch

per-sample loss ℓᵢbatch loss L (one gradient step each)

The gradient of the batch loss is the mean of the per-sample gradients, so the batch loss has to be complete before the backward pass can run. Learning rate 0.1, plain SGD, no momentum. The second epoch starts from the weights the first epoch produced, with the same batch order so the effect of the update is visible.