Batch Size

Batch size is a metric representing the number of training samples used in a single parameter update during machine learning or deep learning training. Rather than processing all data at once, a model learns in fixed-size chunks called batches. Batch size is one of the most important hyperparameters, significantly affecting training speed, stability, and model accuracy. For example, if there are 1,000 training samples and the batch size is set to 100, training proceeds in 10 passes per epoch. The loss function and gradient are computed for each batch, and the model's parameters are updated after each. Types of batch training: • Mini-batch learning: The most common approach — batches of tens to hundreds of samples, balancing stability and computational efficiency. Used in most practical deep learning work. • Batch learning: Uses all data at once for a single gradient update. Training is stable but memory-intensive and impractical for large datasets. • Online learning (SGD): Processes one sample at a time. Computationally light but can be unstable. Key considerations when choosing batch size: • Small batch sizes: More noise, but may improve generalization • Large batch sizes: More stable training, but consumes more resources and may increase overfitting risk The maximum feasible batch size also depends on GPU memory and model size. Batch size is a critical parameter that affects both training efficiency and model accuracy, and must be tuned in conjunction with other factors such as learning rate and number of epochs.