Fig. 5

Synchronous parallelism is a widely used method in distributed deep learning using stochastic gradient descent (SGD) optimization. In this method, multiple worker nodes compute the gradients on different subsets of the data and send them to a parameter server. The parameter server aggregates the gradients and updates the model weights synchronously, which means all workers update their weights simultaneously. This process is repeated for a number of epochs until the model converges