Fig. 3From: Accelerating neural network training with distributed asynchronous and selective optimization (DASO)Global synchronization Schematic of the global synchronization step performed by the global communication group consisting of GPU:A on each node. The network parameters are averaged by each GPU in the group, and the network parameters of each group member are set to the resultBack to article page