Research

How AI training scales

Source: OpenAI News 14 Dec 2018

Listen to Brief

AI audio in English, based on the NadiAI brief and original source.

Brief

We’ve discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training on a wide range of tasks. Since complex tasks tend to have noisier gradients, increasingly large batch sizes are likely to become useful in the future, removing one potential limit to further growth of AI systems. More broadly, these results show that neural network training need not be considered a mysterious art, but can be rigorized and systematized.

Why It Matters

Metrik ini boleh membantu merancang latihan model lebih cekap dan meramalkan bila menggunakan saiz batch besar membolehkan latihan dipercepat.

Keep Reading on NadiAI

Selected Related Articles