Caltech AI Colloquium

Two major advances of the last decade in AI involve language models and diffusion models. However, the remarkable capabilities of such complex models often elude explanation through analytic theory. I will discuss several works in which simple analytic theories can quantitatively explain their performance characteristics. First, for language modeling, we can quantitatively predict for the first time, the power law exponents governing neural scaling laws relating loss to the amount of training data. We show these neural exponents are simply a function of two statistical properties of language itself. Second, for diffusion models, we develop an analytic theory of creativity that explains how they can generate exponentially many novel images from a finite training set by constructing patch mosaics of the training data.  Our analytic theory predicts individual image outputs of trained convolution only diffusion models with high fidelity. Third, we show how co-designing learning during training time and search during test-time scaling can lead to improved mathematical reasoning using language models.  Given time, we will also mention some of our work on constructing, explaining, and controlling digital twins of the brain.