Scale is one key ingredient that has enabled the success of modern LLMs. In this talk I will go over different kinds of parallelisms (model, pipeline, data, sequence, expert) and how they compose with each other to enable training of models with extremely large number of parameters.