Meta's Llama 3 paper puts a frontier training recipe into the prior art, down to the data mix and the annealing schedule

2024-07-23Industry

Meta, "Introducing Llama 3.1: Our most capable models to date," ai.meta.com (July 23, 2024), with the linked paper Llama Team, AI @ Meta, "The Llama 3 Herd of Models," ai.meta.com (July 23, 2024)

Meta released Llama 3.1 in 8-billion, 70-billion and 405-billion parameter versions with downloadable weights, a 128,000-token context window and eight languages, and changed its license so that outputs of the models, including the 405-billion model, may be used to improve other models. The linked paper, "The Llama 3 Herd of Models," states that all three models are publicly released under an updated Llama 3 Community License. The paper discloses the pre-training recipe: a 15.6 trillion token corpus; a final data mix of roughly 50 percent general knowledge, 25 percent mathematical and reasoning data, 17 percent code and 8 percent multilingual text; 3.8 times 10 to the 25th floating point operations of compute on up to 16,000 H100 graphics processors; a two-stage scaling law relating downstream negative log-likelihood to training compute and then to task accuracy; and an annealing phase in which the learning rate is reduced linearly to zero over the final 40 million tokens while the data mix is shifted toward high-quality sources. Post-training sections describe rejection sampling of ten to thirty candidates per prompt scored by a reward model, direct preference optimization with a stated learning rate and beta and an added negative log-likelihood term, and a synthetic code pipeline that produced more than 2.7 million examples checked by static analysis, unit tests and execution feedback. The image, video and speech extensions are described but not released.

What it changesA paper this detailed from a frontier laboratory is enabling prior art for a generation of training technique as of its publication date, and it is the reference examiners will cite against any application on data mixing, learning-rate annealing on curated data, compute-based prediction of downstream accuracy, rejection sampling, preference optimization with a likelihood term, or execution-verified synthetic code. Applications already on file with earlier priority dates are unaffected and have become more valuable for the same reason. For applications filed now, the claim has to recite what differs from the published recipe in a way the examiner can see, so the disclosure meeting with the training team should follow the paper's section headings and ask what the team does differently at each stage. The license change matters on the trade secret side: distillation from the 405-billion model's outputs is now permitted by the licensor, so the patentable subject matter in a product built that way sits in the distillation and fine-tuning pipeline, not in the base model.

More on Prior art, public disclosure and obviousness · Trade secrets and open release

← All recent developments · Search all developments

Burbage Law is the practice of Jon Burbage, a registered patent attorney. 342 United States patents have issued from applications he wrote; the full list is on the Patents page. More about Jon Burbage.

This is general information about a published decision, agency guidance or piece of legislation. It is not legal advice and does not create an attorney-client relationship. Items concerning offices outside the United States are reported for information only and are not counsel on the law of those countries.