OpenAI ships GPT-4 and declines to publish its architecture, size, data or training method

2023-03-14Trade Secrets

OpenAI, "GPT-4," openai.com (Mar. 14, 2023); OpenAI, "GPT-4 Technical Report," cdn.openai.com (Mar. 14, 2023), Section 2, "Scope and Limitations of this Technical Report"

OpenAI released GPT-4, which it describes as "a large multimodal model (accepting image and text inputs, emitting text outputs)" that "passes a simulated bar exam with a score around the top 10% of test takers" where "GPT-3.5's score was around the bottom 10%." Text input is available through ChatGPT to ChatGPT Plus subscribers with a usage cap and through a waitlist for the application programming interface; image input "is still a research preview." The announcement says the base model was trained "using publicly available data (such as internet data) as well as data we've licensed" and that OpenAI and Azure co-designed a supercomputer for the workload, and gives no figures for either. The accompanying technical report states that "GPT-4 is a Transformer-based model pre-trained to predict the next token in a document". In a section titled "Scope and Limitations of this Technical Report," the report states, citing the competitive environment and the safety implications of large-scale models, that "this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar." The report adds that the company is committed to independent auditing of its technologies and plans to share further technical details with additional third parties who can advise on weighing competitive and safety considerations against the scientific value of transparency.

What it changesThe laboratory with the most capable model on the market has chosen trade secret over publication for the model's internals, and that choice is available to any company only on one condition: the same details stay out of its patent applications. A specification that describes the training mixture, the scaling recipe or the architecture to support a claim publishes it eighteen months after filing whether or not the patent issues. A program that files on model internals therefore has to decide, feature by feature, which mechanisms are claimed and disclosed and which are withheld, and the withheld mechanisms have to stay out of the specification entirely, because an enabling disclosure of a secret is a published secret. The division that works in practice is to patent what the product exposes or a competitor can observe, including inference-time behavior, interfaces and data pipelines, and to keep secret what only the training run knows. Write the decision down for each release; it is the record a trade secret claim needs.