Google claims the residual block that lets a layer attend over the outputs of several earlier layers, filed in Greece one Monday and posted to arXiv the next

2026-10-22Filings

Listen · Thinking Makes It So, 2026-10-26 · 10 min

All episodes · Subscribe: Apple Podcasts · Spotify

United States Patent Application Publication No. 2026/0330219 A1, Google LLC, "Attention Neural Networks With Generalized Residual Attention Blocks," Application No. 19/468,910, filed February 3, 2026, claiming priority to Greek Application No. 20250100075 filed February 3, 2025, published October 22, 2026

Claim 1 is a system running an attention neural network whose stack contains generalized residual attention blocks. Each one takes as its input the outputs of two or more preceding blocks rather than only the one just before it, generates a query vector from that input through a query generalized residual block with one or more sets of weights, and applies attention over the input using that query. The dependent claims give the key and value paths the same treatment (claims 8 and 9); spell out the block as layer normalization, a product with a first set of weights shared across the preceding blocks, a sum with a second set that has a subset per preceding block, and a ReLU activation (claims 3 through 6); name multi-head depth-wise cross-attention (claim 12); and let the input grow in dimensionality deeper in the stack (claim 16). The specification reports up to three times fewer training steps to reach comparable perplexity on language and image modeling, and says the first set of weights can start at zero so the block begins as an ordinary residual network. The six inventors posted "DeepCrossAttention: Supercharging Transformer Residual Connections" to arXiv on February 10, 2025, seven days after the Greek filing. As published, claim 2 depends from "the method of claim 1" although claim 1 is a system, and claim 7, which depends from claim 1, refers to "the combination," "the transformed product" and "the second set of weights"; the last two first appear in claims 2 and 3, and no claim in claim 7's chain introduces a combination.

What it changesUpon review, I conclude that the timing of the Greek application, a first filing in Greece by a Mountain View company with inventors in New Jersey, New York and California, one Monday before the preprint went up the next, points to a filing made to put a date in front of the paper. That date matters most outside the United States, where the European Patent Office and China's office have no grace period for a preprint an inventor posts; in the United States the paper was forgiven either way. The lesson for a research group is the one Google's scaling application taught on this page earlier this month: the first filing, normally a United States provisional, goes in before the paper is posted. The other lesson concerns clearance. This claim did not appear in a United States publication until October 22, 2026, twenty months after the paper, so a clearance search run in that interval found the paper and not the claim. Consider whether you should rely on a clearance search on a transformer architecture feature first described in a recent paper. When performing a clearance search now, search by class, not by keywords alone: the paper calls the architecture DeepCrossAttention, and that name appears nowhere in the application, which sits in G06N 3/08 and G06N 3/048. The antecedent problems in claims 2 and 7 are likely to draw a rejection under Section 112(b) of Title 35 of the United States Code, which Google can cure by amendment.

More on What the AI companies are filing · Prior art, public disclosure and obviousness

← All recent developments · Search all developments

Burbage Law is the practice of Jon Burbage, a registered patent attorney. 342 United States patents have issued from applications he wrote; the full list is on the Patents page. More about Jon Burbage.

This is general information about a published decision, agency guidance or piece of legislation. It is not legal advice and does not create an attorney-client relationship. Items concerning offices outside the United States are reported for information only and are not counsel on the law of those countries.