Paul Irish of Google described on the Chrome for Developers blog on 12 November how the Chrome team rebuilt the search engine of the Modern Web Guidance skill three times in six months. The skill is an open-source collection of guides, and its stable version was released the same week. It steers coding agents towards new web platform features instead of old workarounds.

The collection holds more than 200 guides on HTML, CSS and browser APIs. At first, the agent sees only the skill’s description of about 205 tokens. On a frontend task, it searches the guides with a CLI tool and receives ten hits, which take up about 530 tokens. It then loads the guide it has chosen in full, about 800 tokens. Irish said MCP tools either waste tokens or agents do not call them reliably, so the team chose a CLI.

The old model did not know the new APIs

The first version took up 507 MB when installed and needed a C++ build that often failed in cloud sandboxes. The second version ran the all-MiniLM-L6-v2 embedding model in JavaScript. An embedding model converts text into a vector of numbers, and texts with similar meanings get vectors that lie close together.

In production, the search failed. MiniLM, trained around 2018–2020, split new API names such as fetchpriority into meaningless fragments. A similarity threshold of 0.38 also rejected relevant results. A search for ‘confirm destructive action’ returned an empty list, and the agent used a window.confirm() call instead of the dialog element.

The team built a new benchmark from agents’ real searches. Telemetry, which can be switched off with the setting DISABLE_TELEMETRY=1, records only the agent’s 2–5-word search phrase. The team added BM25 keyword search alongside semantic search and raised the number of results to ten. The hit rate – how often the expected guide appears among the top results – rose from 80.6 per cent to 94.7 per cent.

Guide embeddings in the cloud, queries from a lookup table

In the third version, Google’s gemini-embedding-001 model computes the guide embeddings at release time, because the guides do not change between releases. Queries are converted into the same vector space without a neural network. The team used Gemini to compute embeddings for 29,165 words and fine-tuned the word vectors on 33,000 real searches. At runtime, a seven-line loop sums the words’ weighted vectors.

The method is designed for short queries, and, according to telemetry, 86.7 per cent of searches were 2–5 words long. When the team tried using the lookup table to embed whole guides, the embeddings lost their ability to discriminate. The dialog guide was then more similar to the search ‘best pizza in new york’ than to the search ‘dialog’.

The table shows the team’s own measurements of the second and third versions.

Source: Chrome for Developers blog, the team’s own measurements. Version 2 hit rate with tuned hybrid search.

The hit rate improved by only 0.7 percentage points over the tuned second version, so the gains are mainly in size and speed. When queries were embedded in the cloud with the Gemini API, the hit rate was 96.9 per cent.

The skill, its search engine and the benchmark are open source. The search tool runs locally without an API key.