What it is
The process in which a search engine (Google, Bing, Yandex) finds a page, reads it and stores it in its database. Until a page is in that database, it appears neither in search nor in AI assistants' answers.
How we use it
After every deploy the site itself tells Bing and Yandex which pages changed (IndexNow), and we submit important addresses for recrawl in the webmaster tools by hand. Bing picks a page up in 1–3 days, Google in 1–3 weeks.
Where indexing matters to a business
- The site is live, but search does not show it even for the company name.
- The site is moving to a new CMS, and its search rankings must not be lost.
- A staging copy must not end up in search instead of the real site.
- Without indexing no AI assistant will quote the page: ChatGPT, Gemini and Copilot take their answers from search engine indexes.
How long it takes
A page's path: you deploy it and notify search engines; the engine crawls it (Bing within one to three days, Google from a week); the page enters the index; it starts appearing for queries (Google after two to four weeks). Each step is done by someone else's robots on their own schedule. You cannot speed them up, only avoid getting in the way.
How we use it
- Sitemap and robots.txt from the content. On the studio website they are generated by code from the content collections, so a new case gets into the sitemap by itself, in both languages.
- IndexNow on every deploy. The site tells Bing about changed pages itself, and Valery Afanasiev's business-card site does the same after edits. Google does not support the protocol, so important URLs are submitted to it by hand in Search Console.
- Staging stays closed. One environment variable switches the site to staging mode: noindex plus a disallow rule in robots.txt. The museum site demo is closed on three levels: a meta tag, robots.txt and an X-Robots-Tag header that also covers images, video and fonts.
- Moving without losses. In the Joomla to WordPress migration the old-URL map became permanent 301 redirects, and every post and category got a title and description. Indexing is off on the demo and is switched on at the move to the production domain.
- One main domain. For «Дыхание Поволжья» the Latin-letter domain and the typo domain lead to the main one by a permanent redirect.
We set up indexing end to end under Websites.
Common problems
- Closed and forgotten. One line in robots.txt can close the whole site. A block left over from staging is a common reason for "we are not in search".
- Changing URLs. Every changed URL without a redirect is a lost page. We do not change URLs without need.
- Trying to hurry. Recrawl quotas are small: Google takes about ten URLs a day. Pushing hundreds of URLs at once is pointless.
- Not verified in webmaster tools. To submit a sitemap and request a recrawl, the site is verified in Google Search Console and Bing Webmaster Tools. That needs verification codes from the domain owner.
When you do not need it
Private sections, demo stands and customer accounts do not need indexing; on the contrary, they are closed. And if a site lives for one week around an event, waiting for indexing makes no sense: traffic will come from ads or a direct link.