A new site goes live, or an old one moves to a new host, and a few weeks later the client asks why they can no longer find themselves in Google. Very often nothing is broken: one setting from the staging phase is still telling search engines to stay away. This guide covers where that setting can hide, how to check it in a few minutes, and how to make sure it never slips through again.
The usual suspects
- “Discourage search engines from indexing this site”. Under Settings → Reading in WordPress. It is often switched on during development and forgotten at launch. WordPress then asks search engines not to index the site.
- robots.txt. A staging robots.txt with
Disallow: /that was copied along with the site. - A noindex header from the server. Some hosts and staging tools add an
X-Robots-Tag: noindexheader. It does not show up in the page source, only in the response headers. - SEO plugin settings. A page, post type or the whole site set to noindex in the SEO plugin.
- Canonical links to the old domain. After a migration, canonical links that still point to the staging or old address tell Google the real page is somewhere else.
Check it in five minutes
- Open the homepage, view the page source and search for
noindexandcanonical. The canonical link should point to the live address. - Open
/robots.txton the live domain and look forDisallow: /. - Check the response headers of the homepage (the browser’s developer tools, Network tab) for
X-Robots-Tag. - In Google Search Console, inspect the homepage URL. It tells you whether the page may be indexed and why not.
- Check Settings → Reading and the SEO plugin’s site-wide settings.
Fix it and ask Google to come back
Switch off whichever setting is responsible, clear the page cache and check the five points again from a private window. Then request indexing for the homepage and the main pages in Google Search Console and submit the sitemap again. Recovery usually takes days, not weeks, once Google can crawl the site again, but traffic lost in the meantime does not come back.
Make it part of every launch and migration
- Add “search engines allowed, robots.txt, canonical, noindex header” to your launch checklist, and tick it off on the live domain, not on staging.
- Check again a day after launch: some caches and CDNs keep serving the staging version for a while.
- Monitor it afterwards. Plugins and hosts change settings during updates and migrations too.
How Verploy catches it
Every day, Verploy checks the homepage’s search basics on every connected site with Google PageSpeed: title, description, indexing, canonical link, robots.txt, language links and link texts. If the homepage asks Google not to show it, you get an alert, and it clears itself once fixed. The monthly client report includes the SEO score. More on findability in Google.
Want to know the day a client site disappears from Google?
Verploy checks every homepage’s search basics daily and alerts you when a site is hidden from Google. Start free on up to 3 sites, no credit card.