Three concepts in one table: what do they do, and what don't they do?
When your developer mentions a sitemap, robots.txt or canonical tag, they're talking about three different things. A sitemap helps search engines discover important URLs; robots.txt can regulate crawling by specified bots; and a canonical tag indicates the preferred version among similar URLs. None is a "rank first" switch.
| Mechanism | Main role | What it doesn't do | First thing to check |
|---|---|---|---|
| XML sitemap | Lists important, preferred URLs to support discovery. | Not a binding indexing instruction or rankings guarantee. | Does it list real, final, publicly accessible URLs? |
robots.txt | Can restrict crawler access to paths on the website. | Does not reliably remove URLs from Google results or protect access. | Does it accidentally block important pages? |
rel="canonical" | Indicates the primary URL in a group of similar pages. | Not a redirect or an instruction that must always be followed. | Is the target accessible and consistent with internal links? |
noindex | Tells Google not to index a page it can access. | Does not encrypt or protect personal information; Google may not see it if crawling is blocked. | Is it used only where indexing is genuinely unwanted? |
| 301/308 redirect | Takes visitors and crawlers from an old URL to a new, permanent one. | Not a universal solution for 404s or simply a canonical hint. | Does the destination have relevant content and work correctly? |
The point is not to "set up" all five mechanisms but to choose the right solution for the site's actual problem. If a particular subpage isn't indexed, our twelve-status indexing diagnostic helps distinguish errors from intentional exclusions.
What do you want to achieve with this page?
Help it be discovered
A sitemap can signal important URLs, but doesn't guarantee indexing.
Avoid crawling everything
robots.txt controls crawler access, not guaranteed removal from search results.
Prefer one version
Canonical tags indicate the preferred version of duplicate or similar URLs.
XML sitemap: which of your pages should Google discover?
An XML sitemap is a machine-readable list of URLs that a website owner considers important and preferred. It does not replace normal internal linking: pages are also easier to find and use when a menu or relevant article links to them. Google is not obliged to crawl or index a URL just because it appears in a sitemap.
Does a small business website need a sitemap?
Google may discover the pages of a small, well-linked website without one. An XML sitemap can still be useful for maintenance: content management systems often generate it automatically, and its processing can be checked in Search Console. It becomes more important for larger, frequently updated or hard-to-discover sites. The aim isn't to list every technical URL variation.
What should it contain, and how do you check it?
Generally include final, useful, accessible canonical URLs intended for indexing. Don't add test versions, duplicate parameter URLs or old redirected addresses just to increase the count. The <lastmod> element should reflect a real, meaningful change if you supply it at all; automatically assigning a new date every day to unchanged content isn't helpful.
Minimal fictional XML example — not a file to deploy:
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>https://pelda.test/szolgaltatas/</loc>
<lastmod>2026-10-20</lastmod>
</url>
</urlset>The pelda.test domain is for documentation, and the date is fictional. Google ignores the sitemap's <priority> and <changefreq> elements.
Use the Google Search Console Sitemaps report to see whether Google processed the submitted file and reported errors. Processed does not mean every URL in the sitemap has been indexed. Google's guide to creating sitemaps provides more technical detail.
Robots.txt: is crawling permitted?
The robots.txt file is normally found at the root of a site, for example https://pelda.test/robots.txt URL. Its rules apply to crawlers and URL paths, so a mistaken Disallow rule could prevent important sections from being crawled. The file may also be a virtual response generated by your CMS, rather than a separately editable file on the hosting server.
Illustrative fragment, not a live configuration to copy:
User-agent: *
Disallow: /nem-fontos-pelda-utvonal/
Sitemap: https://pelda.test/sitemap.xmlThis only shows the format of the rules. The path in the example is not a real page on the Kaluzsa site. A general Disallow: / can block much more; never overwrite unknown production robots settings based on a template.
Why can't robots.txt hide confidential content?
Robots.txt can restrict crawler requests; it is not access control. A blocked URL can still become known through external links. Contracts, personal data and internal files require real access protection, not merely a request for bots to stay away.
One separate way to exclude a page from indexing is a noindex meta tag or HTTP header. But Googlebot must be able to access the page to see it. Combining a robots.txt block with a hidden noindex can therefore be an ineffective, misleading configuration. See Google's robots.txt documentation for details.
Fictional noindex example, not a setting for live Knowledge Base articles:
<meta name="robots" content="noindex">This simply illustrates the tag. No such restriction will be placed on Knowledge Base articles that are being prepared for later publication. Protecting staging access is a separate operational task.
Canonical: when the same content is accessible at several URLs
Online-shop filters, multiple technical URLs for one article, or www/non-www variants can produce very similar pages. The rel="canonical" signal tells search engines which version you prefer. However, Google may choose another URL if the content or other signals suggest it should.
Fictional HTML example for the preferred page's <head> section:
<link rel="canonical" href="https://pelda.test/szolgaltatas/">This does not redirect visitors. If you permanently replace an old address and want visitors moved to the new location, an appropriate 301 or 308 permanent redirect may be better. The two approaches should not be confused.
Why isn't adding the tag enough?
Check that the canonical destination works, matches the content, appears as the preferred version in the sitemap and receives your internal links. Ideally, redirects, canonical tags and internal links should follow one consistent scheme. Don't make the homepage canonical for every unrelated service page: that sends conflicting signals about important individual content.
What if Google chooses a different canonical?
Open Search Console's URL Inspection tool and compare your declared canonical with Google's selected URL. Review the two pages' content, internal links, redirects and sitemap. It is not necessarily an error if Google selects another primary version of a genuine duplicate. As Google's canonical documentation explains, the tag is a preference signal, not a command.
When it comes to your business
You don't have to set these technical signals alone
Correct sitemaps, robots.txt and canonical tags matter, but a poorly implemented change can exclude valuable pages. Important business URLs are worth reviewing professionally.
Canonical, noindex, robots or 301: when should you use each?
Use these questions when talking to your developer. A problem rarely has one simple technical button: the content, user journey and business objective matter alongside any chosen mechanism.
- New, important page is hard to discover? Check internal links, the sitemap, and whether the URL is genuinely public, preferred and intended for indexing.
- Crawlers cannot access it? Check robots and HTTP rules. A sitemap cannot override a crawl restriction.
- Public page that should not appear in Google's index? In suitable cases,
noindexmay be appropriate. Protect truly sensitive information with actual access controls first. - Several nearly identical URLs? Review canonical tags, content differences, internal links and, if necessary, redirects.
- An old URL has permanently moved to a relevant new address? A 301/308 permanent redirect may be suitable for an actual move.
- Sitemap processed, but one URL missing? Inspect the individual URL's status; resubmitting the sitemap is not an automatic fix.
Five business situations, five different questions
The examples below occur on the site of the same fictional local service business. They are not real Kaluzsa projects. The aim is to frame useful questions for your developer, not to blindly apply code fragments.
| Situation | Next question or action | Mistake to avoid |
|---|---|---|
| Publishing a new, separate service page | Does it have a meaningful internal link, the right final URL and a correct sitemap entry? | "If it's in the XML, Google must index it." |
| One page also loads with parameterised URLs | Do the canonical, internal links and preferred URL agree? | Treating every parameter variation as a separate SEO service page. |
| Public thank-you page should not appear in search | Is a noindex needed on the crawlable page, and is it free of sensitive data? | Using robots.txt to "encrypt" personal details. |
| Permanently replacing an old service URL | Is the new destination genuinely relevant to former visitors? Is a 301/308 required? | Only adding a canonical tag while the old page still loads. |
| Same page accessible via www and non-www | Which is the final version? Do redirects, canonicals, the sitemap and navigation align? | Calling the alternative version a Google penalty or layering contradictory blocks. |
Three quick checks and the most dangerous conflicts
1. Do all technical signals refer to the same page?
Don't list an important service page in your sitemap while blocking it through robots rules and adding noindex to it. Not every discrepancy is necessarily an error, but contradictions between intended behaviour and actual configuration need investigating.
2. Do internal links point to the real final address?
If your menu consistently links to an old URL while a new one is canonical, you're giving visitors and crawlers an unnecessary detour. Likewise, don't keep redirected old URLs in a sitemap when a suitable preferred version exists.
3. Are you trying to use robots.txt instead of data security?
A robots directive cannot stop someone opening a public URL directly. Truly private materials need login or another access-control method. Incorrect crawler blocking can even stop Google reading a noindex signal.
Never change live robots.txt, sitemaps or redirects automatically based on an article. They can affect crawling across entire groups of service pages. Ask for URL-level checks first and keep a rollback plan.
When is a review enough, and when should you call a specialist?
If important pages work, the CMS creates a correct sitemap, pages are internally linked and Search Console reports no significant obstacles, there's no need to invent more technical rules. Maintenance and periodic review may be more useful than over-engineered SEO changes.
If a site accidentally goes live with widespread noindex tags, develops complex redirects, produces an incorrect canonical cluster or blocks crawling of important pages, a technical investigation is justified. The place of technical SEO within the wider work is explained in a separate article. Where necessary, request an SEO review with exact URLs and errors; indexing or rankings still cannot be guaranteed.
Frequently asked questions
Will Google automatically index pages after I submit a sitemap?
No. A sitemap supports discovery; it is not a binding indexing instruction or rankings guarantee.
Can my business website work without a sitemap?
Yes. Google can discover a small, well-linked site without one, though a correct CMS-generated sitemap can still help.
Does robots.txt remove pages from Google results?
It mainly controls crawling. Blocked URLs may still be discovered elsewhere; exclusion requires other tools, and sensitive data needs access protection.
Does every page need a self-referencing canonical?
Self-referencing canonicals are common, useful practice but don't guarantee indexing or replace a sound URL structure.
Can I combine noindex with a robots block?
If robots.txt prevents Googlebot from fetching the page, it may never see the noindex tag. This is therefore an unreliable way to remove content from the index.
Where can I see Google's selected canonical URL?
Use URL Inspection for the relevant Search Console property. Our Search Console for beginners guide explains how to use it.
Sources and professional background
Open official sources
Sitemap basics ↗ — Google Search Central
Create and submit a sitemap ↗ — Search Central
Robots.txt guide ↗ — Search Central
Using noindex ↗ — Search Central
Canonical signals ↗ — Search Central
Related Knowledge Base articles
Next step · Kaluzsa
Don't let technical settings make an important page invisible
If you don't know why a page appears or doesn't appear in Google, let's discuss a technical SEO review.