Definition
robots.txt is a plain text file at the root of a host, such as https://example.com/robots.txt. It tells crawlers which paths they may fetch. Well-behaved crawlers read it before they request anything else on the site.
User-agent: *
Disallow: /cart/
Allow: /cart/help
Sitemap: https://example.com/sitemap.xmlCrawling is not indexing
robots.txt blocks crawling, not indexing. A blocked URL can still be indexed if other pages link to it, usually without a description, because the crawler could not read the page. To keep a page out of results, use noindex and leave the page crawlable. If you block it in robots.txt, the crawler never sees the noindex. The file is also not security: anyone can read it, and not every bot obeys it.
Why it matters
Blocking low-value paths, like carts or endless filter URLs, saves crawl effort for the pages that count, which helps on large sites. See crawl budget. A stray Disallow: / left from staging can hide a whole site, so check the file after every launch. The Sitemap line also points crawlers to your XML sitemap.
Where to see it
Open the file in a browser at /robots.txt. Search Console has a robots.txt report in its settings that shows the files Google found and any problems.
Ask mowteng
If a page is not indexed, ask the assistant to inspect it with gsc_inspect_url and say whether crawling was blocked. For a whole site, use the index coverage audit.
Inspect https://example.com/private/report and tell me whether Google is blocked by robots.txt or whether there is a noindex tag.