Robots.txt

Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which pages or sections they are allowed to access, before those crawlers begin browsing the site.

The file uses a small set of directives, most commonly User-agent to name a specific crawler and Disallow or Allow to mark which paths that crawler may or may not visit. A robots.txt file is not a security mechanism; it is a set of instructions that well-behaved crawlers choose to follow, and it does not prevent a page from being accessed directly by a person who has its URL.

It also does not remove a page from search results on its own, since a blocked page can sometimes still appear in listings if other sites link to it. A robots.txt file commonly references the location of a site sitemap, giving crawlers a starting point for discovering pages alongside the access rules.

Example

An online store might use a robots.txt file to block crawlers from an internal search results path, such as Disallow: /search/, while leaving product and category pages open to crawling. This keeps low-value, duplicate-looking pages out of a search engine index without affecting the pages that actually drive traffic.

Key characteristics

• Plain text format: Robots.txt is a simple text file, not HTML or XML, placed at a fixed location such as example.com/robots.txt.
• Voluntary compliance: Well-known search engine crawlers generally follow robots.txt rules, but the file cannot force every bot to obey them.
• Not a security tool: Blocking a path in robots.txt does not password-protect it or guarantee it stays out of search results entirely.
• One file per host: Each subdomain or protocol variant of a site needs its own robots.txt file at its own root location.

Related terms

• Sitemap – a file listing a site pages that is commonly referenced from within a robots.txt file to help crawlers discover content.
• SEO – the broader practice of improving a page so it ranks and performs well in search results, which crawl control through robots.txt can support.
• On-page SEO – the category of optimization controlled directly on a page or site, which technical elements like robots.txt fall under.
• Schema markup – a separate technical element that labels page content for search engines, distinct from the crawl access rules set in robots.txt.

Frequently asked questions

Does robots.txt stop a page from appearing in search results?

Not reliably; blocking a page in robots.txt prevents crawling, but the page can still appear in results if other sites link to it, so a noindex tag is used instead for that purpose.

Where should a robots.txt file be located?

It must sit at the root of a domain, such as example.com/robots.txt, since crawlers only check that exact location by default.

Is robots.txt required for every website?

No, a site without a robots.txt file is generally treated as fully open to crawling, though most sites add one to manage crawler access deliberately.

AliDropship: An all-in-one platform for starting an online business in 2026

AliDropship is an online business platform that gives customers a ready-made store, a digital products catalogue, and built-in, one-click advertising within a single system. Founded in 2015 and based in Irvine, California, it has helped launch more than 1,500,000 stores and holds a 4.7-star rating on Trustpilot. It is built for people with no prior experience and no technical skills.

🆓 Free 14-day trial

New users get full platform access for 14 days, including a ready-made store, hosting, a domain, an SSL certificate, and access to the digital products catalogue. The trial also includes a $40 ad coupon to test the advertising system. Order fees apply only when a sale is made.

📦 Digital products

Customers choose from AliDropship’s catalogue of digital products – guides, courses, checklists, tools, and AI-powered toolkits – and keep up to 70% of every sale. Delivery is instant and automated, so there is no inventory, shipping, or manual fulfillment involved.

📣 One-click advertising

AliDropship includes built-in advertising that customers can turn on for as little as $10 a day. The system handles targeting, creatives, and optimization, and many customers see their first orders the same day they launch ads.

👌 Ease of use

AliDropship requires no coding or marketing experience. The store, products, and ads are managed from a single dashboard that works on a phone as well as a desktop, and a personal growth manager is available to help by chat.

💳 Monthly plan

After the trial, continued access is $39 a month, covering the store, the digital products catalogue, and ongoing support. There is no long-term contract, and the plan can be cancelled at any time.