A post doing the rounds among shop owners this week makes a simple point: anyone can type /robots.txt after your shop's address, read the lines that say Disallow, and open those pages directly. If one of them is your wholesale price list, an influencer discount page or the collection you are saving for Black Friday, you have not hidden it. You have pointed at it.
This is not a hack and not a platform bug. It is how robots.txt has always worked, and both the standard and Google say so plainly. The fix is quick and costs nothing.
What robots.txt actually does
robots.txt is a small text file that tells search engine crawlers which parts of a site to skip. The internet standard that defines it (RFC 9309) requires it to sit at /robots.txt at the top of your site, which is exactly why anyone can find it. The same standard warns that it is "not a substitute for valid content security measures" and that listing paths in it "exposes them publicly".
Google's own guide is just as direct: robots.txt "is not a mechanism for keeping a web page out of Google". It also points out three things shop owners often miss:
- It only works if the crawler chooses to obey it. Google and other respectable crawlers do; others might not.
- A page you block can still appear in Google if another site links to it. Google just won't read its content.
- To keep a page out of search, Google says to use
noindexor password-protect the page.
So robots.txt is for managing crawler traffic: keeping bots out of your cart, checkout and endless filter combinations. It was never a lock.
The 10-minute check
- Open your own file. In a browser, go to
https://yourshop.com/robots.txt. - Read every Disallow line and ask: would I mind a competitor or a bargain hunter opening this? Lines like these are the ones to worry about:
Lines forUser-agent: * Disallow: /pages/wholesale-prices Disallow: /collections/vip-40-off Disallow: /collections/black-friday-preview Disallow: /pages/influencer-codes/cart,/checkout,/accountor search pages are normal crawl control and fine to keep. - Open each worrying path in a private window, logged out. If the page loads with prices, codes or products on it, it is public to anyone, whatever robots.txt says.
- Search for it. Type
site:yourshop.com wholesale(or the page's name) into Google. If it shows up, someone linked to it and Google listed the address anyway. - Write down each page that loaded, what is on it, and who is supposed to see it. That list is your to-do for the next section.
Where each kind of page should live instead
Wholesale and trade price lists
These are for known customers, so put them behind a customer login. Most shop platforms let you restrict a page or a price list to logged-in customers or a tagged customer group; use whichever your platform offers, and test it in a private window afterwards. If your platform cannot do that, send the price list as a file to approved trade buyers instead of keeping it on the site.
VIP and influencer discount pages
A page that exists only to show a discount is a discount for everyone who finds it. Two safer patterns:
- Give each influencer or VIP their own code with a usage limit and an end date, and don't publish a page that lists the codes.
- If you need a landing page, keep it public but harmless (the offer, no stacked codes), and put the real benefit in the code itself.
Unreleased products and holiday previews
Keep them unpublished or as drafts until launch day. A collection that is live but "hidden" from your menu and listed in robots.txt is live. If your team needs to preview it, use your platform's preview link or a password-protected staging copy.
Test collections and old campaigns
Unpublish or delete them. If an old campaign page has to stay reachable for people who bookmarked it, make sure it no longer carries a live discount.
When you want a page public but out of Google
Some pages are fine for people to see but you would rather not have them in search results, such as a thank-you page or a printable flyer. For those, Google's tool is noindex, either as a tag in the page's head or as an X-Robots-Tag: noindex response header (that one also works for PDFs). Many shop themes and SEO apps have a "hide from search engines" switch that adds it for you.
One catch that trips people up: Google says noindex only works if the page is not blocked in robots.txt, because the crawler has to visit the page to see the tag. So when you add noindex to a page, remove its Disallow line.
Clean up the file itself
Once the sensitive pages are behind a login, unpublished or noindexed, delete their Disallow lines. They no longer do anything useful, and leaving them in place keeps advertising the addresses. Keep the lines that manage crawler traffic (cart, checkout, search, filters). If your platform generates robots.txt for you and you have never edited it, you will probably find nothing to remove; the risk comes from lines someone added by hand or an app added for you.
A habit for the holidays
Before each big sale, repeat the check: open /robots.txt, open every non-obvious path in a private window, and search your own shop's name with the words wholesale, vip and preview. It takes ten minutes and saves you from launching your Black Friday prices a week early.
