[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-post-robots-txt-lists-pages-you-wanted-hidden":3},{"slug":4,"title":5,"body":6,"summary":7,"tags":8,"author":14,"cover_url":15,"published_at":16,"seo_title":17,"seo_description":18,"reading_minutes":19,"related":20},"robots-txt-lists-pages-you-wanted-hidden","Your robots.txt lists the pages you wanted hidden: protecting wholesale prices, VIP discounts and unreleased products on a small shop","\u003Cp>A post doing the rounds among shop owners this week makes a simple point: anyone can type \u003Ccode>\u002Frobots.txt\u003C\u002Fcode> after your shop's address, read the lines that say \u003Ccode>Disallow\u003C\u002Fcode>, and open those pages directly. If one of them is your wholesale price list, an influencer discount page or the collection you are saving for Black Friday, you have not hidden it. You have pointed at it.\u003C\u002Fp>\n\u003Cfigure data-post-media=\"6add495b6f13dd01b60bf909\">\u003Cvideo src=\"https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6add495c6f13dd01b60bf90f-0-bbb86d36.mp4\" autoplay muted loop playsinline preload=\"metadata\">\u003C\u002Fvideo>\u003C\u002Ffigure>\n\u003Cp>This is not a hack and not a platform bug. It is how robots.txt has always worked, and both the standard and Google say so plainly. The fix is quick and costs nothing.\u003C\u002Fp>\n\u003Ch2>What robots.txt actually does\u003C\u002Fh2>\n\u003Cp>robots.txt is a small text file that tells search engine crawlers which parts of a site to skip. The internet standard that defines it (RFC 9309) requires it to sit at \u003Ccode>\u002Frobots.txt\u003C\u002Fcode> at the top of your site, which is exactly why anyone can find it. The same standard warns that it is \"not a substitute for valid content security measures\" and that listing paths in it \"exposes them publicly\".\u003C\u002Fp>\n\u003Cfigure data-post-media=\"6add40026f13dd01b60bf687\">\u003Cimg src=\"https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6add40026f13dd01b60bf68c-0-106600a4.png\" alt=\"A robots.txt file acts more like a signpost than a locked door.\" loading=\"lazy\">\u003Cfigcaption>A robots.txt file acts more like a signpost than a locked door.\u003C\u002Ffigcaption>\u003C\u002Ffigure>\n\u003Cp>Google's own guide is just as direct: robots.txt \"is not a mechanism for keeping a web page out of Google\". It also points out three things shop owners often miss:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>It only works if the crawler chooses to obey it. Google and other respectable crawlers do; others might not.\u003C\u002Fli>\n\u003Cli>A page you block can still appear in Google if another site links to it. Google just won't read its content.\u003C\u002Fli>\n\u003Cli>To keep a page out of search, Google says to use \u003Ccode>noindex\u003C\u002Fcode> or password-protect the page.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>So robots.txt is for managing crawler traffic: keeping bots out of your cart, checkout and endless filter combinations. It was never a lock.\u003C\u002Fp>\n\u003Ch2>The 10-minute check\u003C\u002Fh2>\n\u003Col>\n\u003Cli>\u003Cstrong>Open your own file.\u003C\u002Fstrong> In a browser, go to \u003Ccode>https:\u002F\u002Fyourshop.com\u002Frobots.txt\u003C\u002Fcode>.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Read every Disallow line\u003C\u002Fstrong> and ask: would I mind a competitor or a bargain hunter opening this? Lines like these are the ones to worry about:\n\u003Cpre class=\"code-block\">\u003Ccode class=\"hljs\">User-agent: *\nDisallow: \u002Fpages\u002Fwholesale-prices\nDisallow: \u002Fcollections\u002Fvip-40-off\nDisallow: \u002Fcollections\u002Fblack-friday-preview\nDisallow: \u002Fpages\u002Finfluencer-codes\u003C\u002Fcode>\u003C\u002Fpre>\nLines for \u003Ccode>\u002Fcart\u003C\u002Fcode>, \u003Ccode>\u002Fcheckout\u003C\u002Fcode>, \u003Ccode>\u002Faccount\u003C\u002Fcode> or search pages are normal crawl control and fine to keep.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Open each worrying path in a private window\u003C\u002Fstrong>, logged out. If the page loads with prices, codes or products on it, it is public to anyone, whatever robots.txt says.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Search for it.\u003C\u002Fstrong> Type \u003Ccode>site:yourshop.com wholesale\u003C\u002Fcode> (or the page's name) into Google. If it shows up, someone linked to it and Google listed the address anyway.\u003C\u002Fli>\n\u003Cli>\u003Cstrong>Write down\u003C\u002Fstrong> each page that loaded, what is on it, and who is supposed to see it. That list is your to-do for the next section.\u003C\u002Fli>\n\u003C\u002Fol>\n\u003Ch2>Where each kind of page should live instead\u003C\u002Fh2>\n\u003Ch3>Wholesale and trade price lists\u003C\u002Fh3>\n\u003Cp>These are for known customers, so put them behind a customer login. Most shop platforms let you restrict a page or a price list to logged-in customers or a tagged customer group; use whichever your platform offers, and test it in a private window afterwards. If your platform cannot do that, send the price list as a file to approved trade buyers instead of keeping it on the site.\u003C\u002Fp>\n\u003Cfigure data-post-media=\"6add40026f13dd01b60bf691\">\u003Cimg src=\"https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6add40026f13dd01b60bf696-0-ffb2da9e.png\" alt=\"Sensitive pages should be protected by actual authentication, not just crawler instructions.\" loading=\"lazy\">\u003Cfigcaption>Sensitive pages should be protected by actual authentication, not just crawler instructions.\u003C\u002Ffigcaption>\u003C\u002Ffigure>\n\u003Ch3>VIP and influencer discount pages\u003C\u002Fh3>\n\u003Cp>A page that exists only to show a discount is a discount for everyone who finds it. Two safer patterns:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Give each influencer or VIP their own code with a usage limit and an end date, and don't publish a page that lists the codes.\u003C\u002Fli>\n\u003Cli>If you need a landing page, keep it public but harmless (the offer, no stacked codes), and put the real benefit in the code itself.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch3>Unreleased products and holiday previews\u003C\u002Fh3>\n\u003Cp>Keep them unpublished or as drafts until launch day. A collection that is live but \"hidden\" from your menu and listed in robots.txt is live. If your team needs to preview it, use your platform's preview link or a password-protected staging copy.\u003C\u002Fp>\n\u003Ch3>Test collections and old campaigns\u003C\u002Fh3>\n\u003Cp>Unpublish or delete them. If an old campaign page has to stay reachable for people who bookmarked it, make sure it no longer carries a live discount.\u003C\u002Fp>\n\u003Ch2>When you want a page public but out of Google\u003C\u002Fh2>\n\u003Cp>Some pages are fine for people to see but you would rather not have them in search results, such as a thank-you page or a printable flyer. For those, Google's tool is \u003Ccode>noindex\u003C\u002Fcode>, either as a tag in the page's head or as an \u003Ccode>X-Robots-Tag: noindex\u003C\u002Fcode> response header (that one also works for PDFs). Many shop themes and SEO apps have a \"hide from search engines\" switch that adds it for you.\u003C\u002Fp>\n\u003Cp>One catch that trips people up: Google says \u003Ccode>noindex\u003C\u002Fcode> only works if the page is \u003Cstrong>not\u003C\u002Fstrong> blocked in robots.txt, because the crawler has to visit the page to see the tag. So when you add noindex to a page, remove its Disallow line.\u003C\u002Fp>\n\u003Ch2>Clean up the file itself\u003C\u002Fh2>\n\u003Cp>Once the sensitive pages are behind a login, unpublished or noindexed, delete their Disallow lines. They no longer do anything useful, and leaving them in place keeps advertising the addresses. Keep the lines that manage crawler traffic (cart, checkout, search, filters). If your platform generates robots.txt for you and you have never edited it, you will probably find nothing to remove; the risk comes from lines someone added by hand or an app added for you.\u003C\u002Fp>\n\u003Ch2>A habit for the holidays\u003C\u002Fh2>\n\u003Cp>Before each big sale, repeat the check: open \u003Ccode>\u002Frobots.txt\u003C\u002Fcode>, open every non-obvious path in a private window, and search your own shop's name with the words \u003Ccode>wholesale\u003C\u002Fcode>, \u003Ccode>vip\u003C\u002Fcode> and \u003Ccode>preview\u003C\u002Fcode>. It takes ten minutes and saves you from launching your Black Friday prices a week early.\u003C\u002Fp>\n\u003Cfigure data-post-media=\"6add40026f13dd01b60bf69b\">\u003Cimg src=\"https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6add40026f13dd01b60bf6a0-0-43ce22ef.png\" alt=\"Establishing a pre-sale security routine prevents accidental leaks of holiday pricing.\" loading=\"lazy\">\u003Cfigcaption>Establishing a pre-sale security routine prevents accidental leaks of holiday pricing.\u003C\u002Ffigcaption>\u003C\u002Ffigure>\n\u003Ch2>Sources\u003C\u002Fh2>\n\u003Cul>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdevelopers.google.com\u002Fsearch\u002Fdocs\u002Fcrawling-indexing\u002Frobots\u002Fintro\">Google Search Central: Introduction to robots.txt\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fdevelopers.google.com\u002Fsearch\u002Fdocs\u002Fcrawling-indexing\u002Fblock-indexing\">Google Search Central: Block Search indexing with noindex\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fwww.rfc-editor.org\u002Frfc\u002Frfc9309.html\">RFC 9309: Robots Exclusion Protocol\u003C\u002Fa>\u003C\u002Fli>\n\u003Cli>\u003Ca href=\"https:\u002F\u002Fx.com\u002Fpaulpagepilot\u002Fstatus\u002F2112877309086621950\">The post that started the conversation (X)\u003C\u002Fa>\u003C\u002Fli>\n\u003C\u002Ful>","robots.txt is a public file, and every Disallow line in it is a signpost. A 10-minute check of your own shop's robots.txt, and where wholesale price lists, VIP discount pages and test collections should live instead: behind a login, a password, or unpublished.",[9,10,11,12,13],"seo","shop-security","robots-txt","black-friday","ai-assisted","Async.Shop","https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6add40006f13dd01b60bf681-0-868b4326.png","2026-10-25T02:19:11.65Z","robots.txt shows the pages you wanted hidden: fix it on your shop","robots.txt is public, and Disallow lines point at your wholesale prices and VIP discounts. A 10-minute check and where those pages should live instead.",5,[21,33,43],{"slug":22,"title":23,"type":24,"summary":25,"tags":26,"author":14,"cover_url":30,"published_at":31,"updated_at":32},"shop-your-own-store-like-an-ai-agent-pre-black-friday-test-script","Shop your own store the way an AI agent would: a pre-Black-Friday test script","blog","Ask ChatGPT, Claude and Copilot your customers' top questions about your shop, log every wrong or missing answer, and fix the source before Black Friday.",[27,28,29,12,13],"ecommerce","shopify","ai-agents","https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6ae48cd437fa83c1536d8c1a-0-09c57ab4.png","2026-10-31T04:02:11.962Z","2026-10-31T04:02:11.963Z",{"slug":34,"title":35,"type":24,"summary":36,"tags":37,"author":14,"cover_url":41,"published_at":42,"updated_at":42},"shopify-mcp-servers-shopper-ai-what-to-check","What Shopify's MCP servers let a shopper's AI do in your store, and what to check first","Shopify's docs now split what a shopper's AI assistant can do in your store across several servers. A plain-language map of what each exposes, what the docs leave unsaid, and which policy pages to tidy before an assistant reads them aloud.",[28,38,39,40,13],"ai-shopping-agents","mcp","store-policies","https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6ae1ba761edd7a39ec513077-0-3172c55c.png","2026-10-29T03:19:16.591Z",{"slug":44,"title":45,"type":24,"summary":46,"tags":47,"author":14,"cover_url":52,"published_at":53,"updated_at":54},"etsy-bans-ai-patterns-recipes-what-sellers-must-change","Etsy now bans AI-made patterns, recipes and instructions: what pattern and printable sellers must change, and what AI use is still allowed","On 22 October Etsy banned AI-generated patterns, recipes and instructions, banned bot-made listings and asked for one unedited photo of the real item. A checklist for pattern, recipe and printable sellers: what is at risk, what to fix first, and what AI use is still fine.",[48,49,50,51,13],"etsy","marketplace-policy","digital-products","product-photos","https:\u002F\u002Fmedia.stufio.com\u002Fmedia\u002Fasyncshop\u002Fmediagen\u002F6a\u002F6ae038cf1edd7a39ec50fa8a-0-03819514.png","2026-10-27T07:25:13.068Z","2026-10-27T07:25:13.069Z"]