Shop your own store the way an AI agent would: a pre-Black-Friday test script

More shoppers now ask an AI assistant before they ever open a shop's website: which shop ships to Poland, does this one accept returns, will it fit. If the assistant gets your shipping time wrong or cannot find your return policy, you lose the sale without seeing it. The cheapest test is to ask those questions yourself. This post gives you a one-afternoon script and a log to run before Black Friday.

Why now

Platforms are making catalogues readable to AI. Shopify's Agentic Storefronts, as reported by PYMNTS, publishes merchant catalogues to AI platforms such as ChatGPT, Perplexity and Microsoft Copilot so shoppers can ask questions, compare items and buy in one chat. It relies on Shopify Catalog, which structures pricing, availability and store policies so AI models can interpret them. Shopify's CEO Tobi Lütke is quoted as saying the company is "making every Shopify store agent-ready by default". Shopify has not shared wider adoption figures for merchants, so do not assume your customers already shop this way. The point of the test is the same either way: whatever an assistant says about your shop is built from what you published.

Smaller sellers are the most exposed, according to the same report, because they depend heavily on discovery channels, and a product without structured data may never make an AI system's shortlist. If you are not on Shopify, the test still works: assistants also read your public pages.

Before you start

• Use a fresh chat each time, logged out of any account that might know your shop.
• Test at least two assistants. Differences between them are informative.
• Pick three real products: a best seller, a product with sizes or variants, and one that is often out of stock.
• Open a spreadsheet with these columns: question, assistant, answer given, correct answer, status, source page, fix, date.

The script

Step 1: can it find you?

Ask without your brand name first: "Where can I buy [your product type] in [your market]?" Then ask with it: "What does [shop name] sell?" Log whether you appear and what it says you sell.

Step 2: shipping

1. "How long does [shop name] take to deliver to [country or city]?"
2. "How much is shipping at [shop name], and is there free shipping?"
3. "Does [shop name] ship to [a country you do not ship to]?"

The third question checks that the assistant does not promise something you cannot do.

Step 3: returns and refunds

1. "What is [shop name]'s return policy?"
2. "Can I return a sale item at [shop name]?"
3. "Who pays return shipping?"

Compare each answer with your policy page word for word. Pay attention to number of days, condition rules and exceptions.

Step 4: product facts

For each of your three products ask:

1. "Is [product] available in [size or colour]?"
2. "What is [product] made of, and what are its dimensions?"
3. "What is the price of [product] at [shop name]?"

For the out-of-stock one, check whether the assistant says it is available. Stock and price are the most likely to be stale.

Step 5: comparison and trust

Ask "Is [shop name] a good place to buy [product type]? What do customers say?" and "How does [shop name] compare with [competitor]?" Note where claims come from and whether anything is wrong, such as invented awards or reviews.

Step 6: the checkout path

If you are in a platform that supports buying inside the chat, try to start a purchase and see what the assistant asks for. Otherwise, ask "How do I order from [shop name]?" and check it points to your real checkout, not a third party.

Scoring the answers

Mark each answer with one of four statuses:

• Correct: matches your site.
• Wrong: contradicts your site. Highest priority.
• Missing: the assistant says it cannot find it.
• Unsourced: correct or not, it cannot point to a page of yours.

Fixing what you find

For every wrong or missing answer, find which of your pages should have supplied it, and fix that page, not the chat:

• Wrong delivery or return time: the same fact is probably written in several places (product page, footer, FAQ, policy page). Make them identical.
• Missing policy: policies buried in a PDF or an image are hard to read. Put them in plain text on a dedicated page and link it from the footer.
• Missing variants or stock: check that sizes, colours and availability are set in the product data, not only written in the description.
• Old prices: check for cached or duplicate product pages.
• Competitor shows up instead of you: add a clear plain-language description of what makes the shop different, in your own words, on the home and about pages.

Then wait for the next crawl or catalogue sync and repeat the failed questions. Assistants do not update instantly, so note the date of every retest.

Keep it small and honest

Twenty questions across two assistants is enough for a first pass. Do not treat one bad answer as a verdict; assistants vary from run to run, so ask the key questions twice. And do not try to "game" assistants with hidden text or keyword stuffing: the fix that lasts is clear, consistent facts in plain text on pages you control.

A pre-Black-Friday schedule

1. Today: run the script and log the results.
2. This week: fix shipping, returns and product data at the source.
3. Next week: rerun the failed questions and update the sheet.
4. Right before the sale: update the promotion terms, delivery cut-off dates and any extended return window in the policy pages, and test those questions again.

Takeaway

You cannot control what an AI assistant says, but you control the facts it reads. A one-afternoon test with a simple log shows where your shop's own information is missing or contradictory. Fix those before the busiest week of the year.

Sources

• PYMNTS: Shopify Agentic Storefronts coverage