ChatGPT's Product Recommendations Change 80.2% When Search is Enabled vs Disabled (Study of 20,000 Responses)
A few months ago Rand Fishkin of SparkToro asked an interesting question on LinkedIn about how LLMs choose answers, and how much impact the citations have on the recommendations.
This sent me down a rabbit hole, analyzing 20,000 ChatGPT responses to better understand how much ChatGPT’s product recommendations change when search is enabled vs disabled.
Key Findings
- When search is enabled in ChatGPT, the product recommendations are 80.2% different. That means only 19.8% of products recommended when search was disabled were also recommended when search was enabled.
- Even for the products that were recommended 100% of the time when search was disabled, only 15.8% also appeared when search was enabled.
- ChatGPT included 5.2 products per response when search is enabled, and 6.2 products per response with search disabled.
- After running each ChatGPT prompt 10 times, on average ChatGPT returned a total of 19 unique products per prompt with search enabled and 21.8 unique products per prompt with search disabled.
We also looked to see how much overlap there was for products that were recommended more often vs less often. By running each prompt 10x, we could see the recommendation frequency (Visibility Score) for each product.
| Visibility Score | # Products w/ Search | # Products w/o Search | Overlap Count | % Overlap |
|---|---|---|---|---|
| >10% | 19,028 | 21,786 | 4,037 | 19.8% |
| >20% | 9,465 | 10,911 | 2,045 | 20.1% |
| >30% | 6,399 | 7,501 | 1,405 | 20.2% |
| >40% | 4,675 | 5,708 | 1,026 | 19.8% |
| >50% | 3,480 | 4,467 | 808 | 20.3% |
| >60% | 2,580 | 3,535 | 635 | 20.8% |
| >70% | 1,877 | 2,697 | 464 | 20.3% |
| >80% | 1,350 | 2,017 | 324 | 19.2% |
| >90% | 846 | 1,391 | 192 | 17.2% |
| 100% | 437 | 756 | 94 | 15.8% |
Interestingly enough, the overlap was the lowest for products that were recommended 100% of the time. Only 15.8% of products that were recommended 100% of the time appeared with both search enabled and disabled.
I would’ve assumed the opposite trend. If a product was recommended 100% of the time, I would’ve guessed it would likely still be a strong recommendation even with ChatGPT search enabled. That was not the case.
Methodology
Here’s a summary of how this study was conducted.
- We created a list of 1,000 prompts someone might ask when looking for product recommendations. Example: “What is the best protein powder?”
- Each prompt was run 10x each with search disabled to see all the unique products that were recommended. This also allowed us to see the % frequency each product was recommended in ChatGPT. If a product only appeared in 3/10 runs, it was given a “Visibility Score” of 30%.
- Each prompt was also run 10x each with search enabled.
- We canonicalized the product names so a product with multiple product names was only counted as a single product. For example, “Optimum Nutrition Gold Standard Whey”, “ON Gold Standard Whey”, and “Optimum Nutrition - Gold Standard Whey” were all counted as the same product.
- We analyzed the data to see exactly how many products were recommended both with search enabled vs disabled.
Understanding Search Enabled vs Disabled within ChatGPT
When search is disabled in ChatGPT and it doesn’t search the web, it relies exclusively on all the articles, books, and information it was initially trained on some time ago. But when you enable search, ChatGPT augments its initial training knowledge with relevant articles, product pages, and other info on the web to help improve the answer quality.
Given that the recommended products differ by 80.2% when search is enabled, this tells us that the retrieved sources have a huge impact on product recommendations. This also matches what we found in another study of 10,000 ChatGPT responses which examined the correlation between ChatGPT’s product recommendations and being mentioned in the cited sources.
The more often a product was mentioned in the cited sources, the more often it was recommended (0.4 Pearson correlation). This uses the same methodology we described before of running each prompt 10x to see the % frequency (Visibility Score) each product was recommended.
What This Means for eCommerce Store Owners
The takeaway is simple: Getting mentioned in the cited sources is one of the most impactful things you can do to get recommended more often in ChatGPT.
Trying to influence the training data by scattering brand mentions across the web is a bit of a lost cause. Instead, focus on getting mentioned in the articles that are already being cited for the prompts you want to appear in.