Groups found
| Group | User-agent lines | Rules |
|---|---|---|
| R1 | * | Disallow: /search |
| R2 | GPTBot, OAI-SearchBot | Disallow: / |
| R3 | Google-Extended | Allow: / |
Other line kept as is: Sitemap: https://northwind-recipes.example/sitemap.xml
Token check
Google's robots.txt guide says a user agent "can match only one rule set" (the first, most specific group that matches it). So a crawler with its own group does not also follow the * group.
| Token | Group used | Current rules | Your goal | Result |
|---|---|---|---|---|
| OAI-SearchBot | R2 (named) | Disallow: / | allow everything except /search | Conflict |
| GPTBot | R2 (named) | Disallow: / | block all | Match |
| Google-Extended | R3 (named) | Allow: / | block /premium/ only | Conflict |
| CCBot | R1 (falls back to *) | Disallow: /search | block all | Conflict |
Conflicts
- OAI-SearchBot (R2): currently 'Disallow: /', so it is blocked from the whole site. OpenAI says "OAI-SearchBot is for search" and that "Sites that are opted out of OAI-SearchBot will not be shown" in ChatGPT search results. Your goal is to show in ChatGPT search, so this conflicts. OpenAI also says "Each setting is independent of the others", so you can allow OAI-SearchBot and still block GPTBot.
- Google-Extended (R3): currently 'Allow: /'. Your goal is to block /premium/ only. Google describes Google-Extended as a robots.txt product token with no separate HTTP user agent, so the rule goes in a group named Google-Extended.
- CCBot: no group names it, so it falls back to R1 (*), which only blocks /search. Your goal is to block all. CCBot's own behavior is not confirmed in my sources; I keep the token exactly as you wrote it.
GPTBot (R2, 'Disallow: /') already matches "block all".
Counts
| Item | Count |
|---|---|
| Tokens checked | 4 |
| Match | 1 |
| Conflict | 3 |
| Groups in current file | 3 |
Proposed robots.txt (held)
HELD: not published until you approve this exact file. Every existing unrelated line (the * group and the Sitemap line) is kept. GPTBot and OAI-SearchBot are split into separate groups so they can have different rules.
User-agent: *
Disallow: /search
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /search
User-agent: Google-Extended
Disallow: /premium/
User-agent: CCBot
Disallow: /
Sitemap: https://northwind-recipes.example/sitemap.xml
Two notes: OAI-SearchBot has its own group, so it needs its own 'Disallow: /search' line to keep /search out; Google says rules are case-sensitive, so '/premium/' must match your real path's case. robots.txt states crawl preferences; it does not remove content already collected or indexed.
Questions for you
- Is the premium section's path exactly /premium/ (same letters and case)?
- Do you want other AI crawlers listed? I can only add tokens you name, and their behavior is not confirmed in my sources.
Needs your decision
- Approve the proposed robots.txt above before it is edited or published (HELD).