# Since we added our own lines for the AI bots, they reach areas that were blocked | Illia BizTech

> After a dedicated block for the AI crawlers, the very areas that were blocked start showing up in their requests.

Source: https://illia-biztech.de/en/forum/seit-wir-eigene-zeilen-fur-die-ki-bots-eingetragen-haben-greifen-sie-auf-mehr-zu-als-vorher

1. [Home](/en)
2. [Forum](/en/forum)
3. Since we added our own lines for the AI bots, they reach areas that were blocked

# Since we added our own lines for the AI bots, they reach areas that were blocked

After a dedicated block for the AI crawlers, the very areas that were blocked start showing up in their requests.

* ##### Asked by  
QuietBeacon448
* ##### Asked on  
September 25, 2026

We are an independent car repair shop in Wendschott with a customer portal. Our robots.txt has carried a block on that internal area for years, written under the asterisk that covers all bots. A few weeks ago we added dedicated lines for the AI crawlers, each with a single block of its own.

Now I see exactly those bots in the internal area the asterisk is supposed to block. We only added something, we removed nothing.

#### Answers

![Portrait of Illia Belichenko, technical specialist at Illia BizTech](/_ipx/f_webp,q_75/https://cms.illia-biztech.de/uploads/small_Illia_Belichenko_author_81f88864f0.png)

##### Illia Belichenko · Founder & Technical Lead

✓ Accepted answer September 29, 2026 

You did remove something all the same, namely the reach of the asterisk rules over those bots. Only one group is ever valid for a given crawler. The crawler looks for the group with the most specific matching user agent, and Google's robots.txt specification states plainly that user-agent-specific groups and the global asterisk group are not combined.

The result is the reverse of the intention. The moment a bot finds its own name in the file, only its own block applies to it. Everything under the asterisk is void for that bot, including the old block on the internal area.

The repair is to repeat the rules in every block. They cannot be taken as given. Every named bot gets all the disallows meant for it written into its own block. Several groups sharing one name are merged internally into one.

Files like these grow over years and are rarely read end to end. What your robots.txt actually permits today, bot by bot, is one of the things the [AI visibility report](https://illia-biztech.de/en/ki-sichtbarkeitsbericht) from Illia BizTech checks.

##### 

#### [One shared campaign for Dresden and Bautzen, the budget goes almost entirely to Dresden](/en/forum/eine-gemeinsame-kampagne-fur-dresden-und-bautzen-das-budget-geht-fast-komplett-nach-dresden)

September 29, 2026

[](/en/forum/eine-gemeinsame-kampagne-fur-dresden-und-bautzen-das-budget-geht-fast-komplett-nach-dresden)

#### [We were closed on Buß- und Bettag and Google showed us as open. Who puts that right?](/en/forum/am-bu-und-bettag-hatten-wir-zu-und-google-zeigte-uns-als-geoffnet-wer-stellt-das-richtig)

September 28, 2026

[](/en/forum/am-bu-und-bettag-hatten-wir-zu-und-google-zeigte-uns-als-geoffnet-wer-stellt-das-richtig)

#### [Forty pages for questions nobody types and the AI generates itself](/en/forum/vierzig-seiten-fur-fragen-die-niemand-eintippt-und-die-ki-selbst-erzeugt)

September 27, 2026

[](/en/forum/vierzig-seiten-fur-fragen-die-niemand-eintippt-und-die-ki-selbst-erzeugt)

#### [A provider wants to put us in the AI overview permanently and brought a contract](/en/forum/ein-anbieter-will-uns-dauerhaft-in-die-ki-ubersicht-bringen-und-legt-einen-vertrag-vor)

September 27, 2026

[](/en/forum/ein-anbieter-will-uns-dauerhaft-in-die-ki-ubersicht-bringen-und-legt-einen-vertrag-vor)

#### [Assistants name us without linking us, and the report says zero](/en/forum/assistenten-nennen-unseren-namen-verlinken-uns-aber-nicht-und-im-bericht-steht-null)

September 25, 2026

[](/en/forum/assistenten-nennen-unseren-namen-verlinken-uns-aber-nicht-und-im-bericht-steht-null)

**Share** 

**Topics** 
* robots.txt
* AI crawlers