Developer Hub » Robots File

Link: https://support.brilliantdirectories.com/support/solutions/articles/12000075850

To edit the default version of the Robots.txt file navigate to Developer Hub >> Robots File:




The default file includes the following rules:


User-agent: Googlebot
Disallow: /api/
Allow: /

User-agent: Mediapartners-Google
Disallow: /api/
Allow: /

User-agent: Googlebot-Image
Disallow: /api/
Allow: /
    
User-agent: Yahoo-MMCrawler
Disallow: /api/
Allow: /

User-agent: Yahoo-slurp
Disallow: /api/
Allow: /

User-agent: slurp
Disallow: /api/
Allow: /
    
User-agent: Msnbot
Disallow: /api/
Allow: /

User-agent: MSNBot-media
Disallow: /api/
Allow: /

User-agent: Bingbot
Disallow: /api/
Allow: /

User-agent: Bing Preview bot
Disallow: /api/
Allow: /
    
User-agent: Twitterbot
Disallow: /api/
Allow: /
    
User-agent: *
Disallow: /



To add to the file, click inside the text field and add the necessary rules.  

See How To Add Bots To Your Robots.Txt File (Ahrefs Example).



The following button replaces any value imputed to the default version of the file: