How To: Prevent Search Engines From Indexing a SecureAuth Realm

Version Affected: All

 

Overview

An external-facing SecureAuth realm is a publicly accessible web page, and like any other public website, it can be discovered and indexed by search engine crawlers unless told otherwise. This article explains how to use a robots.txt file to control which search engines, if any, are allowed to crawl the realm.

Configure robots.txt

1. On the IIS server, open D:\inetpub\wwwroot\robots.txt in a text editor. Create the file if it doesn't already exist.
2. Add the directives for the behavior you want, then save the file. For example, to allow every search engine to crawl the site:

User-agent: *
Disallow:

To block every search engine from crawling the site:

User-agent: *
Disallow: /

To block one specific search engine (for example, Ezooms) while still allowing others to crawl the site:

User-agent: Ezooms
Disallow: /

Special Considerations

Some commonly referenced search engine User-agent names, for use in the directives above:

• Top 3 U.S. search engines: Googlebot, Yahoo! Slurp, bingbot
• Commonly blocked crawlers: AhrefsBot, Baiduspider, Ezooms, MJ12bot, YandexBot

SecureAuth Knowledge Base Articles provide information based on specific use cases and may not apply to all appliances or configurations. Be advised that these instructions could cause harm to the environment if not followed correctly or if they do not apply to the current use case.

Customers are responsible for their own due diligence prior to utilizing this information and agree that SecureAuth is not liable for any issues caused by misconfiguration directly or indirectly related to SecureAuth products.