Advertisements

Stay away from my search result pages [insert search engine name here] bot!!


imageNow, this is a bit overkill, but, there are almost as many different ways a search crawler makes use of a robots.txt file as there are search engines (this may be highly over-exaggerated, but anyways…).

Now, one thing you probably do not want with your public facing site, is for the search engine to waste it’s time crawling your search pages. You don’t exactly want a high page rank for your site’s search results, do you?

What to do, what to do?

Well, if your search results pages happen to live under /search/pages/results.aspx, here is an example. This again is a bit overkill, but it should get the job done. Now the search engines can focus on what you want to be getting searched for – your content!

Some search bots allow for wildcards, some are case insensitive, some are case sensitive – hence the number of variations below. Add this into your robots.txt, and you should be good to go.

Disallow: /search/pages/results.aspx
Disallow: /Search/Pages/Results.aspx
Disallow: /Search/Pages/results.aspx
Disallow: /Search/pages/Results.aspx
Disallow: /search/Pages/Results.aspx
Disallow: /search/pages/Results.aspx
Disallow: /Search/pages/results.aspx
Disallow: /search/Pages/results.aspx
Disallow: /search/
Disallow: /Search/
sallow: /search/pages/
Disallow: /Search/Pages/
Disallow: /Search/pages/
Disallow: /search/Pages/
Disallow: /*Results.aspx
Disallow: /*results.aspx

Any additions? Please share them here in the comments!

Advertisements

About Geoff Varosky
Geoff Varosky is a Senior Architect for BlueMetal Architects, based out of Watertown, MA. He has been architecting and developing web based applications his entire career, and has been working with SharePoint for the past 13 years. Geoff is an active member of the SharePoint community, Co-Founder and Co-Organizer of the Boston Area SharePoint Users Group, co-founder for the Boston Office 365 Users Group, co-organizer for SharePoint Saturday Boston and speaks regularly at SharePoint events and user groups.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: