Not all search engines want the social network’s results mixed with theirs
San Francisco, CA -- (SBWIRE) -- 10/04/2012 -- San Francisco, CA- Although Twitter recently updated its robots.txt file opening up millions of pages to be crawled, there is doubt as to whether major search engines are happy about the mingling.
Twitter’s robots.txt changes allow Google, Bing, Yahoo, Yandex and other bots to crawl through some of its search results pages.
The social network is opening its content search results in the form of tweets and hashtags while preventing bots from crawling for search results in the form of users, videos and images.
The change will help those looking for Twitter pages such as the recently popular #olympics hashtag page.
The news is great for internet users hoping to find topical, interesting content. For example, a search for #emmys during the recent Emmy Awards or #olympics during the London Summer Games would have turned up all manner of official and unofficial content relating to both subjects.
From the SEO point of view, Twitter’s strong domain presence should allow it to rank well for certain queries.
However, the question remains – do search engines want to index Twitter’s search results?
Google in particular has stressed that it does not want to display search results pages among its own search results.
Meanwhile, the following webmaster guideline indicates webmasters should not do what Twitter is doing:
“Use robots.txt to prevent crawling of search results pages or other auto-generated pages that don’t add much value for users coming from search engines.”
Google and Twitter have fought for more than a year about crawling and indexing. The dispute began in July 2011 when an agreement that gave Google a special feed of Twitter content ended, killing Google’s Realtime Search product.
InternetProminence.com (http://www.internetprominence.com/) is an SEO company. From the first day of its SEO campaign, Internet Prominence sends ranking reports to its clients. Desired rankings are usually delivered within three months through the services of one of the best SEO companies.
Contact Us: 888-4-SBWIRE (US) - 920-321-1250 (International)