Search is undoubtedly the heart of today’s World Wide Web. Everybody loves, or at least can’t live without, Google.
Still at times the need arises to hide parts of a web site from search engines. More often than not it is private content (customers’ account information, contact information for individuals, etc.) that the client does not want visible to the public. It could also be third-party information like ads. Yet another example is duplicated information across the website as it could negatively impact the page rankings in search engines.
In this article we will first go through the technique that allows us to exclude entire pages and directories from the search crawler. Then we will look into the trickier topic of hiding only part of the page content from the search engine spider.