To stop undesirable information during the research indexes, site owners can instruct spiders not to crawl particular documents or directories from the common robots.txt file in the root Listing with the domain. Furthermore, a page could be explicitly excluded from the online search engine's database by utilizing a meta tag https://www.youtube.com/watch?v=y4pScLpx0Gk