.Google.com's John Mueller answered a question concerning why Google.com marks web pages that are disallowed from creeping by robots.txt and also why the it's secure to overlook the related Browse Console documents regarding those creeps.Bot Visitor Traffic To Concern Guideline URLs.The individual asking the inquiry recorded that bots were actually producing links to non-existent inquiry criterion Links (? q= xyz) to pages with noindex meta tags that are additionally obstructed in robots.txt. What caused the question is actually that Google is actually crawling the hyperlinks to those web pages, getting shut out through robots.txt (without seeing a noindex robotics meta tag) then getting reported in Google Look Console as "Indexed, though blocked out by robots.txt.".The individual inquired the observing concern:." But listed here is actually the huge inquiry: why would certainly Google mark webpages when they can not also observe the web content? What is actually the advantage in that?".Google's John Mueller affirmed that if they can not crawl the web page they can not find the noindex meta tag. He also makes an intriguing acknowledgment of the internet site: search operator, encouraging to disregard the results given that the "normal" customers won't see those outcomes.He composed:." Yes, you are actually proper: if our team can not crawl the web page, we can't see the noindex. That stated, if we can't crawl the web pages, at that point there's not a whole lot for our company to index. So while you may see some of those pages with a targeted site:- inquiry, the normal user will not observe them, so I wouldn't bother it. Noindex is also fine (without robots.txt disallow), it simply implies the URLs will certainly find yourself being actually crept (and also find yourself in the Browse Console document for crawled/not indexed-- neither of these statuses trigger issues to the rest of the site). The essential part is actually that you do not make all of them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the limitations in operation the Website: search evolved hunt operator for diagnostic main reasons. One of those reasons is considering that it is actually not connected to the regular search index, it's a different point completely.Google's John Mueller commented on the web site hunt driver in 2021:." The brief answer is actually that a site: query is certainly not suggested to become total, neither utilized for diagnostics reasons.A site query is actually a details sort of search that limits the results to a certain site. It is actually generally simply words internet site, a bowel, and then the web site's domain name.This concern limits the outcomes to a specific website. It's certainly not meant to become an extensive selection of all the pages from that website.".2. Noindex tag without using a robots.txt is actually alright for these type of conditions where a bot is linking to non-existent pages that are actually obtaining uncovered by Googlebot.3. URLs with the noindex tag are going to produce a "crawled/not catalogued" item in Explore Console which those won't have a damaging impact on the rest of the internet site.Go through the question and also answer on LinkedIn:.Why will Google index webpages when they can not also see the information?Featured Graphic by Shutterstock/Krakenimages. com.