.Google.com's John Mueller answered a question about why Google marks pages that are actually disallowed from creeping by robots.txt as well as why the it is actually safe to neglect the related Search Console records regarding those creeps.Crawler Website Traffic To Inquiry Criterion URLs.The person inquiring the inquiry chronicled that bots were developing links to non-existent question criterion URLs (? q= xyz) to web pages with noindex meta tags that are likewise shut out in robots.txt. What caused the concern is actually that Google.com is actually crawling the hyperlinks to those webpages, acquiring blocked through robots.txt (without noticing a noindex robots meta tag) at that point acquiring reported in Google.com Browse Console as "Indexed, though shut out through robots.txt.".The person talked to the adhering to inquiry:." Yet below is actually the significant concern: why will Google.com mark web pages when they can not also view the web content? What's the benefit during that?".Google's John Mueller verified that if they can't creep the page they can't find the noindex meta tag. He likewise helps make an exciting reference of the web site: search operator, advising to dismiss the outcomes due to the fact that the "ordinary" users won't observe those results.He composed:." Yes, you're proper: if our company can't creep the webpage, our team can't observe the noindex. That stated, if our team can't crawl the web pages, after that there's not a great deal for our company to index. Thus while you may find some of those pages along with a targeted internet site:- concern, the ordinary consumer won't view all of them, so I would not fuss over it. Noindex is also fine (without robots.txt disallow), it merely indicates the URLs will find yourself being actually crept (as well as end up in the Look Console document for crawled/not indexed-- neither of these conditions create concerns to the remainder of the website). The fundamental part is actually that you do not produce all of them crawlable + indexable.".Takeaways:.1. Mueller's answer verifies the restrictions being used the Internet site: hunt advanced hunt operator for diagnostic main reasons. Some of those main reasons is actually considering that it's certainly not hooked up to the regular search index, it is actually a different thing completely.Google.com's John Mueller discussed the web site hunt operator in 2021:." The short response is that an internet site: query is actually certainly not suggested to become full, neither used for diagnostics purposes.An internet site query is actually a certain type of hunt that restricts the end results to a specific website. It's essentially just the word web site, a digestive tract, and afterwards the website's domain name.This question restricts the outcomes to a specific site. It is actually certainly not suggested to become an extensive assortment of all the pages from that internet site.".2. Noindex tag without using a robots.txt is actually alright for these kinds of conditions where a robot is connecting to non-existent pages that are receiving uncovered by Googlebot.3. Links along with the noindex tag will definitely generate a "crawled/not listed" entry in Look Console and that those won't possess a damaging result on the remainder of the web site.Review the inquiry and respond to on LinkedIn:.Why would certainly Google.com index webpages when they can't also find the information?Featured Photo through Shutterstock/Krakenimages. com.