Why is my disallowed page still showing up in Google?
Because robots.txt controls crawling, not indexing. If another site links to your page, Google can list the URL in its results without ever fetching it, which is why those entries appear with no description. The fix is the opposite of what people assume: you must allow the crawler to fetch the page, and put a noindex meta tag or an X-Robots-Tag header on it. Once Google has seen the noindex and dropped the page, you can block it in robots.txt if you also want to save crawl budget.
Can I use robots.txt to hide sensitive pages?
No, and attempting it makes things worse. The file is served publicly at a predictable URL, so anyone curious about your site opens it first. Listing a path there is an announcement, not a defence, and it is a well-known reconnaissance step. Anything that must not be seen needs authentication. Anything that must not be indexed needs a noindex tag. robots.txt is a politeness request to well-behaved crawlers and nothing more.
Do all crawlers obey robots.txt?
The major search engines do, and they are consistent about it. Everyone else is a matter of choice, because the file is advisory and there is no enforcement. Scrapers, spam bots, and vulnerability scanners routinely ignore it, and several AI training crawlers have been observed doing so. If a bot is causing you real load, block it at the server or the firewall, since a line in a text file cannot stop anything that has decided not to read it.
Where exactly does the file go?
At the root of every host you want to control, so https://example.com/robots.txt. A file placed in a subdirectory has no effect. Each subdomain is a separate host and needs its own copy, so blocking something on the main domain does nothing for blog.example.com or shop.example.com. The protocol matters too, so the http and https versions of a host are technically separate.
Should I still list my sitemap in robots.txt?
Yes, it costs nothing and it helps crawlers that have not been told about the sitemap any other way. The Sitemap directive takes an absolute URL and can appear anywhere in the file, independent of any user-agent group. It is not a substitute for submitting the sitemap in Search Console and Bing Webmaster Tools, which also gives you an error report, but it means any crawler reading your robots.txt can find it.