As a web designer, certainly you have files and folders that you don’t want indexed on Google, or any other search engine for that matter. Well, you’re in luck, because there’s an easy way to make that happen.
Search engine spiders look to a file called robots.txt to tell it what it should not index. So if you want to hide any information on your website from indexing, simply create a text file and call it ‘robots’.
When you populate that text file, you do so like this:
User-agent: *
Disallow: /folder/
Disallow:/anotherFolder/file
Or, perhaps you’re in the middle of the web design and want to show it to your customer but you want to block the entire website from search engines until it is finished, you can use:
User-agent: *
Disallow: /
That’s all there is to it. Good luck with all your web design projects!
The Aston Weber Group plays an active role in the betterment of the global community by donating 10% of all profits from our web design and application work to the Red Cross.