Crawling through Umbraco with Robots
The robots.txt file’s main purpose is to tell robots (Google Bot, Bing Bot, etc.) what to index for their search engine, and also what not to. Usually you want most of your website crawled by Google, such as blog posts, product pages, etc., but most websites will have some pages/sections that shouldn’t be indexed or listed in search engines.
Crawling through Umbraco with Sitemaps
Websites come in all shapes and sizes. Some are fast, some are beautiful, and some are a complete mess. Whether it's a high-quality site is irrelevant if people can’t find it, but search engines are here to help. Though the competition to get on first page is tough, this series will dive into some common practices to make your website crawlable.
Updated Ajax + SEO Guidelines
When you build an Ajax based website and want to be SEO friendly, there are a couple of techniques you have to apply. Back in 2009, Google made a proposal on how to make your Ajax pages crawlable.