What is a Sitemap?
A Sitemap is a list of all URLs that one domain contains. Well actually you decide how many URLs your sitemap will contain but in general it should contain all available URLs. In the past, sitemaps where a useful help for search engines to find all available URLs of a domain. But nowadays search engine like Google have way better performances, sitemaps may not be that important anymore. Nevertheless sitemaps are useful tools to regulate crawling!
Kinds of Sitemaps
When talking about sitemaps we have to differentiate the different kinds of sitemaps. Basically sitemaps differ by the data format. So there are HTML Sitemaps and XML Sitemaps.
HTML Sitemaps look like a normal website that includes a list of many other websites. You can use them to simplify the navigation of your website. Visitors have the possibility to see all the URLs you have on your website at one sight. In the Google Webmaster Hangout of Friday, 26th February 2016 Google┬┤s Johannes M├╝ller pointed out that HTML Sitemaps are only necessary if the navigation structure of your website is very complicated.
“Sometimes it can definitely make sense to have these kind of HTML sitemaps, which are essentially kind of a mapping of your category and your detail pages. Especially if we can’t crawl a website normally otherwise. So if you have a really complicated navigation structure, maybe if you have pages that are almost connected just through search forms rather than a logical structure, than at least having one place where we understand the structure of the site, based on the links that can really help us. […] We use HTML sitemaps to understand the structure of the site to find the URLs. But there is obviously a lot more information in XML sitemaps, things like the change date, for example, that tell us that these pages actually changed, which, if you have a larger website, then letting us know that this section of your site or this specific detail page changed recently it makes it a lot easier for us to crawl there and to actually update our index, based on that change. So that’s something you wouldn’t be able to do with an HTML sitemap. It’s almost like an HTML sitemap can be comparable to a normal site’s navigation. And if you already have a good site navigation, then you probably don’t need an HTML sitemap for search engines, at least.”
XML Sitemaps are made for search engines. They look like a typical XML file and as you can read in the quote of Johannes M├╝ller they include a lot more important information to Google than HTML Sitemaps. You can place the change date, to tell search engines when the last change was made and if it is necessary to crawl that URL again soon. Keep in mind that your website has a specific crawl budget. There is also the possibility to include the priority of each URL (from 1.0 to 0.0), which specifies the importance of the URL. The higher the priority the higher the crawl rate, in theory. The following picture shows the typical structure of a XML Sitemap.
Submit Sitemaps to Search Engines
The next thing is to let search engines know about the existence of your sitemap. There are several ways to do this. The first and easiest way is to make an entry in your robots.txt file. It should look like this:
This entry shows any crawler where to find your sitemap. The other way is to submit the sitemap via Webmaster Tools like Google┬┤s Search Console or Bing Webmaster Tools.
In Google Search Console on the left menu select Crawling > Sitemaps. The important thing here is, that only Google crawlers will see this information.
How to create a Sitemap
There is really no need to create a sitemap on your own. Depending on the amount of URLs your website contains this can take hours. But don┬┤t worry, there are many tools that will do all the work for you.
In case you have a WordPress website there are many excellent plugins that will generate sitemaps for you. Some of them will also ping search engines every time you upload a new post. I recommend the following ones:
In case you do not have a WordPress website, you may just google something like free sitemap generator. There will show up plenty of good websites.