Google have issued new information on their ability to index dynamic URLs, with some surprising recommendations that could impact on SEO. Interestingly, they state that they can index any dynamic URL and generally recommend against rewriting URLs.
To clarify, a static URL is usually a hand coded HTML document that requires manual updating and uploading. Obviously this can prove time consuming for site owners, especially with large sites where content changes frequently. Dynamic URLs are generated automatically by data drawn from content management systems (CMS), and are distinguishable by the use of a “?” in the URL. The ? signals that a query string has started, with query string parameters separated by the characters & and =. The various parameters define which piece of CMS data is displayed.
In the past, Google stated that it had difficulty crawling URLs with more than two parameters, session IDs and in particular the parameter &ID=, making static URLs a more desirable choice for SEOs. But programmers had methods of making dynamic URLs appear static, using the Apache module mod_rewrite or using an IIS patch like ISAPI Rewrite for Microsoft platforms. This allowed large dynamically generated sites to have URLs that appeared static.
Rewriting dynamic URLs to appear static can be time consuming and expensive, with potential knock on effects, which is possibly why Google have moved to clarify their position on the matter.
Google state that they have made progress in being able to crawl and index dynamic URLs, and that having dynamic URLs “does not imply a significant disadvantage in terms of indexing and ranking”. They even go as far as saying dynamic URLs are favourable, so they can analyse URL structures themselves without any parameters hidden and therefore don’t lose vital information.
Common myths such as dynamic URLs being un-crawlable and URLs with less than three parameters being OK are exposed as being untrue; dynamic URLs can be crawled with any number of parameters. Google however do clarify that URLs should be kept short whether static or dynamic.
Overall Google recommend that it’s not necessary to rewrite dynamic URLs, but there are several reasons why it is still a good idea to do this:
- Google often announce their improved ability to crawl and index content they struggled with before, such as Flash and content behind forms, but until these are fully proven and operational it is better to err on the side of caution and stick to a tried and trusted practice
- Google are secretive over their ranking algorithms including the impact of keyword use in URLs; achieving any impact from this technique would often involve URL rewriting
- Tidy, user friendly URLs can improve click through rates as opposed to messy dynamic URLs with multiple parameters, especially if relevant keywords are used
- Where URL rewrites aren’t necessary it is still usually good practice to block them from crawlers, especially when the URLs could create duplicate content
- You should consider the ability of all search engines to crawl and index URLs, not just Google
Overall, the marketing benefits of clean, keyword rich URLs make rewriting URLs worthwhile even if Google claim they can fully index any dynamic URL.