Duplicate content and SEO issues

[Elaine Helm]

Search engines currently index our pages at several different addresses, leading to lower page rank and the possibility of duplicate content penalties.

Part of this is because the article links generated on pages using profiles include extra profile IDs and category IDs. For example, this story www.heraldnet.com/article/20081227/NEWS01/712279886 can also be accessed at www.heraldnet.com/article/20081227/NEWS01/712279886/1046/COMM0607.

We also have several persistent URLs for subsidiary sites, such as www.enterprisenewspapers.com (the domain for http://www.heraldnet.com/section/ETP) and www.seattleschild.com (the domain for http://www.heraldnet.com/section/SCM). I'm not sure how we can ensure search engines "see" only the persistent URL version, rather than the section page at our main domain. But the bigger problem here is that every story can be accessed from our main domain and all subsidiary domains. For example: www.heraldnet.com/article/20090329/NEWS01/703299919 and http://www.enterprisenewspapers.com/article/20090329/NEWS01/703299919.

Has anyone else dealt with these issues? I know we can probably tweak our robots.txt file to hone in on the URLs we want to be indexed and prevent the others from being crawled.

I've also read a little bit about the new canonical URL tag option (see post from Google Webmaster Central blog). Has anyone tried this?

[Espen Holmen]

What we have done to deal with this is that we have added the meta header canonical to all article pages like this:



This way at least Google will see that it's the same story and punish you less for having duplicates.

When it comes to using multiple domain names we have set up rewrites so that if you go to http://www.heraldtribune.net/article/20090604/ARTICLE/906041076/205... you will be redirected to the same article on heraldtribune.com.

[Elaine Helm]

Thanks, Espen. Ning stripped out the code, but I have explored the canonical tag as part of our strategy for minimizing duplicates.

The URL redirects work in the case of .com, .net, .org and other domain extensions. I'm primarily concerned with persistent URLs for different brands using the same Saxotech/Publicus implementation. In our case, we have the main brand, HeraldNet.com, as well as three other brands using the same database. Front pages for the subsidiary sites are set up like section pages, e.g. EnterpriseNewspapers.com is really www.heraldnet.com/section/ETP. I don't think redirects will help in that case, will they?