Tuning Google sitemaps
We’ve just taken another looking at Google sitemaps for the first time in a long while. We’ve had mixed resulting using the sitemaps. Changes that we’ve made were taking weeks to come through. So with all the frustration at the way Google indexes our site, we decided to write a little program that parses our weblogs and looks at our crawler traffic. Its been incredibly useful to see what the google crawler is upto.
Here’s how it helped us, as we refine our SEO for our recent photos page, we got to see the effects to the crawler and on the site.
Our Google sitemap had this :
<url>
<loc>http://www.zoomin.co.nz/?recent/photos</loc>
<changefreq>hourly</changefreq>
<priority>1.0</priority>
</url>
And this was our traffic before the changes
2008-04-10 /?recent/photos MSN [65.55.209.80]
2008-04-10 /?recent/photos Yahoo [67.195.37.116]
2008-04-10 /?recent/photos Yahoo [67.195.52.85]
2008-04-09 /?recent/photos Yahoo [67.195.52.85]
2008-04-08 /?recent/photos Yahoo [67.195.52.85]
2008-04-08 /?recent/photos Yahoo [74.6.29.187]
2008-04-07 /?recent/photos Yahoo [67.195.37.116]
2008-04-06 /?recent/photos Yahoo [67.195.37.94]
2008-04-06 /?recent/photos Yahoo [67.195.37.116]
2008-04-06 /?recent/photos Google [66.249.65.206]
2008-04-06 /?recent/photos Yahoo [74.6.29.187]
2008-04-05 /?recent/photos Yahoo [67.195.37.94]
2008-04-05 /?recent/photos Yahoo [67.195.37.99]
We changed the changefreq to always and it seems to have had an affect.
<url>
<loc>http://www.zoomin.co.nz/?recent/photos</loc>
<changefreq>always</changefreq>
<priority>1.0</priority>
</url>
Using the always change frequency tag denotes that a page is dynamic and changes all the time. Our photo page doesn’t change all the time, but it wasn’t getting indexed by google as often as the content has been changing.
The change had an effect within 24 hours of changing our sitemap.
Displaying 1-25 of 70
Date Place Crawler
2008-04-13 /?recent/photos Google [66.249.65.206]
2008-04-13 /?recent/photos Google [66.249.65.206]
2008-04-13 /?recent/photos Yahoo [67.195.58.157]
2008-04-13 /?recent/photos Yahoo [67.195.37.116]
2008-04-13 /?recent/photos Yahoo [67.195.52.85]
2008-04-12 /?recent/photos Google [66.249.65.206]
2008-04-12 /?recent/photos Yahoo [67.195.37.116]
2008-04-12 /?recent/photos Yahoo [67.195.52.85]
2008-04-12 /?recent/photos Google [66.249.65.206]
2008-04-11 /?recent/photos Yahoo [67.195.37.116]
We’re currently thinking about making our seo tool opensource, as its really useful for medium sized sites to figure out what Google is upto.