If you’re not using WP Super Cache and Yoast SEO on your WordPress site you should be. HOWEVER, I’ve finally figured out why I was experiencing XML sitemap issues. It turns out caching messes with sitemaps.
Seems pretty obvious now that I think about it… doh!
I was getting all sorts of weird errors on Google Search Console. I’d get URL not found errors, or have files blocked by robots.txt but the URLs were odd strings that shouldn’t have been there in the first place. Google would index images in an odd manner. I’d look at the sitemaps themselves and they weren’t nicely formatted pages like you’d expect. Instead they were lists of URLs running together in a long string.
I use the Yoast SEO plugin to generate my sitemaps. I recommend it for being straightforward yet flexible. Also, it supports things like portfolio post types which I use on this site. This issue also persisted with sitemaps produced by Jetpack and got the exact same thing. I can’t say whether this issue would also persist with other caching plugins, such as W3 Total Cache, but I expect it would.
So, somewhere in the process of WP Super Cache creating the static files it must include the sitemaps in that process. Something about the cached versions of those sitemap files is messed up and causes them to display incorrectly. It must also throw the cat among the pigeons with Google and other web crawlers. Knowing what has caused the error – the fix is pretty straightforward.
To resolve this issue, we need to tell WP Super Cache not to cache the sitemap files. Here’s how:
- Open your WordPress dashboard, navigate to Settings > WP Super Cache
- Click the Advanced tab
- Scroll down the page to this section:
- Add to this list the file name for each XML sitemap file whatever tool you are using produces. It is common for there to be a sitemap index file that points to other sitemaps that deal with posts, pages, images or other post types separately.
If your sitemap address is http://www.example.com/sitemap.xml then you only need to put sitemap.xml in this box.
- Click Save Strings
When I did this, it resolved the issues where the sitemaps displayed incorrectly immediately. The search engines are likely to need a few days to catch up.