Ever since Panda 2.5 struck in late September and mid October, I’ve noticed more e-mails, tweets and posts that express concern over the Google Webmaster Tools Crawl Error reports.
Many people are wondering if this report has something to do with their sudden drop in the rankings. Others have been bothered by this for awhile, but are not sure how to handle it.
I don’t know about you, but I have tons of crawl errors in my reports, but they aren’t errors I’ve generated.
They’re caused by people who have attempted to link to my site and misspelled the URL, thus generating the error in my Google Webmaster Tools reports.
And if the person is using some kind of automated system or CMS, then it often results in multiple errors because the URLs have been dynamically generated incorrectly over and over again on different pages.
Here’s why you are seeing these errors…
As the Google spider roams the web and finds a link that points to your website URL, it tries to crawl that link.
If there is no content there, the Google Webmaster Tools returns a 404 and shows this error in your Crawl Error report.
The good news is Google tells us on the Webmaster Central blog that there’s no need to worry about errors that don’t exist on your own site. They state and I quote…
We don’t know which URLs are important to you vs. which are supposed to 404, so we show you all the 404s we found on your site and let you decide which, if any, require your attention.
What’s a bit misleading about that statement is where they say “…we show you all the 404s we’ve found on your site.” Well, the problem is these errors are not physically on our sites. So I can see why that report causes confusion.
Nevertheless, it’s a good idea to stay on top of this, but no need to panic over broken URLs that don’t exist on your site. Just fix the internal errors that you can control.
Potential Solution
If you notice a common misspelling of a certain URL in your reports, you could create a 301 redirect with your host to make sure that link gets redirected to the proper page.
I personally have found no reason to do this since my errors seem to be all over the place in terms of spelling. Nevertheless, you may find this works for you.
So if you’re worried about this report impacting your site’s reputation with Google in a negative way, worry no more.
Google clearly states this is nothing to worry about and you can (according to their blog) safely ignore the 404s generated by external sites.
Hi Lisa, Great article…
I changed my blog content quite drastically about 6 month ago and left about 140 urls with 404 errors.
Google has listed these in my webmaster tools. I have downloaded then ans slowly over time asked Google to remove them.
I recently found a free WordPress 404 redirect plugin that redirects all 404 page back to the home page.
I know this is a shortcut but the old URL’s have no relevance to any of the new content so redirecting them to my home is the best option.
So Don’t need a 404 page, anything not found goes to my homepage…
Is this good or a bad thing?
I don’t think it’s going to hurt with Google, but I wonder how useful that is from a user perspective? For example if I’m searching for a specific thing and I’m redirected to your homepage, unless the answer is right there, I’m not sticking around.
Probably better to redirect 404’s to a page with a search box so they can at least search for what they’re looking for.
Hello,
Please check this,
Hello,
My site getting around 5000 ‘404’ errors. Please check the attached image.
http://i.imgur.com/dtMGM6h.jpg
1. Am having tag, but I don’t have such url.
Google webmaster link
“tag/768×1024/download/fd.php?file=paul-2/paul-2-960×540.jpg”
But Actual one will be
“MY_DOMAIN_NAME/download/fd.php?file=paul-2/paul-2-960×540.jpg”
Authors,
Google webmaster link
“author/kelly/download/fd.php?file=robot-spider-01/robot-spider-1680×1260.jpg”
But Actual one will be
“MY_DOMAIN_NAME/download/fd.php?file=robot-spider-01/robot-spider-1680×1260.jpg”
Category
Google webmaster link
“category/flowers/download/fd.php?file=butterfly-nature/nature-butterfly-1152×864.jpg”
But Actual one will be
“MY_DOMAIN_NAME/download/fd.php?file=butterfly-nature/nature-butterfly-1152×864.jpg”
How to solve this 404 Errors. Please suggest me
Very useful. My biggest frustration is the huge amount of 404 errors generated by 3rd party sites linking to non-existent pages. I’m not even talking about links that have any value, I’m talking about sites like Swagbucks.com! Grrr!!!
Every error should be an important error to fix!
You should not ignore crawl errors but instead, use your .htaccess to properly redirect to a similar page to improve user experience while still captering the intended traffic and link juice. At Amorca.com, we know it is very difficult to control how others link to your site but you can control the outcome. We had several links from other sites that were mispelled but instead of leaving them alone and be directed to a 404 page, we decided to use a Redirect 301 instead and redirect those mispelled links to similar pages.
Now, one thing not to do is redirecting non-existent pages to home page instead of a good custom 404 because it might create other more problem like soft 404s.
If the page does not really exist and you do not have a similar page to redirect to using Redirect 301, then we recommend the use of a Redirect 410 (which is also treated same as a 404 by Google) but do not forget to create a custom 410 page listing all your important main pages or include a search box, link to a sitemap.
We recently got a call from someone having problem using Redirect 301 and when we looked at their .htaccess, we found the following problem which was giving a 302 (Moved Temporarily) as follow:
HTTP/1.1 302 Moved Temporarily
After using the proper Redirect 301 syntax, the server was returning the proper response code, which is:
HTTP/1.1 301 Moved Permanently
The problem was that they had the following Redirect 301 syntax on the same domain:
Redirect 301 /oldurl.html http://domain.com/newurl.html
In summary, you need to visit your Google Webmaster Tools (GWT) and pay close attention to the Crawl error report and fix them accordingly with a proper Redirect 301 (moved) or a Redirect 410 (gone). So treat every error as an important error and fix it accordingly no matter how and where the error originated from.
I hope this helps
Absolute URLs or Relative URLs? WordPress and any php based website produces relative urls within its engine.
Absolute URL’s look like this: http://www.yourdomainurl.com/index.php
Relative URL’s look like this: /index.php
Google robots classify relative urls as error. So there has to be some upgrade of google’s robots to learn what are relative urls and not to conclude these are false urls.
Hi to all.
I’ve got a phpBB forum on an Apache server. For some months Google reported to me a lot of 404 errors cause it appends an extra string in the page url. Example:
http://mydomain/viewtopic.php?f=333&t=777 becomes
http://mydomain/viewtopic.php/viewtopic.php?f=333&t=777
Same thing with the viewforum links.
Anyone has an idea about the cause of this strange problem?
Thanks a lot!
At first my trouble with 404 was that my site could not make the errors properly when they were needed. After resolving that, I was plagued by 404 errors from pages, possibly for linking to pages before making them or something getting corrupted. Now I have custom 404 errors for each damaged page.
thanks for the info. I was worried about the crawl errors but will not worry anymore
I have found 404 errors that I caused myself before. I was re-designing a site, and while cleaning up the categories and some old post, I broke the links by putting the post in new cats or trashing them.
Another cause is changing your permalinks structure with not having a redirect plugin in place and running before you make the changes.
The basic plugin I use now is called; Redirection by John Godley for WP. It is easy to use and has working well with all my 404 and 301 re-directs. Simply add the 404 listing and then the working good listing and Bingo, all is well.
Steve
Thanks for this tip 😀
I once accidently linked a website without http and this resulted in alot of 404 not found errors, I will try out your solution now 🙂
Yesterday I came across a really weird kind of crawl error… my site’s page is mastikorner.com/ati-aslam but the google was crawling mastikorner.com/atif-aslam/atif.. :S
wtf !
Yesterday I came across a really weird kind of crawl error… my site’s page is mastikorner.com/ati-aslam but the google was crawling mastikorner.com/atif-aslam/atif.. :S
wtf !
We had changed the permalink structure on our site. Later I experienced 404 errors at my Webmaster profile. WordPress do state they are redirecting them, but I had to installed a plugin to do it properly and get those errors disappear – Can’t recall – I think it was “WordPress 404” maybe!
It Normally happens due to change in permalink but as per my experience one can use 301 redirection and even resubmitting the sitemap after removing all those articles . Broken link checker plugin is also very useful in such cases.
Nice post, most of the errors I seem to get are from sites pointing back to my site to an outdated link which im hoping wont be causing me issues as we will obviously always have outdated links and have no way of removing the links back to us?
really freaked out by those errors , thanx for the post
Thank you for this amazing post you shared
i was wondering the same and came to the same conclusion. I too take the same action you do, nothing! I find OCD stat checking a very low ROI activity.
Thanks for clearing this up, Lisa!