Ever since Panda 2.5 struck in late September and mid October, I’ve noticed more e-mails, tweets and posts that express concern over the Google Webmaster Tools Crawl Error reports.
Many people are wondering if this report has something to do with their sudden drop in the rankings. Others have been bothered by this for awhile, but are not sure how to handle it.
I don’t know about you, but I have tons of crawl errors in my reports, but they aren’t errors I’ve generated.
They’re caused by people who have attempted to link to my site and misspelled the URL, thus generating the error in my Google Webmaster Tools reports.
And if the person is using some kind of automated system or CMS, then it often results in multiple errors because the URLs have been dynamically generated incorrectly over and over again on different pages.
Here’s why you are seeing these errors…
As the Google spider roams the web and finds a link that points to your website URL, it tries to crawl that link.
If there is no content there, the Google Webmaster Tools returns a 404 and shows this error in your Crawl Error report.
The good news is Google tells us on the Webmaster Central blog that there’s no need to worry about errors that don’t exist on your own site. They state and I quote…
We don’t know which URLs are important to you vs. which are supposed to 404, so we show you all the 404s we found on your site and let you decide which, if any, require your attention.
What’s a bit misleading about that statement is where they say “…we show you all the 404s we’ve found on your site.” Well, the problem is these errors are not physically on our sites. So I can see why that report causes confusion.
Nevertheless, it’s a good idea to stay on top of this, but no need to panic over broken URLs that don’t exist on your site. Just fix the internal errors that you can control.
Potential Solution
If you notice a common misspelling of a certain URL in your reports, you could create a 301 redirect with your host to make sure that link gets redirected to the proper page.
I personally have found no reason to do this since my errors seem to be all over the place in terms of spelling. Nevertheless, you may find this works for you.
So if you’re worried about this report impacting your site’s reputation with Google in a negative way, worry no more.
Google clearly states this is nothing to worry about and you can (according to their blog) safely ignore the 404s generated by external sites.
Even though google web masters tried to assure us that there is nothing to worry about we could only hope this to be so. From other point of view we need to do our best to clean all the crawling errors which we can in order to improve our ranking results.
Greets apartments for the Olympics
I never pay attention to this type of errors. It takes too much time to solve and it’s not that important. At least for my sites.
I am having 100s of errors in Google Webmaster Tools. Looking to try 301 redirection plugin!
what exactly seems to be the problem, my friend ? Perhaps I can be of assistance 🙂
This is a bit of a relief. It has been bugging me for awhile if those errors are affecting the SE rank of my site. I guess, as long as you are able to fix all the broken links within, then there is nothing to worry about external broken links that goes into your site.
I was just struggling with these errors and they were driving me nuts until I found your article.
Thanks for the help,now I am stress free :-d
I had a client last year with over 10,000 crawl errors – his e-commerce site was a mess but after working thru the errors his performance turned around quite quickly. I agree that Google is easy to get on with if you are playing the right game.
this information is very helpful for my blog boz i am new in blogging.
thanks
Anyone have any experience with Soft-404 errors on non-error pages on Google webmaster?
I’m getting a lot of “Soft-404 (404-like content)” crawl errors in my account. The links all actually work, but google’s crawler doesn’t seem to like them, which I doubt will do me many favours.
The problem seem to be mostly with one script in particular, which is a redirect script as follows:
Thank you for this enlightening post you shared. I did encountered this error thing and it bothers and confuses me. But not anymore, I will safely ignore the 404s generated by external sites. What a relief!
So guys, have you find solutions that are working to these errors? Can anyone confirm them, I’m having trouble with mine.
Great info on Crawl Error Report. I didn’t have thorough knowledge on how to deal with it. Glad I learned about the solution from this post.
Great explanation on this.. I always thought I was doing something wrong.. Thank you for this post, now I know it isn’t my fault!
no… You’re not the only one 🙂
What if I’m getting these errors with Blogger hosted blog ? how do i fix them ?
Thank you for your timely blog. I have meet this kind of bad things some days ago, but I still did not know why and how to deal with it. Now maybe I do not need to worry it too much, thank you 😀
Hi Lisa,
From time to time I checked these G. crawling errors, whether they are real or not. Sometimes they were real 404 errors, and after that I decided to fix it with a special WordPress plugin for redirection.
Hmmm – good tips. I no longer use GA but good tips nonetheless.
On a related note, if you can find 404 errors from other sites, it makes for a good link-building opportunity if you have a page you can replace their broken link with…only trouble is you need to hunt these opportunities down with Xenu Link Sleuth and the like, but 404’s = easy linking.
(Lots of people don’t do it since you need to actually contact the webmaster and convince them, but it’s rather painless.)
Dear Lisa, actually I was looking for the post that explains about Google Crawl. Happy finding this post and I request you to please suggest if there are any other post similar to this one explaining more about Crawling things.
Thank you. I thought it was a sitemap or robots.txt misconfiguration
Thanks so much for the insight, Lisa. I also got a bit worried about having so many errors in my blog, especially since many of the 404 errors were not related to my posts or pages. Reading here that I can just safely ignore it is definitely a relief.
great blog….. some of my sites have the same problems but roughly an hour a week i spend time cleaning up and refreshing my sites, this helps and also pings google reminding them sites of your sites…so otheres do the same?
I don’t understand the concept of getting your link crawled. What’s the point?
There’s too much to check, and I think I’ll just ignore crawl errors in my blog. I just think that if you have a good blog, that’s enough.
Anyway, I’ve seen with Panda a lot of panic (as example with the crawl errors) with my friends, while in the past there was a relaxed way (probably the best) to blog. Hope things will somehow go back to that, while trying to have a good blog anyway.
Hi,
Thanks for sharing! I just got started learning about the online industries and google spiders. I don’t really know much, but please answer this question. I am currently working on my website and I have a order button on my site is that is not set up to redirect it to a order link. By having that button on my website and when people click on it, it direct them to the 404 page because is not completely set up. Does google find that an error and will drop down my rank?
Hope to here from you soon!
stopping by to wish everybody in this wonderful blog post by Lisa a very blessed, Thanksgiving gobble gobble turkey day
Thanks for posting this information.When Google crawls through your website, it cannot modify, edit or add any content to your website. The crawling refers to Google taking notes of various things about your websites, like metadata, links, back links, keywords and stuff.
Your timing is something else. I’ve been meaning to look into the 404 errors showing up.
Thanks again Lisa
you’re not the only one, friend 🙂
Hi Lisa, I’m not sure I understand how to know which crawl errors are really found on my site versus the ones on other sites – or how to fix them when necessary. Any advice on that?
I just go through the Crawl Error report and view the links in the “Linked From” report to make sure those bad links aren’t on any of my pages.
I’m always game for learning from others, especially, when it comes to learning about 404 Google trawl errors and learning from female CEO’s as well. I’ll be mindful from now on to check my webmasters stats more often 🙂
thanks for your help in this blog…I was wondering the same and also came to the same conclusion. I mostly find OCD stat checking has a low ROI activity, do others find the same?
I have already quit using Google. I’ve been experiencing same problems with you. Thanks for posting this information.
Thank you very much for this info, it did have me worried. I look forward to learning more from you in the future! Thanks again!
I have these issues with some of my sites but they are not a problem, an hour a week or month dedicated to cleaning up and refreshing your sites helps and also pings google reminding them your sites exist and are been monetized regularly.
The word crawl may be misleading here, as is to so many people. When Google crawls through your website, it cannot modify, edit or add any content to your website. The crawling refers to Google taking notes of various things about your websites, like metadata, links, back links, keywords and stuff. Google makes a note of all this, and saves it in their database, so that they can rank your website in their page ranks.
The green underlines that you see in your website, or a blog page is for info links in-text advertising. It is similar to ad sense, just differing in the style of displaying advertisements. The green underlines, when hovered by a mouse, pops up the info links advertisements.
I also noticed that on my account, seems my images are using the wrong link even though I checked all of them.
I think one could use plugin like Redirection for wordpress to overcome such kind of errors or maybe add some codes to .htaccess which would do 301 redirects for such links. Thanks a lot for clearing this up as I always thought that Google Webmaster showed those 404 links which actually existed in our own sites rather than those that people have linked to.
Hey Lisa,
Very informative article on Google 404 crawling errors and 301 redirect for resolution. I never knew that people entering in a website’s wrong URL could cause errors to show. Thanks again 🙂
Yay, no 404’s but thanks for the heads up :).
good job Brad 🙂
my blog pr drop from pr2 to pr1..is it because of this crawl error?
dont have many erors just indexing slow down
Hey Lisa,
This is an awesome reminder, Lisa. By your help, it gives me the knowledge of checking the update now. My first time, tackling this kind of error. Thanks for this article. Gonna sneak for your future posts.
I noticed this also in some of my sites. I was worried because it might contribute to my ranking.
Thanks Liza for sharing it with us.
Great and timely post. I too was wondering the same and came to the same conclusion. I too take the same action you do, nothing! I find OCD stat checking a very low ROI activity.
To be honest I didn’t check for google crawl errors after the update. I should go and check now 🙂
did you see anything different in your Google Webmaster tools dashboard? 🙂
I noticed a small jump in errors on google webmaster, but not too many. But, if they were to suddenly jump up now I am at least prepared.
Cool. I wasn’t super worried about this because I haven’t gotten too many errors, but now I know there is no need to worry at all. Thanks.
I’ve had issues as well. SERP ranking has been very inconsistent as well. Any tips on this matter?
my recommendation would be just to keep going and keep the content fresh, in efforts of getting good search engine rankings. That’s pretty much it 🙂
All I can say pretty much in terms of search engine positioning and ranking is to keep your content fresh on a consistent basis. This way, when search engine algorithms change, you won’t be that much affected, because your content is relevant and always up-to-date 🙂
keep writing and distributing press releases, with links in the author resource section pointing back to your site. If you write and distribute one press release per day, you’ll experience a good number of back links in the long one pointing back to you, and getting endless traffic from your published press releases also, as well as good search engine rankings. Engaging on forums and commenting frequently on blog posts on other sites are also assured means of getting good positioning on search engine results pages. If you have any questions, please let me know how I may be of service in helping you 🙂
I try to remember to check for crawl errors for my blog in Google’s Webmaster tools from time to time.
Last week, I reviewed the errors and noticed that some were coming from my sitemap, which means that the 404 errors were generated from my site.
After doing some research I noticed that all pages ending in ‘te’ were causing 404 pages. After looking into the issue further, I realized I had a redirect issue within my .htaccess file. I fixed the error and everything is working again.
I tried at one time to keep up with the 404 errors by creating a 301 redirect, but I grew tired of the issue when I started to get more links to my posts.
While it is important to look at the 404 crawl errors, I really only concern myself with those that are generated by my blog, itself.
Hey Lisa,
Dave from ages ago chiming in again, I’ve opted into the world of consulting and been having so much fun doing I forgot to keep up with the rest of the world as I used to.
Love the new design…
So on to the issue errors are on everyone’s site and making it perfect is nearly impossible, but I agree it’s more important to only worry about what you can control and not what you can’t.
It seems this rule applies to everything in life even Google.
I believe if people really concentrate on being relevant and create their own content and not try out anything gray hat they will likely never have an issue with Google.
The biggest thing I see out there is poor site construction:
No title tags, meta descriptions, no H tags at all or duplicates of of titles, descriptions and even H title tags.
I really think Google is pretty easy to get along with if you just follow the rules and not try and find an easy way to do it.
If it were easy everyone would do it and there wouldn’t be an opportunity.
Have Fun
Dave
PS Get Best Practices from industry leaders like SEO Moz, Matt Cutts just to name a couple.
Dave,
I do agree with you when you said “Google is pretty easy to get along with”. Pretty much in a nutshell, it’s all about content, and how often you keep the content relevant and updated. If anyone can do that on a daily basis, Google will always stay in love with you 🙂
Excellent post Lisa. I was just thinking about a domain/website that I let expire where I had created some links from it to other sites. I was wondering if I needed to do something.
Your post is timely.
On a side note, since Panda, I’ve kinda quit visiting Google webmaster tools so much. It’s a bit discouraging to see the negative impact Panda has had that is reflected in webmaster central.
hi Jordy,
sounds like a good idea what you put out. Would you be doing that in efforts of increasing search engine optimization and page rank to other sites? 🙂