I was just reading my feeds and this post from Search Engine Roundtable caught my eye. If you’ve ever had to take your site down for some time, like more than 5 hours you will probably have worried about how search engine spiders may treat the down time.
A Google Groups thread has a response from Google’s Berghausen, a Google Webmaster Central representative. Berghausen recommends that you serve up “a ‘503 Service Unavailable’ with a ‘Retry-After’ header indicating when you expect your site to be back up.” That is, if you can serve up that server message while your site is down.
So what is the definition of a 503?
The server is currently unable to handle the request due to a temporary overloading or maintenance of the server. The implication is that this is a temporary condition which will be alleviated after some delay. If known, the length of the delay MAY be indicated in a Retry-After header. If no Retry-After is given, the client SHOULD handle the response as it would for a 500 response.
Note: The existence of the 503 status code does not imply that a server must use it when becoming overloaded. Some servers may wish to simply refuse the connection.
Berhausen goes on to say that “Googlebot will not index your error page, and will come back looking for updates some time after the date specified in the ‘Return-After’ header.” This is a great undocumented feature of Google’s crawling ability.