<pedrocorreia.net ⁄>
 

<Optimize your crawling and indexing ⁄ >




clicks: 2519 2519 2009-09-08 2009-09-08 goto tecnologia myNews tecnologia  Bookmark This Bookmark This


The Internet is a big place; new content is being created all the time. Google has a finite number of resources, so when faced with the nearly-infinite quantity of content that's available online, Googlebot is only able to find and crawl a percentage of that content. Then, of the content we've crawled, we're only able to index a portion.

URLs are like the bridges between your website and a search engine's crawler: crawlers need to be able to find and cross those bridges (i.e., find and crawl your URLs) in order to get to your site's content. If your URLs are complicated or redundant, crawlers are going to spend time tracing and retracing their steps; if your URLs are organized and lead directly to distinct content, crawlers can spend their time accessing your content rather than crawling through empty pages, or crawling the same content over and over via different URLs.

In the slides above you can see some examples of what not to do-real-life examples (though names have been changed to protect the innocent) of homegrown URL hacks and encodings, parameters masquerading as part of the URL path, infinite crawl spaces, and more. You'll also find some recommendations for straightening out that labyrinth of URLs and helping crawlers find more of your content faster, including:



este é só um excerto do artigo, para aceder ao artigo completo, clique no link em baixo:
this is just a small excerpt from the article, to access the full article please click in the link below:

http://googlewebmastercentral.blogspot.com/2009/08/optimize-your-crawl...




Subscribe News RSS  Subscribe News Updates by E-mail





myNews <myNews show="rand" cat="tecnologia" ⁄>

MongoDb Architecture new ...

NOSQL has become a very heated topic for large web-scale deployment where scalability and semi-struc (...)

clicks: 16645 16645 2012-05-14 2012-05-14 goto url (new window) horicky.blogspo... goto myNews tecnologia


A Hopefully Fair and Useful Comparison of Haskell Web Frameworks new ...

Recently there has been a lot of discussion and questions about the differences between the big thre (...)

clicks: 13260 13260 2012-05-13 2012-05-13 goto url (new window) softwaresimply.... goto myNews tecnologia


Retina graphics for your website new ...

retina.js is an open source script that makes it easy to serve high-resolution images to devices wit (...)

clicks: 13109 13109 2012-05-11 2012-05-11 goto url (new window) retinajs.com/ goto myNews tecnologia


What every programmer should know about memory, Part 1 new ...

In the early days computers were much simpler. The various components of a system, such as the CPU, (...)

clicks: 14371 14371 2012-05-10 2012-05-10 goto url (new window) lwn.net/Article... goto myNews tecnologia


The State Of HTML5 Video

HTML5 has entered the online video market, which is both exciting and challenging for developers in (...)

clicks: 5441 5441 2012-02-14 2012-02-14 goto url (new window) longtailvideo.c... goto myNews tecnologia


A Tour of Amazon's DynamoDB

Amazon's recent release of DynamoDB, a database whose name is inspired by Dynamo, the key-value data (...)

clicks: 5208 5208 2012-02-13 2012-02-13 goto url (new window) paperplanes.de/... goto myNews tecnologia


Google BigQuery Service: Big data analytics at Google speed

Rapidly crunching terabytes of big data can lead to better business decisions, but this has traditio (...)

clicks: 4870 4870 2011-11-30 2011-11-30 goto url (new window) googlecode.blog... goto myNews tecnologia


Video: CSS3 Secrets: 10 things you might not know about CSS3

A number of prominent front-end developers delivered helpful talks at this year's Fronteers 2011 con (...)

clicks: 5741 5741 2011-11-30 2011-11-30 goto url (new window) css.dzone.com/a... goto myNews tecnologia