In the SEO world, crawling means following your links and “crawling” around your website. When bots come to your website (any page), they follow other linked pages also on your website. This is one reason why we create site maps, as they contain all of the links in our blog and bots can use them to look deeply into a website.

What is Indexing? In terms, indexing is the process of adding webpages into a Web Search Engine. Depending upon which meta tag you used (index or NO-index), Website Search Engine bots will crawl and index your pages. A no-index tag means that that page will not be added into the web search’s index.

There are millions of websites on this earth, and Most people are left constantly wondering why their articles aren’t getting indexed.
Let’s take a look at some major factors which play some important roles at the backend of crawling and indexing.
The following factors can negatively affect site’s crawl ability:

  • Slow server responses or a significant number of errors
  • A significant number of low value–add pages. These can include:
    • Many versions of a page with URL parameters that offer useless filtering, false navigation, or session identifiers
    • Duplicate content
    • Low-quality pages
    • Dirty pages
  • Long redirect chains
  • Long page-load times that may timeout
  • Nonstrategic use of noindex and nofollow tags
  • Pages served up through AJAX without links in the page source
  • Blocking bots from crawling JavaScript and CSS files
  • “Dirt” in your sitemap

Share This

1 comment:

  1. Thank you for sharing this! What you have shared is very helpful and informative. Would love to see more updates from you.

    SEO Service


Creative Commons License
Blogger Book is licensed under a Creative Commons Attribution-NonCommercial 3.0 Unported License.