Skip to main content

Search Engines Not Indexing Your Internal Pages? Here Is a Trick You Can Do About It

By VJ
Some bloggers have a problem—only their blog’s home page is indexed in search engines and not any internal pages (or not all internal pages). You may submit a sitemap in Google, Yahoo, and in Live Webmaster Tools for the site’s internal pages to be indexed.

You may wish to check which of your pages are actually indexed by using the ‘site’ operator: Search in Google and Live.com:

site:http://yoursitename.com

This tells you how many pages from your site are actually there in the index of the search engine. If only a few of them are indexed, you have to either manually submit each page or submit a sitemap with all pages (How to Submit a Sitemap for Blogger?)

Google Indexing

Google actually indexes all your pages by following links. It is quite easy to be found by Google, since it normally indexes a page if there is a relevant link coming into it. And as the PageRank increases, the indexing becomes faster. Which means, your newer pages will be found even without incoming links.

For even faster indexing, you can change the crawl rate of the Google bot, by going to:

Google Webmaster Tools->Your Site Profile->Tools->Set Crawl Rate.

If you have the option for faster crawling, you may set it. However, make sure that your web host is robust enough to stand faster crawling. Sometimes, the crawling may cause high data transfer and failure at the hosting side.

Live.com Indexing

With Live, I have a little trouble getting fully indexed. If you check CuteWriting with the site operator, you will see it is not indexed fully in Live.com. I was a bit lax about it and didn’t actually submit my site manually. I believed that with quality incoming links that I have, Live would easily find out CuteWriting and index it fully. But it turned out that was not the case. A few days ago, I found that only 18 of my pages were indexed in Live. Right now, I am in a quest to get CuteWriting fully indexed by them.

Every webmaster trying to get into Live search engine, should have an account in Live Webmaster Tools. It’s inevitable. Also, post your queries to the Live Webmaster Tools forums. You can submit a sitemap to Live, just as with Google. For this, create an XML sitemap and submit it at Live Webmaster Tools.

The Trick to Be Found

Though search engine bots can go deep into a site following links, they miss links all the time. Pages on the home page or one link deeper are likely to be found faster by search bots. Here is a trick you can do for this.

About this WP Premium template here, the archive widget, which has links to all posts is placed in a JavaScript snippet. Search engines do not index anything inside the JavaScript widget. They cannot literally see what it is. So, don’t place any link you wish to have indexed within a JavaScript code. Since my archives widget is placed inside JavaScript on the sidebar, it wasn’t naturally found by Live.

Live search bot has a notorious way of not going so deep into a site. So, it found effectively fewer pages from CuteWriting and thus didn’t index it fully.

So, to get indexed fully, you have to point to the entire posts in the blog from the home page itself or one link deeper. For that, I manually placed each month’s archive pages on the side bar. Not in any JavaScript widget. Just as plain links that search engines can follow. This is exactly like placing a sitemap in normal websites.

Another thing you should be knowing about is the robots.txt text file, which is there in every website (ideally). (Know more about link analysis with robots.txt). In Blogger, the search label pages (the category links of course) are disallowed from search engines with robots.txt file. So, those links will not be indexed normally by Google or Yahoo. I found that Live still doesn’t care (it indexed some of my category pages).

So, don’t place links to category pages. It will not be followed and indexed by search bots. The monthly archive pages on Blogger are not disallowed for any search bot. So, they will be followed and crawled fully. Since each month’s archive page holds all post URLs made in that month, they will be crawled and indexed completely. Look at the bottom of my sidebar, where you can see the links to all months. Live bot now needs to crawl just one link deeper to find all of my posts.

Copyright © Lenin Nair 2008
Labels:

Comments

  1. AmolNovember 12, 2008 at 6:38 AM

    The tips you provided to index pages in search engines were really useful.
    I now follow your blog.Your blog is a great resource.
    Thanks

    ReplyDelete
  2. Kash MFebruary 19, 2009 at 4:57 AM

    I recently found that Blogger.com is not restricting all the label pages from getting indexed. Monthly archive pages are having no such issues.

    This is such a waste of Link juice as Blogger label pages can play a vital role in improving internal link structure.

    ReplyDelete

Post a Comment

Comments are moderated very strictly

Popular posts from this blog

Power of Short Sentences

By VJ
Post dedicated to Thomas Hardy (see History Today below). There are monster sentences like the one you encounter as the first paragraph of Oliver Twist by Charles Dickens . One of my friends, whom I am getting equipped for his IELTS ( what is this? ), told me that the examination recommends long sentences. In writing classes also, I guess it’s longer sentences most tutors promote. But indubitably shorter sentences are more powerful . We will see why. Take a long sentence for instance: Tom Cruise, one of the finest actors in the whole world, is perhaps the most powerful celebrity to exist ever according to Time Magazine, but many people still dispute this fact and point out that there are more powerful and popular actors than Cruise, though they were unsuccessful in providing the total number of fans, who liked the films of those actors. This is a long sentence and it is very confusing . Though it has a logical construction and conveys a meaning, it falters in many occasions and seems ...
1 comment
Read more

Creative Writing: Crafting Characters With Emotional Appeal in Mind

By VJ
When you read the greatest fiction works ever, have you ever asked what was so compelling about them that you not only kept reading it, but you ended up reading all other major works of the writer? It may well be because the writer touched your emotional quotient quite a bit. Every reader has a unique taste . Some like to read suspense thrillers , some tender love stories, and some others dark horror and bloodshed stories . That’s why there are all sorts of genres out there. When a writer gives you what exactly you want, you will keep reading. Here we come to the emotional appeal. Character Imperfection Perfect characters may not always be the upshot of a writer’s deliberation. It may well be due to ignorance . Usually the upcoming writers take it for granted that if they create perfect characters, they will be able to garner a bigger audience . It is not true. You have to ask yourself what a character would do in a particular situation. Perfect characters—perfect gunmen, perfect...
1 comment
Read more

En Dash, Em Dash, and Hyphen

By VJ
We have three types of dashes in use: The hyphen, En Dash, and the Em Dash. In this post, we will see how to use them all correctly. Hyphen (-) The hyphen is the minus key in Windows-based keyboards. This is a widely used punctuation mark. Hyphen should not be mistaken for a dash . Dash is different and has different function than a hyphen. A hyphen is used to separate the words in a compound adjective, verb, or adverb. For instance: The T-rex has a movement-based vision. My blog is blogger-powered. John’s idea was pooh-poohed. The hyphen can be used generally for all kinds of wordbreaks . En Dash (–) En Dash gets its name from its length. It is one ‘N’ long (En is a typographical unit that is almost as wide as 'N'). En Dash is used to express a range of values or a distance: People of age 55–80 are more prone to hypertension. Delhi–Sidney flight was late by three hours. In MS Word, you can put an En Dash either from the menu, clicking Insert->Symbol or by the key-combinatio...
11 comments
Read more