Google webmaster guidelines > Technical guidelines > Guideline one of five in this category states...
"Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site."
Definitions-
browser - You are seeing this web page in a browser. A web browser displays web pages. Internet Explorer, Firefox, and Netscape are the names of popular internet browsers.
text browser - A text browser is like a web browser but only displays text (no images, no videos). Text browsers are often used to help those with disabilities browse the internet.
Lynx - Lynx is the name of a text browser that is free.
search engine spider - Also known as a "crawler" or a "bot", a search engine spider follows links to web pages and then reads and retains the information it finds. This information eventually becomes the "copy" of a website in a search engine index. This process is often referred to as "crawling" the web. "Googlebot" is the name of the search engine crawler that is most used by Google.
fancy features - Any web technology or language that may disrupt a search engine spider from reading your page.
Examples and Explanations
This guideline tells us "search engine spiders see your site much as Lynx would" and it tells us that if a text browser has a problem displaying your site then search engine spiders may have problems crawling your site.
A search engine crawler must be able to "crawl" your site in order for it to be indexed in search engines like Google. In fact the Google webmaster guidelines mention several different times the importance of search engine crawlers being able to crawl your site.
This guideline is stating it again, and offers you a suggestion of how to check your site for crawler friendliness.
It is worth mentioning that if your site is written in HTML that validates then crawlers probably have no problem with your site. This guideline mentions "fancy features" and if you do not know what things like "JavaScript, cookies, session IDs, frames, DHTML, or Flash" are then you probably are not using them.
For those of you who are using such features, seeing your site in a text browser is highly recommended to give you an idea of how a search engine spider is seeing your site.
Examine your site -
Examining your website using Lynx.
To examine your site in Lynx you must download and install it to your computer. In resources below I have a link to the place where I downloaded it from. When I went to the Lynx homepage, I found it somewhat unclear, so I looked around until I found a place to get Lynx that gave some instructions and was more clear to a general user.
The first thing I noticed when I used Lynx was that it was like an old DOS window. For those of you who were not using computers in the "early days" when DOS was used often, I can tell you it will seem a bit confusing.
First off, you will not be able to use your mouse. All navigation is done by using your keyboard.
Here is what the Google home page looks like in Lynx:
Now that looks quite different doesn't it?
The reason it looks so different is because a text browser shows a very simple version of a webpage. Let us compare Google in Lynx, to Google the way we are used to seeing it.
The most obvious difference is the big friendly and colorful "Google" that we are used to seeing in the middle of the Google homepage. If you look at the first picture instead of seeing the Google logo, you will only see the word "Google" written in black near the top of the page.
This is because text browsers do not show images. The big colorful Google logo we are used to seeing is an image. The reason that it says "Google" in the text browser is because that is the ALT text that Google has chosen to use for this logo.
Now if you were to navigate through this page in the text browser, you would find that each link works. You will also find that all the text is displayed, and that the search form works. This means that all the elements of the Google home page work also in text browsers. Congratulations Google! Your home page does not pose a problem to search engine crawlers.
If your website can be navigated through in a text browser, then search engine crawlers can navigate it as well.
A simpler way to check your pages is through a "spider simulator" which shows you an approximation of what a search engine crawler might see on you site.
If you check your site on a spider simulator you will be able to detect most of the obvious problems (like text or links not being visible. See our spider simulator. Using this tool you will see many factors of your website like the title, text , links and keyword density of your website.
Key Concepts:
- Make sure that search engine spiders are able to see your site correctly.
- If you use "fancy" elements on your website, ensuring that your website is seen correctly by search engine spiders requires checking your site using a text browser.
Resources
From Google:
Google help pages -
Does Google index dynamic pages?
Does Google index sites that use Macromedia Flash?
Does Google index sites with frames?
The help section for crawling
From other sources:
Our spider simulator - get an idea of how a search engine spider sees your site:
This is where I downloaded Lynx from (they have instructions, and this version worked with my Windows XP):
Download and instructions for Lynx browser
A very detailed side by side comparison of two flash sites. One flash site can be seen by search engine spiders, the other can not. This article tell you how to make your flash page more spider friendly. A great detailed explanation:
How to Optimize A Page That Has Flash
A great list of resources for optimizing Flash content:
Optimizing Flash content in your webpages
Home | Guidelines | About us | Site Map
Copyright 2006 - 2008 Patrick Sexton | This website is not affiliated with Google Inc.
