posted by mark in Winery SEO Guide
Last time, we explored the basics of HTML, the behind-the-scenes language of the web. Let’s talk a little bit more about search engines and how they read websites.
How Do Search Engines Read Websites?
In part 3 of this SEO guide there are two fairly short sections on engines and robots reading your site. I’d like to explore more about what the engines are looking for, so we can dig a little deeper on what you can do to help robots, and users, understand your message.
Because the rules of how websites are set up are fairly well defined, robots tend to behave like extremely thorough users. Let me explain – any website is made up of pages of HTML and links between those pages. Some websites today have additional functionality that allows the pages to animate, or to change as you use them, but in general the basic web contains pages of HTML, with links between those pages. As a user of a website, you visit a site, you read the content, and you find a part of the page that references a topic that you want to know more about. Assuming there’s a link to a page deeper into the site that has information on that topic, you follow that link.
You are unlikely to follow every link on the page. There are probably too many, or you’re too busy, or not every link is relevant to what you’re looking for. A robot is not so picky. Its only task is to visit a page, then find every link on that page and follow it. Each link leads to a page, which the robot will then visit and find every link and follow it. And so on. As it does this, it stores a copy of the contents of each page so that the algorithms the engine has built can analyze those copies to figure out what they’re about.
Two things may be apparent here, given what is happening. If for some reason when a robot visits your page, there is no content (for example if your site is down or if in some cases the page requires a user to complete an action to read the content), then the page will have reduced its ability to rank. There is no content. Most engines are pretty forgiving here, and they’ll come back to visit again, so it’s not a terrible worry if this happens. Also, and much more importantly, if you have a page that is not linked to on your site, the robot cannot and will not find it.
In general, the first page that an engine will visit on your site is your home page. It will follow the process above, and if you have created a page but not linked to it anywhere, the robot cannot find where it is. It will not have the ability to rank because it is effectively invisible to the robots. Note that robots also follow links that go between sites. In fact, links between sites are very important for SEO (review the Site Authority and Links section of part 4). If you have created a page on your site, and you do not link to it from anywhere on your site, however another site does link to it, in general it will be found. I say in general because the other page must have already been found (linked to).
The important point here is that, after you write all kinds of great content, it’s important to link to it so that it can be found by users of your site. Ultimately, that also means the robots can find it, and that could potentially lead to more users from the engines directly.