How (and Why) Search Engines Render Pages
There’s an interesting twist in the way we think about indexing – and that’s rendering.
When we think of page ranking, we usually think of indexing.
That is, we generally think of when a search engine has:
- Discovered a page through sitemaps or by crawling and then visited the page for indexing.
- Gathered all content through the page source.
- Started ranking the page for queries.
Historically, this has been the most important set of steps in the process as it is the trigger for rankings.
However, indexing is not the last step in the discovery process.
I would say its weight will decrease over time while the final step – rendering – will increase and I guess it will completely replace the indexed version.
Indexing vs rendering: what’s the difference?
Essentially, the difference between indexing and rendering can be illustrated by these two images:
This is indexing:
This is the rendering:
It is essentially the same content as it would be when indexed (HTML) and rendered (Chrome).
Why is this important?
Rendering is more important than you think.
Essentially, the reason this is important is that the rendering provides the truth.
With the code, a search engine can understand what a page is about and roughly what is happening there.
With rendering, they can understand user experience and much more about what content should be prioritized.
With rendering, they can answer questions such as:
- Is the content hidden behind a click?
- Does an ad fill the page?
- Is the content that appears towards the bottom of the code actually displayed towards the top or in the navigation?
- Is a page slow to load?
All of these questions, and many more, are answered during rendering.
These answers are important to fully understand a page and how it should be ranked.
When does rendering occur?
In 2018, rendering took weeks.
Unsurprisingly, it takes much less time now.
Seconds in fact.
Around 6:20 p.m. in the audio here you will hear the message from Google Martin Split just answer this question.
How long until Google renders a page?
Support is 5 seconds, and within minutes 90% of indexed pages will pass through the render queue.
It should be noted that this is a queue, not necessarily a render.
That is, if you are on the positive side of the media set that starts within 5 seconds, your page will be to start rendered in 5 seconds, although it may not complete within that time.
If the render starts in 4 seconds but takes 30 seconds to complete, it would be considered among those counted in the plus side of the average set.
We’ve come a long way in 2 years, from weeks to seconds.
Bing works differently.
When I questioned their Web Ranking & Quality project manager, Frédéric Dubut, he replied:
Same answer as before, between minutes and forever 🙂 but I can confirm that we are trying to prioritize rendering for URLs submitted through the API.
—Frederic Dubut (@CoperniX) August 3, 2020
The “before” he is referring to was my Tweet from last September:
I would say the same – sometimes it’s days, it can be weeks and in extreme cases it can also be never. Ultimately, it’s a trade-off between the cost of rendering the page and the value we find in rendering it.
—Frederic Dubut (@CoperniX) September 3, 2019
Presumably they too sped things up, although I don’t have more recent confirmation in time.
So the short answer as to when rendering takes place is “after indexing” and the timeline is variable but short, which basically means that search engines will understand the content and context of a page before rendering. fully understand how it is to prioritize, but in most cases the mismatch is moot.
A big leap forward came in May 2019 when Googlebot’s Web Rendering Service (WRS) component was updated.
Until then, the web rendering service was using Chrome version 41.
In May 2019, the web rendering service was upgraded to Evergreen, which means it uses the most recent version of Chrome for rendering (in a few weeks anyway).
Essentially now when your page is rendered by Google, it is rendered more or less as you would see it in your browser.
What does a web rendering service do?
I wanted to quickly answer a question that I hadn’t quite thought about until I realized I was completely thinking about it.
You are welcome to laugh at me for the evidence of the hiccups in my brain.
Let’s first see where a web rendering service gets its instructions and how.
Here’s basically the render lifecycle:
- A page is discovered via sitemap, crawler, etc.
- The page is added to the list of pages to crawl on a site when the crawl budget is available.
- Page content is crawled and indexed.
- The page is added to the render queue.
- The rendered page.
So, a critical and unspoken part of the process is the render queue.
When a page reaches the top of the render queue, the engine sends it what’s called a headless browser.
This is the step I struggled with.
A headless browser is a browser without a graphical user interface.
For some reason, I had a hard time understanding how it worked.
For example, how can Google know what’s there if it’s not displayed graphically?
The obvious answer is:
“The bot doesn’t have eyes either so…uh…yeah.”
During this mental hiccup, I accepted it as a “browser light” that renders the page so the search engine now understands what appears where and how on a page – even if they don’t have to. eyes to see it.
When all goes well, the rendered version will appear the same to Googlebot as it does to graphical browsers, and if it doesn’t, it’s likely because the page relies on an unsupported feature, such a user permission request, or one of the wrong scripts or other resources.
What about pre-rendering?
Basically, it’s an ethical form on cloaking, where you make a copy of the page as it would appear in the DOM, and pass it to search engines, to make sure they see the same content that a user does when stopping to index content.
“Ask and you will sometimes receive” a response from Google.
And it was one of those times.
The answer was:
You usually don’t need it now.
— 🍌 John 🍌 (@JohnMu) August 2, 2020
Which is great news for anyone using Puppeteer or another pre-rendering library.
I know I’ve seen cases of the pre-rendering system crashing without error notification, causing a whole lot of headaches (read: pages falling off the index).
If we don’t need pre-rendering, we don’t have to worry about such things.
Of course, the key word here was “usually”.
So if you’re considering disabling your pre-rendering system, I have to recommend that you stop the system from working on a handful of pages and wait to see what happens when they’re fetched.
Does Google see the content as it is displayed?
If so, you may be able to stop pre-rendering altogether.
Rendering gives engines the ability to prioritize content based on how a human would likely interact with a page.
It lets the engine know how content is positioned in a browser and how visible different elements are. So when they try to judge or prioritize content or assess usability, they are working with the same product as a visitor.
The future is made
The timeshift change and Mueller’s pre-render statement put the writing on the wall.
Indexing, if we think it’s likely to become a functionally irrelevant step from an SEO perspective, with rendering taking center stage when it comes to web content discovery.
In this article, we have tried to broadly cover what rendering is.
This may leave you with a number of questions. And it should.
To answer that, I’m going to direct you to some important resources.
I cannot recommend highly enough following the links below:
Featured image: Adobe Stock edited by author
Jumper Image: Adobe Stock, edited by author
All screenshots taken by the author