Understand How Meta Tags Affect Search Engine Crawlers

We all talk about on-page and off-page SEO and how useful it is for boosting the ranking potential of your site. But today, we’re going to dig even deeper—into the nitty-gritty technical aspects of meta tags and their impact on your SERP rankings.

There are a lot of different meta tags, and each one has a specific purpose, often related to telling search engine crawlers and browsers how to interpret your site. But not all meta tags are created equal—some impact how your content appears in the SERP while others tell crawlers and browsers how to interpret your page.

Whether you’re building a new page or optimizing an existing one, understanding how to use meta tags and which ones to focus on can have a significant impact on how search engines and website visitors interact with your content.

What Are Meta Tags?

Meta tags are HTML elements in the head section of your website. Search engines use them to learn specific information about your page, such as the title and description of the content. While these meta tags won’t be displayed to readers on your website directly, they are a vital piece of any technical SEO audit.

Example <head> section from What is Search Intent and Why Does it Matter.

Each of these tags provides an important piece of information about how web crawlers should parse the metadata on your website. Basically, it’s a way to tell search engines, and sometimes browsers, how to interpret the content you’ve created.

Let’s break down some of the important meta tags in the screenshot example:

<meta charset="utf-8" />

This defines the character encoding for your website. It tells browsers and search engines how to interpret the code/content of your website at the base level. Basically, it’s defining the language your website uses. UTF-8, also known as Unicode, is the character encoding that covers almost all of the characters and symbols in the world.

<title>What is Search Intent and Why Does it Matter?</title>

The title of the page will appear as the title in the SERP.

<meta name="description" content="Understanding search intent helps you create relevant and engaging content that's both interesting for readers and ranks well in the SERP. Read how it works" />

This tag summarizes your page's content and will display below the title in the SERP and will tell searchers whether or not the page will help answer their query. Just keep in mind that Google can also pull content from your page directly for various different search features.

The title and meta description are probably the most important tags you define on a page-by-page basis, as they’re going to be unique to the content that’s displayed. In the screenshot, we also see the following meta tags:

<meta http-equiv="X-UA-Compatible" content="IE=edge" />

<meta name="HandheldFriendly" content="True" />

<meta name="viewport" content="width=device-width, initial-scale=1.0" />

All three of these impact how your entire page will be displayed on different browsers or mobile devices.

  • X-UA-Compatible tells Microsoft Edge to display the page in the highest mode possible.
  • HandheldFriendly tells browsers and web crawlers that your site uses mobile markup—meaning that the page is optimized for mobile browsers.
  • Viewport tells browsers how to control the page's dimensions and scaling when your site is viewed on different devices (desktop, phone, tablet, etc.).

And the list goes on. There are a lot more meta tags that tell browsers and crawlers how to interpret your website, which we’ll dig into in the next section.

The Difference between SEO Meta Tags and Open Graph Meta Tags

There are tags in our screenshot that include OG in the property designation:

<meta property="og:site_name" content="The Official SpyFu Blog" />

<meta property="og:type" content="article" />

<meta property="og:title" content="What is Search Intent and Why Does it Matter?" />

It’s important to include these tags in your blog content to ensure that sharing it is a seamless experience for readers. If you don’t, when a reader shares your content their post will only include the URL for your article itself, no title, image, or description will appear.

Including these tags can drive traffic back to your site and promote your brand.

How Search Engine Crawlers Use Meta Tags

Think of meta tags as a kind of roadmap for search engine crawlers. Whenever a bot lands on your website, its goal is to learn as much as possible and bring that data back to its respective search engine for indexing. Proper meta tag usage ensures that you’re telling these bots the correct information in a way that’s easy for them to parse.

We can break down the different meta tags based on what they tell crawlers about your website. There are meta tags that:

  • tell crawlers how to interact with your website
  • tell search engines how to display information in the SERP
  • tell browsers and devices how to display your site when it’s accessed

Meta Tags That Impact How Crawlers Interact With Your Website

As soon as a crawler lands on your site, the bot needs to understand how to interact with it. If you’re not defining certain meta tags in the <head> section of your website, it’s impossible for web crawlers to understand the information they’re meant to read. Some of these elements will be defined at the site level, like charset and hreflang, but others will define how specific pages should be crawled.

Charset Meta Tag

<meta charset="utf-8" />

We referenced this earlier, but this meta tag is one of the first things a crawler needs to know when they access your website. That’s why it’s the first meta tag you’ll see in the <head> section of your website. Without it, both crawlers and browsers won’t be able to render your site.

Hreflang Meta Tag

There are basically two ways to define the language that’s being used on your website. The first is a universal HTML tag that appears at the top of your website’s source code:

Example HTML lang tag on What is Search Intent and Why Does it Matter.

This is a simple way to define the content of a website that is static across the entirety of the content you produce. The other way is through the use of an hreflang meta tag, which is more often used for indexing a multi-lingual website.

<link rel="alternate" href="http://example.com/" hreflang="en" />

<link rel="alternate" href="http://example.fr/" hreflang="fr" />

This example includes a website that has both an English and a French version. By defining which language is used for each site, you’re telling Google to index each individually. That ensures that the English variant will appear in English search results and the French variant in French results.

While this may not be pertinent for every website, it’s important to understand how language and country origin impact the indexing (and therefore rankability) of a given website. If you’re using the hreflang tag, just make sure to follow all Google guidelines for designating the correct country. The more countries you’re translating content for, the more complicated it will get.

Now, let’s move on to the tags that define how your website should be crawled.

Robots Meta Tag

<meta name="robots" content="noindex" />

Sometimes you may need to tell crawlers to bypass certain pages of your website for indexing. Whether those pages contain sensitive data or are just not built to show up in the SERP, applying a robots meta tag with the noindex designation will make any crawlers ignore that page.

Just make sure you’re not using this on a page that you want people to be able to find through search. This meta tag will remove the page from a search engine’s index entirely.

Keywords Meta Tag

<meta name="keywords" content="SEO,Competitor Analysis, SpyFu">

The keywords meta tag was originally created to define a set of relevant keywords for specific pages of your website. We say "was" because it’s an outdated method of SEO optimization that dates back to 1995 when search engines weren’t nearly as powerful as they are today.

We don’t recommend you use this meta tag on your website because, ultimately, it serves no purpose. Google et al. will crawl your site content directly and make a decision about what keywords are relevant to the page itself. But it is still something you might see in older websites that have been around since the early 2000s. It helps to be familiar with what you need as well as what you don't need.

Canonical Meta Tag

<link rel="canonical" href="https://www.spyfu.com/blog/what-is-search-intent/" />

This meta tag tells search engines which page is the master copy of any given page. It’s used to cut down on the potential for duplicate content and tells the search engine which version of your page should be displayed in the SERP. You should define this on every page of your blog, even when syndicating content to other publications like LinkedIn or Medium, so you don’t end up cannibalizing your rankings with multiple versions of the same page.

Using this tag also helps you cut down on the time it takes for crawlers to read your entire site, which mitigates any negative impact on site speed while they move through every page.

Site Verification Meta Tag

<meta name="google-site-verification" content="your verification string">

Site verification is a process where you claim ownership over a given page of your website. The site verification meta tag is not the only way to verify your site with Google and is therefore used more as a backup verification method.

If you’re worried about verifying your ownership of the site, you’ll be able to receive a specific verification string from Google to plug into the code directly. It will ensure that the content is linked to the correct Google account of your choosing.

Meta Tags That Impact How Your Website Appears in the SERP

Whenever someone searches for a particular question or keyword, search engines have to decide what results are the best match to answer that question. Once they’ve ranked the results, they then need to display them on the SERP. That’s where your title, description, and URL meta tags come into play. By defining these elements in the <head> section of your website, you’re telling the search engine exactly what information you want to be displayed.

SERP results for What is Search Intent and Why Does it Matter.

This will look familiar to everyone, but it wouldn’t be possible without defining the proper meta tags for each page of your website.

Meta Tags That Define How Your Website is Rendered

These meta tags will help you manage how your website appears in the different browsers and devices that searchers use. Many will be set directly at the website level, but you still need to understand how they work to ensure your technical SEO is the best it can be.

Viewport Meta Tag

<meta name="viewport" content="width=device-width, initial-scale=1.0" />

The viewport tag is one of the core components of mobile responsiveness. This meta tag gives browsers the information they need to render your site correctly across multiple different devices.

It’s important to define it properly. Failing to do so can create a poor experience when people browse your website, which can negatively impact your rankings in the SERP. It’s one of the best tools you have for ensuring a seamless experience regardless of how a user interacts with your website.

Refresh Meta Tag

<meta http-equiv="refresh" content="X">

This meta tag is a method for telling a given page to refresh every X number of seconds. Spam websites often use this tactic to increase the amount of ad revenue they can gain from a given page by updating the ads that are displayed a number of times per each site visit.

The only practical example of using a refresh like this would be for interstitial pages, like those that say “This page will redirect automatically after 15 seconds,” which are commonly used in the post-purchase experience.

We don’t recommend using this as a redirect method for your website because it can negatively impact your rankings if crawlers see a number of pages refreshing in this way. There are also a lot of other methods to refresh that are much better for SEO.

Now that you understand a bit more about how these different meta tags impact the SEO of your website, we’ll break down some best practices for how to use them effectively across the board.

Best Practices for Defining the SEO Meta Tags on Your Website

Meta tags dictate much of how web crawlers interact with your website, so you need to get them right. If you’re not using the correct syntax or ignoring character limits, you run the risk of negatively impacting your rankings and confusing searchers at the same time.

While most CMS platforms will have built-in functionality to help you define these meta tags, at least at the website level, it’s important to check each page of your website before it’s published or after it’s updated. Doing so will ensure that, even with your careful consideration of these meta tags, you don’t run into any issues.

There are also plugins like Yoast for WordPress that make the process of updating meta tags easier. These kinds of third-party plugins are a great choice if you’re not comfortable interacting with code directly on your website. In some cases they guide you toward best practices for SEO.

As with any HTML, the syntax you use is set. Use resources like W3 to double-check how to write these meta tags before going live with any updates.

Character limits are also important, especially for your meta title and description. If you’re not careful, you’ll end up truncating important information, so searchers can’t find it. It is generally accepted that a meta title should be no more than 60 characters, and a meta description should be no more than 158 characters. Take this example, where the second sentence of the meta description is cut off abruptly without providing any information.

Example of a truncated meta description on Google.

That said, on mobile, search engines will truncate your meta descriptions, so you’ll want to aim for less than 120 characters.As long as you’re following these best practices, you should have no problem giving web crawlers the information they need to properly index your site. That, in turn, will give searchers a better overall experience interacting with your website in the SERP.

Meta Tags Tell Search Engines Important Information Quickly

When you use the right meta tags in the right way on your website, they tell search engines everything they need to know about your website. Having that data readily available makes it easy for crawlers to parse the data and bring it back to be indexed.

Optimizing your meta tags can, therefore, positively impact your ranking ability and provide searchers with a better overall SERP experience. Don’t let misplaced or poorly constructed meta tags cause trouble for the SEO of your website.