Post by account_disabled on Mar 5, 2024 9:08:50 GMT
Some content on your site is not necessary for search engine indexing. To prevent some pages from being indexed, you can use a robots meta tag or x-robots-tag. However, it is not uncommon for the robots and robots.txt meta tags to be used incorrectly. This results in messy, conflicting directives that don't achieve the desired result: preventing a page from being indexed. In this guide we want to help you understand how to use the robots and x-robots-tag meta tags to control the indexing of your website content and to identify the most commonly made errors. Specifically, we will see: What is a robots Meta tag? Understand robots Meta tag attributes and directives Robots Meta tag code examples Use robots Meta tags to control snippets What is an X‑Robots-Tag? How to set Meta tag robots and X-Robots-Tag Common mistakes with Meta robots What is a robots Meta tag? A robots Meta tag, also known as a robots tag, is a piece of HTML code that is placef a web page and used to control how search engines crawl and index the URL.
Here's what a robots meta tag looks like in a page's source codllow you to instruct Venezuela Phone Number search engines on how you want them to handle the page and whether or not to include it in the index. What are Meta tag robots for? Meta robot tags are used to control how Google indexes the content of your web page. This includes decisions such as: Whether or not to include a page in search results Whether or not to follow links on a page (even if it is blocked for indexing) The request not to index images on a page The request not to show cached results of the web page in SERPs The request not to show a snippet (meta description) for the page in the SERPs To understand how to use the robots meta tag, we need to look at the different attributes and directives. We'll also share code examples that you can take and insert into your page header to request search engines to index your page in a certain way. Do a technical audit of your site with Semrush's Site Audit tool Try it for free! → ADS illustration Understand robots Meta tag attributes and directives Using the robots meta tag is quite simple, once you understand how to set the two attributes: name and content.
Both of these attributes are required, so you need to set a value for each. Let's look at these attributes in more detail. Name The name attribute controls which crawlers and bots (user-agents, also called UA) must follow the instructions contained in the robots meta tag. To instruct all crawlers to follow the instructions, use: name="robots" In most scenarios, we recommend using this as the default, but you can use as many different robots meta tags as you need to specify instructions for different crawlers. When you want to train multiple crawlers, you simply need to use multip name="googlebdreds of different user agents. The most common are: Google: Googlebot (you can see a full list of Google crawlers here ) Bing: Bingbot (you can see a full list of Bing crawlers here ) DuckDuckGo: DuckDuckBot Baidu: Baiduspider Yandex: YandexBot Contents The content attribute is used to provide instructions to the specified user-agent. It's important to know that if you don't specify a robots meta tag on a web page, the default is to index the page and follow all links (unless they have a rel="nofollow" attribute specified inline).
Here's what a robots meta tag looks like in a page's source codllow you to instruct Venezuela Phone Number search engines on how you want them to handle the page and whether or not to include it in the index. What are Meta tag robots for? Meta robot tags are used to control how Google indexes the content of your web page. This includes decisions such as: Whether or not to include a page in search results Whether or not to follow links on a page (even if it is blocked for indexing) The request not to index images on a page The request not to show cached results of the web page in SERPs The request not to show a snippet (meta description) for the page in the SERPs To understand how to use the robots meta tag, we need to look at the different attributes and directives. We'll also share code examples that you can take and insert into your page header to request search engines to index your page in a certain way. Do a technical audit of your site with Semrush's Site Audit tool Try it for free! → ADS illustration Understand robots Meta tag attributes and directives Using the robots meta tag is quite simple, once you understand how to set the two attributes: name and content.
Both of these attributes are required, so you need to set a value for each. Let's look at these attributes in more detail. Name The name attribute controls which crawlers and bots (user-agents, also called UA) must follow the instructions contained in the robots meta tag. To instruct all crawlers to follow the instructions, use: name="robots" In most scenarios, we recommend using this as the default, but you can use as many different robots meta tags as you need to specify instructions for different crawlers. When you want to train multiple crawlers, you simply need to use multip name="googlebdreds of different user agents. The most common are: Google: Googlebot (you can see a full list of Google crawlers here ) Bing: Bingbot (you can see a full list of Bing crawlers here ) DuckDuckGo: DuckDuckBot Baidu: Baiduspider Yandex: YandexBot Contents The content attribute is used to provide instructions to the specified user-agent. It's important to know that if you don't specify a robots meta tag on a web page, the default is to index the page and follow all links (unless they have a rel="nofollow" attribute specified inline).