SEO Basics: Meta Tags, Sitemaps and Robots.txt Checklist

By Muhammad Rehan Baig · October 14, 2026 · 8 min read

If you run a small website, such as a local business, a portfolio, a blog or a side project, you do not need an agency to get the technical basics of SEO right. A handful of tags in your pages, two small files at the root of your site and a free Google account cover most of what search engines need to find, understand and display your pages properly. This checklist explains each piece, what it actually does (and does not do), and the mistakes that quietly hold small sites back.

How search engines see your site

It helps to know the three stages a page goes through:

1. Crawling: a search engine's bot discovers your URL, usually through links or a sitemap, and downloads the page.
2. Indexing: the search engine analyzes the content and decides whether to store the page in its index, and which URL represents it.
3. Ranking and display: when someone searches, the engine picks relevant pages from the index and builds the result snippet you see.

The tags and files in this guide each help one of those stages. Robots.txt and sitemaps guide crawling, canonical and robots meta tags guide indexing, and titles, descriptions and structured data shape how you appear in results. None of them can make a thin page rank, but getting them wrong can stop a good page from ranking at all.

Part 1: The tags in your page's <head>

The title tag

The <title> is the most important tag on the page. Search engines use it as a strong clue to what the page is about, and Google usually shows it as the clickable headline in results.

• Make every title unique across your site. Duplicate titles make it hard for search engines (and people) to tell pages apart.
• Put the main topic first, then your brand: "Emergency Plumber in Springfield | Smith Plumbing".
• Keep it reasonably short. There is no official character limit; Google truncates titles that are too wide for the result, so around 50–60 characters is a practical target.
• Write for humans. Repeating keywords looks spammy, and Google may replace a title it considers unhelpful with text from elsewhere on the page.

The meta description

The meta description is a short summary of the page. Google has said it does not use it as a ranking signal, but it often uses it as the snippet text under your title, which affects whether people click.

• Write one or two sentences that tell searchers exactly what they will get, ideally with a reason to choose you.
• Aim for roughly 120–160 characters so it is not cut off mid-thought.
• Make it unique for each important page.
• Expect Google to rewrite it sometimes to match a specific query. That is normal.

Before you publish, paste your title and description into the SERP Snippet Preview to see how they are likely to look and where they might be cut off.

The tags that are easy to forget

• Viewport: <meta name="viewport" content="width=device-width, initial-scale=1"> makes your page render properly on phones. Google primarily uses the mobile version of pages for indexing, so this matters.
• Canonical: <link rel="canonical" href="https://example.com/page/"> tells search engines which URL is the main version when the same content is reachable at several addresses (with and without www, with tracking parameters, with or without a trailing slash). Google treats it as a strong hint rather than a command.
• Robots meta tag: <meta name="robots" content="noindex"> keeps a page out of search results. Use it for thank-you pages, internal search results and test pages.
• Language: set <html lang="en"> so browsers and assistive technology know the page language.
• Open Graph tags: og:title, og:description and og:image control the preview card when someone shares your link on social media and messaging apps. They do not affect rankings, but they affect clicks. The Open Graph Generator builds them.

One tag you can skip: the meta keywords tag. Google has ignored it for ranking for many years.

Here is what a solid head section looks like for a typical page:

<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Emergency Plumber in Springfield | Smith Plumbing</title>
  <meta name="description" content="Licensed plumbers available 24/7 in Springfield. Upfront pricing, no call-out fee on weekdays. Call or book online.">
  <link rel="canonical" href="https://smithplumbing.example/emergency-plumber/">
  <meta property="og:title" content="Emergency Plumber in Springfield">
  <meta property="og:image" content="https://smithplumbing.example/img/van.jpg">
</head>

If you would rather not write these by hand, the Meta Tag Generator produces the whole block from a simple form.

Part 2: robots.txt

Robots.txt is a plain text file that tells crawlers which parts of your site they may request. It must live at the root of the host, for example https://example.com/robots.txt; a robots.txt file in a subfolder is ignored. A simple one looks like this:

User-agent: *
Disallow: /admin/
Disallow: /search
Allow: /

Sitemap: https://example.com/sitemap.xml

The User-agent line says which crawlers the rules apply to (* means all), Disallow and Allow list path prefixes, and the Sitemap line points crawlers to your sitemap using a full URL. The Robots.txt Generator helps you put one together without syntax mistakes.

Three robots.txt misunderstandings

1. Disallow does not mean "do not index." Robots.txt controls crawling, not indexing. A blocked URL can still appear in results, usually without a description, if other pages link to it. To keep a page out of search results, let it be crawled and add a noindex robots meta tag; if the page is blocked, crawlers never see the noindex.
2. It is not a security feature. Anyone can read your robots.txt, and listing secret folders in it advertises them. Protect private areas with passwords instead.
3. One wrong line can hide your whole site. Disallow: / under User-agent: * blocks everything. It is sometimes left over from a staging site, so check yours after every launch or redesign.

Part 3: XML sitemaps

An XML sitemap is a list of the URLs you want search engines to know about. It follows a shared protocol supported by the major search engines and looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2026-10-01</lastmod>
  </url>
  <url>
    <loc>https://example.com/services/</loc>
    <lastmod>2026-09-12</lastmod>
  </url>
</urlset>

Sitemaps matter most for new sites with few incoming links, large sites and pages that are not well linked internally. For a small, well-linked site, a sitemap mostly speeds up discovery, but it is cheap insurance.

• List only canonical, indexable pages that return a normal 200 response. Leave out redirects, error pages, noindex pages and duplicate URLs.
• Use full, absolute URLs with the exact protocol and domain your site uses.
• Keep lastmod honest. Google uses it only when it is consistently accurate, so update it when the content meaningfully changes, not on every build.
• Skip priority and changefreq if you like. Google has said it ignores both.
• Mind the limits: one sitemap file can hold up to 50,000 URLs and 50 MB uncompressed. Larger sites split their URLs across several sitemaps and list them in a sitemap index file.

The XML Sitemap Generator turns a list of URLs into a valid file. Upload it to your site's root, reference it in robots.txt, and submit it in Google Search Console.

Part 4: Structured data

Structured data is extra, machine-readable information about a page, written in the shared schema.org vocabulary. Google recommends the JSON-LD format, a small script block in the page. It can make pages eligible for rich results, such as review stars, product prices or event dates, and it helps search engines connect your business name, address and logo.

Good starting types for small sites include Organization or LocalBusiness on your home page, Article or BlogPosting on posts, Product on product pages and BreadcrumbList for site navigation. The Schema Markup Generator builds the JSON-LD for common types.

Keep expectations realistic. Structured data must describe content that is actually visible on the page, and it makes a page eligible for rich results without guaranteeing them. Google has also narrowed some features over time: since 2023, FAQ rich results are shown only for a small set of authoritative government and health sites, and how-to rich results have been retired. Test your markup with Google's Rich Results Test before relying on it.

Part 5: On-page basics beyond the head

• One clear H1 per page that matches the topic, followed by logical H2 and H3 subheadings.
• Short, readable URLs with words separated by hyphens, such as /emergency-plumber/ rather than /page?id=482. The Slug Generator converts titles into clean URL slugs.
• Descriptive alt text on images, written for someone who cannot see them. It helps accessibility and image search.
• Internal links with meaningful anchor text, like "see our water heater repair prices" rather than "click here."
• HTTPS everywhere, with HTTP and non-preferred hostnames redirecting to the main version.
• Fast pages. Google's Core Web Vitals measure loading (Largest Contentful Paint, good at 2.5 seconds or less), responsiveness (Interaction to Next Paint, good at 200 milliseconds or less) and visual stability (Cumulative Layout Shift, good at 0.1 or less). Oversized images are the most common culprit on small sites; see our guide on compressing images without losing quality.

Part 6: Set up Google Search Console

Search Console is free and shows you what Google actually sees. After verifying your site:

1. Submit your sitemap in the Sitemaps report.
2. Check the Pages (indexing) report for pages that are excluded and why.
3. Use URL Inspection on important pages to see the canonical Google chose and whether the page is indexed.
4. Watch the Performance report for the queries that bring visitors, and improve titles and descriptions for pages with many impressions but few clicks.

Bing offers a similar free service, Bing Webmaster Tools, and accepts the same sitemap.

The checklist

Key takeaways

• Title tags matter most; meta descriptions do not affect rankings but do affect clicks.
• Robots.txt controls crawling, not indexing; use noindex to keep a page out of results.
• Sitemaps should list only canonical, indexable URLs with honest lastmod dates.
• Structured data makes pages eligible for rich results but never guarantees them.
• Search Console shows you what Google actually sees, so check it after every change.