afaqrashid.com  ·  portfolio · 2026 ed.

Next.js robots.txt: Why You Shouldn't Block /_next/

By Afaq Rashid ·

Plenty of Next.js sites have this in their robots.txt. Mine did.

User-agent: *
Allow: /
Disallow: /_next/

It looks tidy. /_next/ sounds like build output, and build output sounds like something a search engine has no reason to read. But that folder is how your pages reach a browser, and Google renders pages in a browser.

What lives under /_next/

  • /_next/static/ holds your compiled JavaScript and CSS, plus fonts and imported media. The folders inside it can change between Next.js versions and hosts, so don't write rules for them.
  • /_next/image is the default image optimization route. Every next/image on your site loads through it, unless you've set a custom loader or turned optimization off.

None of this is private. Anyone who opens your site downloads these files, so blocking them hides nothing.

What the rule does to Google

Google crawls a URL, renders the page in headless Chromium, then indexes the rendered HTML. It reads robots.txt first and skips any URL you've disallowed.

So with /_next/ blocked, Google gets your HTML and not much else. Its JavaScript docs are direct: "Google Search won't render JavaScript from blocked files or on blocked pages." Its robots.txt docs warn that if you block resources a page needs, Google "won't do a good job of analyzing pages that depend on those resources."

Images take a hit too. A next/image points to a URL under /_next/image, and robots.txt is one of the methods Google documents for keeping images out of search results. A blanket rule does that to every optimized image on the site.

Why nobody notices

Next.js sends real HTML from the server, so your text and links are in the first response. The page still gets indexed and can still rank for your name.

Your scores won't warn you either. My site scored 100 on all four Lighthouse categories with this rule in place.

The damage shows up elsewhere. Google analyzes a page without its styles, and anything that only appears after JavaScript runs is at risk.

The fix

Delete the line. If you have routes that crawlers shouldn't request, block those by name.

import type { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/", disallow: "/private/" },
    sitemap: "https://example.com/sitemap.xml",
  };
}

Be careful with /api/ as well. If a page loads its content from an API route in the browser, blocking that route keeps the content from Google for the same reason.

Check your own site

In Search Console, open URL Inspection, run a live test and open the tested page. You get a screenshot of what Google rendered and a list of the resources it loaded. Google's help page says visual differences there "might be the result of resources that are blocked".

For a quicker check, paste your robots.txt into the free checker linked below. It runs in your browser and shows which Next.js paths are blocked and which line is doing it.

robots.txt isn't a privacy tool

If the goal was to hide something, this rule was never going to do it. Google says robots.txt "is not a mechanism for keeping a web page out of Google", and a blocked URL can still appear in results without a description.

To keep a page out of search, use noindex or a password. Don't block that page in robots.txt as well, because Google has to crawl it to see the noindex.

Related

Sources