afaqrashid.com  ·  portfolio · 2026 ed.

Next.js robots.txt or sitemap.xml 404? 5 Causes I Tested

By Afaq Rashid ·

If /robots.txt or /sitemap.xml on your Next.js site returns a 404, redirects to a login page, or shows a file you didn't expect, it's one of five causes. I tested each one on a fresh Next.js 16.4.4 app with next build and next start.

1. Your proxy or middleware catches it

This is the one I'd check first on any site with sign-in. A proxy.ts (middleware.ts before Next.js 16) that sends signed-out visitors to /login also sends crawlers there, because they don't have a session either. With this matcher:

// proxy.ts
export function proxy(req: NextRequest) {
  if (!req.cookies.get("session") && req.nextUrl.pathname !== "/login") {
    return NextResponse.redirect(new URL("/login", req.url));
  }
}

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico).*)"],
};

both files redirected:

/robots.txt   307 -> /login
/sitemap.xml  307 -> /login

Add both paths to the exclusions, and they return 200 again:

export const config = {
  matcher: ["/((?!_next/static|_next/image|favicon.ico|robots.txt|sitemap.xml).*)"],
};

Any other file crawlers need should be on that list too, such as llms.txt or an IndexNow key file.

2. The file isn't at the top of app/

robots.ts has to sit directly in app/, or src/app/ if you use a src folder. I moved it into a route group, app/(site)/robots.ts, and /robots.txt returned a 404. The build passed without a warning.

3. There's a robots.txt in public/ as well

I put app/robots.ts and a different public/robots.txt in the same project. The two servers did different things:

next dev               500: A conflicting public file and page file was found for path /robots.txt
next build && start    200, serves public/robots.txt

So in production, the file you edit in app/robots.ts can be ignored with no error at all. Keep one. If you generate robots.txt in code, delete public/robots.txt. The same goes for a public/sitemap.xml next to app/sitemap.ts.

4. Your sitemap.ts is in a subfolder

A sitemap file serves at the path of the folder it's in. app/blog/sitemap.ts gave me /blog/sitemap.xml with a 200, and /sitemap.xml stayed a 404.

That's allowed, but nothing will find it at the root. Either move it to app/sitemap.ts, or list its real URL in robots.txt:

// app/robots.ts
export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: "*", allow: "/" },
    sitemap: "https://example.com/blog/sitemap.xml",
  };
}

5. robots.txt points at a sitemap that isn't there

The sitemap line in robots.txt is just a URL. In my test, robots.ts named https://example.com/sitemap.xml, but the project had no app/sitemap.ts, so that address was a 404.

Open the sitemap URL from your live robots.txt in a browser. It should return XML from your production domain, not a 404, a login page or a localhost address.

Check it from outside

Test without your cookies, the way a crawler arrives:

curl -sI https://example.com/robots.txt | head -1
curl -sI https://example.com/sitemap.xml | head -1

Both should say 200. A 307 or 308 usually means a proxy or a redirect rule caught the file. Once it loads, paste it into the robots.txt checker below to see whether it blocks Google from your Next.js JavaScript, CSS, fonts or images.

Related

Sources