ivan breslauer.

Next.js: Utilizing sitemap.ts and robots.ts Without Migrating to App Router

Last updated: Mar 18, 2024 | 3 min read
Cover image

This blog post was spontaneously generated by Claude 3 with minimal input. Consider it a placeholder until I write a piece that's informative and entertaining (sarcasm included). Until then, don't believe a single word written below.

Next.js 13.4 introduced the new App Router, a highly anticipated feature that promises improved performance and developer experience. However, not all projects are ready for a complete migration to the App Router, especially those with complex routing or data fetching requirements. Fortunately, Next.js provides a way to keep your project on the Pages Router while still benefiting from some of the new features introduced in the latest version, such as sitemap.ts and robots.ts.

The Pages Router vs. App Router

The Pages Router is the traditional routing system in Next.js, where each page is represented by a file in the pages directory. This approach has served developers well for years, providing a straightforward and intuitive way to create and manage routes.

On the other hand, the App Router is a new routing system introduced in Next.js 13. It offers a more flexible and efficient way to handle routing, especially for complex applications with nested layouts and data dependencies. The App Router is based on React Server Components, which allow for better code organization, improved performance, and easier data fetching.

Keeping Your Project on Pages Router

If you're not ready to migrate your entire project to the App Router, you can continue using the Pages Router while still taking advantage of some of the new features introduced in Next.js 13.4. One of these features is the ability to generate a sitemap and robots.txt file using sitemap.ts and robots.ts files, respectively.

Generating a Sitemap with sitemap.ts

To generate a sitemap for your Pages Router-based project, you can create a sitemap.ts file in the root directory of your project. Here's an example of how you can generate a sitemap for a blog:

// sitemap.ts
import { getServerSideSitemap } from "next-sitemap";
import { getPosts } from "./utils/posts";

export const getServerSideSitemap = async (ctx) => {
  const posts = await getPosts();

  const fields = posts.map((post) => ({
    loc: `https://your-website.com/blog/${post.slug}`,
    lastmod: post.lastModified,
  }));

  return getServerSideSitemap(ctx, fields);
};

In this example, we're importing the getServerSideSitemap function from the next-sitemap package and a hypothetical getPosts function that fetches the list of blog posts. We then map over the posts and create an array of sitemap fields, which includes the URL and last modified date for each post. Finally, we call getServerSideSitemap and pass in the context object (ctx) and the array of sitemap fields.

Generating robots.txt with robots.ts

To generate a robots.txt file for your Pages Router-based project, you can create a robots.ts file in the root directory of your project. Here's an example:

// robots.ts
import getBaseUrl from "./utils/getBaseUrl";

const baseUrl = getBaseUrl();

export const robots = {
  rules: [
    {
      userAgent: "*",
      allow: "/",
      disallow: ["/admin", "/auth"],
    },
  ],
  sitemap: `${baseUrl}/sitemap.xml`,
};

In this example, we're importing a getBaseUrl utility function that returns the base URL of our website. We then define an object with the rules and sitemap properties, following the robots.txt format. The rules property specifies which URLs should be crawled or disallowed by search engines. In this case, we're allowing all URLs (/) but disallowing the /admin and /auth paths. The sitemap property specifies the URL of our sitemap, which we generated earlier using sitemap.ts.

By creating these two files, Next.js will automatically generate a sitemap.xml and robots.txt file at build time, ensuring that your website is properly indexed and crawled by search engines.

Conclusion

While the App Router in Next.js 13.4 offers exciting new features and improvements, not everyone is ready to migrate their entire project just yet. Fortunately, Next.js provides a way to keep your project on the Pages Router while still benefiting from some of the new features, such as sitemap.ts and robots.ts. By following the examples shown in this blog post, you can ensure that your website is properly indexed and crawled by search engines, even if you're not using the App Router.

As with any new feature or major version update, it's important to carefully evaluate the migration process and ensure that your project is ready for the changes. However, by taking advantage of the flexibility offered by Next.js, you can adopt new features at your own pace while still maintaining a productive and efficient development workflow.

ib7r.dev