The problem with transforms

Zod's .transform() method is great for one-way data conversion:

const stringToNumber = z.string().transform(val => parseFloat(val));
stringToNumber.parse("42"); // 42

But what if you need to go both ways? Say, you're storing dates as ISO strings in a database but want to work with Date objects in your app.

const stringToDate = z.string().transform(str => new Date(str));
const dateToString = z.date().transform(date => date.toISOString());

// Two separate schemas, manually kept in sync
stringToDate.parse("2024-01-15T10:30:00.000Z"); // Date
dateToString.parse(new Date()); // "2024-01-15T10:30:00.000Z"

This works, but it's brittle. You need to keep track of two schemas and remember that they are intended as inverses. You need to manually verify that the output type of one matches the input type of the other. If you change one, you have to remember to update the other.

Introducing codecs

Codecs are a new Zod API for defining bidirectional transformations between two types. You specify an input schema, output schema, and transformation functions in both directions:

const stringToDate = z.codec(
  z.iso.datetime(),  // input schema: ISO string
  z.date(),          // output schema: Date object
  {
    decode: isoString => new Date(isoString), // string → Date
    encode: date => date.toISOString(),       // Date → string
  }
);

You can process data in both directions using the new top-level .decode() and .encode() methods:

stringToDate.decode("2024-01-15T10:30:00.000Z"); // Date
stringToDate.encode(new Date("2024-01-15"));     // "2024-01-15T00:00:00.000Z"

Note — For bundle size reasons, these new methods have not added to Zod Mini schemas. Instead, this functionality is available via equivalent top-level functions.

// equivalent at runtime
z.decode(stringToDate, "2024-01-15T10:30:00.000Z");
z.encode(stringToDate, new Date());

This is particularly important when you are using Zod to map data back and forth between two different domains. One common use case is to convert data to/from a serializable format like JSON into a richer JavaScript representation (with Date, bigint, etc).

Async

The transformation functions can be async.

const asyncCodec = z.codec(z.string(), z.number(), {
  decode: async str => Number(str),
  encode: async num => num.toString(),
});

The usual "safe" and "async" variants exist:

syncCodec.encode("42");
syncCodec.safeEncode("42");
await asyncCodec.encodeAsync("42");
await asyncCodec.safeEncodeAsync("42");

Composability

Codecs can be composed inside other schemas, just like any other schema. There are no special rules.

const queryParams = z.object({
  before: stringToDate,
  after: stringToDate
})

queryParams.encode({
  before: new Date(),
  after: new Date()
});
// => { before: string, after: string }

.parse() vs .decode()

Let's compare the existing .parse() APIs to .decode(). .parse() is equivalent to .decode() at runtime.

// equivalent at runtime
stringToDate.parse("2024-01-15T10:30:00.000Z"); 
stringToDate.decode("2024-01-15T10:30:00.000Z");

Though they're identical at runtime, their type signatures differ in an important way. While .parse() accepts unknown, decode expects a strongly-typed inputs.

stringToDate.parse(12345); 
// No TypeScript error but fails at runtime

stringToDate.decode(12345);
// ❌ TypeScript error: Argument of type 'number' is not assignable to parameter of type 'string'

Here's a diagram demonstrating the differences:

This is a highly requested feature unto itself.

• #3860 Add strongly typed parse function
• #1748 Typed input for parse methods
• #3978 Type-safe parsing with known input types
• #1892 Strongly typed decode function

How encoding works

Most Zod schemas in the universe don't perform any kind of transformation. Their inferred input and output types are identical. For these schemas, there is no difference between parsing/decoding and encoding.

const mySchema = z.object({
  name: z.string()
});

// no difference
mySchema.parse({ name: "colinhacks" });
mySchema.decode({ name: "colinhacks" })
mySchema.encode({ name: "colinhacks" })

A small number of APIs cause the input and output types to diverge. In these scenarios, the runtime behavior of .decode()/.encode() also differ.

Codecs

This is an obvious one. During .decode(), the decode function runs. During .encode(), the encode function runs. Simple.

Transforms ⚠️

This is the #1 rule of .encode(): you can't use .transform(). That API is inherently unidirectional. If your schema contains any transforms, attempting an "encode" operation with it will throw a runtime error. You'll need to refactor to use z.codec().

const schema = z.string().transform(val => val.length);

schema.encode(5); 
// ❌ ZodEncodeError: Encountered unidirectional transform during encode

Pipes

Note — Codecs are actually implemented as a subclass of ZodPipe augmented with "interstitial" transform logic.

Pipes reverse their order during encoding, from A → B to B → A. That said, pipes are typically used in conjunction with transforms, so "vanilla" pipes are rarely useful in the context of encoding. Prefer z.codec() everywhere.

Refinements

All checks (.refine(), .min(), .max(), etc.) are still executed in both directions.

const schema = stringToDate.refine((date) => date.getFullYear() > 2000, "Must be this millenium");

schema.encode(new Date("2000-01-01"));
// => Date

schema.encode(new Date("1999-01-01"));
// => ❌ ZodError: [
//   {
//     "code": "custom",
//     "path": [],
//     "message": "Must be this millenium"
//   }
// ]

To avoid unexpected errors in your custom .refine() logic, Zod performs two "passes" during .encode(). The first pass ensures the input type conforms to the expected type (no invalid_type errors). If that passes, Zod performs the second pass which executes the refinement logic.

This approach means all parsing & refinement logic runs in exactly the reverse order during encoding. Even "mutating refinements" like z.string().trim() or z.string().toLowerCase() work as expected.

const schema = z.string().trim();

schema.decode("  hello  ");
// => "hello"

schema.encode("  hello  ");
// => "hello"

Default/prefault

Default and prefault values are only applied in the forward direction.

const withDefault = z.string().default("hello");

withDefault.decode(undefined); // "hello"
withDefault.encode(undefined); // ❌ ZodError

This is by design. When you add a default, the input becomes string | undefined but the output stays string. As such, undefined isn't considered a valid input to .encode().

Catch

Similarly, .catch() values are only applied in the forward direction.

Stringbool

Note — Stringbool pre-dates the introduction of codecs in Zod. It has since been internally re-implemented as a codec.

The z.stringbool() API converts string values ("true", "false", "yes", "no", etc.) into boolean. By default, it will convert true to "true" and false to "false" during .encode()..

const stringbool = z.stringbool();

stringbool.decode("true");  // => true
stringbool.decode("false"); // => false

stringbool.encode(true);    // => "true"
stringbool.encode(false);   // => "false"

If you specify a custom set of truthy and falsy values, the first element in the array will be used instead.

const stringbool = z.stringbool({ truthy: ["yes", "y"], falsy: ["no", "n"] });

stringbool.encode(true);    // => "yes"
stringbool.encode(false);   // => "no"

Official codecs

Zod doesn't provide any predefined codecs out of the box. Instead, the docs provide some "canonical" codec implementations you can copy/paste into your projects as needed. These have all been tested internally.

• stringToNumber
• stringToInt
• stringToBigInt
• numberToBigInt
• isoDatetimeToDate
• epochSecondsToDate
• epochMillisToDate
• jsonCodec
• utf8ToBytes
• bytesToUtf8
• base64ToBytes
• base64urlToBytes
• hexToBytes
• stringToURL
• stringToHttpURL
• uriComponent
• stringToBoolean

Some selected examples are below.

stringToBigInt

const stringToBigInt = z.codec(z.string(), z.bigint(), {
  decode: str => BigInt(str),
  encode: bigint => bigint.toString(),
});

stringToBigInt.decode("12345");  // 12345n
stringToBigInt.encode(12345n);   // "12345"

jsonCodec

const jsonCodec = z.codec(z.string(), z.json(), {
  decode: (jsonString, ctx) => {
    try {
      return JSON.parse(jsonString);
    } catch (err: any) {
      ctx.issues.push({
        code: "invalid_format",
        format: "json_string",
        input: jsonString,
        message: err.message,
      });
      return z.NEVER;
    }
  },
  encode: value => JSON.stringify(value),
});

You can pipe jsonCodec into other schemas for additional validation:

const UserFromJson = jsonCodec.pipe(z.object({ 
  name: z.string(), 
  age: z.number() 
}));

UserFromJson.decode('{"name":"Alice","age":30}');  // { name: "Alice", age: 30 }
UserFromJson.encode({ name: "Bob", age: 25 });     // '{"name":"Bob","age":25}'

base64ToBytes

const base64ToBytes = z.codec(z.base64(), z.instanceof(Uint8Array), {
  decode: base64String => z.core.util.base64ToUint8Array(base64String),
  encode: bytes => z.core.util.uint8ArrayToBase64(bytes),
});

base64ToBytes.decode("SGVsbG8=");  // Uint8Array([72, 101, 108, 108, 111])
base64ToBytes.encode(bytes);       // "SGVsbG8="

For further reading, see the Zod 4.1 release notes and the Codecs documentation page.

{
  // package.json
  "llms": "https://docs.cool-library.com/llms.txt",
  "llmsFull": "https://docs.cool-library.com/llms-full.txt",
  "mcpServer": "https://mcp.cool-library.com",
}

I've implemented this in Zod and encourage my fellow library authors to do the same. Moreover, I encourage agentic IDEs/CLIs to implement first-party support for this convention.

If you add support to your library or agent, please fill out this form.

Details

1. The llms field is a URL pointing to an llms.txt file. If you aren't familiar, this is a proposed convention for documentation sites to provide an AI-friendly "sitemap" to AI agents. It's a simple spec: basically just a bunch of categorized Markdown links.
2. The llmsFull field is a URL pointing to an llms-full.txt file containing the full documentation for the library. Typically this is a concatenation of all the .md/.mdx documentation pages in the docs.
3. The mcpServer field is a URL pointing to an MCP server. For a typical library, this server would offer a simple Q&A tool that is capable of answering questions about the library's API/usage. This can use RAG over the library's docs, GitHub issues, Discord server, etc.

Do agents already look at these fields?

No. This is a proposed convention. As more libraries implement this, there is more incentive for agents (IDEs, CLIs, etc) to implement first-party support. On the flip side of the coin, if any agentic CLI/IDE implements support, there is more incentive for libraries to implement this convention. There's a chicken-and-egg problem.

Fortunately, you can probably get your agent of choice to support this with some custom instructions in your AGENTS.md file.

<!-- AGENTS.md -->
For each package in my package.json, read its package.json and look for the `llms`, `llmsFull`, and `mcpServer` fields. If found, use these resources to work with that library:

- `llms`: Use this URL to fetch the llms.txt file, which contains categorized Markdown links to the library's documentation. Use this as a sitemap to understand what documentation is available and how it's organized.

- `llmsFull`: Use this URL to fetch the llms-full.txt file, which contains the complete documentation for the library. Use this for comprehensive understanding of the library's API, features, and usage patterns.

- `mcpServer`: Connect to this MCP server URL to access specialized tools for answering questions about the library. Use this server's Q&A capabilities when I need specific help with the library's API, troubleshooting, or best practices. If possible, prompt the user to add this server to their MCP server registry.

Why not AGENTS.md?

This doesn't solve the same problem.

AGENTS.md is for telling agents how to work inside your codebase. It's like a contributor's guide: how to run tests, run builds, lint, update configs, etc. It gives the agent best practices for modifying a codebase.

This is a qualitatively different problem that a library trying to advertise it's own functionality to an agent. The codebase is already installed in node_modules; agents shouldn't be modifying it.

An agent doesn't care about the package's internals, it only cares about the public API. Like any developer, it doesn't want to read your code to learn how to do stuff—that's hard, time-consuming, and expensive. Having some well-known package.json fields that have an established semantic meaning is a cleaner solution with higher signal-to-noise ratio.

If you maintain a library that implements this convention, fill out this form.

EDIT: A previous version of this post recommended publishConfig, operating under the mistaken belief that it could be used to override "exports" during npm publish. As it turns out, npm only uses "publishConfig" to override certain .npmrc fields like registry and tag, whereas pnpm has expanded its use to override package metadata like "main", "types", and "exports". There are a number of reasons you may not wish to strongly couple your deployment logic to pnpm (detailed in the publishConfig section below). My updated recommendation is to use a custom export condition plus customConditions in tsconfig.json.

Jump into the code: https://github.com/colinhacks/live-typescript-monorepo

In development, your TypeScript code should feel "alive". When you update your code in one file, the effects of that change should propagate to all files that import it instantaneously, with no build step. This is true even for monorepos, where you may not be importing things from a file, but from a local package.

- import { Fish } from "../pkg-a/index";
+ import { Fish } from "pkg-a"

This is vital to TypeScript's value proposition.

This post explains a few strategies you can use to make your TypeScript monorepo feel more alive. Refer to the corresponding repo where you can play around with the code for each strategy: https://github.com/colinhacks/live-typescript-monorepo

The repo contains three subdirectories, each containing a monorepo with the following file structure:

.
├── package.json
├── packages
│   ├── pkg-a
│   │   ├── README.md
│   │   ├── index.ts
│   │   ├── package.json
│   │   └── tsconfig.json
│   └── pkg-b
│       ├── README.md
│       ├── index.ts
│       ├── package.json
│       └── tsconfig.json
├── pnpm-lock.yaml
├── pnpm-workspace.yaml
├── tsconfig.base.json
└── tsconfig.json

This is a pnpm monorepo (pnpm-workspace.yaml) with two packages, pkg-a and pkg-b. pkg-b has a dependency on pkg-a. Each package has a tsconfig.json that extends a tsconfig.base.json in the root of the monorepo.

Here is a quick rundown of the solutions. Don't worry if there are terms you're not familiar with, everything is explained in the breakdown.

1. Use project references ("references" in tsconfig.json)
2. Use publishConfig in package.json to specify .ts file in development and .js file in production. Requires pnpm.
3. Configure compilerOptions.paths in tsconfig.json to override resolution for local package names.
4. Recommended Define a custom conditional export condition in package.json#exports.

Note that I'm explaining all the solutions I found for the sake of education, but the recommended solution is #5 (custom export conditions) so feel free to jump ahead if you're just looking for a solution!

A primer: runtime vs. static

The fundamental annoyance here is this: Node.js has an algorithm for module resolution. When it sees an import from a bare specifier like "pkg-a", it scans up the directory tree checking for a directory called "pkg-a" in each node_modules folder it encounters. Once it finds "pkg-a", it reads the package.json inside and uses the main and exports to figure out how to resolve the bare specifier to a file on disk.

The TypeScript server does almost the same thing, though it uses the "types" field (or the "types" export condition in "exports") to find the declaration file corresponding to a particular package. There are also lots of ways to hack TypeScript's module resolution algorithm. (We'll get to that in a bit.)

But in development, we want things to behave differently. We want the TypeScript server to look at our "raw" .ts files when resolving imports to other local packages in our monorepo, not the compiled declaration files. Similarly, when we execute our local code (say, when running tests) we want it to "run" our TypeScript source code. You shouldn't need to rebuild your project before running tests.

So we need a way to hijack module resolution both statically (for TypeScript) and at runtime (for Node.js or tools like Vitest). We also need to make sure both of these things are hijacked in a way that they agree with each other. If TypeScript is looking at our src/index.ts files but Node.js is still importing lib/index.js, the types may not reflect the runtime behavior of the code. You may have seen this referred to as static-runtime disagreement.

Okay, enough background. Let's get into the details.

1. Project references

The code for this is under the project-references subdirectory. Project references are a TypeScript feature that make it easier to split a large TypeScript codebase into chunks that are typechecked separately by the TypeScript server. This can be a huge performance win for large codebases.

In our monorepo, pkg-b has a dependency on pkg-a. So in our packages/pkg-b/tsconfig.json, we can add a reference to pkg-a like so:

{
  "references": [{"path": "../pkg-a"}]
}

Unfortunately the "references" field is not inherited when you use "extends", so you have to declare it in every package's tsconfig.json. If your monorepo packages have a lot of interconnection, this can get unwieldy fast.

You may also notice that "references" will end up mirroring the "dependencies" list in package.json. These will need to be kept in sync to work as expected. There are tools (nx) that try to automate this, but the need to run a command to re-generate configs starts to feel like a "build step" in itself…and that's what we're trying to avoid.

There are long-gestating efforts to make this more ergonomic, but there seems to be little movement on this front.

The final nail in the coffin is the difficulty of incorporating "references" into runtime module resolution. It's purely a TypeScript feature, and no tools incorporate this into their module resolution. So in all likelihood, you'd need to use one of the approaches below in conjunction with project references to achieve true "live types" with runtime .ts resolution.

There are absolutely scenarios where project references are indispensable. If you have a large enough monorepo, project references may become necessary to avoid re-typechecking the entire codebase when you make a change! But for non-giant projects, they aren't necessary and introduce too much complexity and potential footguns.

2. "publishConfig" in package.json

This approach requires pnpm to work! The publishConfig field behaves very differently between npm publish and pnpm publish.

One obvious solution is just to have each package's package.json point to our .ts files in package.json.

{
  "name": "pkg-a",
  "main": "./src/index.ts",
  "types": "./src/index.ts",
  "exports": {
    ".": {
      "import": "./src/index.ts",
      "require": "./src/index.ts",
      "types": "./src/index.ts"
    },
    "./package.json": "./package.json"
  }
}

This introduces an obvious and immediate problem. We can't publish our raw .ts files to npm without breaking most tools. They need to be properly transpiled to JavaScript first. When we run npm publish, we need these fields to point to the appropriate lib/index.js and lib/index.d.ts files.

Many people have written custom build scripts that will duplicate package.json and rewrite these fields before publishing. Fortunately the good folks at pnpm have given us a better option: publishConfig.

{
  "name": "pkg-a",

  // development config
  "exports": "./src/index.ts",

  // production config
  "publishConfig": {
    "main": "./lib/index.js",
    "types": "./lib/index.d.ts",
    "exports": {
      ".": {
        "import": "./lib/index.js",
        "require": "./lib/index.js",
        "types": "./lib/index.d.ts"
      },
      "./package.json": "./package.json"
    }
  }
}

When you run pnpm publish, pnpm will read the publishConfig field in package.json and use those values to override the top-level values for "main", "exports", and "types".

While npm supports a field called publishConfig, it only lets you set npm config settings like registry and tag. Sad.

In essence, our top-level exports, main, types, etc. now only apply in development, so we can point these to raw .ts files! TypeScript will happily parse these files, as will any modern bundler, framework, or a tool like tsx.

For simplicity, I've only set one field: exports, and I'm setting it to a simple string value. This has some benefits:

1. It's clean! You don't need to redundantly declare a big "exports" object in both the top-level package.json and "publishConfig". (Though you still can if you rely on subpath imports.)
2. It's safe! By specifying "exports" as a single string, no subpath imports are allowed. That level of strictness is good. It means you can't use subpath imports internally that would be illegal using the configuration specified in publishConfig.
3. It just works at runtime. Tools like tsx, esbuild, Vite, etc. can all happily resolve this import using just the single exports key. You don't need "main" unless you're using Node.js 10 or earlier in your development environment.
4. Similarly, TypeScript is happy. I lied a bit earlier…it doesn't need a special "types" field. It's more than happy to fall back to "exports" and pull type signatures out of a regular .ts source file.

The big downside is the reliance on pnpm. That means if you ever run npm publish by accident, you'll accidentally publish a package that isn't runnable by Node.js 💀 You also can't rely on popular GitHub Actions like JS-DevTools/npm-publish since those use npm. This obstacle is surmountable with some tweaks to CI and diligence around publishing, but it is an important gotcha.

3. "paths" in tsconfig.json

TypeScript provides another way to "hijack" module resolution: the compilerOptions.paths.

{
  "compilerOptions": {
    "paths": {
      "pkg-a": ["./packages/pkg-a/src/index.ts"],
      "pkg-b": ["./packages/pkg-b/src/index.ts"]
    }
  }
}

The compilerOptions.paths option overrides TypeScript's normal module resolution. Any import that matches a key in paths will be resolved to the corresponding file. (If you specify multiple paths, TypeScript will use the first one that exists on your file system.)

This should be added to the tsconfig.json for every package in your monorepo. You can avoid redundancy by having all of your package tsconfigs extend a shared tsconfig.base.json.

Remember, we also need to incorporate this into our runtime module resolution.

The popular tsx tool by @privatenumbr does this automatically. This is the best way to run a TypeScript file with Node.js.

$ npm install -g tsx
$ tsx src/index.ts

You can also use tsx in conjunction with Node.js's --import flag.

$ npm install tsx
$ node --import tsx src/index.ts

In the Vite/Vitest ecosystem, there is a popular plugin to do the same called vite-tsconfig-paths.

This solution can be a bit fiddly, and requires some diligence in how you configure per-package tsconfigs. It should also be noted that the TypeScript team kinda hates tsconfig.paths, and there are a lot of ways to shoot yourself in the foot with it.

4. liveDev mode in tshy

The tshy (TypeScript Hybridizer) tool is an opinionated tool by the creator of npm that makes it simple to build ESM and CommonJS packages from your TypeScript source code.

It recently added support for a liveDev mode that will hardlink your TypeScript source code into ./dist/esm and ./dist/commonjs directories. To set this up, install tshy into devDependencies.

$ pnpm add tshy --dev

Add the following "tshy" config to your package.json:

{
  "tshy": {
    "liveDev": true
  }
}

Then run tshy in your package directory.

$ npx tshy

For simplicity, add "tshy" as the "build" script in each of your workspaces. Then you can run this script in each workspace with one command from the root of your workspace. (The exact command depends on your package manager.)

This lets VS Code discover your live TypeScript source code without any additional package.json configuration! And tools like TypeScript and Vitest will be able to resolve your workspace imports to the .ts files with no additional configuration at all!

The downside is that this requires running tshy in each of your workspaces packages, which starts to feel like a build step.

The upside is that you only need to do this once! Once your src files are hard-linked into dist, you can edit them like normal and those changes are automatically reflected in dist. (This is how hard links work.)

The downside is that you'll need to re-run tshy each time you add a new TypeScript file (due to how hard links work…). Running tshy --watch can mitigate the "new file" problem, but for the purposes of this repo, I'm avoiding any solutions that require a file system watcher.

5. Custom conditions in "exports"

This lets you specify your TypeScript files in package.json#exports under a custom export condition of your choosing.

The mostly widely-utilized export conditions are import (for specifying ESM code), require (for specifying CJS code), and types (for specifying type definitions).

// package.json
{
  "name": "pkg-a",
  "exports": {
    "*": {
      "import": "./lib/index.js",
      "require": "./lib/index.cjs"
    }
  }
}

But you're allowed to use any string you like as a custom export condition! Export conditions are intended as an open-ended mechanism for users to specify import entrypoints for specific runtimes, bundlers, or other tooling. For instance, "deno", "bun", and "workerd" (Cloudflare Workers) all support custom conditions so libraries can ship build that are specific to those runtimes.

Here's how a custom condition might look in your package.json. There's nothing special about the string "@colinhacks/zod" here! It could be anything.

// package.json
{
  "name": "pkg-a",
  "exports": {
    "*": {
      "import": {
        "@colinhacks/source": "./src/index.ts", // must be first, order matters!
        "default": "./lib/index.js",
        "types": "./lib/index.d.ts"
      },
      "require": {
        "@colinhacks/source": "./src/index.ts",
        "default": "./lib/index.cjs",
        "types": "./lib/index.d.ts"
      }
    }
  }
}

You should put your custom condition first. Order matters! It's important that your custom condition is the first one in the list (even before "types").

By default, neither TypeScript nor Node.js pays attention to this new "@colinhacks/source" export condition. You need to tell them to incorporate it into their respective module resolution algorithms.

To tell TypeScript, add "@colinhacks/source" to the customConditions field in tsconfig.json for all packages in your monorepo. (You can avoid redundancy by having all of your package tsconfigs extend a shared tsconfig.base.json.)

{
  "compilerOptions": {
    // include this in the tsconfig for all packages
    "customConditions": ["@colinhacks/source"]
  }
}

Node.js can accept custom conditions via the --conditions flag.

node --import=tsx --conditions=@colinhacks/source ./src/index.ts

In the Vite/Vitest ecosystem, this can be configured with resolve.conditions setting.

// vite.config.json
export default {
  resolve: {
    conditions: ['@colinhacks/source'],
  },
};

A note on condition naming

I chose "@colinhacks/source" as the condition name for the sake of uniqueness. You want to pick a condition name that will only be defined in your workspace packages only. If you choose something generic like "source", you may have dependencies with the same condition defined in their package.json. In this case, VS Code will resolve imports of that package using its "source" files, instead of using it's pre-compiled .d.ts files from "types". This can hurt performance of the TypeScript server and slow down your editor.

Conclusion

Overall, my recommendation is #5: custom export conditions.

• It's clean, easy to configure, and works well with modern tooling.
• It doesn't require you to use pnpm, which is a non-starter for many projects.
• You don't need to worry about keeping runtime and TypeScript configurations in sync: you just set "exports" once using your custom condition, then tell your other tools (including TypeScript itself) to pay attention to that condition.

As usual the "comments section" for this post is on Twitter. Feel free to make comments or ask questions in the replies to this tweet:

new blog post 👇 it breaks down a few approaches to configuring "live types" in TypeScript monorepos. you should never need to run `build` while developing!

1. tsconfig paths
2. custom export conditions
3. publishConfig (*my recommended solution)https://t.co/sf6F3CfcsQ

— Colin McDonnell (@colinhacks) May 31, 2024

Happy monorepo hacking!

import {z} from 'zod';

z.string().email();

A lot of people think every technically valid email address should pass validation by that schema. I used to think that too.

Over the years I've merged several PRs to make the email regex more "technically correct". Most recently, I merged a PR that adds support for IPv6 addresses as the domain part of an email. They look like this and I hate them:

jonny@[ipv6:7e95:0559:10f2:21e9:9dab:7309:c116:ca3b]

Turns out that PR also broke plain old subdomains: user@subdomain.domain.com. That means Zod currently fails to parse my mom's current email address but Jonny up there can parse his freakish IPv6 email.

This caused a spiritual crisis and made me re-evaluate my whole stance on what z.string().email() should do. Zod's users are mostly engineers who are building apps. When you're building an app, you want to make sure your users are providing normal-ass email addresses. So that's what z.string().email() is going do.

So I rewrote the regex from scratch to be simple and reasonable. Here's what I came up with:

/^(?!\.)(?!.*\.\.)([a-z0-9_'+\-\.]*)[a-z0-9_+\-]@([a-z0-9][a-z0-9\-]*\.)+[a-z]{2,}$/i;

Let's break that down in plain English:

The username (AKA "local part")

• can use letters, numbers, and these special characters: _'+-.
• can't have two dots in a row
• can't start or end with a dot

The domain

• must consist of at least 2 dot-separated segments consisting of letters, numbers, and hyphens
• can't have two dots in a row
• must end with a "TLD" that's 2+ letters

It is not trying to be RFC 5322 compliant. It's not going to check if the TLD is real. And it's not going to implement any of this craziness:

• No "printable" characters: wtf-!#$%&'*/=?^_{|}~@mail.com
• No quoted local parts: "zod is cool"@mail.com
• No comments: regexiscool(kinda)@mail.com
• No IPv4: billie@[1.2.3.4]
• No IPv6: jonny@[ipv6:7e95:0559:10f2:21e9:9dab:7309:c116:ca3b]
• No emoji: 👎@mail.com

But for reasonable people, it'll do.

Spoiler alert: GitHub Pages and GitHub actions are not involved.

I recently set out to build a proper documentation site for Zod with a short and eminently reasonable list of goals:

1. The site should be generated from the README. Zod's README.md is the "source of truth".
2. The site should automatically update whenever the master branch is updated.
3. I didn't want to worry about building or maintaining a complex build workflow in GitHub Actions, if possible.

Despite the eminent reasonability, I had a hard time finding a simple solution. After a lot of research and false starts, I've found what I think is the lowest-effort, highest-reward way to publish a documentation site for your GitHub-hosted project.

Let's jump in.

1. Add this index.html to your repo

Create a file called index.html in your project root and paste the following contents into it.

Replace all occurrences of user/repo with your project's name. And maybe write up a slightly more detailed meta description :)

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>user/repo</title>
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="description" content="user/repo is cool!" />
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0" />
    <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify/lib/themes/vue.css" />
    <style>
      .markdown-section { max-width: 700px; }
    </style>
  </head>
  <body>
    <nav
      style="display: flex; flex-direction: row; align-items: center; justify-content: space-between;"
    >
      <!-- <ul><li href="/link">Link 1</li></ul> -->
    </nav>
    <div id="app"></div>
    <script>
      window.$docsify = {
        subMaxLevel: 1,
        maxLevel: 3,
        auto2top: true,
        repo: 'user/repo',
        routerMode: 'history',
        // homepage: "README.md"
      };
    </script>
    <script src="//cdn.jsdelivr.net/npm/docsify@4.12.2/lib/docsify.min.js"></script>
    <script src="//cdn.jsdelivr.net/npm/prismjs@1.28.0/prism.min.js"></script>
  </body>
</html>

Those 1150 bytes are doing a lot, but most importantly it's loading Docsify. Docsify is a documentation generator. You may recognize it from its goofy looking gradient cover pages with big pill-shaped buttons.

I'm not a fan of these cover pages, so we won't be using one here. All in all, though, I think Docsify is extraordinarily well designed.

It also differs from some other options like Docusaurus and Gitbook in that it's a fully client-side documentation generator. Your Markdown files don't get pre-compiled to HTML ahead of time; instead, Docsify fetches it asynchronously, parses the contents in-browser, and injects the rendered HTML into the page.

"The horror! Multiple round-trip requests before the first meaningful paint!?" Turns out, it's not a big issue. Modern globe-spanning ultra-fast CDNs for static content, this still feels near-instant. Google agrees; the Zod site gets a 98 Performance score on Lighthouse.

2. Run locally

There's no build step required with Docsify. You could deploy your index.html and a README.md to any static site hosting service and it would work. We're going to be using Netlify to deploy this site, but before we get to that, let's see how the site is looking. We can use Netlify's CLI to start a simple static site server.

$ npx netlify dev
◈ Netlify Dev ◈
   ┌─────────────────────────────────────────────────┐
   │                                                 │
   │   ◈ Server now ready on http://localhost:8888   │
   │                                                 │
   └─────────────────────────────────────────────────┘

Open http://localhost:8888 in your browser, and you should see a rendered version of your README.md!

If netlify dev detects that you're using a framework, this may not work. In that case you'll need to disable framework detection and tell Netlify to act as a simple static file host.

npx netlify dev --framework "#static"

If your primary Markdown file isn't called README.md, you can specify a different name with the homepage setting in the Docsify configuration.

window.$docsify = {
  subMaxLevel: 1,
  maxLevel: 3,
  auto2top: true,
  repo: 'user/repo',
  routerMode: 'history',
  homepage: 'home.md', // <-- add this
};

3. Configure rewrites

Docsify using client-side routing to determine which Markdown file to load and render. In other words, it's a pure single-page application; index.html loads Docsify, which then assumes control over your browsers URL bar, similar to React Router or other client-side routers.

If your project/site contains several Markdown files, you may encounter a strange behavior. Consider the following file structure:

├── index.html
├── README.md
└── CONTRIBUTING.md

If you start the development server and type localhost:8888/CONTRIBUTING into the URL bar, you'll see a 404. That's because there's no HTML file called CONTRIBUTING.html. We need all incoming requests to render index.html! Then Docsify will read the URL and load the appropriate Markdown file.

We can achieve this with a Netlify feature called rewrites. (All static file hosts have an equivalent feature.) Create a file called _redirects with no extension. (With Netlify, both redirects and rewrites are declared in _redirects.)

$ touch _redirects

Then paste in the following:

/*            /             200

This tells Netlify that all incoming requests (/*) should render the homepage (/). However, unlike a redirect, rewrites do not change the URL. Basically Netlify will sneakily render index.html under the hood, without telling the browser.

Another solution is to use a hash router. In fact, this is the default behavior for Docsify. Unfortunately, it makes the URL messy; the URL would look like domain.com/#/CONTRIBUTING instead of simply domain.com/CONTRIBUTING. I prefer clean URLs so I chose the rewrites approach.

Now when we run npx netlify dev, we can open localhost:8888/CONTRIBUTING directly without seeing a 404.

4. Deployment

Firebase used to be my go-to for static file hosting, but these days Netlify leads the pack on developer experience. It only takes a couple minutes to set up Github-linked static site hosting.

First, let's push our new files to master.

$ git add .
$ git commit -am "Add docs site"
$ git push origin master

Then configure Netlify.

1. Log in or create an account at netlify.com.
2. From the dashboard, click Add new site > Import an existing project.
3. Authenticate with GitHub (or GitLab or Bitbucket) and select your repository.

When prompted for a "Branch to deploy" select master (the default) unless your primary branch has a different name.

Under "Build settings" you can simply leave everything blank:

• "Base directory" — leave blank (defaults to the project root)
• "Build command" — leave blank (no build step necessary!)
• "Publish directory" — leave blank (defaults to the project root)

There is virtually no configuration required, because we're simply hosting static files from the root directory of our master branch. Couldn't be easier!

Note that we're deploying the entire contents of our repository to Netlify. You could write a build command to delete all unwanted files but I didn't bother. The contents of my repo are already public on the internet, so…🤷‍♂️

Proceed with the deployment. Once Netlify finishes deploying, Netlify will provide a *.netlify.app subdomain for your site.

5. Final touches

There's plenty more to do before your site is truly ready.

• Connect a custom domain
• Add a robots.txt to your root directory
• Add links to the top <nav> in index.html
• Add Open Graph meta tags
• Add Twitter meta tags and a banner image
• Generate a favicon (RealFaviconGenerator is great)

But I'll leave that as an exercise for the reader.

First, we all switched back to relational DBs after recovering from an embarrassingly long collective delusion that NoSQL was cool. Then TypeScript started its meteoric rise after years of quiet development. Shortly thereafter GraphQL was announced and quickly became the de facto way to implement typesafe APIs.

TLDR: I published a new library called tRPC that makes it easy to build end-to-end typesafe APIs without code generation, by leveraging the power of modern TypeScript. To jump straight to the tRPC walkthrough, click here.

GraphQL's TypeScript problem

GraphQL's biggest competitive advantage is that it's a spec, not a library. It's designed to be universal and language-agnostic. If you use different languages for your client and server, then by all means stick with GraphQL! It remains the best language-agnostic way to build typesafe data layers.

But the vast majority of projects that use GraphQL are JavaScript-based. The graphql NPM module has 5M downloads/week vs ~300k) for the graphene Python module and ~230k for the graphql Rubygem. It's not a perfect metric, but the signal is clear.

And a growing fraction of JavsScript projects are TypeScript-based. As TypeScript approaches near-universal adoption in the JS ecosystem, that means most GraphQL-based projects will soon be TypeScript-based (if they aren't already).

Here's the thing: most people use GraphQL as a massively over-engineered way to share the type signature of their API with your client. What if you could import that type signature directly into your client using plain ole import syntax? Well…you can.

A demonstrative example

We're going to build the world's simplest typesafe API—no codegen or GraphQL required. There's only one endpoint, with the following type signature:

getUser(id: string) => { id: string, name: string }

For the sake of simplicity, the API is implemented as an Express router. It should be easy to tell what's going on even if you aren't familiar with it.

#1 Define getUser function

// server/routes.ts
export const getUser = async (id: string) => {
  return { id, name: 'Bilbo' };
};

#2 Implement a simple RPC endpoint

We set up a special /rpc endpoint that accepts POST requests with a payload of type { endpoint: string; arguments: any[] }. The endpoint property indicates the function to call (in our case, the only option is "getUser"). We take the arguments array, pass it into the appropriate function, and res.send the results.

// server/router.ts
import express from 'express';
import * as routes from './routes';

type Payload = { endpoint: string; arguments: any[] };

const app = express();
app.post('/rpc', async (req, res) => {
  const payload: Payload = req.body;
  if (payload.endpoint === 'getUser') {
    const user = await routes.getUser(...payload.args);
    return res.status(200).send(user);
  }

  return res.status(404).send('Endpoint not found');
});

A totally skippable aside about RPC

With a REST APIs, you use different URLs (/users, /user/:id) and HTTP request types (e.g GET, POST, etc) to access different functionality. For instance GET /users will fetch all users, GET /user/abc123 will fetch a single user, and POST /user will create a new user.

RPC strips all of that away. Endpoints are just named functions that accept some arguments and return a value. The RPC ethos is to make API calls look and feel like calling a function (albeit one that's defined on a remote server).

GraphQL is an RPC protocol that's been augmented with a schema language and a mechanism for client-side field selection.

#3 Import the type signature of getUser into the client

This is where the magic happens. We directly import the type signature of our API into out client. Yes, it really is this easy.

// client/index.tsx
import type { getUser } from '../server/routes.ts';

Note that this works no matter where routes.ts lives in your file system! It could be in a different Git repo; it could be on your external hard drive. As long as the TypeScript file exists on your computer, you can import it from any other TypeScript file.

"But wait, I can't just import stuff from a different repo!"

This is often true…when you're importing code. For instance, Create React App (CRA) prohibits you from importing anything from outside of your src directory.

But that doesn't apply here. We're using import type syntax, which is a purely static construct. That is, it can only import type signatures, but not data, code, or any runtime construct. For most build systems powered by Babel (including CRA), import type statements (and all type annotations) just get stripped out before the bundling step. If you're using Webpack with Babel, Parcel, or Rollup, this Just Works™. Don't believe me? Play around with this sample repo where you can see this approach working just fine in an unejected CRA project.

You can use import type to import a type from any .ts or .tsx file anywhere on your filesystem. Period. This will work even if your client and server are in different repositories in different parts of your filesystem.

Of course, if multiple developers are working on the same codebase, you'll need to standardize the relative locations of client and server repos, so relative imports work on everyone's machines. You can either establish a convention ("make sure the server and client repos are in the same folder") or use something like Git Submodules.

"But wait, isn't importing stuff from the server into the client a security vulnerability?!"

Yes—when you're importing code. But not here! Remember, we're using the import type syntax. It was introduced in early 2020 (TypeScript 3.8) and it comes with a very important guarantee:

All import type declarations are removed completely when you compile your TypeScript code into a JavaScript bundle, no matter what. So yes — you can safely use import type without worrying about accidentally importing sensitive data from your server.

#4 Make an API call

Now that we've imported the type signature of getUser, how do we make strongly typed API call? Answer: a bit of TypeScript magic. (Don't worry, you don't need to understand how this works! Just know that it's possible!)

// client/index.tsx
import type { getUser } from '../server/routes.ts';

type API = {
  getUser: getUser;
  // add additional routes here...
};

const query = <Endpoint extends keyof API>(
  endpoint: Endpoint,
  ...args: Parameters<API[Endpoint]>
): ReturnType<API[Endpoint]> => {
  return fetch('http://localhost:3000/api/rpc', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ endpoint, arguments: args }),
  }).then((response) => response.json()) as any;
};

We now have a fully typesafe query function, which you use like this:

const user = await query('getUser', 'user_abc73bd39ef');
// => Promise<{ id: "user_abc73bd39ef", name: "Bilbo" }>

It's a function called query. The first argument is the name of the endpoint; the second is the input of the function. (As implemented each RPC endpoint can only accept a single input.)

TypeScript validates that the endpoint name is valid…

…typechecks the arguments…

…and strongly types the result.

And if you update the definition of getUser on your server, the new type signature automatically propagates to your client at the speed of TypeScript, no code generation required. The developer experience is unreal.

By using a next-generation ORM to fetch data in your endpoints, you get most of the value of GraphQL — end-to-end typesafety, easy nested fetching, field selection, etc — in a TypeScript-native way.

If you want, you can use this approach exactly as described; all the code above is available as a Next.js project in this repo. But there's a better way!

Introducing tRPC!

Around 6 months ago, I distilled these ideas into a proof-of-concept library called tRPC. Since then, the inimitable @alexdotjs has turned it into a full-fledged library. Lets see how to implement our getUser example with tRPC. Spoiler alert: it's mindblowingly easy.

If you want to jump straight to the README, check it out at github.com/trpc/trpc — and leave a star if you're feeling generous!

Create a router

// server/index.ts
import * as trpc from '@trpc/server';

const appRouter = trpc.router();
export type AppRouter = typeof appRouter;

Add a query endpoint

Let's implement getUser.

// server/index.ts
import * as trpc from '@trpc/server';

export const appRouter = trpc.router().query('getUser', {
  input: String,
  async resolve(req) {
    req.input; // string
    return { id: req.input, name: 'Bilbo' };
  },
});

A few things to note:

• You add "endpoints" by calling the .query method. The first argument is a procedure name. The second is an object containing two properties: input and resolve.

• The input is a function that validates the input. For simplicity, all tRPC procedures accept a single argument. The type signature of this argument is inferred from the return type of the input function.

  I recommend defining your inputs using a schema definition library (I hear Zod is pretty cool 🙃). Far too few developers rigorously validate incoming API payloads; this is a huge security risk and source of bugs. So tRPC has validation baked-in from the start.

  For simplicity, I'm just using the built-in JavaScript String constructor, which accepts any value and converts it into a string.

• The endpoint logic is defined in the resolve function. A "pseudorequest" is passed into it, which contains input (automatically typed as string in this case) and ctx (more on that later). tRPC infers the return type of your resolver. So make sure the returned value is statically typed! We recommend using a typesafe ORM like Prisma or TypeORM that automatically type the results of database operations; otherwise you'll need to write TypeScript types manually.

What about mutations?

Like GraphQL, tRPC makes a distinction between queries and mutations. Add mutations to your router with the mutation method:

// server/index.ts
import * as trpc from '@trpc/server';
import { z } from 'zod';
export const appRouter = trpc
  .router()
  .query('getUser', {
    input: String,
    async resolve(req) {
      req.input; // string
      return { id: req.input, name: 'Bilbo' };
    },
  })
  .mutation('createUser', {
    input: z.object({ name: z.string() }),
    async resolve(req) {
      return await prisma.user.create({
        data: req.input,
      });
    },
  });

The syntax for the mutation is identical to query. tRPC only makes a distinction because query payloads are cached differently than mutation payloads.

Chainable methods

Important note: when you add multiple endpoints to a router, you must chain the calls to query() and mutation()! This lets tRPC infer the type signature of your whole API. If you're familiar with Express you may be tempted to do this:

export const appRouter = trpc.router();

// ❌ DON'T DO THIS!! ❌

appRouter.query('getUser', {
  input: String,
  async resolve(req) {
    req.input; // string
    return { id: req.input, name: 'Bilbo' };
  },
});

This doesn't work at all. All router methods are immutable—they return an entirely new instance of TRPCRouter, instead of modifying the current instance. So you must chain these methods.

Merging routers

At some point, you'll want to split up your tRPC router into multiple files. If all methods must be chained together, how is this possible?

tRPC provides an easy way to merge routers. I recommend creating a root appRouter and several "child routers" that you merge into it.

import { userRouter } from './routers/user';
import { postRouter } from './routers/post';

export const appRouter = trpc
  .createRouter()
  .merge('users.', userRouter) // prefix userRouter endpoints with "users."
  .merge('posts.', postRouter); // prefix postRouter endpoints with "posts."

All procedure names will be prefixed with the string literal you pass as the first argument of .merge. So a createUser mutation on userRouter will be re-named to users.createUser. This is made possible by the incredible string literal templates feature introduced in TypeScript 4.1!

Handing incoming requests

Now lets use our router to handle some actual HTTP requests. tRPC ships "adapters" that easily let you generate Express middleware or Next.js API routes from your tRPC router.

Next.js integration

// pages/api/trpc/[trpc].ts
import { createNextApiHandler } from '@trpc/server/adapters/next';

const appRouter = /* ... */;

export default createNextApiHandler({
  router: appRouter,
  createContext: (opts) => {
    return { bearer: opts.req.headers.authorization };
  },
});

Express integration

If you're using Express, do this instead:

import { createExpressMiddleware } from '@trpc/server/adapters/express';

const appRouter = /* ... */;

const app = express();

app.use(
  '/trpc',
  createExpressMiddleware({
    router: AppRouter,
    createContext: () => null, // no context
  })
);

app.listen(80);

Koa, Hapi, Nest.js

Coming soon (and PRs welcome)!

Request context

All server adapters accept createContext function, where you can generate a ctx object from the "raw" HTTP request and response objects (available as opts.req and opts.res). This ctx is available in all your resolvers.

Client-side usage

So what about the client side? The @trpc/client module generates a typesafe "tRPC client"—analagous to the query method we implemented in our toy example. You can import the the type signature of your entire API at once with import type { AppRouter } from '...'. Then pass it into createTRPCClient, along with the URL of your API:

// pages/index.ts
import { createTRPCClient } from '@trpc/client';
import type { AppRouter } from './api/[trpc]';

const trpcClient = createTRPCClient<AppRouter>({
  url: 'http://localhost:3000/api',
});

trpcClient.query('getUser', 'user_1238823498');
// => Promise<{ id: string; name: string }>

React integration

If you use React, you can use the @trpc/react package to convert your tRPC client into React hooks powered by React Query. This gets you a ton of cool features like request invalidation, SSR, and cursor-based pagination! 🚀

import { createReactQueryHooks } from '@trpc/react';

// path to your backend server where your AppRouter is defined
import type { AppRouter } from '../server/trpc';

export const trpc = createReactQueryHooks<AppRouter>();

Vue and Svelte

Get in touch if you'd like to build dedicated bindings for those frameworks! Create an issue on the GitHub repo.

Wrapping up

I've been using various iterations of tRPC in production for nearly a year, and it truly feels like living in the future. I'm immensely proud of what tRPC has become. Checkout the docs (and leave a star! 🤩) on the GitHub repo.

There are a lot of other features I could highlight, but that'll do for now. Huge shoutout to @alexdotjs, who picked up a fledgling proof-of-concept four months ago and carried it to the finish line!

The background

The reason Zod 2 has been in beta for so long (well, one of the reasons anyway!) is that I’ve been increasingly unsatisfied with Zod’s implementation of transformers. Transformers are a mechanism within Zod to convert data from one type to another. For the sake of rigor, I decided it was paramount for transformers to encapsulate both pre- and post-transform validation; to create one, you gave it an input schema A, and output schema B, and a transformation function (arg: A)=>B.

Multiple issues have cropped up that have led me to re-evaluate the current approach. At best, transformers are a huge footgun and at worst they’re fundamentally flawed.

Context

In version 1, a Zod schema simply let you define a data type you'd like to validate. Under the hood, it magically inferred the static TypeScript type for your schema. More technically, there is a base class (ZodType) that tracked the inferred type in a generic parameter called Type.

Transformers break this assumption. The accept an Input of one type and return an Output of a different type. To account for this, every Zod schema now had to track both an input and output type. For non-transformers, Input and Output are the same. For transformers only, these types are different.

Let's look at stringToNumber as an example:

const stringToNumber = z.transformer(z.string(), z.number(), (val) =>
  parseFloat(val)
);

For stringToNumber, Input is string and Output is number. Makes sense.

What happens when you pass a value into stringToNumber.parse?

1. The user passes a value into stringToNumber.parse
2. The transformer passes this value through the parse function of its input schema (z.string()). If there are parsing errors, it throws a ZodError
3. The transformer takes the output from that and passes it into the transformation function (val => parseFloat(val))
4. The transformer takes the output of the transformation and validates it against the output schema (z.number())
5. The result of that call is returned

Here's the takeaway: for a generic transformer z.transformer(A, B, func), where A and B are Zod schemas, the argument of func should be the Output type of A and the return type is the Input type of B. This lets you do things like this:

const stringToNumber = z.transformer(z.string(), z.number(), (val) =>
  parseFloat(val)
);
const numberToBoolean = z.transformer(
  z.number(),
  z.boolean(),
  (val) => val > 25
);
const stringToNumberToBoolean = z.transformer(
  stringToNumber,
  numberToBoolean,
  (val) => 5 * val
);

The problems with transformers

After implementing transformers, I realized transformers could be used to implement another much-requested feature: default values. Consider this:

const stringWithDefault = z.transformer(
  z.string().optional(),
  z.string(),
  (val) => val || 'trout'
);
stringWithDefault.parse('marlin'); // => "marlin"
stringWithDefault.parse(undefined); // => "trout"

Voila, a schema that accepts string | undefined and returns string, substituting the default "trout" if the input is ever undefined.

So I implemented the .default(val:T) method in the ZodType base class as below (partially simplified)

default(def: Input) {
  return ZodTransformer.create(this.optional(), this, (x: any) => {
    return x === undefined ? def : x;
  });
}

Do you see the problem with that? I didn’t. Neither did anyone who read through the Transformers RFC which I left open for comment for a couple months before starting on implementation.

This implementation quickly gets wonky when you try to pass an existing transformer in as the input or output. In other words, transformers aren't composable. Let's see what happens if we try to set a default value on stringToNumber.

stringToNumber.default('3.14');

// EQUIVALENT TO
const defaultedStringToNumber = z.transformer(
  stringToNumber.optional(),
  stringToNumber,
  (val) => (val !== undefined ? val : '3.14')
);

defaultedStringToNumber.parse('5');
/* { ZodError: [
  {
    "code": "invalid_type",
    "expected": "string",
    "received": "number",
    "path": [],
    "message": "Expected string, received number"
  }
] */

Let’s walk through why this fails. The input ("5") is first passed into the transformer input (stringToNumber.optional()). This converts the string "5" to the number 5. This is then passed into the transformation function. But wait: val is now number | undefined, but the transformer function needs to return a string. Otherwise, if we pass 5 into stringToNumber.parse it’ll throw. So we need to convert 5 back to "5". That may seem easy in this toy example but it’s not possible in the general case. Zod can’t know how to magically undo the transformation function.

In practice, the current definition of default in ZodType shouldn’t have even been possible. The only reason the type checker didn’t catch this bug is because there are a regrettable number of anys floating around in Zod. It’s not a simple matter to switch them all to unknowns either; I’ve had to use any in several instance to get type inference and certain generic methods to work properly. I’ve tried multiple times to reduce the number of anys but I’ve never managed to crack it.

It’s possible this is a one-off issue. I could find some other way to implement .default() that doesn’t involve transformers. Unfortunately this isn’t even the only problem in Zod’s implementation.

The .transform method

Initially the only way to define transformers was with z.transformer(A, B, func). Eventually I implemented a convenience method you can use like this:

z.string().transform(z.number(), (val) => parseFloat(val));

// equivalent to
z.transformer(z.string(), z.number(), (val) => parseFloat(val));

Some users were executing multiple transforms in sequence without changing the actual data type:

z.string()
  .transform(z.string(), (val) => val.toLowerCase())
  .transform(z.string(), (val) => val.trim())
  .transform(z.string(), (val) => val.replace(' ', '_'));

To reduce the boilerplate here, it was recommended that I overload the method definition to support this syntax:

z.string()
  .transform((val) => val.toLowerCase())
  .transform((val) => val.trim())
  .transform((val) => val.replace(' ', '_'));

If the first argument is a function instead of a Zod schema, Zod should assume that the transformation doesn’t transform the type. In other words, z.string().transform((val) => val.trim()) should be equivalent to z.string().transform(z.string(), (val) => val.trim()). Makes sense.

Consider using this method on a transformer:

stringToNumber.transform(/* transformation_func */);

What type signature do you expect for transformation_func?

Most would probably expect (arg: number)=>number. Some would expect (arg: string)=>string. Neither of those are right; it’s (arg: number)=>string. Yes really. The transformation function expects an input of number (the Output of stringToNumber) and a return type of number (the Input of stringToNumber). This type signature is a natural consequence of a series of logical design decisions, but the end result is dumb. Well done, self.

Intuitively, you should be able to append .transform(val => val) to any schema. Surely passing in the identity function to a method called .transform() should be a no-op? Unfortunately due to how transformers are implemented, that's not always possible.

More complicated examples

These strange issues composability issues also make it difficult to write any generic functions on top of Zod (of which .transform and .default are two examples). Others have encountered similar issues, like here and here. For the sake of simplicity, I won't break down the nuances of those cases; the underlying issue is the composability problem I described above.

When should defaults even be applied?

Strangely, there doesn't appear to be a consensus expectation on whether default values should be "applied" pre- or post-transformations. Consider the simple stringToNumber example; should stringToNumber.default(/* value */) accept a number or a string?

As implemented in Zod 2 it should accept string (the Input of the output schema of stringToNumber). In this case, the schema looks at the value passed into parse, checks if it's undefined, and, if so, hot-swaps in the default value. That value is then passed through all the transformers and refinements downstream (in this case, being converted to a number) by the non-optional stringToNumber. Remember, as implemented, this is how stringToNumber.default() returns:

stringToNumber.default(val);

// equivalent to

z.transformer(stringToNumber.optional(), stringToNumber, (val) =>
  val !== undefined ? val : '3.14'
);

But some users expect stringToNumber.default to accept a number; they assume that a default value "short-circuits" the parsing logic. If you pass undefined into .parse(), the parsing should "return early" and return the value. In this case, taht value should be a number because that's the ultimate expected return type of the transformer returned by stringToNumber.default(val).

A path forward

Note from March 2021: the approach described below also had issues! Ultimately the implementation used in Zod 3 was slightly different.

I just created an RFC describing a plan to improve this state of affairs. The idea was to have each Zod schema contain a list of post-parse transformation functions. When a value is passed into .parse, Zod will type check the value, then pass it through the transform chain. This is the approach Yup uses.

const schema = z
  .string()
  .transform((val) => val.length)
  .transform((val) => val > 100);

type In = z.input<typeof schema>; // string
type Out = z.input<typeof schema>; // boolean

Unlike before, Zod doesn’t validate the data type between each transform. We’re relying on the TypeScript engine to infer the correct type based on the function definitions. In this sense, Zod is behaving just like I intended; it’s acting as a reliable source of type safety that lets you confidently implement the rest of your application logic — including transforms. Re-validating the type between each transform is overkill; TypeScript’s type checker already does that.

Each schema will still have an Input (the inferred type of the schema) and an Output (the output type of the last transform in the chain). But because we’re avoiding the weird hierarchy of ZodTransformers everything behaves in a much more intuitive way.

One interesting ramification is that you could interleave transforms and refinements. Zod could keep track of the order each mod/refinement was added and execute them all in sequence:

const schema = z
  .string()
  .transform((val) => parseFloat(val))
  .refine((val) => val > 25, { message: 'Too short' })
  .transform((val) => `${val}`)
  .refine((val) => /^\d+$/.test(val), { message: 'No floats allowed' });

We'll have to wait and see how this turns out!

Update from March 2021

This is the general approach being used in the new Zod 3 alpha release. There are some small implementational details — refinements and transforms are now "housed" within the ZodEffects wrapper class instead of being attached directly to the schema.

A note on default values: Zod no longer uses transforms to implement default values. Transforms are only capable of implementing post-parse defaults: parse the input, then swap in the default value if the result is undefined. But I've decided that pre-parse defaults make more sense (and are more expected by users). That is — if the input is undefined swap in the default value, and then finish the parsing process (which includes refining and transforming). As such, the defaulting logic now lives in the ZodOptional class.

Edit: 2022 The issues described here have been largely solved by layouts in Next.js 13.

I recently set out to implement a single-page application (SPA) on top of Next.js.

To clarify, I'm using the term "SPA" in a very specific way. I'm referring to the "classical" SPA model: an application that handles all data fetching, rendering, and routing entirely client-side.

Turns out, Next.js does not make this easy.

To actually learn how to build a single-page application on top of Next.js, checkout my tutorial! Building a single-page application with Next.js.

Why not use Next's router

Next's built-in router isn't as flexible as React Router! React Router lets you nest routers hierarchically in a flexible way. It's easy for any "parent" router to share data and provide an outer layout to its "child" routes. This is true for both top-level routes (e.g. /about and /team) and nested routes (e.g. /settings/team and /settings/user).

This isn't possible with Next's built-in router! Instead, all shared state and layout must be initialized in your custom _app.tsx component. It's equivalent to building an app with a single, top-level React Router. Nominally, Next.js lets you define "nested routes" (e.g. pages/settings/team.tsx), but in practice it "flattens" all those routes. The only code that's shared is the stuff you can fit into _app.tsx.

There are some established patterns for circumventing this limitation, notably those described in this post by Adam Wathan. If you are dead-set on using Next's routing system, you should look at those approaches. But in my opinion they're agregiously hacky.

Why is this so hard?

I finally got React Router to work inside Next.js — check out my tutorial for details — but there were some thorny issues along the way. The guidance in the documentation is…scant, to put it mildly. It boils down to "use a custom App".

That's it. Not particularly enlightening. Or detailed.

I found this to be pretty strange. Aren't SPAs still the dominant model for building React-based applications? Didn't Next.js just run a wildly successful conference with 60,000 virtual attendees? Didn't they just raise a \$21M Series A? Am I the only person struggling with this? Why is this so hard?

My answer: Vercel doesn't really want you to build SPAs, at least not with client-side routing. Theyr're a server hosting company, so they want you to use servers. To that end, I believe they consciously emphasize the server-side rendering (SSR) approach and de-emphasize the SPA approach.

What about SSG?

An astute reader may ask "Why would Next.js try to de-popularize SPAs but encourage static site generation (SSG)?" Like SPAs, statically generated sites don't involve servers and thus doesn't make Vercel much money. Good point. This is where my theory veers into conspiracy and madness.

I suspect SSG is a strategic marketing play. A huge fraction of developers will build a static website at some point in their career. Having built their personal website with Next.js, they're more likely to choose Next.js down the road when designing their shiny new SaaS app. And since it's against Vercel's fair use policy to run a commercial enterprise on the free tier, all these new companies will eventually convert to a paid plan. They're playing the long game.

I don't think the SSG functionality in Next.js was created solely for marketing reasons; that's ludicrous. I think Next.js is the best way to build static site; I wrote a 1500-word article to that effect just 5 months ago. In fact, I believe Vercel is laser-focused on improving the developer experience, perhaps more than any other company. That's exactly why I don't believe the lack of documentation on client-side routing is merely an oversight. The strength of the rest of their documentation makes this hole even more glaring.

None of this is bad, per se. Vercel and Next.js originated, in part, as a reaction to the SPA-centricity of React development pre-2016. They're under no obligation to document, advertise, or encourage the usage of SPA paradigm on their platform, especially if it works against their own interests. And all told, their free tier is extremely generous.

But remember that, at the end of the day, Vercel is a hosting company with a vested financial interest in pushing server-side rendering and de-popularizing SPAs. If someone tried to learn full-stack web development by reading the Next.js documentation, they wouldn't even know that SPAs are an option. So keep that in mind as you design your next project!

Addendum: The section below was added shortly after initial publication.

In response to this article the lead maintainer of Next.js, Tim Neutkens, provided this counterpoint on Twitter:

All our marketing and communication around this have been around making sure you don’t use SSR if you don’t need it, e.g. by adding revalidate (incremental generation) and fallback (on-demand generation)

— Tim (@timneutkens) October 29, 2020

This actually sheds light on another interesting aspect of Vercel's SSG push.

Recently Next.js has introduced support for "dynamic static generation" with the fallback flag and incremental static regeneration with the revalidate flag. Tim presented these features as evidence that Next.js is helping developers avoid server-side rendering if it isn't absolutely necessary. But using either of these features requires Vercel to spin up a serverless function for your app. In other words, Next is blurring the line between SSG and SSR.

If you rely on these features, you've left the "pure" SSG paradigm. Your app now requires a server to work as expected — which means Vercel stands to make money! You can no longer simply next build && next export your app and deploy to your static hosting service of choice. Moreover, if your project is commercial in nature, you now must upgrade to a Pro plan (if hosting on Vercel).

Wrap up

If you want to actually learn how to build a single-page application on top of Next.js, checkout my tutorial Building a single-page application with Next.js .

Hopefully I didn't come off as too critical of Vercel. Next.js is phenomenal and I love it. If I didn't I would never have gone through all this trouble!

The "comments section" for this post is this Twitter thread. Jump in and yell at me for all the bad takes in this post!

✨ NEW BLOG POST ✨

It started as a simple "How to build a SPA with Next.js" tutorial.

It turned into a fevered rant about Vercel, Next.js, perverse incentives, and the future of the SPA paradigm. Oops.https://t.co/hrRtKojezH

— Colin McDonnell (@colinhacks) October 29, 2020

If you're into Next.js or TypeScript, follow me on Twitter @colinhacks! I build and maintain open-source tools like Zod (a TypeScript validation library) and tRPC (a tool for building end-to-end typesafe APIs with TypeScript). Or subscribe to the newsletter to get notified when I publish new posts!

Update 2023 — Updated for Next.js 13, React Router 6, and React 18.

Update 2022 — Next.js 13 now supports nested routes via the /app directory. However, it requires a server to render React Server Components, and thus doesn't qualify as a "single-page application" as defined in this post.

I recently set out to implement a single-page application (SPA) on top of Next.js.

To clarify, I'm using the term "SPA" in a very specific way. I'm referring to the "old school" SPA model: an application that handles all data fetching, rendering, and routing entirely client-side.

Turns out, Next.js does not make this easy.

Why not use Next.js routing?

Next.js is not as flexible as React Router! React Router lets you nest routers hierarchically in a flexible way. It's easy for any "parent" router to share data with all of its "child" routes. This is true for both top-level routes (e.g. /about and /team) and nested routes (e.g. /settings/team and /settings/user).

This isn't possible with Next's built-in router! Instead, all shared state and layout must be initialized in your custom _app.tsx component. It's equivalent to building an app with a single, top-level React Router. This is true despite the fact that Next.js nominally lets you define "nested routes" (e.g. pages/settings/team.tsx). In practice it "flattens" all those routes.

There are complex, hacky ways of achieving this in Next, including several described here by Adam Wathan. If you are dead-set on using Next's routing system, you should look at those approaches.

Why use Next.js at all?

Because it's awesome! You can split your bundle with dynamic imports, which solves one of the biggest performance issues associated with large SPAs. You can easily deploy to Vercel. You can build your API using API Routes, instead of maintaining separate client and server codebases. Plus there's that snazzy new <Image> component!

But the killer feature of Next.js for me is the zero-config client-server codesharing.

The ability to share code and typings between my client and server is a massive win. The developer experience is 10x better than Lerna and 100x better than Yarn Workspaces (trust me). It's the enabling feature behind my new library github.com/colinhacks/trpc, which lets you build an end-to-end typesafe data layer. Better yet, it abstracts away the API entirely and lets you call your server-side functions directly from your client. I'm biased, but it's extremely cool. Check it out at github.com/colinhacks/trpc! ✨

Implementation

For highly interactive "dashboard-style" SaaS development, the dominant paradigm is still the SPA. Unfortunately Next.js is (nearly) incompatible with the SPA paradigm for one fundamental reason: routing. Next.js ships with its own built-in page-based routing system, whereas the typical SPA relies on client-side routing, typically using a library like react-router.

Below I will walk through all the errors I encountered while trying to use React Router inside a Next app and how I fixed them. For demonstration purposes, I'll be using the simple router below, which is a simplified version of the "Basic Routing" example from the react-router docs.

import React from 'react';
import {BrowserRouter as Router, Routes, Route, Link} from 'react-router-dom';

export default function App() {
  return (
    <Router>
      <div>
        <ul>
          <li>
            <Link to="/">Home</Link>
          </li>
          <li>
            <Link to="/about">About</Link>
          </li>
        </ul>

        <Routes>
          <Route path="/about" element={<h1>About</h1>} />
          <Route path="/" element={<h1>Home</h1>} />
        </Routes>
      </div>
    </Router>
  );
}

Browser history needs a DOM

Let's start by creating a new Next.js app, installing React Router (npm install react-router-dom), and pasting the router code above into pages/index.tsx. When you run next dev you'll immediately get a not-so-subtle Invariant failed: Browser history needs a DOM error.

This happens because Next.js tries to pre-render your pages on the development server in development mode. React Router, on the other hand, requires access to the global window object provided by the browser. Because window isn't available in the server environment, React Router craps out.

To fix this, we need to (drumroll please) use a custom App! Add a file called pages/_app.tsx (or pages/_app.js if you're a monster). Let's try a naive approach to fixing this problem. If window isn't defined, return null; otherwise, render the page.

import {AppProps} from 'next/app';
import {useEffect, useState} from 'react';

function App({Component, pageProps}: AppProps) {
  if (typeof window === 'undefined') return null;
  return <Component {...pageProps} />;
}
export default App;

Hydration failed

Reload the page and you'll see a new error.

When you load a page of your Next.js app, Next.js 1) tries to pre-render it on the server, 2) sends the result to the browser, and 3) "re-hydrates" the page in the browser. Re-hydration means the page is re-rendered again on the browser and compared against the version that was rendered on the server. If they disagree, React throws an error.

Let's get a little more clever.

import {AppProps} from 'next/app';
import {useEffect, useState} from 'react';

function App({Component, pageProps}: AppProps) {
  const [render, setRender] = useState(false);
  useEffect(() => setRender(true), []);
  return render ? <Component {...pageProps} /> : null;
}
export default App;

This approach useState and useEffect. When Next.js renders a page on the server, it doesn't execute any useEffect calls; it just renders the page using default values and returns the result.

When this page is loaded by the browser will do the same thing: render the page using default values. Because the default value of render is false, the browser will initially render null. This is great, because it agrees with the server-rendered version of the page.

Immediately after, the browser will execute the useEffect call, which will set render to true. The page will now re-render in its full glory. We do waste a little time rendering null initially, but it's a microscopic amount of time: under a millisecond.

404 Page Not Found

React Router and the Next.js router can interact in strange and unexpected ways. To understand the problem, you need to understand the different between client-side routing and "server-side routing".

Currently pages/index.tsx implements a router with 2 "pages": the home page (/) and an about page (/about).

Try this:

1. Go to localhost:3000
2. Click the "About" link

You should see the URL changes to localhost:3000/about and we see the About "page". Great! But:

3. Refresh the page

Now we get a 404. What gives? localhost:3000/about worked just fine before the reload.

As far as Next.js is concerned, the homepage (localhost:3000) is the only page that exists. Once that page loads, react-router siezes control of the URL bar. When you click a react-router <Link>, it changes the URL programmatically, but the page is never fully reloaded.

But when you refresh the page, the browser asks Next.js to provide the content for localhost:3000/about. Since we haven't implemented pages/about.tsx, Next.js gives up and throws a 404.

The solution: Rewrites!

This feature was made available in Next.js 9.5. Rewrites are like "URL aliases"; you can tell Next.js to remap some "source" URL to a different "destination" URL. You can configure them in next.config.js file (if that file doesn't exist you should create it in the root of your project). Add the following contents:

// next.config.js
module.exports = {
  async rewrites() {
    return [
      {
        source: '/:any*',
        destination: '/',
      },
    ];
  },
};

This rewrite rule maps all incoming requests (/:any*) to the homepage (/).

Now, if you restart the server and navigate to localhost:3000/about your index.tsx homepage should render. Yay!

Mixing client- and server-side rendering

Important note: Next.js ignores this rewrite rule if the incoming request corresponds to an existing page in the pages directory. For instance, if you add pages/about.tsx later, Next.js will stop rewriting /about to the homepage.

This is actually really cool! For starters, it lets you use API Routes without any additional configuration. But more importantly, it lets you mix-and-match the SPA, SSR, and SSG paradigms at will! For instance, to add a /settings page that is server-side rendered:

1. Create and implement pages/settings.tsx
2. Add a getServerSideProps fetcher
3. Add a link to /settings using the <Link> component from next/link (not the one from react-router-dom!)

Now you have a single server-side rendered page inside your SPA! Can't do that with create-react-app 🙃

Wrap up

Big shout out to Tanner Linsley's GitHub Gist on converting from CRA to Next.js, which inspired and informed this post.

You can see ALL the code I used here in the demo repo at github.com/colinhacks/nextjs-spa.

The "comments section" for this post is this Twitter thread. Chime in! I love discussing Next.js, TypeScript, open-source software, and just about everything else!

✨ NEW BLOG POST ✨

It started as a simple "How to build a SPA with Next.js" tutorial.

It turned into a fevered rant about Vercel, Next.js, perverse incentives, and the future of the SPA paradigm. Oops.https://t.co/hrRtKojezH

— Colin McDonnell (@colinhacks) October 29, 2020

If you're into Next.js or TypeScript, follow me on Twitter @colinhacks! I build and maintain open-source tools like Zod (a TypeScript validation library) and tRPC (a tool for building end-to-end typesafe APIs with TypeScript). Or subscribe to the newsletter to get notified when I publish new posts!

Feb 24, 2021: a previous version of this post contained a bug where a new cookie would be created each time you signed out and back in again. These (identical) cookies would stack up cause problems. If you're having issues, pass the { path: '/' } option to nookies.set call — details below.

I had a hell of a time figuring out how to get Firebase Auth, Next.js, tokens, cookies, and getServerSideProps to play nice together. I'm hoping this breakdown will spare you the same suffering!

To jump straight to the sample repo, head to https://github.com/colinhacks/next-firebase-ssr 🤙

I'm currently building a new app. My goal is to do all data fetching inside getServerSideProps , just like the bad ole days of HTML templating. There are a lot of reasons for this.

• It reduces the cognitive overhead compared to the classic JAMstack approach. The client-server dichotomy is gone — it's just a server! So I can deploy my entire application with a single command.
• It solves the stale bundle problem. With a fully client-side rendered app, an user may use your application for days or weeks without doing a full page refresh. If the API changes underneath them, they may send API payloads that are no longer supported.
• It strikes a great balance between dynamism and search engine optimization.
• The user experience is unbeatable. With good caching and performant fetching, you can swap out the ubiquitous SPA spinner for snappy page loads.

As usual, Dan Abramov says it best:

We’ve reached peak complexity with SPA. The pendulum will swing a bit back to things on the server. But it will be a new take — a hybrid approach with different tradeoffs than before. Obviously I’m thinking React will be a part of that wave.

— Dan Abramov (@dan_abramov) August 3, 2020

It seems like the SSR-centric approach to building apps with Next.js is gaining steam fast. So I'm extremely confused why the official Next.js with-firebase-authentication example project doesn't demonstrate how to do it! It only demonstrates how to make authenticated POST requests to an API, not authenticate users inside getServerSideProps .

So below I explain how to use Next.js and Firebase Auth to:

• sign in users (duh)
• generate ID tokens
• store those ID tokens as a cookie
• auto-refresh the cookie whenever Firebase refreshes the ID token (every hour by default)
• implement authenticated routes
• authorize the user in getServerSideProps
• redirect unauthenticated users to a login page from within getServerSideProps

Let's dive in.

#1 Configure your Firebase JS SDK

You've probably done this already. Should look something like this:

// firebaseClient.ts

import * as firebase from 'firebase/app';
import 'firebase/auth';

if (typeof window !== 'undefined' && !firebase.apps.length) {
  firebase.initializeApp({
    apiKey: 'APIKEY',
    authDomain: 'myproject-123.firebaseapp.com',
    databaseURL: 'https://myproject-123.firebaseio.com',
    projectId: 'myproject-123',
    storageBucket: 'myproject-123.appspot.com',
    messagingSenderId: '123412341234',
    appId: '1:1234123412341234:web:1234123421342134d',
  });
  firebase.auth().setPersistence(firebase.auth.Auth.Persistence.SESSION);
}

export { firebase };

The firebase.apps.length check is a clever way of preventing Next.js from accidentally re-initalizing your SDK when Next.js hot reloads your application!

#2 Configure your Firebase Admin SDK

You've probably done this already too. Check out the Firebase docs for guidance. It'll look something like this:

// firebaseAdmin.ts

import * as firebaseAdmin from 'firebase-admin';

// get this JSON from the Firebase board
// you can also store the values in environment variables
import serviceAccount from './secret.json';

if (!firebaseAdmin.apps.length) {
  firebaseAdmin.initializeApp({
    credential: firebaseAdmin.credential.cert({
      privateKey: serviceAccount.private_key,
      clientEmail: serviceAccount.client_email,
      projectId: serviceAccount.project_id,
    }),
    databaseURL: 'https://YOUR_PROJECT_ID.firebaseio.com',
  });
}

export { firebaseAdmin };

Alternatively you can store the secret.json file elsewhere on your file system and tell Firebase where you find it using the GOOGLE_APPLICATION_CREDENTIALS environment variable. For details check out the Firebase Admin setup documentation here.

#3 Create a Context and provider

Now we're getting to the meat and potatoes. We're going to use React Context to implement the authentication logic. Let's start by writing a simple provider:

import { createContext } from 'react';

const AuthContext = createContext<{ user: firebase.User | null }>({
  user: null,
});

export function AuthProvider({ children }: any) {
  const [user, setUser] = useState<firebase.User | null>(null);

  // handle auth logic here...

  return (
    <AuthContext.Provider value={{ user }}>{children}</AuthContext.Provider>
  );
}

In the AuthProvider component we initialize a user state and pass it into our React component hierarchy using React Context. If you haven't seen this pattern before, I recommend reading the official docs on Context before continuing.

Listen for token changes

Now let's add a useEffect hook that initializes our Firebase authentication listener:

import nookies from 'nookies';

const AuthContext = createContext<{ user: firebase.User | null }>({
  user: null,
});

export function AuthProvider({ children }: any) {
  const [user, setUser] = useState<firebase.User | null>(null);

  useEffect(() => {
    return firebase.auth().onIdTokenChanged(async (user) => {
      if (!user) {
        setUser(null);
        nookies.set(undefined, 'token', '', { path: '/' });
      } else {
        const token = await user.getIdToken();
        setUser(user);
        nookies.set(undefined, 'token', token, { path: '/' });
      }
    });
  }, []);

  return (
    <AuthContext.Provider value={{ user }}>{children}</AuthContext.Provider>
  );
}

This is where the magic happens. We're hooking into the firebase.auth().onIdTokenChanged event listener; if you've never used it, it's identical to onAuthStateChanged but it also fires when the user's ID token is refreshed.

In the onIdTokenChanged callback, we check if the user is still signed in. If they are, we set the user with setUser .

Here's the most important part: we also set a token cookie that contains the user's ID token. (To accomplish this, I'm using the excellent nookies package by the great and powerful @maticzav.)

Feb 24, 2021: passing the { path: '/' } option to nookies.set prevents your app from creating a new token each time this code runs!

Now all outgoing requests — both API requests and page navigations! — will contain the user's ID token as a cookie! There's an example of this further down the page.

Token refresh

A previous version of this post incorrectly assumed that Firebase automatically refreshes the ID token on an hourly basis. As it turns out, that's not true. Firebase only does so if it is maintaining an active connect to Firestore or Realtime Database. If you aren't using one of these, services we need to refresh the tokens ourselves.

Below is the final version of our AuthProvider. Note the new useEffect hook that automatically force-refreshes the Firebase token every 10 minutes.

import nookies from 'nookies';

const AuthContext = createContext<{ user: firebase.User | null }>({
  user: null,
});

export function AuthProvider({ children }: any) {
  const [user, setUser] = useState<firebase.User | null>(null);

  // listen for token changes
  // call setUser and write new token as a cookie
  useEffect(() => {
    return firebase.auth().onIdTokenChanged(async (user) => {
      if (!user) {
        setUser(null);
        nookies.set(undefined, 'token', '', { path: '/' });
      } else {
        const token = await user.getIdToken();
        setUser(user);
        nookies.set(undefined, 'token', token, { path: '/' });
      }
    });
  }, []);

  // force refresh the token every 10 minutes
  useEffect(() => {
    const handle = setInterval(async () => {
      const user = firebaseClient.auth().currentUser;
      if (user) await user.getIdToken(true);
    }, 10 * 60 * 1000);

    // clean up setInterval
    return () => clearInterval(handle);
  }, []);

  return (
    <AuthContext.Provider value={{ user }}>{children}</AuthContext.Provider>
  );
}

Why use React Context?

You may be wondering why you need to use React Context at all. Let's consider a more naive approach to writing this hook.

// DONT DO THIS

export const useAuth = () => {
  const [user, setUser] = useState<firebase.User | null>(null);

  useEffect(() => {
    return firebase.auth().onIdTokenChanged(async (user) => {
      if (user) {
        setUser(user);
      } else {
        setUser(null);
      }
    });
  }, []);

  return { user };
};

This actually works fine. However it would create a new user state variable and a new listener every time you use the hook. We want to make sure the user variable is the same everywhere throughout our application to avoid difficult-to-debug synchronization issues. We only want a single reference to the currently signed in user that is shared throughout our app.

#4 Add your provider to a custom App

The documentation for setting up a Custom App page with Next.js are at https://nextjs.org/docs/advanced-features/custom-app. It's the ideal place to add "wrapper" code that you want to exist on all other pages — like Context providers.

// pages/_app.tsx

import type { AppProps } from 'next/app';
import { AuthProvider } from '../auth';

function App({ Component, pageProps }: AppProps) {
  return (
    <AuthProvider>
      <Component {...pageProps} />
    </AuthProvider>
  );
}
export default App;

#5 Create the useAuth hook

Now that the context and provider is set up, our actual useAuth hook is dead simple:

export const useAuth = () => {
  return useContext(AuthContext);
};

#6 Check the token in getServerSideProps

Below is a complete working example of how to implement an authenticated route. If the user isn't properly signed in (e.g. if the token cookies doesn't exist or if the token verification fails) the user is redirected to the /login page.

Otherwise, you can confidently fetch and return data belonging to the user!

// /pages/authenticated.tsx
import React from 'react';
import nookies from 'nookies';
import { InferGetServerSidePropsType, GetServerSidePropsContext } from 'next';

import { firebaseAdmin } from '../firebaseAdmin';

export const getServerSideProps = async (ctx: GetServerSidePropsContext) => {
  try {
    const cookies = nookies.get(ctx);
    const token = await firebaseAdmin.auth().verifyIdToken(cookies.token);

    // the user is authenticated!
    const { uid, email } = token;

    // FETCH STUFF HERE!! 🚀

    return {
      props: { message: `Your email is ${email} and your UID is ${uid}.` },
    };
  } catch (err) {
    // either the `token` cookie didn't exist
    // or token verification failed
    // either way: redirect to the login page
    ctx.res.writeHead(302, { Location: '/login' });
    ctx.res.end();

    // `as never` prevents inference issues
    // with InferGetServerSidePropsType.
    // The props returned here don't matter because we've
    // already redirected the user.
    return { props: {} as never };
  }
};

export default (
  props: InferGetServerSidePropsType<typeof getServerSideProps>
) => (
  <div>
    <p>{props.message}</p>
  </div>
);

The full solution

If you've made it this far you must think the ideas in this post are pretty useful! Consider retweeting this to get the word out: 🙌

Wrote up a guide to authenticated server-side rendering with Nextjs and Firebase Auth.

There' s a surprising lack of documentation or sample code for this use case, given that it may well become a best practice over the next couple years. 1/nhttps://t.co/22hVqp8Fy5

— Colin McDonnell (@colinhacks) August 17, 2020

The full code of this project — plus some other goodies! a sign up form! a logout button! — is available at https://github.com/colinhacks/next-firebase-ssr. Clone that repo to hit the ground running. 🏃🏽‍♂️💨

By the way, I'm currently working on an extremely rad library that lets you define your API with TypeScript and auto-generate a strongly typed SDK that you can safely use on the client. I think it could be a gamechanger for full-stack TypeScript/Next.js development. Join the mailing list below to stay updated.

Check out the HN discussion here.

This article has been translated into Japanese and Russian.

The existing open-source funding models don't work well for small projects.

Big projects — operating systems, frameworks, CMSs, or fully self-hostable applications — are in a privileged position to extract more value from their users, especially corporate ones. Since entire APIs and products are built on top of them, they inspire enough appreciation (or, more likely, fear of obsolescence!) to net a sustainable income from one-off or monthly donations.¹

But most OSS projects aren't big. The typical project on GitHub is better described as a utility — an apt term, given the infrastructural role these projects play in the world. It is a small tool that does one thing really well. Over the course of building a complex application, you may end up using dozens of these utilities — and they'll save you hundreds of hours of development time.

Unfortunately, utilities like these rarely bring in any meaningful amount of money from donations, no matter how widely used or beloved they are. Consider react-router. Even with 41.3k stars on GitHub, 3M weekly downloads from NPM, and nearly universal adoption in React-based single-page applications, it only brings in ~\$17k of donations annually.

The root of the problem is that open-source donations are made on a per-project basis². To support a project via GitHub Sponsors or OpenCollective, you must create yet another auto-renewing monthly subscription for each project you want to support. Moreover, at the moment of truth ("I'm gonna sponsor X!") it's easy to "logic" yourself out of donating ("But what if it gets replaced by some new hotness next week?!"). This has a huge dampening effect on total donations to open source. And in the end, only the projects that are massively, insanely, indisputably useful get funded. And those projects are usually the "big stuff" — frameworks, self-hostable software, etc.

A new model is needed; one that works for the small- and medium-sized projects, not just the huge ones. So I propose a novel(-ish) approach to OSS sustainability.

Introducing "sponsor pools"

1. Every month, you donate some amount into a "wallet".
2. Your funds are then distributed to the projects in your "sponsor pool". Your sponsor pool is just the set of open-source projects you want to support.
3. Adding new projects to your pool should require one click — as easy as starring the repo on GitHub.

That's it. It's hardly ingenious, which is why it's surprising that no major player in OSS has implemented it for facilitating open source donations.³

This would achieve the holy grail of OSS sustainability: the one-click sponsorship.⁴ Once a person has funded their sponsor pool, it would take a single click to financially support another project. The marginal cost — both psychological and financial — of supporting additional projects would drop to zero. That's a game changer.

Why this works better

Why do I think this will dramatically increase the total amount donated to open source? To answer that, consider a hypothetical question.

How much of your income would you be willing to donate to open-source software every year?

I suspect the typical overpaid HN lurker reading this post (👋) came up with a number in the hundreds of dollars. Compare that with how much you’re actually donating — is there a difference? There is for me. That's because there's currently no way to make a donation to the abstract concept of "open source software". But with sponsor pools, donation amounts will reflect an honest answer to the question posed above.

GitHub

The best case scenario, in my opinion, is for GitHub to natively support this model as an extension of GitHub Sponsors. It's where most projects live so it's best positioned to create a zero-friction donation system like this.

Of course, if GitHub implemented something like this, they would likely divorce the donation mechanism from stars. Perhaps instead, users who have created and funded a sponsor pool can be presented with an "Add to Pool" button in place of the current "Sponsor" button.

GitHub has a financial incentive to switch to this approach — sort of. They're currently eating the cost of all processing fees for transactions on GitHub Sponsors. So if you sponsor a project at $1/mo, the maintainer gets $1/mo…and GitHub pays \$0.30 to the credit card companies. Larger donation sizes means GitHub pays a proportionally smaller amount as fees.

Note that I said proportionally smaller. If more total donations are being processed (which is the goal of this proposal!), they may still end up paying more in total fees, even if the ratio of fees-to-donations is smaller. If the sponsor pool concept is massively successful — say, cumulative donations of $1B annually — GitHub will be eating nearly $20M in card fees. I suspect that would ruffle some feathers at Microsoft.

As a final addendum — I'd love to see embeddable "badges" that are unlocked at different levels of donorship. Imagine a world where you see a "GitHub Sponsor Gold Badge" in the footer of someone’s website, indicating that they donate, say, \$1000+ annually to open source. Clicking on this badge would bring you to a trusted site that lets you verify the claim. Here's my quick-and-dirty attempt at a badge design:

Some may cry "virtue signaling" but approaches like this are a proven way to establish positive reinforcement loops that a) increase awareness and b) encourage more folks to donate to open source.

Wrapping up

This is a hard problem with a vast array of potential solutions. I don't mean to criticize any existing approaches; they've all done a lot for maintainers, including myself, and I don't mean to minimize that. There may also be major implementational or regulatory hurdles to building something like this that I'm not aware of. This is merely a hypothetical exercise.

I write proposals like this one every month or so. Put your email in the box to get future proposals delivered to your inbox. ✌️

Footnotes

¹ Some examples of financially successful large projects.

>

• operating systems: Red Hat, Elementary OS
• frameworks: Livewire, Next.js
• CMSs: Ghost, Wordpress
• Full-featured applications: Sentry, Blender, Discourse

² It's worth noting that currently GitHub Sponsorships actually correspond to a given user or organization, NOT a particular repository or project.

I think it's important that the sponsorship pools themselves are comprised of projects, not users or organizations. This gets complicated if multiple entities/organizations are listed as the maintainers of a particular project. There would need to be a mechanism in place to split the incoming donation money among the set of maintainers as they see fit.

³ It's been pointed out to me that Flattr uses this exact model! Unfortunately they're not intended for open source software specifically and simply don't have enough awareness to really move the needle. For this model to reach it's potential, it needs to be implemented by a visible, trusted, established player.

Other organizations trying interesting things: Snowdrift ("crowdmatching"…very cool concept), Flossbank (a Tidelift-like model), Gitcoin.

⁴ The even holier grail: a zero click sponsorship! An example of this approach is Brave Browser's token-based micropayments system. Brave lets you fund your "wallet" with various cryptocurrencies, which are then distributed to creators proportional to the time you spend on their site.

It’s a great idea, but surveillance is necessary to gather the requisite information. This is implemented ethically by Brave, but that’s only possible because they control the full browser experience. For this to work across browsers would require an extension with full access to your browsing patterns. So probably a non-starter.

A maintainer of Emotion published a very thoughtful response to a version of this post! I encourage you to check that out here. The essay below has modified to address some of his points.

Emotion is my favorite CSS-in-JS library.

It's easy to define style classes — both inline or in separate files. You can powerfully compose classes in powerful ways with the cx utility (the Emotion equivalent of Jed Watson's classnames). You sprinkle in your styles using the standard className attribute. There's no need to modify your markup/JSX — as it should be! You only need to install a single module (yarn add emotion). And there's no complicated Babel plugin or config file to set up.

import { css, cs } from 'emotion';

const redBorder = css({ border: '1px solid red' });
const blueText = css({ color: 'blue' });

const MyComp = () => {
  return <div className={cx(redBorder, blueText)}>hello</div>;
};

I'm currently building a Tailwind-style CSS-in-JS utility styling library with a maniacal focus on brevity and developer experience. If you want to hear about that when it's ready, join my mailing list.

The "but"

But unfortunately everything I just said only applies to the "vanilla" emotion module (docs), not the confusingly-named @emotion/core module.

@emotion/core is the React-centric Emotion wrapper that gives you some extra goodies, specifically server-side rendering and theming. It's also the official recommendation of the Emotion project for any React project.

So why exactly is Emotion recommending this for React devs?

Minimal upsides

The main three advertised benefits of using @emotion/core are server-side rendering (SSR), theming, and customizability. Let's dig into those.

Out-of-the-box SSR

There's no denying this is a remarkable technical achievement. Getting SSR to "just work" with both Next.js, Gatsby, and the classic ReactDOMServer.renderToString approach is very impressive. I'd be lying if I claimed to understand the complexities involved.

I don't have data on this, but — in my experience — SSR isn't a consideration for a hefty majority of React projects. If you started a project/website in the last 7 years where SEO/SEO/pageload speed/bundle size was an important design consideration, you probably didn't choose React. Website builders, static site generators, and HTML templating still dominate that arena. Take it from someone who got torn apart on HN for advocating the use of React/Next.js for personal developer websites 😘

For the people who do need SSR, the guidance is a little slim.

Next.js

There's no explicit documentation from Next.js on how to set up SSR with vanilla emotion. Next.js provides a sample project here. Notably, this project a) has a very uninformative Readme and b) is built with @emotion/core! So it's not immediately obvious that the approaches on display will even transfer to a vanilla project.

Enough buildup. Here's the internet's first comprehensive guide to setting up SSR with vanilla Emotion and Next.js:

1. yarn add emotion-server
2. create _document.tsx in your pages directory and copy this gist into it. ⚠️ This gist is intended for Emotion v10. Version 11 is the latest version. ⚠️
3. ok ur done

Gatsby

For completeness, here's some instructions for Gatsby users, too.

1. yarn add gatsby-plugin-emotion
2. add 'gatsby-plugin-emotion' to your plugins list in gatsby-config.js

If you're using @emotion/core to avoid the complexities of SSR configuration, you may want to reconsider.

Theming

In the era of React Context and Hooks, there's no reason for libraries to use prop or high-order components to handle theming. Emotion provides a useTheme hook, but it still requires adding an additional library (emotion-theming).

This isn't a controversial claim; the next version of Emotion will explicitly recommend using a Context/Hook based solution, so I won't belabor this point.

Even Context/Hooks might be overkill for many projects. Just define your theme values as variables and import them into components as needed. If you're using TypeScript here's some code to get you started:

// theme.ts
export const primaryColor = "blue";
export const serif = `'Encode Sans', Times New Roman, Times, serif`;

// anydamnfile.ts
import { css } from 'emotion';
import * as theme from './theme.ts';

export const MyComponent = ()=>{
  return <p className={css({ color: theme.primaryColor, fontFamily: theme.serif })}>
}

If you want to import your theme with a useTheme hook, here's a workable implementation that took me several seconds to write:

import * as theme from './theme.ts';
export const useTheme = () => theme;

Customization

Edit: This section was added after the fact to address a concern raised by the maintainer of Emotion in his response to an earlier version of this post.

@emotion/core provides a CacheProvider component that lets you customize low-level aspects of it's behavior. This customization isn't possible with vanilla emotion. I'll let the maintainer of Emotion explain it:

Because [@emotion/core] doesn't give you string class names back, but rather just opaque objects, the rule insertion is lazy. This might not be as interesting on its own, but it allows us to customize injection through <CacheProvider>. Use cases for this are not as common - but they exist and once you ever come across one then you might be very thankful for the path we've chosen…Main things that can be customized:

• prefixing rules
• parser plugins
• container element (very important for rendering into iframes)
• nonce (very important for CSP)

— Andarist (Github)

If you absolutely need this degree of customizability, then @emotion/core is probably right for you.

For everyone else, let's look at the downsides.

Serious downsides

The css prop

Emotion recommends using their non-standard css prop to style your components, instead of React's built-in className. This causes me immeasurable emotional pain.

This approach destroys the portability of your React components. Your components are now unusable in any codebase that isn't configured to use @emotion/core.

The portability and encapsulation of React components is one of the most powerful and wondrous achievements in web development in the last last decade. Don't give that up without a good reason!

Installation woes

Unfortunately to get that non-native css prop to work, Emotion core entirely replaces your project's JSX parser. It substitutes the built-in React.createElement function with Emotion's custom jsx function.

There are a couple ways to set this up.

Option #1: install the @emotion/babel-preset-css-prop Babel plugin and add it to your .babelrc. If you're using Create React App, this isn't impossible. If you're using TypeScript, you probably don't have a .babelrc in your project.

If you're in one of those buckets, there's option #2: copy these two lines at the top of every React component you want to style with Emotion:

/** @jsx jsx */

import { jsx } from '@emotion/core';

If your TypeScript config or linter doesn't allow unused imports, you'll have to disable those rules to get rid of the warning. Check out this issue if you want to see dozens of TypeScript users being sad about this.

Lack of composability

Perhaps the most damning issue with @emotion/core is that it makes the simple things harder. If you want to define a new class or use cx, you have to wrap your component with the ClassNames render prop. But with @emotion/core, these basic functions — found in nearly all CSS-in-JS libraries — requires you to modify your markup. In my humble opinion, requiring markup modifications is a cardinal sin for a styling library.

Here's the example from the top of this post, reimplemented with @emotion/core:

import { ClassNames } from '@emotion/core';

const MyComp = () => {
  return (
    <ClassNames>
      {({ css, cx }) => {
        const redBorder = css({ border: '1px solid red' });
        const blueText = css({ color: 'blue' });

        return <div className={cs(redBorder, blueText)}>hello</div>;
      }}
    </ClassNames>
  );
};

Wrapping up

I understand how this happened. Vanilla emotion was undoubtedly getting flooded with GitHub issues by frustrated developers hitting against subtle limitations of its design. @emotion/core fixes these issues. But because @emotion/core is now the officially recommended approach for all React projects (the vanilla option isn't even mentioned on the Readme anymore), I suspect thousands of developers using it who would be better served by plain ol' emotion.

And finally: a huge thank you to the Emotion team for all their exceptional work and contributions to the open-source community.

I'm an open sourcer and ride-or-die React fan. Put your email in the box to hear when I write new things.

Check out the HN roast discussion here! 🤗

This article has been translated into Russian.

I recently set out to build my personal website — the one you're reading now, as it happens!

Surprisingly, it was harder than expected assemble a "tech stack" that met my criteria. My criteria are pretty straightforward; I would expect most React devs to have a similar list. Yet it was surprisingly hard to put all these pieces together.

Given the lack of a decent out-of-the-box solution, I worry that many developers are settling for static-site generators that place limits on the interactivity and flexibility of your website. We can do better.

Let's quickly run through my list of design goals:

React (+ TypeScript)

I want to build the site in React and TypeScript. I love them both wholeheartedly, I use them for my day job, and they're gonna be around for a long time. Plus writing untyped JS makes me feel dirty.

I don't want limitations on what my personal website can be/become. Sure, at present my site consists of two simple, static blog posts. But down the road, I may want to build a page that contains an interactive visualization, a filterable table, or a demo of a React component I'm open-sourcing. Even something simple (like the email newsletter signup form at the bottom of this page) was much more pleasant to implement in React; how did we use to build forms again?

Plus: I want access to the npm ecosystem and my favorite UI, animation, and styling libraries. I sincerely hope I never write another line of raw CSS ever again.

Good authoring experience

If it's annoying to write a blog post, I won't do it. That's a regrettable law of the universe. Even writing blog posts in plain HTML — a bunch of <p> tags in a div — is just annoying enough to bug me. The answer: Markdown of course!

Static site generators (SSGs) like Hugo and Jekyll provide an undeniably wonderful authoring experience. All you have to do is touch a new .md file in the proper directory and get to writing. Unfortunately all Markdown-based SSGs I know of are too restrictive. Mixing React and Markdown on the same page is either impossible or tricky. If it's possible, it likely requires some plugin/module/extension, config file, blob of boilerplate, or egregious hack. Sorry Hugo, I'm not going to re-write my React code using React.createElement like it's 2015.

Well, that doesn't work for me. I want my website to be React-first, with a sprinkling of Markdown when it makes my life easier.

Static generation

As much as I love the Jamstack, it doesn't cut it from an SEO perspective. Many blogs powered by a "headless CMS" require two round trips before rendering the blog content (one to fetch the static JS bundle and another to fetch the blog content from a CMS). This degrades page load speeds and user experience, which accordingly degrades your rankings on Google.

Instead I want every page of my site to be pre-rendered to a set of fully static assets, so I can deploy them to a CDN and get fast page loads everywhere. You could get the same benefits with server-side rendering, but that requires an actual server and worldwide load balancing to achieve comparable page load speeds. I love overengineering things as much as the next guy, even I have a line. 😅

My solution

I describe my final architecture design below, along with my rationale for each choice. I distilled this setup into a website starter/boilerplate available here: https://github.com/colinhacks/devii. Below, I allude to certain files/functions I implemented; to see the source code of these, just clone the repo git clone git@github.com:colinhacks/devii.git

Next.js

I chose to build my site with Next.js. This won't be a surprising decision to anyone who's played with statically-rendered or server-side rendered React in recent years. Next.js is quickly eating everyone else's lunch in this market, especially Gatsby's (sorry Gatsby fans).

Next.js is by far the most elegant way (for now) to do any static generation or server-side rendering with React. They released their next-generation (pun intended) static site generator in the 9.3 release back in March 2020. In the spirit of using technologies in the spring of their life, Next.js is a no-brainer.

Here's a quick breakdown of the project structure. No need to understand every piece of it; but it may be useful to refer to throughout the rest of this post.

.
├── README.md
├── public // all static files (images, etc) go here
├── pages // every .tsx component in this dir becomes a page of the final site
|   ├── index.tsx // the home page (which has access to the list of all blog posts)
|   ├── blog
|       ├── [blog].md // a template component that renders the blog posts under `/md/blog`
├── md
|   ├── blog
|       ├── devii.md // this page!
        ├── whatever.md // every MD file in this directory becomes a blog post
├── components
|   ├── Code.tsx
|   ├── Markdown.tsx
|   ├── <various others>
├── loader.ts // contains utility functions for loading/parsing Markdown
├── node_modules
├── tsconfig.json
├── package.json
├── next.config.js
├── next-env.d.ts
├── .gitignore

TypeScript + React

Both React and TypeScript are baked into the DNA of Next.js, so you get these for free when you set up a Next.js project.

Gatsby, on the other hand, has a special plugin for TypeScript support, but it's not officially supported and seems to be low on their priority list. Also, after messing with it for an hour I couldn't get it to play nice with hot reload.

Markdown authoring

Using Next's special getStaticProps hook and glorious dynamic imports, it's trivial to import a Markdown file and pass its contents into your React components as a prop. This achieves the holy grail I was searching for: the ability to easily mix React and Markdown.

Frontmatter support

Every Markdown file can include a "frontmatter block" containing metadata. I implemented a simple utility function (loadPost) that loads a Markdown file, parses its contents, and returns a TypeScript object with the following signature:

type PostData = {
  path: string; // the relative URL to this page, can be used as an href
  content: string; // the body of the MD file
  title?: string;
  subtitle?: string;
  date?: number;
  author?: string;
  author_image?: string;
  tags?: string[];
  banner_image?: string;
  thumb_image?: string;
};

I implemented a separate function loadPosts that loads all the Markdown files under /md/blog and returns them as an array (PostData[]). I use loadPosts on this site's home page to render a list of all posts I've written.

Medium-inspired design

I used the wonderful react-markdown package to render Markdown as a React component. My Markdown rendered component (/components/Markdown.tsx) provides some default styles inspired by Medium's design. Just modify the style pros in Markdown.tsx to customize the design to your liking.

GitHub-style code blocks

You can easily drop code blocks into your blog posts using triple-backtick syntax. Specify the programming language with a "language tag", just like GitHub!

To achieve this I implemented a custom code renderer (/components/Code.tsx) for react-markdown that uses react-syntax-highlighter to handle the highlighting. So this:

// pretty neat huh?
const test = (arg: string) => {
  return arg.length > 5;
};

turns into this:

// pretty neat huh?
const test = (arg: string) => {
  return arg.length > 5;
};

RSS feed generation

An RSS feed is auto-generated from your blog post feed. This feed is generated using the rss module (for converting JSON to RSS format) and showdown for converting the markdown files to RSS-compatible HTML. The feed is generated during the build step and written as a static file to /rss.xml in your static assets folder. It's dead simple. That's the joy of being able to easily write custom build scripts on top of Next.js's getStaticProps hooks!

SEO

Every blog post page automatically populated meta tags based on the post metadata. This includes a title tag, meta tags, og: tags, Twitter metadata, and a link tag containing the canonical URL. You can modify/augment this in the PostMeta.ts component.

Static generation

You can generate a fully static version of your site using yarn build && yarn export. This step is entirely powered by Next.js. The static site is exported to the out directory.

After its generated, use your static file hosting service of choice (Firebase Hosting, Vercel, Netlify) to deploy your site.

Insanely customizable

There's nothing "under the hood" here. You can view and modify all the files that provide the functionality described above. Devii just provides a project scaffold, some Markdown-loading loading utilities (in loader.ts), and some sensible styling defaults (especially in Markdown.tsx).

To start customizing, modify index.tsx (the home page), BlogPost.tsx (the blog post template), and Markdown.tsx (the Markdown renderer).

Get started

Head to the GitHub repo to get started: https://github.com/colinhacks/devii. If you like this project, leave a ⭐️star⭐️ to help more people find Devii! 😎

To jump straight into the code, clone the repo and start the development server like so:

git clone git@github.com:colinhacks/devii.git mysite
cd mysite
yarn
yarn dev

I write essays like this one every month. Put your email in the box to get future proposals delivered to your inbox.

Update 2023 — Many of the problems with other libraries have been solved since this post was originally published.

There are a handful of wildly popular validation libraries in the Javascript ecosystem with thousands of stars on GitHub.

When I started my quest to find the ideal schema declaration/validation library, I assumed the hardest part would be identifying the Most Great option from a sea of excellent ones.

But as I dug deeper into the many beloved tools in the ecosystem, I was surprised that none of them could provide the features and developer experience I was looking for.

tl;dr: I made a new Typescript validation library that has static type inference and the best DX this side of the Mississippi. To jump straight to README, head over to https://github.com/colinhacks/zod

A pinch of context

I'm building an API for a healthcare application with Typescript. Naturally I want this mission-critical medical software to be rock-solid, so my goal is to build a fully end-to-end typesafe data layer.

All data that passes from the client to the server AND server to client should be validated at runtime. Moreover, both the client and server should have static type definitions of all payloads, so I can catch more errors at compile-time.

I also don't hate myself, so I don't want to keep my static types and runtime type validators in sync by hand as my data model changes. Which means we need a tool that supports 🚀StAtIc TyPe InFeReNcE!!🚀 vuvuzela sounds

Let's look at our options!

The existing options

Joi (14.3k stars)

Doesn't support static type inference. Boo.

Yup (8.6k stars)

Yup is jquense's popular, Joi-inspired validation library that was implemented first in vanilla JS, with Typescript typings added later.

Yup supports static type inference! But with a small caveat: the typings are wrong.

For instance, the yup package treats all object properties as optional by default.

const schema = yup.object({
  asdf: yup.string(),
});
schema.validate({}); // passes

Yet the inferred type indicates that all properties are required. 😕

type SchemaType = yup.InferType<typeof schema>;

// returns { asdf: string }
// should be { asdf?: string }

Yup also mis-infers the type of "required" arrays.

const numList = yup.array().of(yup.string()).required();

// .required() is used to indicate a non-empty list
numList.validateSync([]); // fails

// yet the inferred type doesn't reflect this
type NumList = yup.InferType<typeof numList>;
// returns string[]
// should be [string,...string[]]

Finally, Yup doesn't explicitly support generic union and intersection types. This is a recurring source of frustration for the Yup community.

These may sound like nitpicks. But it's very uncool for the inferred type to not actually reflect the actual type of the validator it came from.

Moreover, there are no plans to fix these issues in Yup's type inference because those changes would be backwards compatible. See more about this here.

io-ts (2.7k stars)

Finally, a library that was designed for Typescript from the ground up!

Look, io-ts is excellent library. It's creator, gcanti, has done more than anyone to bring proper higher-order functional programming to Typescript with his fp-ts library.

But in my situation, and I think many others', io-ts prioritizes functional programming purity at the expense of developer experience. Functional purity is a valid and admirable design goal, but it makes io-ts particularly hard to integrate into an existing codebase with a more procedural or object-oriented bias. It's difficult to progressively integrate io-ts without entirely refactoring your code with a more functional flavor.

For instance, consider how to define an object schema with optional properties in io-ts:

const A = t.type({
  foo: t.string,
});

const B = t.partial({
  bar: t.number,
});

const C = t.intersection([A, B]);

type C = t.TypeOf<typeof C>;
/*
{
  foo: string;
  bar?: number | undefined
}
*/

You must define the required props with a call to t.type({...}), define your optional props with a call to t.partial({...}), and merge them with t.intersection().

Spoiler alert: Here's the equivalent in my new library:

const C = z.object({
  foo: z.string(),
  bar: z.number().optional(),
});

type C = t.TypeOf<typeof C>;
/* {
  foo: string;
  bar?: number | undefined
} */

io-ts also requires the use of gcanti's functional programming library fp-ts to parse results and handle errors. From the io-ts docs, here is the most basic example of how to run a validation:

import * as t from 'io-ts';
import {pipe} from 'fp-ts/lib/pipeable';
import {fold} from 'fp-ts/lib/Either';

// failure handler
const onLeft = (errors: t.Errors): string => `${errors.length} error(s) found`;

// success handler
const onRight = (s: string) => `No errors: ${s}`;

pipe(t.string.decode('a string'), fold(onLeft, onRight));
// => "No errors: a string"

The functional approach here is alienating for developers who aren't accustomed to it.

Again: fp-ts is fantastic resource for developers looking to keep their codebase strictly functional. But depending on fp-ts necessarily comes with a lot of intellectual overhead; a developer has to be familiar with functional programming concepts, fp-ts's nomenclature, and the Either monad to do a simple schema validation. It's alienating for devs who don't have a functional background, and it drives them to libraries like Yup, which returns incorrect types.

Introducing Zod

So, in the tradition of many a nitpicky programmer, I decided to build my own library from scratch. What could go wrong?

The final result: https://github.com/colinhacks/zod.

Zod is a validation library designed for optimal developer experience. It's a Typescript-first schema declaration library with rigorous (and correct!) inferred types, incredible developer experience, and a few killer features missing from the existing libraries.

• Uses Typescript generic inference to statically infer the types of your schemas
• Eliminates the need to keep static types and runtime validators in sync by hand
• Has a composable, declarative API that makes it easy to define complex types concisely

Zod was also designed with some core principles designed to make all declarations as non-magical and developer-friendly as possible:

• Fields are required unless explicitly marked as optional (just like Typescript!)
• Schemas are immutable; methods (i.e. .optional()) return a new instance.
• Zod schemas operate on a Parse, don't validate! basis!

To jump straight to README, head over to https://github.com/colinhacks/zod. If you're feeling frisky, leave a star 🌟👍

Primitives

import * as z from 'zod';

const stringSchema = z.string(); // => ZodType<string>
const numberSchema = z.number(); // => ZodType<number>
const booleanSchema = z.boolean(); // => ZodType<boolean>
const undefinedSchema = z.undefined(); // => ZodType<undefined>
const nullTypeSchema = z.null(); // => ZodType<null>

Parsing

// every ZodType instance has a .parse() method
const stringSchema = z.string();
stringSchema.parse('fish'); // => "fish"
stringSchema.parse(12); // throws Error('Non-string type: number');

Type inference

Like in io-ts, you can extract the Typescript type of any schema with {'z.TypeOf<>'}.

const A = z.string();
type A = z.TypeOf<typeof A>; // string

const u: A = 12; // TypeError
const u: A = 'asdf'; // compiles

We'll include examples of inferred types throughout the rest of the documentation.

Objects

// all properties are required by default
const dogSchema = z.object({
  name: z.string(),
  age: z.number(),
  neutered: z.boolean(),
});

type Dog = z.TypeOf<typeof dogSchema>;
/* equivalent to:
type Dog = {
  name:string;
  age: number;
  neutered: boolean;
} 
*/

const cujo = dogSchema.parse({
  name: 'Cujo',
  age: 4,
  neutered: true,
}); // passes, returns Dog

const fido: Dog = {
  name: 'Fido',
  age: 2,
}; // TypeError: missing required property 'neutered'

Arrays

const dogsList = z.array(dogSchema);

dogsList.parse([{name: 'Cujo', age: 3, neutered: true}]); // passes
dogsList.parse([]); // passes

Plus you can explicitly define a non-empty array schema, something io-ts doesn't support.

// Non-empty lists

const nonEmptyDogsList = z.array(dogSchema).nonempty();
nonEmptyDogsList.parse([]); // throws Error("Array cannot be empty")

Unions (including nullable and optional types)

Zod includes a built-in z.union method for composing "OR" types.

const stringOrNumber = z.union([z.string(), z.number()]);

stringOrNumber.parse('foo'); // passes
stringOrNumber.parse(14); // passes

Unions are the basis for defining nullable and optional values.

/* Optional Types */ // "optional string" === the union of string and undefined
const A = z.union([z.string(), z.undefined()]);
A.parse(undefined); // => passes, returns undefined
type A = z.TypeOf<typeof A>; // string | undefined

There is also a shorthand way to make a schema "optional":

const B = z.string().optional(); // equivalent to A

const C = z.object({
  username: z.string().optional(),
});
type C = z.TypeOf<typeof C>; // { username?: string | undefined };

/* Nullable Types */ const D = z.union([z.string(), z.null()]);

const E = z.string().nullable(); // equivalent to D
type E = z.TypeOf<typeof D>; // string | null

You can create unions of any two schemas.

/* Custom Union Types */
const F = z.union([z.string(), z.number()]).optional().nullable();
F.parse('tuna'); // => tuna
F.parse(42); // => 42
F.parse(undefined); // => undefined
F.parse(null); // => null
F.parse({}); // => throws Error!

type F = z.TypeOf<typeof F>; // string | number | undefined | null;

Intersections

Intersections are useful for creating "logical AND" types.

const a = z.union([z.number(), z.string()]);
const b = z.union([z.number(), z.boolean()]);

const c = z.intersection(a, b);
type c = z.TypeOf<typeof C>; // => number

const neverType = z.intersection(z.string(), z.number());
type Never = z.TypeOf<typeof stringAndNumber>; // => never

This is particularly useful for defining "schema mixins" that you can apply to multiple schemas.

const HasId = z.object({
  id: z.string(),
});

const BaseTeacher = z.object({
  name: z.string(),
});

const Teacher = z.intersection(BaseTeacher, HasId);

type Teacher = z.TypeOf<typeof Teacher>;
// { id:string; name:string };

Object merging

In the examples above, the return value of z.intersection is an instance of ZodIntersection, a generic class that wraps the two schemas passed in as arguments.

But if you're trying to combine two object schemas, there is a shorthand:

const Teacher = BaseTeacher.merge(HasId);

The benefit of using this shorthand is that the returned value is a new object schema (ZodObject), instead of a generic ZodIntersection instance. This way, you're able to fluently chain together many .merge calls:

// chaining mixins
const Teacher = BaseTeacher.merge(HasId).merge(HasName).merge(HasAddress);

Tuples

These differ from arrays in that they have a fixed number of elements, and each element can have a different type.

const athleteSchema = z.tuple([
  // takes an array of schemas
  z.string(), // name
  z.number(), // jersey number
  z.object({
    pointsScored: z.number(),
  }), // statistics
]);

type Athlete = z.TypeOf<typeof athleteSchema>;
// type Athlete = [string, number, { pointsScored: number }]

Recursive types

You can define a recursive schema in Zod, but because of a limitation of Typescript, their type can't be statically inferred. If you need a recursive Zod schema you'll need to define the type definition manually, and provide it to Zod as a "type hint".

interface Category {
  name: string;
  subcategories: Category[];
}

const Category: z.ZodType<Category> = z.lazy(() => {
  return z.object({
    name: z.string(),
    subcategories: z.array(Category),
  });
});

Category.parse({
  name: 'People',
  subcategories: [
    {
      name: 'Politicians',
      subcategories: [{name: 'Presidents', subcategories: []}],
    },
  ],
}); // passes

Function schemas

Zod also lets you define "function schemas". This makes it easy to validate the inputs and outputs of a function without intermixing your validation code and "business logic".

You can create a function schema with z.function(args, returnType) which accepts these arguments.

• args: ZodTuple The first argument is a tuple (created with z.tuple(\[...\]) and defines the schema of the arguments to your function. If the function doesn't accept arguments, you can pass an empty tuple (z.tuple([])).
• returnType: ZodType The second argument is the function's return type. This can be any Zod schema.

const args = z.tuple([
  z.object({nameStartsWith: z.string()}),
  z.object({skip: z.number(), limit: z.number()}),
]);

const returnType = z.array(
  z.object({
    id: string(),
    name: string(),
  })
);

const FetcherFactory = z.function(args, returnType);

z.function returns a higher-order "function factory". Every "factory" has .validate() method which accepts a function as input and returns a new function. The returned function automatically validates both its inputs and return value against the schemas provided to z.function. If either is invalid, the function throws. This lets you confidently execute business logic in a "validated function" without worrying about invalid inputs or return types, mixing your validation and business logic, or writing duplicative types for your functions. Here's an example.

const validatedQueryUser = FetchFunction.validate((filters, pagination) => {
  // the arguments automatically have the appropriate types
  // as defined by the args tuple passed to `z.function()`
  // without needing to provide types in the function declaration

  filters.nameStartsWith; // autocompletes
  filters.ageLessThan; // TypeError

  // Typescript statically verifies that value returned by
  // this function is of type { id: string; name: string; }[]

  return 'salmon'; // TypeError
});

const users = validatedQueryUser(
  {
    nameStartsWith: 'John',
  },
  {
    skip: 0,
    limit: 20,
  }
); // => returns { id: string; name: string; }[]

This is particularly useful for defining HTTP or RPC endpoints that accept complex payloads that require validation. Moreover, you can define your endpoints once with Zod and share the code with both your client and server code to achieve end-to-end type safety.

Onwards and upwards

I think there's a LOT of room for improvement in how we handle the complexity of data modeling, validation, and transport as developers. Typescript, GraphQL, and Prisma are huge steps towards a future where our tooling can provide guarantees of data integrity. But there's still a long way to go.

If you like what you see, head over to https://github.com/colinhacks/zod and leave a star.

I'm an open sourcer and ride-or-die TypeScripter. Put your email in the box to hear when I publish new essays or open source libraries.