
I maintain Zod, which is a schema library. Zod let's people declare an email schema like this:

```ts
import {z} from 'zod';

z.string().email();
```

A lot of people think every _technically valid_ email address should pass validation by that schema. I [used to think that](https://github.com/colinhacks/zod/issues/639#issuecomment-917223045) too.

Over the years I've merged several PRs to make the email regex more "technically correct". Most recently, I merged a PR that adds support for _IPv6 addresses_ as the domain part of an email. They look like this and I hate them:

```
jonny@[ipv6:7e95:0559:10f2:21e9:9dab:7309:c116:ca3b]
```

Turns out that PR also broke plain old subdomains: `user@subdomain.domain.com`. That means Zod currently fails to parse my _mom's current email address_ but Jonny up there can parse his freakish IPv6 email.

This caused a spiritual crisis and made me re-evaluate my whole stance on what `z.string().email()` should do. Zod's users are mostly engineers who are building apps. When you're building an app, you want to make sure your users are providing normal-ass email addresses. So that's what `z.string().email()` is going do.

So I rewrote the regex from scratch to be simple and reasonable. Here's what I came up with:

```ts
/^([A-Z0-9_+-]+\.?)*[A-Z0-9_+-]@([A-Z0-9][A-Z0-9-]*\.)+[A-Z]{2,}$/i;
```

Let's break that down in plain English:

**The username** (AKA "local part")

- can use these characters: `[a-zA-Z0-9-._]`
- can't have two dots in a row
- can't start or end with a dot

**The domain**

- can use these characters: `[a-zA-Z0-9-.]`
- must have one or more dots in it
- can't have two dots in a row
- can't start or end with a dot
- can't start or end with a hyphen
- must end with a "TLD" that's 2+ letters, only `[a-zA-Z]` allowed

It is _not_ trying to be [RFC 5322](https://datatracker.ietf.org/doc/html/rfc5322) compliant. It's not going to check if the TLD is real. And it's not going to implement any of this craziness:

- No "printable" characters: `wtf-!#$%&'*/=?^_{|}~@mail.com`
- No quoted local parts: `"zod is cool"@mail.com`
- No comments: `regexiscool(kinda)@mail.com`
- No IPv4: `billie@[1.2.3.4]`
- No IPv6: `jonny@[ipv6:7e95:0559:10f2:21e9:9dab:7309:c116:ca3b]`
- No emoji: `👎@mail.com`

But for reasonable people, it'll do.


An email regex for reasonable people

> Spoiler alert: GitHub Pages and GitHub actions are not involved.

I recently set out to build a proper documentation site for [Zod](https://github.com/colinhacks/zod) with a short and eminently reasonable list of goals:

1. The site should be generated from the README. Zod's `README.md` is the "source of truth".
2. The site should automatically update whenever the `master` branch is updated.
3. I didn't want to worry about building or maintaining a complex build workflow in GitHub Actions, if possible.



Despite the eminent reasonability, I had a hard time finding a simple solution. After a lot of research and false starts, I've found what I think is the lowest-effort, highest-reward way to publish a documentation site for your GitHub-hosted project.

Let's jump in.

## 1. Add this `index.html` to your repo

Create a file called `index.html` in your project root and paste the following contents into it.

> Replace all occurrences of `user/repo` with your project's name. And maybe write up a slightly more detailed `meta` description :)

```
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <title>user/repo</title>
    <meta http-equiv="X-UA-Compatible" content="IE=edge,chrome=1" />
    <meta name="description" content="user/repo is cool!" />
    <meta name="viewport" content="width=device-width, user-scalable=no, initial-scale=1.0, maximum-scale=1.0, minimum-scale=1.0" />
    <link rel="stylesheet" href="//cdn.jsdelivr.net/npm/docsify/lib/themes/vue.css" />
    <style>
      .markdown-section { max-width: 700px; }
    </style>
  </head>
  <body>
    <nav
      style="display: flex; flex-direction: row; align-items: center; justify-content: space-between;"
    >
      
    </nav>
    <div id="app"></div>
    <script>
      window.$docsify = {
        subMaxLevel: 1,
        maxLevel: 3,
        auto2top: true,
        repo: 'user/repo',
        routerMode: 'history',
        // homepage: "README.md"
      };
    </script>
    <script src="//cdn.jsdelivr.net/npm/docsify@4.12.2/lib/docsify.min.js"></script>
    <script src="//cdn.jsdelivr.net/npm/prismjs@1.28.0/prism.min.js"></script>
  </body>
</html>
```

Those 1150 bytes are doing a lot, but most importantly it's loading [Docsify](https://docsify.js.org/). Docsify is a documentation generator. You may recognize it from its goofy looking gradient cover pages with big pill-shaped buttons.

![Docsify sample homepage](/docsify.png)

> I'm not a fan of these cover pages, so we won't be using one here. All in all, though, I think Docsify is extraordinarily well designed.

It also differs from some other options like Docusaurus and Gitbook in that it's a _fully client-side_ documentation generator. Your Markdown files don't get pre-compiled to HTML ahead of time; instead, Docsify fetches it asynchronously, parses the contents in-browser, and injects the rendered HTML into the page.

_"The horror! Multiple round-trip requests before the first meaningful paint!?"_ Turns out, it's not a big issue. Modern globe-spanning ultra-fast CDNs for static content, this still feels near-instant. Google agrees; the [Zod site](https://zod.dev) gets a 98 Performance score on Lighthouse.

![98 performance on Lighthouse](/lighthouse.png)

## 2. Run locally

There's no build step required with Docsify. You could deploy your `index.html` and a `README.md` to any static site hosting service and it would work. We're going to be using [Netlify](https://www.netlify.com/) to deploy this site, but before we get to that, let's see how the site is looking. We can use Netlify's CLI to start a simple static site server.

```bash
$ npx netlify dev
◈ Netlify Dev ◈
   ┌─────────────────────────────────────────────────┐
   │                                                 │
   │   ◈ Server now ready on http://localhost:8888   │
   │                                                 │
   └─────────────────────────────────────────────────┘
```

Open `http://localhost:8888` in your browser, and you should see a rendered version of your `README.md`!

> If `netlify dev` detects that you're using a framework, this may not work. In that case you'll need to disable framework detection and tell Netlify to act as a simple static file host.
>
> ```sh
> npx netlify dev --framework "#static"
> ```

> If your primary Markdown file isn't called `README.md`, you can specify a different name with the `homepage` setting in the Docsify configuration.
>
> ```js
> window.$docsify = {
>   subMaxLevel: 1,
>   maxLevel: 3,
>   auto2top: true,
>   repo: 'user/repo',
>   routerMode: 'history',
>   homepage: 'home.md', // <-- add this
> };
> ```

## 3. Configure rewrites

Docsify using client-side routing to determine which Markdown file to load and render. In other words, it's a pure single-page application; `index.html` loads Docsify, which then assumes control over your browsers URL bar, similar to React Router or other client-side routers.

If your project/site contains several Markdown files, you may encounter a strange behavior. Consider the following file structure:

```
├── index.html
├── README.md
└── CONTRIBUTING.md
```

If you start the development server and type `localhost:8888/CONTRIBUTING` into the URL bar, you'll see a 404. That's because there's no HTML file called `CONTRIBUTING.html`. We need all incoming requests to render `index.html`! Then Docsify will read the URL and load the appropriate Markdown file.

We can achieve this with a Netlify feature called [rewrites](https://docs.netlify.com/routing/redirects/). (All static file hosts have an equivalent feature.) Create a file called `_redirects` with no extension. (With Netlify, both redirects and rewrites are declared in `_redirects`.)

```
$ touch _redirects
```

Then paste in the following:

```
/*            /             200
```

This tells Netlify that all incoming requests (`/*`) should render the homepage (`/`). However, unlike a _redirect_, rewrites do not change the URL. Basically Netlify will sneakily render `index.html` under the hood, without telling the browser.

> Another solution is to use a _hash router_. In fact, this is the default behavior for Docsify. Unfortunately, it makes the URL messy; the URL would look like `domain.com/#/CONTRIBUTING` instead of simply `domain.com/CONTRIBUTING`. I prefer clean URLs so I chose the rewrites approach.

Now when we run `npx netlify dev`, we can open `localhost:8888/CONTRIBUTING` directly without seeing a 404.

## 4. Deployment

Firebase used to be my go-to for static file hosting, but these days Netlify leads the pack on developer experience. It only takes a couple minutes to set up Github-linked static site hosting.

First, let's push our new files to `master`.

```sh
$ git add .
$ git commit -am "Add docs site"
$ git push origin master
```

Then configure Netlify.

1. Log in or create an account at [netlify.com](https://www.netlify.com/).
2. From the dashboard, click `Add new site > Import an existing project`.
3. Authenticate with GitHub (or GitLab or Bitbucket) and select your repository.

When prompted for a `"Branch to deploy"` select `master` (the default) unless your primary branch has a different name.

Under "Build settings" you can simply leave everything blank:

- `"Base directory"` — leave blank (defaults to the project root)
- `"Build command"` — leave blank (no build step necessary!)
- `"Publish directory"` — leave blank (defaults to the project root)

![Netlify configuration settings](/netlify_build.png)

There is virtually no configuration required, because we're simply hosting static files from the root directory of our `master` branch. Couldn't be easier!

> Note that we're deploying the entire contents of our repository to Netlify. You could write a build command to [delete all unwanted files](https://answers.netlify.com/t/can-i-exclude-certain-files-like-readme-md/15416) but I didn't bother. The contents of my repo are already public on the internet, so...🤷‍♂️

Proceed with the deployment. Once Netlify finishes deploying, Netlify will provide a `*.netlify.app` subdomain for your site.

## 5. Final touches

There's plenty more to do before your site is truly ready.

- Connect a custom domain
- Add a `robots.txt` to your root directory
- Add links to the top `<nav>` in `index.html`
- Add [Open Graph](https://ahrefs.com/blog/open-graph-meta-tags/) meta tags
- Add [Twitter](https://developer.twitter.com/en/docs/twitter-for-websites/cards/overview/markup) meta tags and a banner image
- Generate a favicon ([RealFaviconGenerator](https://realfavicongenerator.net/) is great)

But I'll leave that as an exercise for the reader.

From README to documentation site in 10 minutes

In the mid-2010s, the pendulum of web development swung massively in the direction of increased type safety.

First, we all switched back to relational DBs after recovering from an embarrassingly long collective delusion that NoSQL was cool. Then TypeScript started its meteoric rise after years of quiet development. Shortly thereafter GraphQL was announced and quickly became the de facto way to implement typesafe APIs.

> TLDR: I published a new library called tRPC that makes it easy to build end-to-end typesafe APIs without code generation, by leveraging the power of modern TypeScript. To jump straight to the tRPC walkthrough, [click here](#trpc).

### GraphQL's TypeScript problem

GraphQL's biggest competitive advantage is that it's a _spec_, not a library. It's designed to be universal and language-agnostic. If you use different languages for your client and server, then by all means stick with GraphQL! It remains the best language-agnostic way to build typesafe data layers.

But the vast majority of projects that use GraphQL are JavaScript-based. The `graphql` NPM module has [5M downloads/week](https://www.npmjs.com/package/graphql) vs [~300k](https://pypistats.org/packages/graphene)) for the `graphene` Python module and [~230k](https://bestgems.org/gems/graphql) for the `graphql` Rubygem. It's not a perfect metric, but the signal is clear.

And a growing fraction of JavsScript projects are TypeScript-based. As TypeScript approaches near-universal adoption in the JS ecosystem, that means **most** GraphQL-based projects will soon be TypeScript-based (if they aren't already).

Here's the thing: most people use GraphQL as a massively over-engineered way to share the **type signature** of their API with your client. What if you could import that type signature _directly into your client_ using plain ole `import` syntax? Well...you can.

## A demonstrative example

We're going to build the world's simplest typesafe API—no codegen or GraphQL required. There's only one endpoint, with the following type signature:

`getUser(id: string) => { id: string, name: string }`

For the sake of simplicity, the API is implemented as an Express router. It should be easy to tell what's going on even if you aren't familiar with it.

### #1 Define `getUser` function

```ts
// server/routes.ts
export const getUser = async (id: string) => {
  return { id, name: 'Bilbo' };
};
```

### #2 Implement a simple RPC endpoint

We set up a special `/rpc` endpoint that accepts POST requests with a payload of type `{ endpoint: string; arguments: any[] }`. The `endpoint` property indicates the function to call (in our case, the only option is `"getUser"`). We take the `arguments` array, pass it into the appropriate function, and `res.send` the results.

```ts
// server/router.ts
import express from 'express';
import * as routes from './routes';

type Payload = { endpoint: string; arguments: any[] };

const app = express();
app.post('/rpc', async (req, res) => {
  const payload: Payload = req.body;
  if (payload.endpoint === 'getUser') {
    const user = await routes.getUser(...payload.args);
    return res.status(200).send(user);
  }

  return res.status(404).send('Endpoint not found');
});
```

> **A totally skippable aside about RPC**

> With a REST APIs, you use different URLs (`/users`, `/user/:id`) and HTTP request types (e.g GET, POST, etc) to access different functionality. For instance `GET /users` will fetch all users, `GET /user/abc123` will fetch a single user, and `POST /user` will create a new user.

> RPC strips all of that away. Endpoints are just named functions that accept some arguments and return a value. The RPC ethos is to make API calls look and feel like calling a function (albeit one that's defined on a remote server).

> GraphQL is an RPC protocol that's been augmented with a schema language and a mechanism for client-side field selection.

### #3 Import the type signature of `getUser` into the client

This is where the magic happens. We **directly import** the type signature of our API into out client. Yes, it really is this easy.

```tsx
// client/index.tsx
import type { getUser } from '../server/routes.ts';
```

Note that this works no matter where `routes.ts` lives in your file system! It could be in a different Git repo; it could be on your external hard drive. As long as the TypeScript file exists on your computer, you can import it from any other TypeScript file.

**_"But wait, I can't just import stuff from a different repo!"_**

This is often true...when you're importing code. For instance, Create React App (CRA) prohibits you from importing anything from outside of your `src` directory.

But that doesn't apply here. We're using `import type` syntax, which is a **purely static construct**. That is, it can only import _type signatures_, but not data, code, or any runtime construct. For most build systems powered by Babel (including CRA), `import type` statements (and all type annotations) just get stripped out before the bundling step.
If you're using Webpack with Babel, Parcel, or Rollup, this Just Works™. Don't believe me? Play around with this [sample repo](https://github.com/colinhacks/import-type-cra) where you can see this approach working just fine in an unejected CRA project.

You can use `import type` to import a type from any `.ts` or `.tsx` file anywhere on your filesystem. Period. This will work even if your client and server are in different repositories in different parts of your filesystem.

> Of course, if multiple developers are working on the same codebase, you'll need to standardize the relative locations of client and server repos, so relative imports work on everyone's machines. You can either establish a convention ("make sure the `server` and `client` repos are in the same folder") or use something like Git Submodules.



**_"But wait, isn't importing stuff from the server into the client a security vulnerability?!"_**

Yes—when you're importing code. But not here! Remember, we're using the `import type` syntax. It was introduced in early 2020 ([TypeScript 3.8](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-3-8.html#type-only-imports-and-export)) and it comes with a very important guarantee:

![import type never leaves remnants at runtime](/import-type.png)

All `import type` declarations are removed **completely** when you compile your TypeScript code into a JavaScript bundle, no matter what. So _yes_ — you can safely use `import type` without worrying about accidentally importing sensitive data from your server.



### #4 Make an API call

Now that we've imported the type signature of `getUser`, how do we make strongly typed API call? Answer: a bit of TypeScript magic. (Don't worry, you don't need to understand how this works! Just know that it's possible!)

```tsx
// client/index.tsx
import type { getUser } from '../server/routes.ts';

type API = {
  getUser: getUser;
  // add additional routes here...
};

const query = <Endpoint extends keyof API>(
  endpoint: Endpoint,
  ...args: Parameters<API[Endpoint]>
): ReturnType<API[Endpoint]> => {
  return fetch('http://localhost:3000/api/rpc', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify({ endpoint, arguments: args }),
  }).then((response) => response.json()) as any;
};
```

We now have a **fully typesafe** `query` function, which you use like this:

```ts
const user = await query('getUser', 'user_abc73bd39ef');
// => Promise<{ id: "user_abc73bd39ef", name: "Bilbo" }>
```

It's a function called `query`. The first argument is the _name_ of the
endpoint; the second is the input of the function. (As implemented each
RPC endpoint can only accept a single input.)

TypeScript validates that the endpoint name is valid...

![invalid endpoint name error](/trpc-invalid-endpoint.png)

...typechecks the arguments...

![invalid arguments](/trpc-invalid-arg.png)

...and strongly types the result.

![typed payload](/trpc-typed-payload.png)

And if you update the definition of `getUser` on your server, the new type signature **automatically** propagates to your client at the speed of TypeScript, no code generation required. The developer experience is unreal.

By using a next-generation ORM to fetch data in your endpoints, you get most of the value of GraphQL — end-to-end typesafety, easy nested fetching, field selection, etc — in a TypeScript-native way.







If you want, you can use this approach exactly as described; all the code above is available as a Next.js project in [this repo](https://github.com/colinhacks/typesafe-api-no-graphql). But there's a better way!

<a name="trpc"></a>

## Introducing tRPC!

Around 6 months ago, I distilled these ideas into a proof-of-concept library called tRPC. Since then, the inimitable [@alexdotjs](https://twitter.com/alexdotjs) has turned it into a full-fledged library. Lets see how to implement our `getUser` example with tRPC. Spoiler alert: it's mindblowingly easy.

> If you want to jump straight to the README, check it out at [github.com/trpc/trpc](github.com/trpc/trpc) — and leave a star if you're feeling generous!

### Create a router

```ts
// server/index.ts
import * as trpc from '@trpc/server';

const appRouter = trpc.router();
export type AppRouter = typeof appRouter;
```

### Add a query endpoint

Let's implement `getUser`.

```ts
// server/index.ts
import * as trpc from '@trpc/server';

export const appRouter = trpc.router().query('getUser', {
  input: String,
  async resolve(req) {
    req.input; // string
    return { id: req.input, name: 'Bilbo' };
  },
});
```

A few things to note:

- You add "endpoints" by calling the `.query` method. The first argument is a _procedure name_. The second is an object containing two properties: `input` and `resolve`.

- The `input` is a function that **validates the input**. For simplicity, all tRPC procedures accept a _single argument_. The type signature of this argument is inferred from the return type of the `input` function.

  I recommend defining your inputs using a schema definition library (I hear [Zod](https://github.com/colinhacks/zod) is pretty cool 🙃). Far too few developers rigorously validate incoming API payloads; this is a huge security risk and source of bugs. So tRPC has validation baked-in from the start.

  For simplicity, I'm just using the built-in JavaScript `String` constructor, which accepts any value and converts it into a string.

- The endpoint logic is defined in the `resolve` function. A "pseudorequest" is passed into it, which contains `input` (automatically typed as `string` in this case) and `ctx` (more on that later). tRPC **infers the return type** of your resolver. So make sure the returned value is statically typed! We recommend using a typesafe ORM like Prisma or TypeORM that automatically type the results of database operations; otherwise you'll need to write TypeScript types manually.

### What about mutations?

Like GraphQL, tRPC makes a distinction between queries and mutations. Add mutations to your router with the `mutation` method:

```ts
// server/index.ts
import * as trpc from '@trpc/server';
import { z } from 'zod';
export const appRouter = trpc
  .router()
  .query('getUser', {
    input: String,
    async resolve(req) {
      req.input; // string
      return { id: req.input, name: 'Bilbo' };
    },
  })
  .mutation('createUser', {
    input: z.object({ name: z.string() }),
    async resolve(req) {
      return await prisma.user.create({
        data: req.input,
      });
    },
  });
```

The syntax for the `mutation` is **identical** to `query`. tRPC only makes a distinction because `query` payloads are cached differently than `mutation` payloads.

### Chainable methods

Important note: when you add multiple endpoints to a router, you **must chain the calls** to `query()` and `mutation()`! This lets tRPC infer the type signature of your whole API. If you're familiar with Express you may be tempted to do this:

```ts
export const appRouter = trpc.router();

// ❌ DON'T DO THIS!! ❌

appRouter.query('getUser', {
  input: String,
  async resolve(req) {
    req.input; // string
    return { id: req.input, name: 'Bilbo' };
  },
});
```

This doesn't work at all. All router methods are `immutable`—they return an entirely new instance of `TRPCRouter`, instead of _modifying_ the current instance. So you _must chain_ these methods.

### Merging routers

At some point, you'll want to split up your tRPC router into multiple files. If all methods must be chained together, how is this possible?

tRPC provides an easy way to **merge routers**. I recommend creating a root `appRouter` and several "child routers" that you merge into it.

```ts
import { userRouter } from './routers/user';
import { postRouter } from './routers/post';

export const appRouter = trpc
  .createRouter()
  .merge('users.', userRouter) // prefix userRouter endpoints with "users."
  .merge('posts.', postRouter); // prefix postRouter endpoints with "posts."
```

All procedure names will be prefixed with the string literal you pass as the first argument of `.merge`. So a `createUser` mutation on `userRouter` will be re-named to `users.createUser`. This is made possible by the incredible _string literal templates_ feature introduced in [TypeScript 4.1](https://devblogs.microsoft.com/typescript/announcing-typescript-4-1/#template-literal-types)!

### Handing incoming requests

Now lets use our router to handle some actual HTTP requests. tRPC ships "adapters" that easily let you generate Express middleware or Next.js API routes from your tRPC router.

#### Next.js integration

```ts
// pages/api/trpc/[trpc].ts
import { createNextApiHandler } from '@trpc/server/adapters/next';

const appRouter = /* ... */;

export default createNextApiHandler({
  router: appRouter,
  createContext: (opts) => {
    return { bearer: opts.req.headers.authorization };
  },
});
```

#### Express integration

If you're using Express, do this instead:

```ts
import { createExpressMiddleware } from '@trpc/server/adapters/express';

const appRouter = /* ... */;

const app = express();

app.use(
  '/trpc',
  createExpressMiddleware({
    router: AppRouter,
    createContext: () => null, // no context
  })
);

app.listen(80);
```

#### Koa, Hapi, Nest.js

Coming soon (and PRs welcome)!

### Request context

All server adapters accept `createContext` function, where you can generate a `ctx` object from the "raw" HTTP request and response objects (available as `opts.req` and `opts.res`). This `ctx` is available in all your resolvers.

### Client-side usage

So what about the client side? The `@trpc/client` module generates a typesafe "tRPC client"—analagous to the `query` method we implemented in our toy example. You can import the the type signature of your **entire API** at once with `import type { AppRouter } from '...'`. Then pass it into `createTRPCClient`, along with the URL of your API:

```ts
// pages/index.ts
import { createTRPCClient } from '@trpc/client';
import type { AppRouter } from './api/[trpc]';

const trpcClient = createTRPCClient<AppRouter>({
  url: 'http://localhost:3000/api',
});

trpcClient.query('getUser', 'user_1238823498');
// => Promise<{ id: string; name: string }>
```

#### React integration

If you use React, you can use the `@trpc/react` package to convert your tRPC client into React hooks powered by [React Query](https://react-query.tanstack.com/). This gets you a ton of cool features like request invalidation, SSR, and cursor-based pagination! 🚀

```ts
import { createReactQueryHooks } from '@trpc/react';

// path to your backend server where your AppRouter is defined
import type { AppRouter } from '../server/trpc';

export const trpc = createReactQueryHooks<AppRouter>();
```

#### Vue and Svelte

Get in touch if you'd like to build dedicated bindings for those frameworks! Create an issue on [the GitHub repo](https://github.com/trpc/trpc).

## Wrapping up

I've been using various iterations of tRPC in production for nearly a year, and it truly feels like living in the future. I'm immensely proud of what tRPC has become. Checkout the docs (and leave a star! 🤩) on [the GitHub repo](https://github.com/trpc/trpc).

There are a lot of other features I could highlight, but that'll do for now. Huge shoutout to [@alexdotjs](https://twitter.com/alexdotjs), who picked up a fledgling proof-of-concept four months ago and carried it to the finish line!

Building an end-to-end typesafe API — without GraphQL

Zod's v2 has been in beta since September 2020 (around 3 months as of this writing). The people are starting to talk.



## The background

The reason Zod 2 has been in beta for so long (well, one of the reasons anyway!) is that I’ve been increasingly unsatisfied with Zod’s implementation of transformers. Transformers are a mechanism within Zod to convert data from one type to another. For the sake of rigor, I decided it was paramount for transformers to encapsulate both pre- and post-transform validation; to create one, you gave it an input schema `A`, and output schema `B`, and a transformation function `(arg: A)=>B`.

Multiple issues have cropped up that have led me to re-evaluate the current approach. At best, transformers are a huge footgun and at worst they’re fundamentally flawed.





## Context

In version 1, a Zod schema simply let you define a data type you'd like to validate. Under the hood, it magically inferred the static TypeScript type for your schema. More technically, there is a base class (`ZodType`) that tracked the inferred type in a generic parameter called `Type`.

![Screen Shot 2020-12-08 at 5 09 36 PM](https://user-images.githubusercontent.com/3084745/101593584-03bc6f00-39a5-11eb-8a66-5e7e84103290.png)

Transformers break this assumption. The accept an `Input` of one type and return an `Output` of a different type. To account for this, every Zod schema now had to track both an input and output type. For non-transformers, `Input` and `Output` are the same. For transformers only, these types are different.

Let's look at `stringToNumber` as an example:

```ts
const stringToNumber = z.transformer(z.string(), z.number(), (val) =>
  parseFloat(val)
);
```

For `stringToNumber`, Input is `string` and Output is `number`. Makes sense.

What happens when you pass a value into `stringToNumber.parse`?

1. The user passes a value into `stringToNumber.parse`
2. The transformer passes this value through the `parse` function of its _input schema_ (z.string()). If there are parsing errors, it throws a ZodError
3. The transformer takes the output from that and passes it into the transformation function (`val => parseFloat(val)`)
4. The transformer takes the output of the transformation and validates it against the output schema (z.number())
5. The result of that call is returned

Here's the takeaway: for a generic transformer `z.transformer(A, B, func)`, where A and B are Zod schemas, the argument of `func` should be the `Output` type of A and the return type is the `Input` type of B. This lets you do things like this:

```ts
const stringToNumber = z.transformer(z.string(), z.number(), (val) =>
  parseFloat(val)
);
const numberToBoolean = z.transformer(
  z.number(),
  z.boolean(),
  (val) => val > 25
);
const stringToNumberToBoolean = z.transformer(
  stringToNumber,
  numberToBoolean,
  (val) => 5 * val
);
```

## The problems with transformers

After implementing transformers, I realized transformers could be used to implement another much-requested feature: default values. Consider this:

```ts
const stringWithDefault = z.transformer(
  z.string().optional(),
  z.string(),
  (val) => val || 'trout'
);
stringWithDefault.parse('marlin'); // => "marlin"
stringWithDefault.parse(undefined); // => "trout"
```

Voila, a schema that accepts `string | undefined` and returns `string`, substituting the default "trout" if the input is ever undefined.

So I implemented the `.default(val:T)` method in the ZodType base class as below (partially simplified)

```ts
default(def: Input) {
  return ZodTransformer.create(this.optional(), this, (x: any) => {
    return x === undefined ? def : x;
  });
}
```

Do you see the problem with that? I didn’t. Neither did anyone who read through the [Transformers RFC](https://github.com/colinhacks/zod/issues/100) which I left open for comment for a couple months before starting on implementation.

This implementation quickly gets wonky when you try to pass _an existing transformer_ in as the input or output. In other words, transformers aren't composable. Let's see what happens if we try to set a default value on `stringToNumber`.

```ts
stringToNumber.default('3.14');

// EQUIVALENT TO
const defaultedStringToNumber = z.transformer(
  stringToNumber.optional(),
  stringToNumber,
  (val) => (val !== undefined ? val : '3.14')
);

defaultedStringToNumber.parse('5');
/* { ZodError: [
  {
    "code": "invalid_type",
    "expected": "string",
    "received": "number",
    "path": [],
    "message": "Expected string, received number"
  }
] */
```

Let’s walk through why this fails. The input ("5") is first passed into the transformer input (`stringToNumber.optional()`). This converts the string `"5"` to the number `5`. This is then passed into the transformation function. But wait: `val` is now `number | undefined`, but the transformer function needs to return a `string`. Otherwise, if we pass `5` into `stringToNumber.parse` it’ll throw. So we need to convert `5` back to `"5"`. That may seem easy in this toy example but it’s not possible in the general case. Zod can’t know how to magically undo the transformation function.

In practice, the current definition of `default` in ZodType shouldn’t have even been possible. The only reason the type checker didn’t catch this bug is because there are a regrettable number of `any`s floating around in Zod. It’s not a simple matter to switch them all to `unknown`s either; I’ve had to use `any` in several instance to get type inference and certain generic methods to work properly. I’ve tried multiple times to reduce the number of `any`s but I’ve never managed to crack it.

It’s possible this is a one-off issue. I could find some other way to implement `.default()` that doesn’t involve transformers. Unfortunately this isn’t even the only problem in Zod’s implementation.

## The `.transform` method

Initially the only way to define transformers was with `z.transformer(A, B, func)`. Eventually I implemented a convenience method you can use like this:

```ts
z.string().transform(z.number(), (val) => parseFloat(val));

// equivalent to
z.transformer(z.string(), z.number(), (val) => parseFloat(val));
```

Some users were executing multiple transforms in sequence without changing the actual data type:

```ts
z.string()
  .transform(z.string(), (val) => val.toLowerCase())
  .transform(z.string(), (val) => val.trim())
  .transform(z.string(), (val) => val.replace(' ', '_'));
```

To reduce the boilerplate here, it was recommended that I overload the method definition to support this syntax:

```ts
z.string()
  .transform((val) => val.toLowerCase())
  .transform((val) => val.trim())
  .transform((val) => val.replace(' ', '_'));
```

If the first argument is a function instead of a Zod schema, Zod should assume that the transformation doesn’t transform the type. In other words, `z.string().transform((val) => val.trim())` should be equivalent to `z.string().transform(z.string(), (val) => val.trim())`. Makes sense.

Consider using this method on a transformer:

```ts
stringToNumber.transform(/* transformation_func */);
```

What type signature do you expect for `transformation_func`?

Most would probably expect `(arg: number)=>number`. Some would expect `(arg: string)=>string`. Neither of those are right; it’s `(arg: number)=>string`. Yes really. The transformation function expects an input of `number` (the Output of `stringToNumber`) and a return type of `number` (the Input of `stringToNumber`). This type signature is a natural consequence of a series of logical design decisions, but the end result is dumb. Well done, self.

Intuitively, you should be able to append `.transform(val => val)` to any schema. Surely passing in the identity function to a method called `.transform()` should be a no-op? Unfortunately due to how transformers are implemented, that's not always possible.

### More complicated examples

These strange issues composability issues also make it difficult to write any generic functions on top of Zod (of which `.transform` and `.default` are two examples). Others have encountered similar issues, like [here](https://github.com/colinhacks/zod/issues/199) and [here](https://github.com/colinhacks/zod/issues/213). For the sake of simplicity, I won't break down the nuances of those cases; the underlying issue is the composability problem I described above.

## When should defaults even be applied?

Strangely, there doesn't appear to be a consensus expectation on whether default values should be "applied" pre- or post-transformations. Consider the simple `stringToNumber` example; should `stringToNumber.default(/* value */)` accept a number or a string?

As implemented in Zod 2 it should accept `string` (the `Input` of the output schema of `stringToNumber`). In this case, the schema looks at the value passed into parse, checks if it's `undefined`, and, if so, hot-swaps in the default value. That value is then passed through all the transformers and refinements downstream (in this case, being converted to a `number`) by the non-optional `stringToNumber`. Remember, as implemented, this is how `stringToNumber.default()` returns:

```ts
stringToNumber.default(val);

// equivalent to

z.transformer(stringToNumber.optional(), stringToNumber, (val) =>
  val !== undefined ? val : '3.14'
);
```

But some users expect `stringToNumber.default` to accept a number; they assume that a default value "short-circuits" the parsing logic. If you pass `undefined` into `.parse()`, the parsing should "return early" and return the value. In this case, taht value should be a `number` because that's the ultimate expected return type of the transformer returned by `stringToNumber.default(val)`.

## A path forward

> Note from March 2021: the approach described below also had issues! Ultimately the implementation used in Zod 3 was slightly different.

I just created [an RFC](https://github.com/colinhacks/zod/issues/264) describing a plan to improve this state of affairs. The idea was to have each Zod schema contain a list of post-parse transformation functions. When a value is passed into `.parse`, Zod will type check the value, then pass it through the transform chain. This is the approach Yup uses.

```ts
const schema = z
  .string()
  .transform((val) => val.length)
  .transform((val) => val > 100);

type In = z.input<typeof schema>; // string
type Out = z.input<typeof schema>; // boolean
```

Unlike before, Zod doesn’t validate the data type between each transform. We’re relying on the TypeScript engine to infer the correct type based on the function definitions. In this sense, Zod is behaving just like I intended; it’s acting as a reliable source of type safety that lets you confidently implement the rest of your application logic — including transforms. Re-validating the type between each transform is overkill; TypeScript’s type checker already does that.

Each schema will still have an Input (the inferred type of the schema) and an Output (the output type of the last transform in the chain). But because we’re avoiding the weird hierarchy of ZodTransformers everything behaves in a much more intuitive way.

One interesting ramification is that you could interleave transforms and refinements. Zod could keep track of the order each mod/refinement was added and execute them all in sequence:

```ts
const schema = z
  .string()
  .transform((val) => parseFloat(val))
  .refine((val) => val > 25, { message: 'Too short' })
  .transform((val) => `${val}`)
  .refine((val) => /^\d+$/.test(val), { message: 'No floats allowed' });
```

We'll have to wait and see how this turns out!

**Update from March 2021**

This is the general approach being used in the new Zod 3 alpha release. There are some small implementational details — refinements and transforms are now "housed" within the ZodEffects wrapper class instead of being attached directly to the schema.

A note on default values: Zod no longer uses transforms to implement default values. Transforms are only capable of implementing _post-parse defaults_: parse the input, then swap in the default value if the result is `undefined`. But I've decided that _pre-parse defaults_ make more sense (and are more expected by users). That is — if the input is `undefined` swap in the default value, and _then_ finish the parsing process (which includes refining and transforming). As such, the defaulting logic now lives in the ZodOptional class.