# Rejot Documentation ## Getting Started # What is ReJot ReJot is a database-to-database synchronization engine designed for teams operating in distributed architectures. It lets you decouple service communication by allowing data to flow asynchronously between services, using your database's changelog/WAL as the queue. ReJot's asynchronous model improves reliability and reduces latency compared to synchronous inter-service communication. With data contracts expressed in code, developers retain full control over what data is shared and how it is shared. Define what to publish and consume using your database's query language, just like defining tables or writing migrations. ReJot fits naturally into your development workflow. ## How does it work? - Data dependencies in ReJot are modeled through [Data Contracts](/docs/concepts/data-contracts) known as **[Public Schemas](/docs/spec/public-schema)** and **[Consumer Schemas](/docs/spec/consumer-schema)**. These contracts are defined by developers in code. - Public Schemas define how internal datasets are exposed, allowing teams to independently evolve their internal data models while maintaining the contract with consumers. - Consumers at the same time can ingest these data sources by setting up a Consumer Schema. - ReJot connects to the publishing data store's write-ahead-log and pushes changes to Public schemas to the data stores on the consuming side. - An intermediate event store is used as durable storage to store these updates. ## How to get started If you're curious to learn how ReJot works, you cab quickly try it out in your own codebase by walking through our [Quickstart](/docs/start/quickstart) guide. If you would rather see a concrete example of how ReJot integrates into a repository check out our [Example Repositories](/docs/start/examples). # Install import LanguageVariants from "@/components/LanguageVariants.astro"; import Notice from "@/components/Notice.astro"; ## ReJot CLI The ReJot CLI is used to create and validate manifest files, as well as starting sync services. The `rejot-cli` is published to [npmjs.com](https://www.npmjs.com/package/@rejot-dev/cli). You can use your package manager of choice, including **Bun**, to install the CLI. After installation, the CLI should be available on your path. ```bash rejot-cli --help ``` ### Container Image The CLI tool is also available as a [container image](https://github.com/rejot-dev/rejot/pkgs/container/rejot-cli). ```bash docker run --rm ghcr.io/rejot-dev/rejot-cli:main --help ``` ### Build From Source We use Bun as our primary runtime. ```bash git clone git@github.com:rejot-dev/rejot.git cd ./rejot bun install bun run rejot-cli --help ``` ## Model Context Protocol (MCP) The ReJot Model Context Protocol (MCP) server gives LLMs the ability to manipulate, inspect and use ReJot's tooling. The MCP can be installed through your package manager of choice. After installation, the MCP server will be available as `rejot-mcp`. You can configure your editor to use the MCP server by adding the following configuration to your editor's configuration file. ```json { "mcpServers": { "rejot-mcp": { "command": "rejot-mcp", "args": ["--project", "."] } } } ``` Not all editors automatically substitute relative paths, such as "." for the project root. You might have to manually specify the absolute project root. This should work in Cursor. For more information, see the documentation for your editor of choice: - [Cursor](https://docs.cursor.com/context/model-context-protocol#configuring-mcp-servers) - [Windsurf](https://docs.windsurf.com/windsurf/cascade/mcp) - [VS Code + Github Copilot](https://code.visualstudio.com/docs/copilot/chat/mcp-servers#_add-an-mcp-server) ## Manifest JSON Schema To make editing manifest files easier, you can configure JSON Schema in your editor. The schema is included in the `@rejot-dev/contract` npm package. You can find it in either of these locations: - Local path: `node_modules/@rejot-dev/contract/schema.json` - URL: https://unpkg.com/@rejot-dev/contract@latest/schema.json ### VSCode / Cursor Add the following configuration to your `.vscode/settings.json` workspace file: ```json "json.schemas": [ { "fileMatch": ["rejot-manifest.json"], "url": "https://unpkg.com/@rejot-dev/contract@latest/schema.json" } ] ``` ### IntelliJ For IntelliJ-based IDEs, follow the [IntelliJ documentation][custom-schema-intellij] to add custom JSON Schemas to your project. [custom-schema-intellij]: https://www.jetbrains.com/help/idea/json.html#ws_json_schema_add_custom # Examples We have an example repository that demonstrates how ReJot integrates in a Typescript monorepo. For deployment details see the README. - ShopJot Example Application: https://github.com/rejot-dev/example-microservice-ts # Quickstart import { Code } from "astro:components"; import Notice from "@/components/Notice.astro"; import rawSchemaText from "./schema.ts?raw"; import PreWithCopy from "@/components/PreWithCopy.astro"; This guide helps you quickly set up replication between PostgreSQL database instances using ReJot. For demonstration purposes we'll use one Postgres database for both publishing and consuming data and configure it using a single manifest file and with no external event store. ## Prerequisites - [Install the ReJot CLI](/docs/start/install) - [A Postgres database with logical replication enabled](/docs/guides/setup-postgres) ## Create a ReJot Manifest First, we'll have to initialize a [Manifest](/docs/spec/manifest) file. This file contains all configuration that ReJot needs in order to sync between datastores. By default this will create a `rejot-manifest.json` file in the working directory. ```bash rejot-cli manifest init --slug "my-sync-project" ``` ### Add Database Connections Configure connections to your source and destination databases. In this example, we'll use one database as both source and destination, so we will need only one connection. ```bash rejot-cli manifest connection add \ --slug "my-db" \ --type postgres \ --database postgres \ --host localhost \ --password example \ --port 5432 \ --user postgres ``` ### Set Up Data Store Define a data store and attach it to the connection we just created. ```bash # Source data store rejot-cli manifest datastore add \ --connection my-db \ --publication rejot_publication \ --slot rejot_slot ``` Quickly inspect your current manifest config using ```rejot-cli manifest info ``` ### Set Up Event store Next up we'll define an event store, an event store is used by ReJot to store Public Schema events before replicating them to consumers. For demonstration purposes we'll use the same Postgres database connection we created before but generally a separate dedicated database is better suited. The appropriate schemas and tables will be created on first launch. ```bash rejot-cli manifest eventstore add \ --connection my-db ``` ReJot can function without an event store, but this means changes to a Public Schema are not stored durably and will be lost if the sync service is stopped or crashes. For local development this is usually fine. ### Define Public and Consumer Schemas Our public and consumer schemas define what data is exposed from the datastore and how that exposed data should be synced to the destination. For the publishing side, this is done through a simple select query that will be run each time the source table is updated. The select query is responsible for transforming the internal schema to the public schema and must include a `WHERE` clause using the primary key for that table. Note that in the example query, the `:id` placeholder will be the primary key value for incoming updates. Public and Consumer Schemas can be created through Typescript code, create a `schemas.ts` file containing your schemas. See our full [guide on defining schemas](/docs/guides/defining-schemas) for more detailed description of the schema definition process. ### Collect Schemas to Manifest Before your newly defined schemas can be used, they must me collected into your manifest file. You can do this using the `collect` command in the CLI. It might be nice to check your schemas are correct first using the `--check` flag. ```bash rejot-cli collect schemas.ts --check --print ``` If all looks good, you can materialize the schemas into your manifest. ```bash rejot-cli collect schemas.ts --write ``` ## Create Target Table in Destination Before launching the synchronization process, create the target table in your destination database: ```sql CREATE TABLE target_table (id SERIAL PRIMARY KEY, api_key TEXT NOT NULL); ``` ## Start Synchronization Start the synchronization process: ```bash rejot-cli manifest sync ./rejot-manifest.json ``` Any new writes to your source tables in the source datastore should now be transformed and written into the destination datastore! ## Guides # Managing Manifests import Notice from "@/components/Notice.astro"; This guide covers how to create and manage ReJot manifest files using the `rejot-cli manifest` commands. ReJot uses [Manifests](/docs/spec/manifest) as a central configuration file that defines your data synchronization setup. Think of it as a "lockfile" for your data contracts - it specifies what data should be synced, where it should be synced, and in what format. ## Manifest Components A manifest definition contains (a subset of) the following components: - [Data stores](/docs/spec/data-store): Source databases that ReJot monitors for changes - [Event stores](/docs/spec/event-store): Where ReJot stores and manages change events - [Public Schemas](/docs/spec/public-schema): Define what data is exposed from your data stores - [Consumer Schemas](/docs/spec/consumer-schema): Define how data is transformed and stored in target systems - Workspaces: Group multiple manifests into a single project. For AI-native code editors, the [ReJot MCP](/docs/start/install#mcp) server makes managing manifest files effortless. ## Initializing a Manifest To create a new empty manifest file in the current directory, use the following command. Each manifest requires a unique slug, which is used to identify where public schemas are exposed and where sync services can locate them on the network. ```bash rejot-cli manifest init --slug "@my-org/my-sync-slug" ``` ## Setting Up Data and Event Stores Before configuring public and consumer schemas, you need to specify how ReJot should connect to your databases and where the ReJot Event Store is located. For each data store, ReJot will monitor changes to public schemas defined for that data store and push those changes to the event store. While you can configure the event store to use the same database, it's generally recommended to use a separate database. Database connection credentials can be stored in the ReJot Manifest file, but these should only be committed for development setups. See our [Composing Manifests](/docs/guides/composing-manifests) guide for examples on how to split your manifests up for different environments. ### Adding Database Connections Let's start by adding the connection details: ```bash rejot-cli manifest connection add \ --slug "my-datastore" \ --type postgres \ --database postgres \ --host localhost \ --password example \ --port 5432 \ --user postgres ``` Alternatively, you can provide a connection string: ```bash rejot-cli manifest connection add \ --slug "my-datastore" \ --connection-string "postgres://postgres:postgres@example:5432/postgres" ``` ### Configuring Data Stores Next, link a data store to the connection: ```bash rejot-cli manifest datastore add \ --connection my-datastore \ --publication rejot_publication \ --slot rejot_slot ``` There's an (implicit) difference between data stores with or without a publication and slot. Data stores without these can only be used as a destination, while data stores with these can be used as either a source or destination. A data store can be removed from the manifest in the following way: ```bash rejot-cli manifest datastore remove my-datastore ``` ### Configuring Event Stores Event stores are also based on a connection, but they don't require a publication or slot. ```bash rejot-cli manifest eventstore add --connection my-eventstore ``` Similar to data stores, an event store can be removed from the manifest in the following way: ```bash rejot-cli manifest eventstore remove my-eventstore ``` ## Verifying Your Configuration You can use the `info` command to display a parsed version of the manifest. When errors or warnings are detected, they will be printed to the console. ```bash $ rejot-cli manifest info ``` ``` Manifest Configuration: Connections: - my-datastore (postgres) Host: localhost:5432 Database: postgres User: postgres string: postgres://postgres@localhost:5432/postgres Data Stores (Replication Sources): - Connection: my-datastore Publication / slot: rejot_publication / rejot_slot Event Stores (Replication Targets): - Connection: my-eventstore ``` ## Workspaces A ReJot workspace is a collection of manifests that are managed together. It's useful for local development use cases where you're typically only using a single event store and only want to run a single sync service. A manifest can link to other manifests using the `workspaces` field. This field should contain the relative path to other manifest files. ```json { "slug": "@my-org/my-sync-slug", "workspaces": ["./service-one/rejot-manifest.json", "./service-two/rejot-manifest.json"] } ``` The workspace manifest is typically in a parent directory related to other manifests. An example structure might look like this: ``` my-monorepo/ ├── service-one/ │ ├── rejot-manifest.json │ └── ... ├── service-two/ │ ├── rejot-manifest.json │ └── ... └── rejot-manifest.json ``` When using the `collect` command, schemas are collected into the nearest manifest. ## Next Steps Now that you know how to manage your manifest file, you can [create public and consumer schemas](/docs/guides/defining-schemas) # Defining Schemas import { Code } from "astro:components"; import PreWithCopy from "@/components/PreWithCopy.astro"; import Notice from "@/components/Notice.astro"; import rawSchemaText from "./schema.ts?raw"; import rawFullSchemaText from "./finishedSchema.ts?raw"; import PartialCode from "@/components/PartialCode.astro"; import LanguageVariants from "@/components/LanguageVariants.astro"; When using ReJot, you define Public and Consumer schemas right from your codebase. This allows you to group related pieces of code together. In this case you can define your public schemas close to where you define the underlying database tables. This guide will show you how to define (Postgres) schemas in your TypeScript codebase. If you want to use ReJot with a non-TypeScript codebase, you can modify your manifest directly to create schemas, or you can choose to define your schemas in TypeScript anyway. ## Setting Up First, add the required dependencies to your project: ## Creating Public and Consumer Schema Definitions Create a new file called `schemas.ts` that will contain your public and consumer schemas: Make sure you adjust the data store and manifest slugs in this example to the manifest you've created. ### Transformations Transformations are a crucial part of schema definitions that specify how data is transformed between different stages of the sync process. Transformation are used by both the publishing and consuming parties, for publishers, the transformation defines how data from the internal representation should be mapped to the public schema. Conversely on the consuming side the transformation maps the public schema back into an internal representation for the consumer. #### Public Schema Transformations For public schemas, you need to create a SQL transformation for each table referenced in the schema. These transformations: - Take the modified row as input parameters. They can be accessed in column order using positional parameters (e.g., `$1`, `$2`, etc). We also provide support for named parameters, using the name of the column. E.g. `:id`. - MUST return exactly one row that matches the public schema's output structure. Here's an example of a public schema transformation: The row returned from the transformation query must exactly match the public schema's output structure, including casing in column names. In Postgres, when not double-quoting column names, they are always returned in lowercase. It's recommended to always add an explicit "AS" statement, to ensure the casing is correct. E.g. `SELECT id, first_name AS "firstName" FROM my_table`. #### Multi-table transformations A common case is to (partially) de-normalize your data for the public schema, to ensure correct results ReJot should listen to updates to all of these tables. As a data producer you will have to define schema transformations for each table. Note that your public schema transformation must always return one result, in case there is a one-to-many relationship between tables you cannot include them in your ReJot public schema without doing some kind aggregation in your query. For PostgreSQL, use one of the [aggregation functions](https://www.postgresql.org/docs/current/functions-aggregate.html) that are available. #### Consumer Schema Transformations For consumer schemas, transformations handle the insertion of public schema data into the destination data store. These transformations: - Use named parameters (e.g., `:name`, `:country`) to reference fields from the public schema - Must handle conflicts appropriately since ReJot doesn't guarantee exactly-once delivery Here's an example of a consumer schema transformation: ## Materializing Schemas Code-based schemas need to be materialized into your manifest file to be usable by sync services. This is done using the `collect` command: ```bash rejot-cli collect --write --check schemas.ts ``` This command will: - Read your schema definitions from `schemas.ts`. - Generate the corresponding manifest entries. - Check that all consumer schemas are referencing valid public schemas (if `--check` is specified). - Update your manifest file (if `--write` is specified). The manifest includes a `definitionFile` key for each schema, which points to the actual source file and is re-used later on to update collected schemas. If you want to change a schema, edit the file referenced by `definitionFile` and run the collect command again. ## Next Steps Now that you know how to define schemas, you can [run sync services](/docs/guides/running-sync-services) to start and manage sync services using manifests. # Running Sync Services import DarkImage from "@/components/DarkImage.astro"; A sync service is responsible for monitoring changes in data stores, applying necessary transformations, and propagating these changes to consumers as mutation events. When starting a sync service, you can supply any number of manifests. The service will be responsible for all Public schemas defined in those manifests. Consumer schemas that reference Public schemas in other manifests can be retrieved over the network from other sync services. ## Prerequisites - You need to have your manifest(s) ready. For more information, see the [Managing Manifests](/docs/guides/managing-manifests) and [Defining Schemas](/docs/guides/defining-schemas) guides. ## Single Sync Service If you're deploying a single sync service, you can start the service from the `rejot-cli` and pass all your manifests: ```bash rejot-cli manifest sync rejot-manifest.json another-rejot-manifest.json ``` Upon startup, the sync service will verify connections to all data stores and open replication slots to begin monitoring for changes. ## Multiple Sync Services It can be beneficial to set up separate sync services in cases where you expect high load or you prefer to split service by domain. If you have data dependencies between sync services, they can exchange data over the network. For this purpose, each sync service exposes a web server that other sync services can communicate with. You can configure the endpoint that a sync service exposes through the `rejot-cli` arguments. To enable external sync service discovery, you must specify a [Resolver](/docs/reference/resolvers) that helps the sync service locate manifest slugs on the network. The simplest resolver is the environment resolver, where you supply a mapping of manifest to hostname through environment variables: ```bash EXPORT REJOT_SYNC_SERVICE_slug-a=sync-service-a:3000 rejot-cli manifest sync rejot-manifest.json \ --hostname 0.0.0.0 \ --port 3000 \ --resolver env ``` See [Composing Manifests](/docs/guides/composing-manifests) for guidance on how to split up your manifests. # Set Up Postgres Data Store import Notice from "@/components/Notice.astro"; This guide explains how to prepare your PostgreSQL database as a Data Source or Event Store for use with ReJot. ## Enabling Logical Replication The only step absolutely required for ReJot to work is to enable logical replication. Creating custom roles and permissions is recommended for production use cases, but not necessary for development. ReJot will automatically create a replication slot, publication, schemas, and tables when the sync process is started. Logical replication is not enabled by default in PostgreSQL. The configuration that manages logical replication is called [`wal_level`][pg-wal-level], which must be set to `logical`. Note that changing this config requires a database restart. There are two methods to enable this config: either through the `postgresql.conf` file, ```sql wal_level = 'logical' ``` or by setting it through SQL directly: ```sql SHOW wal_level; -- Inspect current setting ALTER SYSTEM SET wal_level = 'logical'; ``` Hosted PostgreSQL providers like Supabase, Neon, or Google Cloud SQL manage this setting via their own dashboards. When given the option to choose from a pooled connection or a dedicated connection, choose the dedicated connection. ## Postgres as a Source Data Store Using a Postgres data store as a Source Data Store means that ReJot will listen to the logical replication stream of that store and apply Public Schema transformations to its messages. ReJot leverages PostgreSQL's logical replication stream using the `pgoutput` native plugin to capture database changes. ### Replication Slots A PostgreSQL replication slot is a mechanism that keeps track of changes replicated from a publication for each subscriber, ensuring no changes are missed even if a subscriber disconnects temporarily. It maintains a record of which changes have been consumed by subscribers and prevents the deletion of Write-Ahead Log (WAL) segments until all subscribers have processed them. In most cases it is **not** necessary to create a replication slot manually. ReJot will create a slot if it doesn't exist yet based on the data store config in your manifest. ### Creating a PostgreSQL Publication A PostgreSQL [Publication][pg-create-publication] is defined on a primary Postgres node and specifies which set of tables in a database can be replicated to other systems. On first starting a Sync service, a publication is automatically created if it doesn't exist yet. This publication includes all tables and is equivalent to: ```sql CREATE PUBLICATION rejot_publication FOR ALL TABLES; ``` If you prefer to control which tables are included in the publication, you can create the publication manually: {/* prettier-ignore */} ```sql CREATE PUBLICATION rejot_publication FOR TABLE some_table, another_table; ``` See the [PostgreSQL docs][pg-create-publication] for more details. ## Postgres as an Event Store To use PostgreSQL as an event store, the user used to connect to the database must be able to create a schema called `rejot_events`. The `rejot-cli` will create these schemas for you on launch. If you don't want to give the `rejot_role` access to create schemas, you'll have to manually create these schemas and assign the following roles to the `rejot_role`. ## Hardening the ReJot Database Role We recommend creating a separate role for ReJot and granting access to that user for production use cases. If you want to get started quickly, you can use an admin user instead. ReJot will create the necessary schemas, databases, and tables automatically. ```sql CREATE ROLE rejot_role WITH REPLICATION LOGIN PASSWORD ''; ``` ### Source Data Store Permissions To be able to read data from a source data store, ReJot must be able to create a replication slot, publication, and must be able to issue `SELECT` statements on the tables being replicated. To apply Public Schema transformations, the ReJot Postgres user must have the `SELECT` permission on the relevant schemas (`public` by default). {/* prettier-ignore */} ```sql GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO rejot_role; ``` ### Destination Data Store Permissions For ReJot's replication to function, we must be able to keep track of the last seen transaction ID and the state of Public Schema transformations. This is done in the `rejot_data_store` schema. When starting the sync service, ReJot will automatically run migrations to create the necessary schema and tables. To do this, write permissions on the `rejot_data_store` schema are required. ### Event Store Permissions Similar to the destination data store, ReJot will automatically run migrations to create the necessary schema and tables needed for the event store. This is the `rejot_events` schema. ### Recommendations For maximum security we recommend creating a separate role for ReJot and granting specific permissions to that role. The replication slot, publication and schemas should be created manually by a system administrator. Then the necessary permissions for the use case should be granted to the ReJot role. ### Reference {/* prettier-ignore */} ```sql -- Create the schemas using a privileged user CREATE SCHEMA IF NOT EXISTS rejot_events; CREATE SCHEMA IF NOT EXISTS rejot_data_store; -- Grant access to these new schemas to your unprivileged user GRANT CREATE, USAGE ON SCHEMA rejot_events TO rejot_role; GRANT CREATE, USAGE ON SCHEMA rejot_data_store TO rejot_role; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA rejot_data_store TO rejot_role; GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA rejot_data_store TO rejot_role; GRANT SELECT, INSERT, UPDATE ON ALL TABLES IN SCHEMA rejot_events TO rejot_role; ``` [pg-wal-level]: https://www.postgresql.org/docs/current/runtime-config-wal.html#GUC-WAL-LEVEL [pg-publication]: https://www.postgresql.org/docs/current/logical-replication-publication.html [pg-create-publication]: https://www.postgresql.org/docs/current/sql-createpublication.html # Composing Manifests import Notice from "@/components/Notice.astro"; When deploying sync services, you'll often need different configurations for various environments (development, staging, production) or manifests that have different owners within the organization. This guide explains strategies for structuring and managing your manifest file collections. - **Database Connections**: Development, staging, and production typically use separate databases with different credentials or hostnames - **Event Store Configurations**: Your event store might differ between development and production - **Secret Management**: Production secrets are typically stored in secure vaults - **Mono-repo vs Multi-repo**: Definitions might exist in completely separate code repositories - **Ownership**: Different stakeholders in the organization might want to control specific schemas ## Environment-Specific Manifest Files The most common reason to split manifest files is to separate environment-specific configuration and secrets into distinct files. ``` /manifests ├── rejot-manifest.base.json ├── rejot-manifest.dev.json ├── rejot-manifest.staging.json └── rejot-manifest.prod.json ``` You'll typically need different connection configurations for each environment, which means your base manifest won't specify any connections: ```json // rejot-manifest.base.json { "connections": [], "dataStores": ..., "publicSchemas": ..., } // rejot-manifest.dev.json { "connections": [{ "slug": "conn-datastore-a", "config": { "host": "localhost", ... } }] } // rejot-manifest.prod.json { "connections": [{ "slug": "conn-datastore-a", "config": { "host": "my-datastore-a.europe-west4.domain.com", ... } }] } ``` When starting a sync service, you can pass multiple manifest files: ```bash # Development rejot-cli manifest sync rejot-manifest.base.json rejot-manifest.dev.json # Production rejot-cli manifest sync rejot-manifest.base.json rejot-manifest.prod.json ``` ## Workspaces Use the [`workspaces`](/docs/spec/manifest) key in your manifest to include other Manifest files automatically. ## Reference # Connectors We currently support PostgreSQL as both source and destination for sync. Our goal is to support most major database providers. | Database | Version | Source | Sink | Backfills | Event store | | ---------- | ---------- | ------- | ------- | --------- | ----------- | | PostgreSQL | ≥ 15 | ✅ | ✅ | Planned | ✅ | | MySQL | | Planned | Planned | Planned | | | MongoDB | | Planned | Planned | Planned | | # Request database connectors Let us know which database you would like to see supported by creating an issue [in our Github issue tracker][github-issue] [github-issue]: https://github.com/rejot-dev/rejot/issues # Language Bindings Language bindings enable developers to define public and consumer schemas using their preferred programming language and existing package management infrastructure. | Language | Contracts (Public/Consumer Schema) | Transformations | | --------------------- | ----------------------------------- | ------------------------------------- | | JavaScript/TypeScript | [@rejot-dev/contract][npm-contract] | [@rejot-dev/adapter-postgres][npm-pg] | Python bindings are currently in development. ## Requesting Additional Language Support If you would like to see support for additional languages, please create an issue in our [GitHub issue tracker][github-issue]. [github-issue]: https://github.com/rejot-dev/rejot/issues [npm-contract]: https://www.npmjs.com/package/@rejot-dev/contract [npm-pg]: https://www.npmjs.com/package/@rejot-dev/adapter-postgres # Resolvers Resolvers are components that determine how to connect to sync services based on manifest slugs. ## Localhost Resolver A resolver for local development that always returns `localhost:port`. This is intended for development purposes only and works with a single sync service. ```bash rejot-cli manifest sync --resolver localhost ``` ## Environment Resolver Resolves hosts using environment variables. The format for environment variables is: ``` REJOT_SYNC_SERVICE_=: ``` Example: ```bash REJOT_SYNC_SERVICE_MY_MANIFEST=my-service:3000 REJOT_SYNC_SERVICE_ANOTHER=another-service:4000 ``` ```bash rejot-cli manifest sync --resolver env ``` ### Kubernetes Resolver A resolver for Kubernetes environments that constructs service DNS names following the Kubernetes service naming convention. The hostname format is:
`rejot-manifest-..svc.cluster.local` ## Concepts # Architecture The overview below shows how ReJot would operate in a microservice architecture where each service has its own data store. - **Sync Engine**: Consumes the write-ahead log of a data store and applies the public schema transformations to the rows mutated in that data store. Stores these public schema events/messages into the event store and handles writing the mutation to the destination data store. - **Event Store**: Durable storage backend for public schema events. ![ReJot Architecture Overview](/content/docs/rejot-architecture.svg) # Data Contracts Data dependencies in ReJot are explicitly modeled, allowing engineering teams to maintain control over how their data is exposed within the organization. By decoupling data producers from consumers, data owners gain the flexibility to modify internal data models without impacting downstream consumers. Data producers can publish their data through a [Public Schema](/docs/spec/public-schema), while consumers can subscribe to datasets through a [Consumer Schema](/docs/spec/consumer-schema). ## Fan-out Replication ReJot is designed to support fan-out replication between data producers and consumers. Each public and consumer schema is versioned to accommodate changes in these data models. Note that this requires an [Event Store](/docs/spec/event-store) in your setup. import DarkImage from "@/components/DarkImage.astro"; ## Manifest Spec # Manifest The manifest is a JSON file that describes all of ReJot's configuration. Manifests are used by Sync Services to understand what data needs to be synced where. Manifests may be distributed over multiple files and usually follow organizational structure. While we document the schema of the manifest here, we suggest to modify the manifest through the `rejot-cli`. See our [guide](/docs/guides/managing-manifests) on how to set up your own manifests. import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; # Data Stores import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; A data store represents a source or destination of data. Data stores are typically databases, and [Connections](/docs/spec/connection) are used to configure clients to these databases. ## Data Store ## Config The configuration of a data store depends on its type, specified by the `connectionType` field. Currently, the following types are supported: ### Postgres Configuration for PostgreSQL-based data stores. # Event Stores import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; ## Event Store Event stores are used by ReJot as durable storage for Public Schema updates. # Public Schemas import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; Public schemas define how data is exposed from a data store managed by ReJot, see our [Defining Schemas](/docs/guides/defining-schemas) guide for more details. ## Public Schema ## Source ## Version ## Config The configuration of a public schema depends on its type, specified by the `publicSchemaType` field. Currently, the following types are supported: ### Postgres Configuration for PostgreSQL-based public schemas. #### Transformations Transformations define how changes in source tables are reflected in the public schema. # Consumer Schemas import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; Consumer schemas define how data is ingested from a public schema into a destination data store, see our [Defining Schemas](/docs/guides/defining-schemas) guide for more details. ## Consumer Schema ## Source Manifest Slug ## Public Schema Reference ## Config The configuration of a consumer schema depends on its type, specified by the `consumerSchemaType` field. Currently, the following types are supported: ### Postgres Configuration for PostgreSQL-based consumer schemas. # Connections Configuration required to setup a client connection to some data store. For supported databases see the [Connectors](/docs/reference/connectors) import ApiSchema from "@/components/schema/ApiSchema.astro"; import manifestSchema from "@/content/manifests/manifest_v0.json"; ## Connection ## Connection Config ## CLI Spec # Collect The `collect` command is used to collect public and consumer schemas from TypeScript files. import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; # Manifest import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; ## Get manifest information The `manifest:info` command is used to display information about a specific manifest. ## Initialize a new manifest The `manifest:init` command is used to initialize a new manifest file at the specified path. ## Synchronize manifest configurations The `manifest:sync` command is used to synchronize manifest configurations. # Manifest Connection import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; ## Add a connection to a manifest The `manifest:connection:add` command is used to add a connection to a manifest. ## Remove a connection from a manifest The `manifest:connection:remove` command is used to remove a connection from a manifest. ## List all connections in a manifest The `manifest:connection:list` command is used to list all connections in a manifest. ## Update a connection in a manifest The `manifest:connection:update` command is used to update a connection in a manifest. # Manifest Datastore import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; ## Add a datastore to a manifest The `manifest:datastore:add` command is used to add a datastore to a manifest. ## Remove a datastore from a manifest The `manifest:datastore:remove` command is used to remove a datastore from a manifest. ## List all datastores in a manifest The `manifest:datastore:list` command is used to list all datastores in a manifest. # Manifest Eventstore import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; ## Add an eventstore to a manifest The `manifest:eventstore:add` command is used to add an eventstore to a manifest. ## Remove an eventstore from a manifest The `manifest:eventstore:remove` command is used to remove an eventstore from a manifest. ## List all eventstores in a manifest The `manifest:eventstore:list` command is used to list all eventstores in a manifest. # Workspace import CliSchema from "@/components/schema/CliSchema.astro"; import { commands } from "@/content/cli/cli-schema_v0.json"; ## Get workspace information The `workspace:info` command is used to display information about the current workspace configuration. ## Initialize a new workspace The `workspace:init` command is used to initialize a new ReJot workspace by creating a root manifest file.