Data Migration

Data evolution is one of the hardest problems in software engineering. ReJot makes moving data between databases or clusters easy while also enabling smoother data migrations. Using our sync engine, we can effortlessly create a read-copy of the new model while keeping writes on the old version until fully verified.

In this use case, we will outline migrating data from a legacy database to a new database. The same steps can be applied to migrating data to a new cluster, or a different schema in the same database.

Background

There are some terms you’ll need to understand before we get started:

• Data Store: A database. The canonical owner of the data you want to migrate. Typically has some sort of schema.
• Data Server: The service that interacts with the data store. We see this as the logical “owner” of the data within the data store. Typically this service is stateless and is running in multiple (load balanced) instances. Contains authentication routines and other business logic.
• Data Client: An application that interacts with the server. This is usually (but not in this use case) a frontend or user-facing application. The client also has a backing data store, but this is usually ephemeral. It can be reconstructed from the data in the server/store.

Context

We’ll be using the “Strangler Pattern” to migrate data from a legacy database to a new database. This pattern involves creating a new data server that interacts with the new database. We’ll then slowly migrate data from the old database to the new one while keeping the old data server running and serving requests. Once the data has been migrated, we’ll switch over to the new data server.

The synchronization engine at the core of ReJot is designed to move data from a data server to a data client. If there are no transformations or access rules in place, this is a 1:1 mapping. Once the client is up-to-date, the client contains a full copy of the data.

Business logic that is part of the data server can choose to atomically change data in the store. A data client does not have this ability. It can only interact with the server through issuing commands. The server then decides to honor the command or not.

Method

The steps to take are (roughly) the following:

1. Create a new data client that follows the data server we want to migrate.
2. (Optional) Setup transformation rules to transform data from the old schema to the new one.
3. Wait for the data client to (almost)catch up to the data server.
4. Switch reads from the data server to the data client.
5. Promote the data client to be the new data server.
6. Get rid of the old data server.

There are some caveats to this method:

• If there are transformation rules in place, the business logic contained in the data server will need to be able to handle the old schema as well as the new schema.
• Since synchronization is an eventually consistent process, there is a small window where the data client may seem to have fully caught up, but there is still some data in transition. Thus, when promoting the client, we need to buffer any modifications that are currently being applied to the original server.
• In the case where the data store does not use commands, and instead writes directly to the store, it will not be trivial to promote the data client. Commands can be buffered, while direct writes can have arbitrary mutations that cannot be buffered. In that case it might be necessary to migrate some business logic first to use the more ReJot-friendly command pattern.

Key Benefits

• Zero Downtime: Keep your system running while migrating data
• Data Integrity: Ensure no data is lost during the migration process
• Verification: Compare old and new data stores before switching
• Rollback Safety: Easily revert to the original system if needed

Common Use Cases

• Database upgrades
• Schema changes
• Storage system migrations
• Cloud migrations
• Database replication setup

Local First

Todo. :^)

Backend-for-Frontend (BFF)

In a service-oriented architecture, the Backend-for-Frontend (BFF) acts as an intermediary layer between the frontend and numerous backend services. Rather than having the frontend interact directly with each backend service, the BFF acts as a single point of entry for the frontend. This moves complexity away from the UI layer, improves latency, and solves over-fetching of data. BFFs are all about data, a perfect fit for ReJot’s synchronization capabilities.

Background

There are some terms you’ll need to understand before we get started:

• Data Store: A database. The canonical owner of the data you want to migrate. Typically has some sort of schema.
• Data Server: The service that interacts with the data store. We see this as the logical “owner” of the data within the data store. Typically this service is stateless and is running in multiple (load balanced) instances. Contains authentication routines and other business logic.
• Data Client: An application that interacts with the server. This is usually (but not in this use case) a frontend or user-facing application. The client also has a backing data store, but this is usually ephemeral. It can be reconstructed from the data in the server/store.

Method

In this use case, we’ll have an extra data client that acts like the Backend-for-Frontend. We assume that the setup before introducing ReJot is a service-oriented architecture. A number of teams own their respective services with a backing data store. The team working on the frontend wants to integrate data from multiple teams before sending it to the end-user.

The product teams have already set up publication rules for their data. The BFF data client is a receiver of data owned by these teams. The data that is shared depends on the type of frontend application we are dealing with. A back-office application would receive more data than a customer facing application.

The BFF business logic code will be responsible for the authentication and authorization of the requests coming in. This is important, as the data client would receives data related to all customers, as opposed to only data of the customer issuing the request. The data client will also forward any mutation commands received from the frontend to the appropriate data server. Note that transformations to these commands can be freely applied, as the team thinks appropriate. The BFF data client can also choose to only receive a subset of the server data.

Overview

The architecture of the BFF setup is quite typical of any architecture that contains a BFF. The difference is in the way data is exchanged between the services. In a traditional setup,

Legal Entity Management

Scalable Business Intelligence