Upcasters or a versioned event store: pros and cons

In a previous article, I wrote a few things about upcasters. One of the significant downsides when implementing an upcaster is that it adds to our application’s technical debt. An alternative technique is the versioned event store (or versioned event stream), where the existing event store is copied and modified. In this post I’ll discuss the pros and cons of both approaches.

Upcasting

In the event sourcing context, upcasting an event means to transform that event from its original to its new structure. Some other libraries or implementations use the term event adapter or event transformer when referring to the same technique. An upcaster is called when the application is reading existing events from an event stream (to reconstruct an aggregate, for example). The upcasting function transforms applicable events before they are forwarded to any event listeners, projections, etc. The immutability of the existing events is guaranteed as they are not touched or modified on disk.

Axon is one of the frameworks that has built-in support for upcasting events.

Pros

• Events continue to be immutable, the complete event history remains intact;
• No corrective events are necessary, we can safely remove old event listeners & code.

Cons

• The in-memory view of the event stream does not match the on-disk state of the stream;
• (Axon specifc) Upcasters work on intermediate event representations (xml);
• Serialization (of events) can be broken;
• Projections are not automatically updated;
• Maintenance burden, upcasters need to be maintained indefinitely (number of upcasters in repository doesn’t decrease in general);
• Performance considerations (depending on the complexity of the upcaster);
• Upcaster chains can be tricky to test (there are multiple combinations);
• Complex upcasters (merging or splitting events, pulling in data from other sources, etc.);
• Snapshots have to be rebuilt or upcast;
• (Axon specific) Can’t upcast events from one aggregate to another.

(Versioned) refactoring of event streams

This is a technique that allows modifications to existing events, while aiming to satisfy some of the concerns around immutability. To achieve this, we simply copy an existing event stream while simultaneously modifying (some) events. This approach is also referred to as Copy and Replace. To perform the Replace, we can re-use existing upcasting logic.

1. Create a new, empty event stream (with a version number suffix added to the name of the stream);
2. Iterate over all events in the existing stream;
3. Apply the upcasting function to applicable events and write the result to the new stream, copy any events that do not have to be modified;
4. Let the application use the new stream (flip a feature toggle, for example);
5. Perform cleanup: remove upcasting function, archive or delete old stream.

Pros

• The in-memory view of the event stream matches the on-disk state of stream;
• There’s no performance or maintenance penalty (after running the refactoring process);
• The conversion/upcasting function can be removed after the event stream is copied.

Cons

• Events are no longer immutable;
• Projections are not automatically updated;
• Serialization can potentially be broken;
• Snapshots will have to be rebuilt;
• Old versions / copies of event stores (streams) should be retained indefinitely if required for auditing;
• If you encounter a bug in an event (or event handler) after the refactoring process, you’ll have to refactor again.

Downtime

In a running system, events will continue to come in while the Copy and Replace process is running. The simple solution to that problem is stopping the application (or switch it to read-only mode), create the new version and finally start the application again. Depending on the environment, traffic and other parameters, that downtime may not be acceptable.

This can be solved in a number of ways, I’ll describe one alternative:

• Use a message queue such as RabbitMQ, or a data streaming platform such as Kafka;
• Publish events, as they are persisted, to the message bus;
• Consume events from the bus, pass them through the upcasting function, and construct the new event stream;
• Switch to the new stream, once it is up to date and all replicas are aware of new version.

I hope this is useful for you, let me know in the comments!