AsyncAPI Online Conference - My Favourites and Learnings

May 5, 2020
Andrea Richiardi

With Elastic Path moving towards the agility, speed and resilience of the microservice architecture I have grown even more hungry for best practices around inter-process communication, having already quite some experience with event-driven systems.

The AsyncAPI specification is one of those excellent efforts around making your life as a Software Engineer easier whilst working on distributed systems.

It goes without saying that when I have first heard that Fran Méndez and Lukasz Gornicki were organizing the AsyncAPI Online Conference, the remote-only no-registration-needed conference, I jumped in without hesitation.

Sure, living on the West Coast meant waking up at 5 AM yet I hope that with the following recap I will be able to testify that it was worth it.

CloudEvents - A First Step Towards Serverless Interop

The first talk I want to mention was a very nice overview, historical context included, about the CloudEvents initiative.

From the main website:

CloudEvents is a specification for describing event data in a common way.

I like to work against specifications: they provide shared vocabulary and structure. They also solve the "What format my data should be?" problem.

That was also the driver at IBM: reconciling the disparate data shapes that the middleware layer might need to handle while routing events from producers to consumers can be a pain and you want to use one data format and forget about it.

We internally use what the spec calls a structured message but what I would like to highlight here is its binary mode:

POST /event HTTP/1.0
Host: example.com
Content-Type: application/json
ce-specversion: 1.0
ce-type: com.bigco.newItem
ce-source: http://bigco.com/repo
ce-id: 3d172eb0-5e8c-48df-a918-a45d9df076fc
{
  "your-normal": "json"
}

The advantage of the header representation is that it is additive - especially important for legacy systems where you want to avoid breaking changes.

After the talk, Fran Mendez also shared a nice blog post on how AsyncAPI and CloudEvents coexist and I am going to attach it here as well.

Post

Talk

Event-Based API Patterns and Practices

One of the biggest benefits of the microservice architecture is that you end up with loosely coupled services carrying out their purpose.

In practice, services require information from one another. Think of the use case of the cart service needing customer information. The obvious way to achieve this is by use of REST interfaces and HTTP calls.

The approach works until your call chains become long or whenever a service needs access to public APIs. At that point, the weaker link of the chain will deteriorate responsiveness and user experience.

This is where asynchronous communication and event backbones, like Apache Kafka, become useful. James' talk is packed with information around this: he not only gives us the rundown of use cases and technologies but also lays out a series of design patterns for us to make use of, including but not limited to the following.

Thin Events and Hypermedia

The definition of Thin Event is very simple and akin to notifications: a minimal piece of data that a service needs to be notified that something has happened.

They are commonly used when you want to make sure your service does not work against stale data and if that's the case trigger a re-fetch.

This is were hypermedia comes into play: what if the event payload could carry a link to the endpoint the service needs to call for syncing up?

Brilliant stuff.

Treat Event Definition as part of your API design

At the end of the day, events and their schema are part of the contract that a service establishes with its consumers.

This is no different from REST API design and James' advice here is to follow the same process that your company already has in place for it.

Offline and Error Recovery Support

When designing your platform it is good practice to make sure asynchronous API consumers can easily troubleshoot failed deliveries.

A very good example of this is GitHub's Recent Deliveries feature.

Orthogonal but equally important to mere observability is recoverability: the design should devise a mechanism for allowing clients to "catch up" when either they go offline or a server-side data center is unavailable.

Talk

Building HTTP/2 with the Mercure Protocol

The previous talks focused mostly on the backend.

Nonetheless, software where asynchronous communication amounts to snappier user experience is touchpoints like web apps and mobile clients.

The state of the art for frontend technology though is not devoid of road bumps.

HTTP Long Polling

This venerable mechanism keeps a connection open from client to server and once the data is ready the server sends it back and closes it. The main problem here is resource consumption: creating a connection is expensive and there is a very difficult balance to achieve in terms of payload size and message frequency.

Server Side Events

This more modern (built-in for most browsers) solution tries to keep a connection open forever. The EventSource JavaScript class is used for both opening and closing the connection to the server, which is the only party allowed to send events. The connection problem is solved but the communication is one-way only.

WebSockets

It was introduced for filling the gap and providing a full-duplex communication channel. It works by "downgrading" an HTTP connection to the lower TCP level (here is the reference). By being lower-level it is somehow harder to secure and requires some level of re-tooling, given you cannot re-use existing mechanisms like authentication.

The folks at Les Tilleuls have had the excellent idea of blending many ideas from the above options and put together the Mercure Protocol.

The main highlights of this initiative are:

  • Built on top of HTTP
    • Sending an event is an HTTP POST - Super Simple! 👍
    • Receiving is subscribing to Server Side Events
    • It capitalizes on existing technologies (JWT, encryption, ...)
  • Full Duplex
  • HTTP/2 - Speedy! 🏃‍♀️💨
  • Supports REST and GraphQL
  • Boasts a production-grade AGPL implementation - the Mercure Hub - written in Go
  • Hypermedia via JSON-LD - Elastic Path approves! ✅

Talk

Conclusion

Given the lengthy post already I am going to give a last shout to Event Modeling with AsyncAPI by my friend and former mentor Matt Bishop.

His talk explores the Event Modeling framework: a highly effective way for development teams and product management to collaborate on event-based systems.

Thanks for reading and enjoy the talks!