Azure Event Grid WebHooks (Part 1)

Few days ago, Microsoft announced the new Event Grid service. The service is described as:
"... a fully-managed intelligent event routing service that allows for uniform event consumption using a publish-subscribe model."
Although not directly related, I see this service as a complement to the serverless offerings provided by Microsoft after Azure Functions and Logic Apps.

Event Grid has many capabilities and scenarios. In brief , it's a service that is capable of listening to multiple event sources using topics and publishing them to subscribers or handlers that are interested in these  events.
Event sources can be Blob storage events, Event hub events, custom events, etc. And subscribers can be Azure functions, logic apps, WebHooks.
In this post I'll focus on pushing WebHooks in a scalable, reliable, pay as you go, and easy manner using Event Grid.

Topics, and WebHooks

Topics are a way to categorize events. A publisher defines topics and sends specific events to these topics. Publishers can subscribe to topics to listen and respond to events published by event sources.

The concept of WebHooks is not new. WebHooks are HTTP callbacks that respond to events that were originated in other systems. For example you can create HTTP endpoints that listen to WebHooks published by GitHub when code is pushed to a specific repository. This creates an almost endless number of integration possibilities.
In this post we'll simulate a blogging engine that pushes events when new posts are published. And we'll create a subscriber that listens to these events.

Creating a topic

The first step to publishing a custom event is to create a topic. As other Azure resources, Event Grid topics are created in resource groups. To create a new resource group named "rg" we can execute this command using Azure CLI v2.0.

az group create --name rg --location westus2

I Chose westus2 region because currently Event Grid has limited region availability. But this changes all the time.

The next step is to create a topic in the resource group. We'll name our topic "blog":

az eventgrid topic create --name blog -l westus2 -g rg

When you run the above command, the response should look like:

{                                                                                                    
  "endpoint": "https://blog.westus2-1.eventgrid.azure.net/api/events",                                                     
  "id": "/subscriptions/5f1ef4e8-6358-4a75-b171-58904114fb57/resourceGroups/rg/providers/Microsoft.EventGrid/topics/blog", 
  "location": "westus2",                                                                                                    
  "name": "blog2",                                                                                                          
  "provisioningState": "Succeeded",                                                                                         
  "resourceGroup": "rg",                                                                                                    
  "tags": null,                                                                                                             
  "type": "Microsoft.EventGrid/topics"                                                                                      
}

Observe the endpoint attribute. Now we have the URL to be used to to push events: https://blog.westus2-1.eventgrid.azure.net/api/events.

Subscribing to a topic

To show the capabilities of the Event Grid, I need to create hundreds of subscribers. You can create your subscribers in any HTTP capable framework. I chose to use AWS Lambda functions + API Gateway hosted in Sydney region. This proves that there is no Azure magic by any means. Just pure HTTP WebHooks sent from Azures data centers in west US to AWS data centers in Sydney.
The details of creating Lambda functions and exposing them using API Gateway aren't relevant to this post, the important thing is to understand that I have an endpoint that listens to HTTP requests on: https://twzm3c5ry2.execute-api.ap-southeast-2.amazonaws.com/prod/{id} and forwards them to AWS Lambda implemented in C#.
The command to create a subscription looks like:

az eventgrid topic event-subscription create --name blogreceiver   --endpoint https://twzm3c5ry2.execute-api.ap-southeast-2.amazonaws.com/prod/   -g rg  --topic-name blog 

I created 100 subscriptions using this simple Powershell script:

while($val -ne 100) { $val++ ;  az eventgrid topic event-subscription create --name blogreceiver$val   --endpoint https://twzm3c5ry2.execute-api.ap-southeast-2.amazonaws.com/prod/$val   -g rg  --topic-name blog}

An important thing to notice which is the security implications of this model. If I was able to specify any URL as a subscriber to my topic, I'd be able to use Azure Event Grid as a DDoS attacking tool. That's why subscription verification is very important.

Subscription verification

To verify that the subscription endpoint is a real URL and is really willing to subscribe to the topic, a verification request is sent to the subscription endpoint when the subscription is created. This request looks like:

[
{
    "Id": "dbb80f11-6fbb-4fc3-9c1f-034f00da3b5f",
    "Topic": "/subscriptions/5f1ef4e8-6358-4a75-b171-58904114fb57/resourceGroups/rg/providers/microsoft.eventgrid/topics/blog",
    "Subject": "",
    "Data": {
        "validationCode": "4fc3f59c-2d03-41f4-b466-da65a81f8ba5"
    },
    "EventType": "Microsoft.EventGrid/SubscriptionValidationEvent",
    "EventTime": "2017-08-20T11:11:00.0101361Z"
}
]

The validationCode attribute has a unique key to identify the subscription request. The endpoint should respond to the verification request with the same code:

{"validationResponse":"3158cb2f-a2c4-46ca-96b0-ae2c8562fa43"}

The subscriber

The subscriber is very simple. It checks whether the request has a validation code. If so, it responds with the validation response. Otherwise it just returns 200 or 202.

    public class Event
    {
        public Data Data { get; set; }
    }
    public class Data
    {
        public string validationCode { get; set; }
    }

    public class Receiver
    {
        public object Handle(Event[] request)
        {
            Event data = request[0];
            if(data.Data.validationCode!=null)
            {
                return new {validationResponse = data.Data.validationCode};
            }
            return "";
        }
    }

Note that the AWS API Gateway is responsible for setting the status code to 200.

 

Pushing events

As I showed above, I created 100 subscribers. Now it's time to start pushing events which is a simple post request but of course this request must be authenticated. The authentication methods supported are Shared Access Signature "SAS" and keys. I'll use the latter for simplicity.

To retrieve the key, you can use the management portal or this command:

az eventgrid topic key list --name blog --resource-group rg

To configure my .net core console application that will push the events, I created 2 environment variables using Powershell:

$env:EventGrid:EndPoint = "https://blog.westus2-1.eventgrid.azure.net/api/events"
$env:EventGrid:Key = "HQI2Ff7MoqlV8RFc/U........."

I created a class to read the configuration variables into an instance of it:

class EventGridConfig
{
    public string EndPoint { get; set; }
    public string Key { get; set; }
}

The rest is simple. Reading the configuration variables, and posting an event to the endpoint.

Configuration = builder.Build();
var config = new EventGridConfig();
Configuration.GetSection("EventGrid").Bind(config);

var http = new HttpClient();
string content = @"
    [
        {
            ""id"": ""123"",
            ""eventType"": ""NewPost"",
            ""subject"": ""blog/posts"",
            ""eventTime"": ""2017-08-20T23:14:22+1000"",
            ""data"":{
                ""title"": ""Azure Event Grid"",
                ""author"": ""Hesham A. Amin""
            }
        }
    ]";

http.DefaultRequestHeaders.Add("aeg-sas-key", config.Key);
var result = http.PostAsync(config.EndPoint, new StringContent(content)).Result;

Now it's Azure's Event Grid turn to push this event to the 100 subscribers.

The result

Running the above console application sends a request to Azure Event Hub. In turn in sends the event to the 100 subscribers I've created.
To see the result. I use AWS API Gateway CloudWatch graphs which show the number of requests to my endpoint. I ran the application few times and the result was this graph:

Requests per minute

Summary

In this post I've shown how to use Azure Event Grid to push WebHooks to HTTP endpoints and how to subscribe to these WebHooks.
In next posts I'll explore more capabilities of Azure Event Grid.