Google
 

Sunday, September 24, 2017

Azure Event Grid WebHooks - Retries (Part 3)

Building distributed systems is challenging. If not carefully designed and implemented, a failure in one component can cause cascading failures that affect the whole system. That's why patterns like Retry and Circuit Breaker should be considered to improve system resilience. In case of sending WebHooks the situation might be even worse as your system is calling a totally external system with no availability guarantees and over the internet which is less reliable than your internal network.
Continuing on the previous parts of this series (Part 1, Part 2) I'll show how to use Azure Event Grid to overcome this challenge.

Azure Event Grid Retry Policy

Azure Event Grid provides a built-in capability to retry failed requests with exponential backoff, which means that in case the WebHook request fails, it will be retried with increased delays.
As per the documentation failed requests will be retried after 10 seconds, and if the request fails again, it will keep retrying after 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, and 1 hour. However these numbers aren't exact intervals as Azure Event Grid adds some randomization to these intervals.
Events that take more than 2 hours to be delivered will be expired. This duration should be increased to 24 hours after the preview phase.
This behavior is not trivial to implement which adds to the reasons why using a service like Azure Event Grid should be considered as an alternative to implementing it's capabilities from scratch.

Testing Azure Event Grid Retry

To try this capability and building on the example used in Part 1, I made a change to the AWS Lambda function that receives the WebHook to introduce random failures:

public object Handle(Event[] request)
{
    Event data = request[0];
    if(data.Data.validationCode!=null)
    {
        return new {validationResponse = data.Data.validationCode};
    }

    var random = new Random(Guid.NewGuid().GetHashCode());
    var value = random.Next(1 ,11);

    if(value > 5)
    {
        throw new Exception("Failure!");
    }

    return "";
}

Lines 9-15 produce almost 50% failure rate. When I pushed an event (as shown in the previous posts) to a 1000 WebHook subscribers, the result was the below chart depicting the number of API calls per minute and number of 500 errors per minute:


Number of requests per minute (Blue) - Number of 500 Errors per minute (Orange)

We can observe the following:
  • The number of errors (orange) is almost half the number of requests (blue)
  • Number of requests  per minute is around 1500 for the first minute. My explanation is that since we have 1000 listeners and 50% failure rate, Azure has made extra 500 requests.
  • After a bit less than 2 hours (not shown in the chart for size constraints) the number of errors has dropped to 5 and no more requests were made. This is due to the expiration period during the preview.

Summary

Azure Event Grid is a scalable and resilient service that can be used in case of handling thousands (maybe more) of WebHook receivers. Whether your solution is hosted on premises or on Azure, you can use this service to offload a lot of work and effort.
I wish that Azure Event Grid could give some insights on how events are pushed and received which would help a lot in troubleshooting as the subscriber is usually not under your control. I hope this will become an integrated part of the Azure portal.
It's worth mentioning that other cloud providers support similar functionality as Event Grid that are worth checking, specifically Amazon Simple Notification Service (SNS) and Google Cloud Pub/Sub. Both have overlapping functionality with Azure Event Grid.

15 comments:

SEO Services in Cochin said...

superb blog post ..

thanks for sharing it with us

Eleftheria Drosopoulou said...

Hello Hesham,

Nice blog! I am editor at Java Code Geeks (www.javacodegeeks.com). We have the JCG program (see www.javacodegeeks.com/join-us/jcg/), that I think you’d be perfect for.

If you’re interested, send me an email to eleftheria.drosopoulou@javacodegeeks.com and we can discuss further.

Best regards,
Eleftheria Drosopoulou

Pavel Co Ebele said...

Hi, Great.. Tutorial is just awesome..It is really helpful for a newbie like me.. I am a regular follower of your blog. Really very informative post you shared here. Kindly keep blogging. If anyone wants to become a .Net developer learn from Dot Net Training in Chennai. or learn thru Dot Net Training in Chennai. Nowadays Dot Net has tons of job opportunities on various vertical industry.
or Javascript Training in Chennai. Nowadays JavaScript has tons of job opportunities on various vertical industry.

PLC Training Chennai said...

PLC Training in Chennai | PLC Training Institute in Chennai | PLC Training Center in Chennai | PLC SCADA Training in Chennai | PLC SCADA DCS Training in Chennai | Best PLC Training in Chennai | Best PLC Training Institute in Chennai | PLC Training Centre in Chennai | Embedded System Training in Chennai | Embedded Training in Chennai | VLSI Training in Chennai | VLSI Training Institute in Chennai

PLC Training Chennai said...

PLC Training in Chennai | PLC Training Institute in Chennai | PLC Training Center in Chennai | PLC SCADA Training in Chennai | PLC SCADA DCS Training in Chennai | Best PLC Training in Chennai | Best PLC Training Institute in Chennai | PLC Training Centre in Chennai | PLC SCADA Training in Chennai | Automation Training Institute in Chennai | PLC Training in Kerala

Embedded Systems Course in Chennai said...

Embedded Training in Chennai | Best Embedded Training in Chennai | Embedded System Training in Chennai | Embedded System Training Institute in Chennai | Best Embedded System Training Institute in Chennai | Embedded Course in Chennai | Embedded System Training Institutes in Chennai | Embedded System Training Center in Chennai | Best Embedded System Training in Chennai | Embedded Systems Training in Chennai | VLSI Training in Chennai | VLSI Training Institute in Chennai

VLSI Training Chennai said...

VLSI Training in Chennai | Best VLSI Training in Chennai | VLSI Training Centres in Chennai | VLSI Courses in Chennai | VLSI Training Courses in Chennai | VLSI Training Institute in Chennai | VLSI Training Institutes in Chennai | Best VLSI Training Institute in Chennai

viswanath said...

AWS Training in Bangalore - Live Online & Classroom
myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.


IOT Training in Bangalore - Live Online & Classroom
IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

Abmel joseph said...

I am really happy with your blog because your article is very unique and powerful for the new reader. Nowadays tons of job opportunities in the various vertical industry. VLSI Training in Chennai

Dharani M said...

Nice information
sas training in Marathahalli

sas training institutes in Marathahalli

mounika said...

Nice post..
aws course in BTM

aws training center in BTM

cloud computing courses in BTM

amazon web services training institutes in BTM

best cloud computing institute in BTM

cloud computing training in btm

aws training in btm

aws certification in btm

best aws training in btm



aws certification training in btm

kayal said...

Thanks for sharing this post
iot courses in Bangalore

internet of things training course in Bangalore

internet of things course in Bangalore

kayal said...

Nice Post
aws course in Bangalore

aws training center in Bangalore

cloud computing courses in Bangalore

amazon web services training institutes in Bangalore

best cloud computing institute in Bangalore

cloud computing training in Bangalore

aws training in Bangalore

aws certification in Bangalore

best aws training in Bangalore

aws certification training in Bangalore

uma said...

nice blog..keep sharing



selenium training centers in Bangalore

best software testing training institutes in Bangalore with placements

automation testing courses in Bangalore

selenium testing course in Bangalore

software testing institutes in Bangalore

selenium training in Bangalore

best selenium training in Bangalore

selenium course in Bangalore

chandana said...

I am really happy with your blog.

best android training center in Marathahalli

best android development institute in Marathahalli

android training institutes in Marathahalli

ios training in Marathahalli

android training in Marathahalli

mobile app development training in Marathahalli