Amazon Simple Notification Service (SNS) vs. Apache Kafka

Using SNS for any notification use case where available is the default and defacto solution. It directly integrates with SES to configure both incoming email and email delivery responses.

Additionally, any notifications, such as CloudWatch alarms, are a good use case for SNS topics and allow us to fan out delivery as needed (pagerduty, email, etc)

For brokering messages, Confluent Kafka is well suited since it offers a managed solution ready to use. Scenarios where the solution is not very well suited are for example, where pricing is an issue. The solution costs quite a lot for basic usage (for example: for 3 clusters, pricing is above 100k$ a year).

Incentivized

• Built in for quicker setup within AWS ecosystem.
• Trusted as you control the users and configuration via IAM and easy access controls.
• Can be sent to S3 simple storage or for long term storage if required.
• Can be used in many regions, same configs.

Incentivized

• Apache Kafka is able to handle a large number of I/Os (writes) using 3-4 cheap servers.
• It scales very well over large workloads and can handle extreme-scale deployments (eg. Linkedin with 300 billion user events each day).
• The same Kafka setup can be used as a messaging bus, storage system or a log aggregator making it easy to maintain as one system feeding multiple applications.

Incentivized

• At times you receive access denied errors which are annoying.
• Rarely do you receive internal failure errors where you can't access the information. It is rare but it does happen.
• You are required to add an MWS Authentication Token every so often. I wish it would pull that information automatically for you so you don't have to go searching for it.

Incentivized

• The Kafka Tool is a community-made Java application that looks and feels from the past century.
• Logging can be confusing. This certainly shows when we have to do troubleshooting.
• Hybrid scenarios - pub/sub, but there are services in and outside a Kubernetes cluster. Then there are a ~3 options, but only 2 (the harder ones) are production-safe.

Incentivized

Kafka has suited our use case very well so far. Going forward we are planning to expand our platform manifold so the load on Kafka and our reliance on Kafka is going to increase only.

It is useful for applications developed using event driven architecture. It helps in tracking and logging the events in a very timely and efficient manner. The dashboards are a little difficult to implement. But overall it is very easy to integrate with other AWS services like Lambda, API GW, S3 and DynamoDB. The permissions to access should be resolved before using it.

Incentivized

Apache Kafka is highly recommended to develop loosely coupled, real-time processing applications. Also, Apache Kafka provides property based configuration. Producer, Consumer and broker contain their own separate property file

Incentivized

The AWS documentation is well maintained and has lots of information which makes it easier for developers to refer to and develop applications in a fast efficient manner. It is well documented with examples which is easy to understand and implement. You can also get help by posting into forums from like-minded developers.

Incentivized

Support for Apache Kafka (if willing to pay) is available from Confluent that includes the same time that created Kafka at Linkedin so they know this software in and out. Moreover, Apache Kafka is well known and best practices documents and deployment scenarios are easily available for download. For example, from eBay, Linkedin, Uber, and NYTimes.

Incentivized

Amazon Simple Notification Service (SNS) is well integrated in AWS and has been there since the early days of public cloud. It is a cost effective and very inexpensive solution to meet the needs of event notifications and custom messaging. Wish to share that there is considerable number of developers who can easily build solutions using AWS SNS. So, training costs are minimal. Other solutions are emerging and we are seeing a great usage especially of Firebase notifications because of its very neat integration with open source cross platform hybrid app frameworks like ionic, xamarin. SNS needs to become better and should have plugin support for the mobile application developers using low code/no code tools too.

Incentivized

Apache Kafka is built for scale. From high throughput and real-time data streaming, it has a strong advantage over RabbitMQ with its low latency. This put Apache Kafka at the forefront as the platform of choice for large datasets messaging and ensuring scalability when data scale up tremendously. RabbitMQ however has its strengths in traditional messaging. Routing and message delivery reliability are the bedrock of RabbitMQ and this is where RabbitMQ excels. In my previous workplace, RabbitMQ was of choice as reliability matters more than scale. In two words. Apache Kafka for scale, RabbitMQ for reliability. And for cloud deployment and large dataset messaging in what I am doing now, Apache Kafka is the default choice.

• Cost of alert calls to the different stakeholders across different geographies have gone down since using Amazon SNS.
• Amazon SNS has saved a lot of time for the employees that they used to spend to call multiple stakeholders so they can now focus more on productive tasks.
• Amazon SNS usage needs prior knowledge of programming.

Incentivized

• Positive: bursts of traffic on special holidays are easy to handle because Kafka can absorb and buffer all the messages we need to process long enough to let an understaffed set of back-end services catch up on processing. Hard to put a number to it but we probably save $5k a month having fewer machines running.
• Positive: makes decoupling the web and API services from the deeper back-end services easier by providing topics as an interface. This allowed us to split up our teams and have them develop independently of each other, speeding up software development.
• Negative: our engineers have made mistakes such as accidentally dropping a few thousand messages due to the CLI being confusing to use, and as a result a customer lost some of their precious data. I'd say that was more our fault than Kafka's though.

Incentivized