Amazon Kinesis

We used Kinesis as the basis for information distribution for a mission-critical business operation. We expected a huge amount of mini events reporting information between a Iot of devices, databases, geo info systems, and web sites, as well as reporting (business intelligence). Kinesis was used as the messaging pipeline (nowadays streaming) that glued everything together.

Amazon Kinesis is a great replacement for Kafka and it works better whenever the components of the solution are AWS based. Best if extended fan-out is not required, but still price-performance ratio is very good for simplifying maintenance.

Amazon Kinesis is being used to stream telemetry data from millions of connected devices on the field - helping to drive command and control use cases and also sensor data for performance monitoring. Our volume is billions of messages per day, real-time, with aggressive requirements on response and processing time. All messages are stored in a big data structure and also two-way communication with a mobile app.

Perfect for real-time data processing and streaming. Also, there's no need for any specific setup - you just start using it immediately and it easily integrates with the rest of AWS capabilities (like Redshift), although integration with Lambda could be better. You can make your overall analytics landscape way simpler with Kineses even if you have non-Amazon solutions like Tableau. It all integrates really well!

We currently use Kinesis as a trigger for Amazon Lambda functions, however we're working at switching back to SQS since Lambda now works with SQS directly. We simply used Kinesis as a temporary method of plumbing requests from one Lambda function to another.

We also use it as a byproduct of using DynamoDB Streams, but only in so far as that's used under-the-hood to link changes from DynamoDB to a Lambda trigger.

It's great for big data applications, where it's not as important to make sure each event is processed, but you're more looking for overall analytics and speed is more important than absolute detail. It's added quite a few features like analytics and other tools specifically designed around big data, but it's not the same thing as a queue system.

Originally we were forced into using Kinesis since Lambda didn't support SQS directly, but now that there is native SQS support for Lambda, we'll be switching almost all of our implementation over to use that instead. Kinesis has its strengths, but monitoring and error retry logic is not one of those.