26. What can you do with Kafka on AWS?

https://www.youtube.com/watch?v=nH03_bnr_xA

Luciano and Eoin explore the wonderful world of data streaming using Kafka on AWS. In this episode we focus mainly on Managed Streaming for Kafka (or MSK) and discuss what are the main differences between MSK and Kinesis. We also explore the main features that MSK provides, its scaling characteristics, pricing and, finally, how MSK works in conjunction with other AWS services.
We conclude the episode by providing a decision tree that should help you to decide whether you should use Kinesis or MSK or avoid streaming services entirely in favor of something like SNS or SQS.

00:00 Intro
01:57 Option for managed Kafka in the cloud
02:50 The feature set for Kafka
06:17 Terminology comparison Kafka vs Kinesis
08:04 Differences between Kafka and Kinesis
09:10 The role of Zookeeper in a Kafka cluster
11:25 Protocols and delivery guarantees
12:05 Throughput, Scalability model and retention
14:15 The Kafka ecosystem
15:15 How to setup MSK
16:43 Security options in MSK
18:35 MSK Serverless (preview)
20:50 Monitoring MSK
22:00 MSK integrations
23:00 Using MSK and Kafka with Lambda
26:00 MSK pricing
28:02 Should I use MSK or Kinesis?
30:18 Resources to deep dive with MSK
31:49 Wrap-up

In this episode, we mentioned the following resources:
- Our previous episode on Kinesis data streams: https://www.youtube.com/watch?v=u_nR6...
- Our series of Event services: https://www.youtube.com/watch?v=CG7uh...
- AWS MSK sizing spreadsheet: https://dy7oqpxkwhskb.cloudfront.net/...
- Should My Startup use Kinesis or MSK? - https://www.youtube.com/watch?v=TJS19...
- Intro to MSK (reinvent talk from 2018) - https://www.youtube.com/watch?v=9nKsw...
- Running Apache Kafka on AWS (by Frank Munz) - https://www.youtube.com/watch?v=HtU9p...
- Cloudonaut - Kinesis versus MSK - https://www.youtube.com/watch?v=kcBAK...

You can listen to AWS Bites wherever you get your podcasts:
- Apple Podcasts: https://podcasts.apple.com/us/podcast...
- Spotify: https://open.spotify.com/show/3Lh7Pzq...
- Google: https://podcasts.google.com/feed/aHR0...
- Breaker: https://www.breaker.audio/aws-bites
- RSS: https://anchor.fm/s/6a3312a0/podcast/rss

Do you have any AWS questions you would like us to address?
Leave a comment here or connect with us on Twitter:
- https://twitter.com/eoins
- https://twitter.com/loige