This is the continuation of the Spark Streaming Basics. I explained the basic stream example, which runs only on one AWS Glue container. The stream producer was Netcat, and the sink was a text file. In this post, the stream producer is still Netcat, but the sink is Kafka. Both Kafka and Spark running on Docker containers.

This is a very basic example created to explain Spark streaming. Spark run on the AWS Glue container locally.

This is a short description of lambda calculus. Lambda calculus is the smallest programming language that is capable of variable substitution and a single function definition scheme. Haskell is the functional programming language based on lambda calculus, which I will explore. I already explained how to use VSCode for Haskell Development to support the code listed here.

Discuss the most possible ways of passing parameters in the python functions.

Python Data classes using collections.namedtuple, typing.NamedTuple and latest @dataclass decorator.