Install Apache Kafka on AlmaLinux 9

In this tutorial, we want to show you to Install and Configure Apache Kafka on AlmaLinux 9.

Apache Kafka (Kafka) is an open-source, distributed streaming platform that enables (among other things) the development of real-time, event-driven applications.

Kafka has three primary capabilities:

  1. It enables applications to publish or subscribe to data or event streams.
  2. It stores records accurately (i.e., in the order in which they occurred) in a fault-tolerant and durable way.
  3. It processes records in real-time (as they occur).

Kafka is a distributed platform—it runs as a fault-tolerant, highly available cluster that can span multiple servers and even multiple data centers. The Kafka topics are partitioned and replicated in such a way that they can scale to serve high volumes of simultaneous consumers without impacting performance. As a result, according to Apache.org, “Kafka will perform the same whether you have 50 KB or 50 TB of persistent storage on the server.”

Steps To Install and Configure Apache Kafka on AlmaLinux 9

To complete this guide, you must log in to your server as a non-root user with sudo privileges and set up a basic firewall. To do this, you can follow our guide on Initial Server Setup with AlmaLinux 9.

Install Required Packages For Kafka

First, you need to prepare your server for installing Kafka. Update and upgrade your local package index with the command below:

sudo dnf update -y && sudo dnf upgrade -y

Then, use the command below to install the required packages, and JDK on AlmaLinux 9:

sudo dnf install wget git unzip java-11-openjdk -y

Set up Apache Kafka on AlmaLinux 9

At this point, you need to download and get the latest release of Kafka.

Download Kafka

Visit the Apache Kafka downloads page and look for the Latest release and get the sources under Binary downloads. Get the one that is recommended by Kafka with the wget command:

sudo wget https://downloads.apache.org/kafka/3.3.2/kafka_2.13-3.3.2.tgz

Then, make a directory for your Kafka under /usr/local directory and switch to it with the following commands:

sudo mkdir /usr/local/kafka-server && sudo cd /usr/local/kafka-server

Next, extract your downloaded file in this directory:

sudo tar -xvzf ~/kafka_2.13-3.3.2.tgz --strip 1

Create Zookeeper Systemd Unit File

At this point, you need to create a Zookeeper systemd unit file for helping in performing common service actions such as starting, stopping, and restarting Kafka.

Zookeeper is a top-level software developed by Apache that acts as a centralized service and is used to maintain naming and configuration data and to provide flexible and robust synchronization within distributed systems. Zookeeper keeps track of the status of the Kafka cluster nodes and it also keeps track of Kafka topics, partitions, etc.

To create the zookeeper systemd unit file, you can use your favorite text editor, here we use vi editor:

sudo vi /etc/systemd/system/zookeeper.service

Add the following content to the file:

[Unit]
Description=Apache Zookeeper Server
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
ExecStart=/usr/local/kafka-server/bin/zookeeper-server-start.sh /usr/local/kafka-server/config/zookeeper.properties
ExecStop=/usr/local/kafka-server/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

When you are done, save and close the file.

Create Systemd Unit File for Kafka

Now you need to create a systemd unit file for Apache Kafka on AlmaLinux 9. To do this, use your favorite text editor, here we use vi:

sudo vi /etc/systemd/system/kafka.service

Add the following content to the file:

Note: Make sure your JAVA_HOME configs are well inputted or Kafka will not start.

[Unit]
Description=Apache Kafka Server
Documentation=http://kafka.apache.org/documentation.html
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
Environment="JAVA_HOME=/usr/lib/jvm/jre-11-openjdk"
ExecStart=/usr/local/kafka-server/bin/kafka-server-start.sh /usr/local/kafka-server/config/server.properties
ExecStop=/usr/local/kafka-server/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

When you are done, save and close the file.

At this point, you need to reload the systemd daemon to apply changes and then start the services by using the commands below:

# sudo systemctl daemon-reload
# sudo systemctl enable --now zookeeper
# sudo systemctl enable --now kafka

Verify your Kafka and Zookeeper services are active and running on AlmaLinux 9:

# sudo systemctl status kafka
Output
● kafka.service - Apache Kafka Server
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset:>
     Active: active (running) since Thu 2023-01-26 02:12:14 EST; 7s ago
       Docs: http://kafka.apache.org/documentation.html
   Main PID: 74791 (java)
      Tasks: 69 (limit: 23609)
     Memory: 332.3M
        CPU: 7.875s
     CGroup: /system.slice/kafka.service
...
# sudo systemctl status zookeeper
Output
● zookeeper.service - Apache Zookeeper Server
     Loaded: loaded (/etc/systemd/system/zookeeper.service; enabled; vendor pre>
     Active: active (running) since Thu 2023-01-26 02:12:09 EST; 42s ago
   Main PID: 74408 (java)
      Tasks: 32 (limit: 23609)
     Memory: 72.1M
        CPU: 2.979s
     CGroup: /system.slice/zookeeper.service
...

Install Cluster Manager for Kafka (CMAK) on AlmaLinux 9

CMAK (previously known as Kafka Manager) is an open-source tool for managing Apache Kafka clusters developed by Yahoo. At this point, you need to clone the CMAK from GitHub by using the command below:

# cd ~ 
# sudo git clone https://github.com/yahoo/CMAK.git
Output
Cloning into 'CMAK'...
remote: Enumerating objects: 6542, done.
remote: Counting objects: 100% (266/266), done.
remote: Compressing objects: 100% (143/143), done.
remote: Total 6542 (delta 150), reused 196 (delta 111), pack-reused 6276
Receiving objects: 100% (6542/6542), 3.96 MiB | 12.41 MiB/s, done.
Resolving deltas: 100% (4211/4211), done.

Configure Cluster Manager for Apache Kafka

At this point, you need to make some configuration changes in the CMAK config file. Open the file with your favorite text editor, here we use vi:

sudo vi ~/CMAK/conf/application.conf

Change cmak.zkhosts=”my.zookeeper.host.com:2181″ and you can also specify multiple zookeeper hosts by comma delimiting them, like so: cmak.zkhosts=”my.zookeeper.host.com:2181,other.zookeeper.host.com:2181. The host names can be ip addresses too.

cmak.zkhosts="localhost:2181"

When you are done, save and close the file.

At his point, you need to create a zip file that can be used to deploy the application. You should see a lot of output on your terminal as files are downloaded and compiled. This will take some time to complete.

# cd ~/CMAK/ 
# ./sbt clean dist

When it is completed, you will get the following output:

Output
[info] Your package is ready in /root/CMAK/target/universal/cmak-3.0.0.7.zip

Change into the directory where the zip file is located and unzip it by using the commands below:

# cd /root/CMAK/target/universal
# unzip cmak-3.0.0.7.zip
# cd cmak-3.0.0.7

Access CMAK Service

When are finished with the previous step, you can run the Cluster Manager for Apache Kafka service on AlmaLinux 9 by using the command below:

bin/cmak

By default, it will choose port 9000, so open your favorite browser and point it to http://ip-or-domain-name-of-server:9000. In case your firewall is running, kindly allow the port to be accessed externally:

sudo firewall-cmd --zone=public --permanent --add-port 9000/tcp

You should see the following interface:

Cluster Manager for Apache Kafka
Cluster Manager for Apache Kafka

Add Cluster From the CMAK

From there, you can easily add the clusters. To do this, click cluster, and add the cluster.

Apache kafka add cluster AlmaLinux 9
Add Cluster

You will be presented with a page as shown below. Fill in the form with the details being requested (Cluster Name, Zookeeper Hosts, etc). In case you have several Zookeeper Hosts, add them delimited by a comma. You can fill in the other details depending on your needs.

Cluster information
Cluster Info

Create a Topic in the CMAK interface

From your newly added cluster, you can click on Topic, and create. You will be required to input all the details you need about the new Topic (Replication Factor, Partitions, and others). Fill in the form then click “Create”.

Cluster topic Kafka
Create Topic

Then, click on Cluster view to see your topics.

Topic info
Topic Summary

From there you can add topics, delete them, config them and etc.

Conclusion

At this point, you have learned to Install and Configure Apache Kafka on AlmaLinux 9.

Hope you enjoy it. You may be like these articles on the Orcacore website:

Install and Configure WordPress on AlmaLinux 9

Install VMware Tools on AlmaLinux 8

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Latest Articles

POPULAR TAGS

Most Popular