Mustafa AKIN

Kafka 2.7.0 Installatin on CentOS 8

2021-01-01T09:00:00+00:00

Introduction

Apache Kafka is a popular distributed message broker designed to efficiently handle large volumes of real-time data. A Kafka cluster is not only highly scalable and fault-tolerant, but it also has a much higher throughput compared to other message brokers such as ActiveMQ and RabbitMQ. Though it is generally used as a publish/subscribe messaging system, a lot of organizations also use it for log aggregation because it offers persistent storage for published messages.

A publish/subscribe messaging system allows one or more producers to publish messages without considering the number of consumers or how they will process the messages. Subscribed clients are notified automatically about updates and the creation of new messages. This system is more efficient and scalable than systems where clients poll periodically to determine if new messages are available.

In this tutorial, you will install and use Apache Kafka 2.7.0 on CentOS 8.

Prerequisites

To follow along, you will need:

One CentOS 8 server and a non-root user with sudo privileges.
At least 4GB of RAM on the server. Installations without this amount of RAM may cause the Kafka service to fail, with the Java virtual machine (JVM) throwing an “Out Of Memory” exception during startup.
OpenJDK 8 installed on your server. To install this version, follow these instructions on installing specific versions of OpenJDK. Kafka is written in Java, so it requires a JVM; however, its startup shell script has a version detection bug that causes it to fail to start with JVM versions above 8.

Step 1 — Creating a User for Kafka and Firewall rules

Firewall Rules

As mentioned above we have to open several ports to allow clients to connect to our Kafka cluster and to allow the nodes to communicate with each other. In CentOS 8 we have to add the corresponding firewall rules.

Let’s start with the firewall rule for ZooKeeper. We have to open ports 2888, 3888 and 2181.

root@bari01$ vi /etc/firewalld/services/zooKeeper.xml

We have to add the following content to the zooKeeper.xml file.

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>ZooKeeper</short>
  <description>Firewall rule for ZooKeeper ports</description>
  <port protocol="tcp" port="2888"/>
  <port protocol="tcp" port="3888"/>
  <port protocol="tcp" port="2181"/>
</service>

For Kafka, we have to open port 9092.

$ vi /etc/firewalld/services/kafka.xml

We have to add the following content to the kafka.xml file.

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>Kafka</short>
  <description>Firewall rule for Kafka port</description>
  <port protocol="tcp" port="9092"/>
</service>

Now we can activate the new firewall rules. Let’s first restart the firewalld service to enforce that all existing service specifications are reloaded.

$ systemctl restart firewalld

We can now permanently add the firewall rules for ZooKeeper and Kafka.

$ firewall-cmd --permanent --add-service=zooKeeper
$ firewall-cmd --permanent --add-service=kafka

After activating the rules we have to restart firewalld.

$ systemctl restart firewalld

Users

Since Kafka can handle requests over a network, you should create a dedicated user for it. This minimizes damage to your CentOS machine should the Kafka server be compromised. We will create a dedicated kafka user in this step, but you should create a different non-root user to perform other tasks on this server once you have finished setting up Kafka.

Logged in as your non-root sudo user, create a user called kafka with the useradd command:

$ sudo useradd kafka -m

The -m flag ensures that a home directory will be created for the user. This home directory, /home/kafka, will act as our workspace directory for executing commands in the sections below.

Set the password using passwd:

$ sudo passwd kafka

Add the kafka user to the wheel group with the adduser command, so that it has the privileges required to install Kafka’s dependencies:

$ sudo usermod -aG wheel kafka

Your kafka user is now ready. Log into this account using su

$ su -l kafka

Now that we’ve created the Kafka-specific user, we can move on to downloading and extracting the Kafka binaries.

Step 2 — Downloading and Extracting the Kafka Binaries

Let’s download and extract the Kafka binaries into dedicated folders in our kafka user’s home directory.

To start, create a directory in /home/kafka called Downloads to store your downloads:

`$ mkdir ~/Downloads

Use curl to download the Kafka binaries:

$ curl "https://downloads.apache.org/kafka/2.7.0/kafka_2.13-2.7.0.tgz" -o ~/Downloads/kafka.tgz

Create a directory called kafka and change to this directory. This will be the base directory of the Kafka installation:

$ mkdir ~/kafka && cd ~/kafka

Extract the archive you downloaded using the tar command:

$ tar -xvzf ~/Downloads/kafka.tgz --strip 1

We specify the --strip 1 flag to ensure that the archive’s contents are extracted in ~/kafka/ itself and not in another directory (such as ~/kafka/kafka_2.11-2.1.1/) inside of it.

Now that we’ve downloaded and extracted the binaries successfully, we can move on configuring to Kafka to allow for topic deletion.

Step 3 — Configuring the Kafka Server

Kafka’s default behavior will not allow us to delete a topic, the category, group, or feed name to which messages can be published. To modify this, let’s edit the configuration file.

Kafka’s configuration options are specified in server.properties. Open this file with vi or your favorite editor:

$ vi ~/kafka/config/server.properties

Let’s add a setting that will allow us to delete Kafka topics. Press i to insert text, and add the following to the bottom of the file:

$ delete.topic.enable = true

When you are finished, press ESC to exit insert mode and :wq to write the changes to the file and quit. Now that we’ve configured Kafka, we can move on to creating systemd unit files for running and enabling it on startup.

Step 4 — Creating Systemd Unit Files and Starting the Kafka Server

In this section, we will create systemd unit files for the Kafka service. This will help us perform common service actions such as starting, stopping, and restarting Kafka in a manner consistent with other Linux services.

Zookeeper is a service that Kafka uses to manage its cluster state and configurations. It is commonly used in many distributed systems as an integral component. If you would like to know more about it, visit the official Zookeeper docs.

Create the unit file for zookeeper:

$ sudo vi /etc/systemd/system/zookeeper.service

Enter the following unit definition into the file:

[Unit]
Requires=network.target remote-fs.target
After=network.target remote-fs.target

[Service]
Type=simple
User=kafka
ExecStart=/home/kafka/kafka/bin/zookeeper-server-start.sh /home/kafka/kafka/config/zookeeper.properties
ExecStop=/home/kafka/kafka/bin/zookeeper-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that Zookeeper requires networking and the filesystem to be ready before it can start.

The [Service] section specifies that systemd should use the zookeeper-server-start.sh and zookeeper-server-stop.sh shell files for starting and stopping the service. It also specifies that Zookeeper should be restarted automatically if it exits abnormally.

Save and close the file when you are finished editing.

Next, create the systemd service file for kafka:

$ sudo vi /etc/systemd/system/kafka.service

Enter the following unit definition into the file:

[Unit]
Requires=zookeeper.service
After=zookeeper.service

[Service]
Type=simple
User=kafka
ExecStart=/bin/sh -c '/home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1'
ExecStop=/home/kafka/kafka/bin/kafka-server-stop.sh
Restart=on-abnormal

[Install]
WantedBy=multi-user.target

The [Unit] section specifies that this unit file depends on zookeeper.service. This will ensure that zookeeper gets started automatically when the kafa service starts.

The [Service] section specifies that systemd should use the kafka-server-start.sh and kafka-server-stop.sh shell files for starting and stopping the service. It also specifies that Kafka should be restarted automatically if it exits abnormally.

Save and close the file when you are finished editing.

Now that the units have been defined, start Kafka with the following command:

$ sudo systemctl start kafka

To ensure that the server has started successfully, check the journal logs for the kafka unit:

$ journalctl -u kafka

You now have a Kafka server listening on port 9092

While we have started the kafka service, if we were to reboot our server, it would not be started automatically. To enable kafka on server boot, run:

$ sudo systemctl enable kafka

Now that we’ve started and enabled the services, let’s check the installation.

Step 5 — Testing the Installation

Let’s publish and consume a “Hello World” message to make sure the Kafka server is behaving correctly. Publishing messages in Kafka requires:

A producer, which enables the publication of records and data to topics.
A consumer, which reads messages and data from topics.

First, create a topic named TutorialTopic by typing:

$ ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic TutorialTopic

You will see the following output:


Created topic TutorialTopic.

You can create a producer from the command line using the kafka-console-producer.sh script. It expects the Kafka server’s hostname, port, and a topic name as arguments.

Publish the string “Hello, World” to the TutorialTopic topic by typing:

$ echo "Hello, World" | ~/kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic TutorialTopic > /dev/null

Next, you can create a Kafka consumer using the kafka-console-consumer.sh script. It expects the ZooKeeper server’s hostname and port, along with a topic name as arguments.

The following command consumes messages from TutorialTopic. Note the use of the –from-beginning flag, which allows the consumption of messages that were published before the consumer was started

$ ~/kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic TutorialTopic --from-beginning

If there are no configuration issues, you should see Hello, World in your terminal:

Hello, World

The script will continue to run, waiting for more messages to be published to the topic. Feel free to open a new terminal and start a producer to publish a few more messages. You should be able to see them all in the consumer’s output.

When you are done testing, press CTRL+C to stop the consumer script. Now that we have tested the installation, let’s move on to installing KafkaT.

Step 6 — Installing KafkaT (Optional)

KafkaT is a tool from Airbnb that makes it easier for you to view details about your Kafka cluster and perform certain administrative tasks from the command line. Because it is a Ruby gem, you will need Ruby to use it. You will also need ruby-devel and build-related packages such as make and gcc to be able to build the other gems it depends on. Install them using yum

$ sudo yum install ruby ruby-devel make gcc patch

You can now install KafkaT using the gem command:

$ sudo gem install kafkat

KafkaT uses .kafkatcfg as the configuration file to determine the installation and log directories of your Kafka server. It should also have an entry pointing KafkaT to your ZooKeeper instance.

Create a new file called .kafkatcfg:

$ vi ~/.kafkatcfg

Add the following lines to specify the required information about your Kafka server and Zookeeper instance:

{
  "kafka_path": "~/kafka",
  "log_path": "/tmp/kafka-logs",
  "zk_path": "localhost:2181"
}

Save and close the file when you are finished editing.

You are now ready to use KafkaT. For a start, here’s how you would use it to view details about all Kafka partitions:

$ kafkat partitions

You will see the following output:

Topic                 Partition   Leader      Replicas        ISRs    
TutorialTopic         0             0         [0]             [0]
__consumer_offsets    0             0         [0]                           [0]
...
...

You will see TutorialTopic, as well as __consumer_offsets, an internal topic used by Kafka for storing client-related information. You can safely ignore lines starting with __consumer_offsets.

To learn more about KafkaT, refer to its GitHub repository.

Step 7 — Setting Up a Multi-Node Cluster (Optional)

If you want to create a multi-broker cluster using more CentOS 8 machines, you should repeat Step 1, Step 4, and Step 5 on each of the new machines. Additionally, you should make the following changes in the server.properties file for each:

The value of the broker.id property should be changed such that it is unique throughout the cluster. This property uniquely identifies each server in the cluster and can have any string as its value. For example, “server1”, “server2”, etc.
The value of the zookeeper.connect property should be changed such that all nodes point to the same ZooKeeper instance. This property specifies the Zookeeper instance’s address and follows the <HOSTNAME/IP_ADDRESS>: format. For example, "203.0.113.0:2181", "203.0.113.1:2181" etc.

If you want to have multiple ZooKeeper instances for your cluster, the value of the zookeeper.connect property on each node should be an identical, comma-separated string listing the IP addresses and port numbers of all the ZooKeeper instances.

if clustur not working check this :

ERROR Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) kafka.common.InconsistentClusterIdException: The Cluster ID jvrDMiqjSWmPEMoAPD9iMQ doesn't match stored clusterId Some(DrtQg1zUTAW4VoYZjb_F7A) in meta.properties. The broker is trying to join the wrong cluster. Configured zookeeper.connect may be wrong.

The Kafka data directory (check config/server.properties for log.dirs property, it defaults to /tmp/kafka-logs) contains a file called meta.properties. It contains the cluster ID. Which should have matched the ID registered to ZK. Either edit the file to match ZK, edit ZK to match the file, or delete the file (it contains the cluster id and the broker id, the first is currently broken and the second is in the config file normally). After this minor surgery, Kafka will start with all your existing data, since you didn’t delete any data file.

Like this: mv /tmp/kafka-logs/meta.properties /tmp/kafka-logs/meta.properties_old

Step 8 — Restricting the Kafka User

Now that all of the installations are done, you can remove the kafka user’s admin privileges. Before you do so, log out and log back in as any other non-root sudo user. If you are still running the same shell session you started this tutorial with, simply type exit.

Remove the kafka user from the sudo group:

sudo gpasswd -d kafka wheel

To further improve your Kafka server’s security, lock the kafka user’s password using the passwd command. This makes sure that nobody can directly log into the server using this account:

$ sudo passwd kafka -l

At this point, only root or a sudo user can log in as kafka by typing in the following command:

$ sudo su - kafka

In the future, if you want to unlock it, use passwd with the -u option:

$ sudo passwd kafka -u

You have now successfully restricted the kafka user’s admin privileges.

jq Command Line

2020-09-30T09:00:00+00:00

./jq

jq is a lightweight and flexible command-line JSON processor.

Json to CSV

$ jq -r '.data[] | [.media.id, .media.name, .url] | @csv' ytblist.json > fileout2.csv

File lists as Json

#!/bin/bash

while getopts u:a:f: flag
do
    case "${flag}" in
        u) username=${OPTARG};;
        h) helper=${OPTARG};;
        f) folder=${OPTARG};;
    esac
done

find $folder -ls | jq -nR '
  # Return an object with useful information
  def gather:
    [splits(" +")] as $in
    | { pathname: $in[-1], entrytype: $in[2][0:1], size: ($in[6] | tonumber) };

  reduce (inputs | gather) as $entry ({};
      ($entry.pathname | split("/") ) as $names
      | if ($entry|.entrytype == "-") then
           ($names[0:-1] + ["items"]) as $p
           | setpath($p; getpath($p) + [{name: $names[-1], size: $entry.size}])
        else . end) '

parameters :

-f foldername start folder name

CentOS Firewall Cmd

2017-08-26T09:00:00+00:00

Centos Firewall Configuration

Services

Adding a Service to specific Zones

The easiest method is to add the services or ports you need to the zones you are using. Again, you can get a list of the available services with the –get-services option:

firewall-cmd --get-services

output

RH-Satellite-6 amanda-client amanda-k5-client bacula bacula-client ceph ceph-mon dhcp dhcpv6 dhcpv6-client dns docker-registry dropbox-lansync freeipa-ldap freeipa-ldaps freeipa-replication ftp high-availability http https imap imaps ipp ipp-client ipsec iscsi-target kadmin kerberos kpasswd ldap ldaps libvirt libvirt-tls mdns mosh mountd ms-wbt mysql nfs ntp openvpn pmcd pmproxy pmwebapi pmwebapis pop3 pop3s postgresql privoxy proxy-dhcp ptp pulseaudio puppetmaster radius rpc-bind rsyncd samba samba-client sane smtp smtps snmp snmptrap squid ssh synergy syslog syslog-tls telnet tftp tftp-client tinc tor-socks transmission-client vdsm vnc-server wbem-https xmpp-bosh xmpp-client xmpp-local xmpp-server

You can get more details about each of these services by looking at their associated .xml file within the /usr/lib/firewalld/services directory. For instance, the SSH service is defined like this:

For instance, if we are running a web server serving conventional HTTP traffic, we can allow this traffic for interfaces in our “public” zone for this session by typing:

sudo firewall-cmd --zone=public --permanent --add-service=http
sudo firewall-cmd --reload

Adding a Port to specific Zones

sudo firewall-cmd --zone=public --permanent --add-port=9090/tcp
sudo firewall-cmd --reload

Robocopy Example Commands

2017-03-21T10:00:00+00:00

Robocopy Examples

Robocopy is a command that is used at the command line to make copies of files and folders. It is also known as Robust File Copy. It comes with windows vista, but was also part of the windows resource kit. It was made to be used for mirroring directories.

Official Documentation

Move all folders and files

SET MoveDirSource=\\path\to\source\
SET MoveDirDestination=D:\path\to\target

MKDIR "%MoveDirDestination%"
FOR    %%i IN ("%MoveDirSource%\*") DO           MOVE /Y "%%i" "%MoveDirDestination%\%%~nxi"
FOR /D %%i IN ("%MoveDirSource%\*") DO ROBOCOPY /MOVE /E "%%i" "%MoveDirDestination%\%%~nxi"

Useful poweshell commands

2017-03-21T09:00:00+00:00

What is PowerShell?

PowerShell is an automation platform and scripting language for Windows and Windows Server that allows you to simplify the management of your systems. Unlike other text-based shells, PowerShell harnesses the power of the .NET Framework, providing rich objects and a massive set of built-in functionality for taking control of your Windows environments.

Official Documentation

Remove Prefix from folder/file name

Get-ChildItem UP-* | foreach {rename-item $_ $_.Name.Replace("UP-", "")}

Find working process

Find working php files :

Get-WmiObject win32_Process | where-object {$_.CommandLine -like "*php*"} | Select-Object ProcessId,CommandLine,ProcessName,CreationDate | Format-List

Output :

ProcessId    : 7340
CommandLine  : "C:\Program Files (x86)\JetBrains\PhpStorm 2016.2.2\bin\PhpStorm64.exe"
ProcessName  : PhpStorm64.exe
CreationDate : 20170320114715.853150+180

Find working php files and disylay on table list

Get-WmiObject win32_Process | where-object {$_.CommandLine -like "*php*"} | Select-Object ProcessId,CommandLine,ProcessName,CreationDate | Format-Table -wrap

Output :

ProcessId CommandLine                                                                                                                                                            ProcessName      CreationDate
--------- -----------                                                                                                                                                            -----------      ------------
     7340 "C:\Program Files (x86)\JetBrains\PhpStorm 2016.2.2\bin\PhpStorm64.exe"                                                                                                PhpStorm64.exe   20170320114715.8
                                                                                                                                                                                                  53150+180
     7772 "C:\Program Files (x86)\JetBrains\PhpStorm 2016.2.2\bin\fsnotifier64.exe"                                                                                              fsnotifier64.exe 20170320114736.5

Kill process working more than 15 min

 (Get-WmiObject win32_Process | where-object {$_.CommandLine -like '*download-file.php*' -and $_.ConvertToDateTime($_.CreationDate) -lt (Get-Date).AddMinutes(-15)}).Invokemethod("terminate", $null)

Capistrano configuration for deploy with gitlab

2017-03-11T10:45:35+00:00

Capistano

Capistrano is written in Ruby, but it can easily be used to deploy any language.

If your language or framework has special deployment requirements, Capistrano can easily be extended to support them.

Configure Server

configure deploy user

sudo groupadd deployers
sudo adduser deploy
sudo usermod -a -G deployers deploy

to give the deployers group the permissions, run the following and edit the /etc/sudoers file: visudo
Add the following line to after the groups:

..
## Allows people in group wheel to run all commands
deploy      ALL=(ALL)NOPASSWD:/bin/chown, /bin/chmod,/bin/mkdir

..

generate ssh-key for deploy user

sudo su - deploy
ssh-keygen -t rsa -b 4096 -C "deploy@example.com"

add new deploy key to gitlab on http://gitlabserver/admin/deploy_keys from new server cat ~/.ssh/id_rsa.pub
enable new deploy-key on gitlab project
copy gitlab-runner key from gitlab-machine su gitlab-runner -c "cat ~/.ssh/id_rsa.pub"
paste gitlab ssh-key to new server via deploy user vi ~/.ssh/authorized_keys
change file mode chmod 644 ~/.ssh/authorized_keys
install git on new server yum -y install git
change project installation folder group chgrp -hR deployers /var/www
change project installation folder permission chmod 775 -R /var/www

first create capistrano config file with cap install command on project root folder.

LEMP-CentOS 7 Installation

2017-03-11T09:45:35+00:00

Prepare Machine

sudo yum install epel-release
sudo yum update

Install Nginx

sudo yum -y install nginx
sudo firewall-cmd --permanent --zone=public --add-service=http 
sudo firewall-cmd --permanent --zone=public --add-service=https
sudo firewall-cmd --reload
sudo systemctl enable nginx
sudo systemctl start nginx

Snmp Basic Configuration

install snmp yum -y install net-snmp net-snmp-utils
keep orginal config file cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.BAK
clear current config file echo "" > /etc/snmp/snmpd.conf and paste below text with changing SERVERHOST

rocommunity  public  *SERVERHOST*
rocommunity  public   127.0.0.1
syslocation  "HUS, DTM"
syscontact  m.akin@medyatakip.com
 
disk /
disk /home
 
includeAllDisks 10%

Open vi /etc/firewalld/services/snmp.xml and copy below text

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>SNMP</short>
  <description>SNMP protocol</description>
  <port protocol="udp" port="161"/>
</service>

reload fierwall firewall-cmd --reload
dd the service to your public zone firewall-cmd --zone=public --add-service snmp --permanent
again reload fierwall firewall-cmd --reload

Install MariaDB

sudo yum install mariadb-server mariadb -y
sudo systemctl enable mariadb
sudo systemctl start mariadb
sudo mysql_secure_installation

Install PHP

PHP Latest Stable For Centos

sudo yum install -y php php-mysql php-fpm php-intl php-common php-opcache php-dom php-mcrypt

PHP 7

rpm -Uvh https://mirror.webtatic.com/yum/el7/webtatic-release.rpm
sudo yum install -y php70w php70w-mysql php70w-fpm php70w-intl php70w-common php70w-opcache php70w-dom php70w-mcrypt

sudo systemctl enable php-fpm
sudo systemctl start php-fpm

Configure PHP

Open sudo vi /etc/php.ini
set ;cgi.fix_pathinfo=1 to cgi.fix_pathinfo=0
set date.timezone = to date.timezone = "Europe/Istanbul"
Open sudo vi /etc/php-fpm.d/www.conf
Find the line that specifies the listen parameter, and change to listen = /var/run/php-fpm/php-fpm.sock
set ;listen.owner = nobody to listen.owner = nobody
set ;listen.group = nobody to listen.group = nobody
set user = apache to user = nginx
set group = apache to group = nginx
restart php-fpm sudo systemctl restart php-fpm

Composer Install

curl -sS https://getcomposer.org/installer | php
chmod +x composer.phar
mv composer.phar /usr/local/bin/composer

Configure Nginx to Process PHP Pages

open sudo vi /etc/nginx/conf.d/default.conf
add below nginx configuration

server {
    listen       80;
    server_name  domainname.com;

    # note that these lines are originally from the "location /" block
    root   /var/www/public;
    index index.php index.html index.htm;
 
    location / {
        try_files $uri $uri/ =404;
    }
    error_page 404 /404.html;
    error_page 500 502 503 504 /50x.html;
    location = /50x.html {
        root /usr/share/nginx/html;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_pass unix:/var/run/php-fpm/php-fpm.sock;
        fastcgi_index index.php;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        include fastcgi_params;
    }
}

be careful about SELINUX