Threat Hunting: Log Monitoring Lab Setup with ELK

Elastic Stack is formerly known as the ELK Stack.

Elk Stack is a collection of free opensource software from Elastic Company which is specially designed for centralized logging. It allows the searching, analyzing, and visualization of logs from different sources. in this guide, we will learn to install Elastic Stack on ubuntu.

To configure ELK Stack in your Ubuntu platform, there are some prerequisites required for installation.

Ubuntu 20.04
Root Privileges

Table of Content

ELK Stack components
Install Java and All Dependencies
Install and configure Elasticsearch
Install and configure Logstash
Install and configure Kibana
Install and configure NGINX
Install and configure Filebeat
Routing Linux Logs to Elasticsearch
Create a Log Dashboard in Kibana
Monitoring SSH entries

ELK Stack components

Elasticsearch: It is a restful search engine that stores or holds all of the collected Data.
Logstash: It is the Data processing component that sends incoming Data to Elasticsearch.
Kibana: A web interface for searching and visualizing logs.
Filebeat: A lightweight Single-purpose Data forwarder that can send data from thousands of machines to either Logstash or Elasticsearch.

Install Java and All Dependencies

Elasticsearch requires OpenJDK available in our machine. Install Java using the below command along with the HTTPS support and wget packages for APT.

<br />
apt install -y openjdk-11-jdk wget apt-transport-https curl

apt install –y openjdk–11–jdk wget apt–transport–https curl

Now, we are going to import Elasticsearch public key into APT. To import the GPG key enter the following command:

<br />
wget -qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

wget –qO – https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add –

Add Elastic repository to the directory sources.list.d by using the following command :

<br />
echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

echo “deb https://artifacts.elastic.co/packages/6.x/apt stable main” | sudo tee –a /etc/apt/sources.list.d/elastic–6.x.list

Install and configure Elasticsearch

Update the system repository

<br />
apt update

apt update

Install Elasticsearch by using the following command:

<br />
apt install elasticsearch

apt install elasticsearch

Next, we configure Elasticsearch.

Elasticsearch listens for traffic on port 9200. We are going to restrict outside access to our Elasticsearch instance so that outside parties cannot access data or shut down the elastic cluster through the REST API. Now we’re going to do some modifications to the Elasticsearch configuration file – elasticsearch.yml.

Enter the following command:

<br />
nano /etc/elasticsearch/elasticsearch.yml

nano /etc/elasticsearch/elasticsearch.yml

Find the line that specifies network.host attribute and uncomment it and add localhost as its value and also uncomment http.port attribute.

<br />
network.host: localhost<br />
http.port: 9200

network.host: localhost

http.port: 9200

Now, start and enable Elasticsearch services.

<br />
systemctl start elasticsearch<br />
systemctl enable elasticsearch

systemctl start elasticsearch

systemctl enable elasticsearch

Let’s verify the status if Elasticsearch.

<br />
systemctl status elasticsearch<br />
curl -X GET “localhost:9200”

systemctl status elasticsearch

curl –X GET “localhost:9200”

By default Elasticsearch is listening on the port 9200 you can also verify it on your web browser by pinging https://localhost:9200

Now Elasticsearch is up and running.

Install and configure Logstash

Logstash used to collect and centralizing logs from different servers using filebeat

First Let’s confirm OpenSSL is running and then install Logstash by running following command:

<br />
openssl version -a<br />
apt install logstash -y

openssl version –a

apt install logstash –y

Edit the /etc/hosts file and add the following line

<br />
nano /etc/hosts

nano /etc/hosts

Where 18.224.44.11 is ip address of server elk-master.

Let’s generate an SSL certificate to secure the log data transfer from the client Rsyslog & Filebeat to the Logstash server.

To do this create a new SSL directory under Logstash configuration directory and navigate into that directory generate an SSL certificate by running following command:

<br />
mkdir -p /etc/logstash/ssl<br />
cd /etc/logstash/<br />
openssl req -subj ‘/CN=elk-master/’ -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout ssl/logstash-forwarder.key -out ssl/logstash-forwarder.crt

mkdir –p /etc/logstash/ssl

cd /etc/logstash/

openssl req –subj ‘/CN=elk-master/’ –x509 –days 3650 –batch –nodes –newkey rsa:2048 –keyout ssl/logstash–forwarder.key –out ssl/logstash–forwarder.crt

Now, we are going to create new configuration files for Logstash named ‘filebeat-input.conf’ as input file from filebeat ‘syslog-filter.conf’ for system logs processing, and ‘output-elasicsearch.conf’ file to define Elasticsearch output.

Navigate to Logstash directory create a file ‘filebeat-input.conf’ in conf.d directory by running command

<br />
cd /etc/logstash/<br />
nano conf.d/filebeat-input.conf

cd /etc/logstash/

nano conf.d/filebeat–input.conf

and paste the following configuration

<br />
input {<br />
  beats {<br />
    port => 5443<br />
    type => syslog<br />
    ssl => true<br />
    ssl_certificate => “/etc/logstash/ssl/logstash-forwarder.crt”<br />
    ssl_key => “/etc/logstash/ssl/logstash-forwarder.key”<br />
  }<br />
}

beats {

port => 5443

type => syslog

ssl => true

ssl_certificate => “/etc/logstash/ssl/logstash-forwarder.crt”

ssl_key => “/etc/logstash/ssl/logstash-forwarder.key”

For the system log data processing, we are going to use a filter plugin named ‘grok’. Create a new conf. file ‘syslog-filter.conf in the same directory

<br />
nano conf.d/syslog-filter.conf

nano conf.d/syslog–filter.conf

and paste the following configuration lines

<br />
filter {<br />
  if [type] == “syslog” {<br />
    grok {<br />
      match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}” }<br />
      add_field => [ “received_at”, “%{@timestamp}” ]<br />
      add_field => [ “received_from”, “%{host}” ]<br />
    }<br />
    date {<br />
      match => [ “syslog_timestamp”, “MMM  d HH:mm:ss”, “MMM dd HH:mm:ss” ]<br />
    }<br />
  }<br />
}

if [type] == “syslog” {

grok {

match => { “message” => “%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}” }

add_field => [ “received_at”, “%{@timestamp}” ]

add_field => [ “received_from”, “%{host}” ]

date {

match => [ “syslog_timestamp”, “MMM d HH:mm:ss”, “MMM dd HH:mm:ss” ]

And at last create a configuration file ‘output-elasticsearch.conf’ for the output of elasticsearch.

nano conf.d/output-elasticsearch.conf

and paste the following configuration

<br />
output {<br />
  elasticsearch { hosts => [“localhost:9200”]<br />
    hosts => “localhost:9200”<br />
    manage_template => false<br />
    index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”<br />
    document_type => “%{[@metadata][type]}”<br />
  }<br />
}

elasticsearch { hosts => [“localhost:9200”]

hosts => “localhost:9200”

manage_template => false

index => “%{[@metadata][beat]}-%{+YYYY.MM.dd}”

document_type => “%{[@metadata][type]}”

And at last, save and exit.

Now start, enable & verify the status of Logstash service.

<br />
systemctl start logstash<br />
systemctl enable logstash<br />
systemctl status logstash

systemctl start logstash

systemctl enable logstash

systemctl status logstash

Install and configure Kibana

Install Kibana by using the following command

<br />
apt install kibana

apt install kibana

We are going to do some modifications to the kibana configuration file.

<br />
nano /etc/kibana/kibana.yml

nano /etc/kibana/kibana.yml

Locate and uncomment the following Attributes

Now start & enable the kibana service:

<br />
systemctl enable kibana<br />
systemctl start kibana

systemctl enable kibana

systemctl start kibana

Install and configure NGINX

Install Nginx and ‘Apache2-utlis’

<br />
apt install nginx apache2-utils -y

apt install nginx apache2–utils –y

Now, create a new virtual host file named Kibana.

<br />
nano /etc/nginx/sites-available/kibana

nano /etc/nginx/sites–available/kibana

and paste the following configuration Into the file.

<br />
server {<br />
    listen 80;<br />
    server_name localhost;<br />
    auth_basic “Restricted Access”;<br />
    auth_basic_user_file /etc/nginx/.kibana-user;<br />
    location / {<br />
        proxy_pass https://localhost:5601;<br />
        proxy_http_version 1.1;<br />
        proxy_set_header Upgrade $http_upgrade;<br />
        proxy_set_header Connection ‘upgrade’;<br />
        proxy_set_header Host $host;<br />
        proxy_cache_bypass $http_upgrade;<br />
    }<br />
}

listen 80;

server_name localhost;

auth_basic “Restricted Access”;

auth_basic_user_file /etc/nginx/.kibana–user;

location / {

proxy_pass https://localhost:5601;

proxy_http_version 1.1;

proxy_set_header Upgrade $http_upgrade;

proxy_set_header Connection ‘upgrade’;

proxy_set_header Host $host;

proxy_cache_bypass $http_upgrade;

Let’s create authentication for the Kibana Dashboard and activate the Kibana virtual host configuration and test Nginx configuration after that enable & restart the Nginx service by using the following command.

<br />
sudo htpasswd -c /etc/nginx/.kibana-user elastic<br />
ln -s /etc/nginx/sites-available/kibana /etc/nginx/sites-enabled/<br />
nginx -t<br />
systemctl enable nginx<br />
systemctl restart nginx

sudo htpasswd –c /etc/nginx/.kibana–user elastic

ln –s /etc/nginx/sites–available/kibana /etc/nginx/sites–enabled/

systemctl enable nginx

systemctl restart nginx

Install and configure Filebeat

We’re going to configure filebeat data shippers on our elk-master server. This will be used to collect data from various sources and transport them to Logstash and Elasticsearch.

Download & Install filebeat by running the following command.

<br />
curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.8.11-amd64.deb

curl –L –O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.8.11-amd64.deb

Let’s repackage the downloaded file by using the following command:

<br />
sudo dpkg -i filebeat-6.8.11-amd64.deb

sudo dpkg –i filebeat–6.8.11–amd64.deb

Next, open the filebeat configuration file named ‘filebeat.yml’

<br />
nano /etc/filebeat/filebeat.yml

nano /etc/filebeat/filebeat.yml

Edit the configuration file:

we’re going to use Elasticsearch to perform additional processing on data collected by filebeat. Therefore, Enable the filebeat prospectors by changing the ‘enabled’ line value to ‘true’.

Next head to the Elasticsearch output section and add the following lines

output.elasticsearch:

<br />
hosts: [“192.168.0.156:9200”]<br />
username: “elastic”<br />
password: “123”<br />
setup.kibana:<br />
host: “192.168.0.156:5601”

hosts: [“192.168.0.156:9200”]

username: “elastic”

password: “123”

setup.kibana:

host: “192.168.0.156:5601”

Enable and configure the Elasticsearch module by running following command

sudo filebeat modules enable elasticsearch

Let’s start filebeat

<br />
sudo filebeat setup<br />
sudo service filebeat start

sudo filebeat setup

sudo service filebeat start

And at last copy the Logstash certificate file – logstash-forwarder.crt – to /etc/filebeat directory by running following command:

<br />
cp /etc/logstash/ssl/logstash-forwarder.crt /etc/filebeat/<br />
sudo service filebeat restart

cp /etc/logstash/ssl/logstash–forwarder.crt /etc/filebeat/

sudo service filebeat restart

To test ELK stack open your browser and browse your server ip address followed by port 5601

<br />
https://localhost:5601

https://localhost:5601

Routing Linux Logs to Elasticsearch

We’re routing logs from rsyslog to Logstash and these logs transferred to Elasticsearch automatically

Routing From Logstash To Elasticsearch

Before routing logs from rsyslog to Logstash firstly we need to set up log forwarding between Logstash and Elasticsearch.

To do this we’re going to create a configuration file for Logstash. To create configuration file head over towards the directory /etc/logstash/conf.d and create a logstash.conf file

<br />
cd /etc/logstash/conf.d<br />
nano logstash.conf

cd /etc/logstash/conf.d

nano logstash.conf

paste the following configuration into the logstash.conf file

<br />
input {                                                                                     <br />
  udp {                                                                                     <br />
    host => “127.0.0.1”                                                                     <br />
    port => 10514                                                                            <br />
    codec => “json”                                                                         <br />
    type => “rsyslog”                                                                       <br />
  }                                                                                          <br />
}    <br />
                                                                                                                                                         <br />
# The Filter pipeline stays empty here, no formatting is done.    <br />
filter { }                                                                                       <br />
# Every single log will be forwarded to ElasticSearch. If you are using another port, you should specify it here.                                                                                             <br />
output {                                                                                    <br />
  if [type] == “rsyslog” {                                                                  <br />
    elasticsearch {                                                                          <br />
      hosts => [ “127.0.0.1:9200” ]                                                         <br />
    }                                                                                       <br />
  } }

input {

udp {

host => “127.0.0.1”

port => 10514

codec => “json”

type => “rsyslog”

}

# The Filter pipeline stays empty here, no formatting is done.

filter { }

# Every single log will be forwarded to ElasticSearch. If you are using another port, you should specify it here.

output {

if [type] == “rsyslog” {

elasticsearch {

hosts => [ “127.0.0.1:9200” ]

}

Restart the Logstash service.

<br />
systemctl restart logstash

systemctl restart logstash

Let’s check that everything is running correctly issue the following command:

<br />
netstat -na | grep 10514

netstat –na | grep 10514

Routing from rsyslog to Logstash

Rsyslog has the capacity to transform logs using templates in order to forward logs in rsylog, head over to the directory /etc/rsylog.d and create a new file named 70-output.conf

<br />
cd /etc/rsyslog.d<br />
nano 70-output.conf

cd /etc/rsyslog.d

nano 70–output.conf

And paste the following configuration into the 70-output.conf file

<br />
# This line sends all lines to defined IP address at port 10514<br />
# using the json-template format.<br />
*.*                         @127.0.0.1:10514;json-template

# This line sends all lines to defined IP address at port 10514

# using the json-template format.

*.* @127.0.0.1:10514;json–template

Now we have log forwarding, create a 01-json-template.conf file in the same folder

<br />
nano 01-json-template.conf

nano 01–json–template.conf

And paste the following configuration into the 01-json-template.conf file

<br />
template(name=”json-template”<br />
  type=”list”) {<br />
    constant(value=”{“)<br />
      constant(value=”\”@timestamp\”:\””)     property(name=”timereported” dateFormat=”rfc3339″)<br />
      constant(value=”\”,\”@version\”:\”1″)<br />
      constant(value=”\”,\”message\”:\””)     property(name=”msg” format=”json”)<br />
      constant(value=”\”,\”sysloghost\”:\””)  property(name=”hostname”)<br />
      constant(value=”\”,\”severity\”:\””)    property(name=”syslogseverity-text”)<br />
      constant(value=”\”,\”facility\”:\””)    property(name=”syslogfacility-text”)<br />
      constant(value=”\”,\”programname\”:\””) property(name=”programname”)<br />
      constant(value=”\”,\”procid\”:\””)      property(name=”procid”)<br />
    constant(value=”\”}\n”)<br />
}

template(name=“json-template”

type=“list”) {

constant(value=“{“)

constant(value=“\”@timestamp\”:\””) property(name=“timereported” dateFormat=“rfc3339”)

constant(value=“\”,\”@version\”:\”1″)

constant(value=“\”,\”message\”:\””) property(name=“msg” format=“json”)

constant(value=“\”,\”sysloghost\”:\””) property(name=“hostname”)

constant(value=“\”,\”severity\”:\””) property(name=“syslogseverity-text”)

constant(value=“\”,\”facility\”:\””) property(name=“syslogfacility-text”)

constant(value=“\”,\”programname\”:\””) property(name=“programname”)

constant(value=“\”,\”procid\”:\””) property(name=“procid”)

constant(value=“\”}\n”)

Restart rsyslog service and verify that logs are correctly forwarded into Elasticsearch.

<br />
systemctl restart rsyslog<br />
curl -XGET ‘http://localhost:9200/logstash-*/_search?q=*&pretty’

systemctl restart rsyslog

curl –XGET ‘http://localhost:9200/logstash-*/_search?q=*&pretty’

Note:- Logs will be forwarded in an index named logstash-*.

Create a Log Dashboard in Kibana

Open your browser and head over to https://localhost:5601 and you should see the following screen.

Go to the management section and create an index pattern called logstash-* and proceed for the next step.

we’ve defined logstash-* as our index pattern. Now we can specify some settings before we create it. In the field of time filter field name choose @timestamp and create an index pattern

Monitoring SSH entries

This one is a little bit special, as we can go into the “Discover” tab in order to build our panel.

When entering the discover tab, select logstash-*

From there, in the fiterbar, put a query filter “programename:ssh*”.

Now we can see every log related to the SSHd service in our machine.

As we can see, now we have direct access to every log related to the SSHd service. we can for example track illegal access attempts or wrong logins.

Similarly, we can monitor various illegal access attempts or wrong logins like ftp, telnet etc…

For example, I took Telnet access to my server from a different machine.

Let’s check what happens on the Kibana dashboard.

Hold tight!

Nice! Now your panel is included in your dashboard.

Author – Vijay is a Certified Ethical Hacker, Technical writer and Penetration Tester at Hacking Articles. Technology and Gadget freak. Contact Here

F1TYM1

Threat Hunting: Log Monitoring Lab Setup with ELK

Table of Content

ELK Stack components

Install Java and All Dependencies

Install and configure Elasticsearch

Install and configure Logstash

Install and configure Kibana

Install and configure NGINX

Install and configure Filebeat

Routing Linux Logs to Elasticsearch

Routing from rsyslog to Logstash

Create a Log Dashboard in Kibana

Monitoring SSH entries

Like this:

Threat Hunting: Log Monitoring Lab Setup with ELK

Table of Content

ELK Stack components

Install Java and All Dependencies

Install and configure Elasticsearch

Install and configure Logstash

Install and configure Kibana

Install and configure NGINX

Install and configure Filebeat

Routing Linux Logs to Elasticsearch

Routing from rsyslog to Logstash

Create a Log Dashboard in Kibana

Monitoring SSH entries

Share this:

Like this: