[ELK Stack] ELK 설치 (Elasticsearch, Logstash, Kibana, ELK install)

 

• ELK 설치 (Elasticsearch, Logstash, Kibana, ELK install)

ELK를 설치하기위해 yum repo를 추가 해주어야 합니다. 아래 명령어들을 복사 + 붙여넣기하여 repo를 추가해주시면 됩니다. 만약, 7점대를 설치하고 싶다면 6.x로 되어있는걸 7.x로 변경하시면 7점대의 버전을 다운로드할 수 있습니다.

참고로 elastic repo가 빠를땐 빠른데, 느릴땐 굉장히 느려서 설치만 ELK 설치만 거의 한시간가량 걸릴수도 있습니다.

# kibana repo
cat <<EOF > /etc/yum.repos.d/kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

# elasticsearch repo
cat <<EOF > /etc/yum.repos.d/elasticsearch.repo
[elasticsearch-6.x]
name=Elasticsearch repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

# logstash repo
cat <<EOF > /etc/yum.repos.d/logstash.repo
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

 

위의 yum repo 설정이 완료 되었다면 openjdk 1.8 이상의 버전이 필요하기 때문에 없다면 아래와 같이 openjdk 설치를 먼저 진행해주시고 이후에 elk 설치를 진행 하시면 됩니다.

yum -y install java-1.8.0-openjdk-headless
yum -y install elasticsearch
yum -y install logstash
yum -y install kibana

systemctl start elasticsearch
systemctl start logstash
systemctl start kibana
systemctl enable elasticsearch
systemctl enable logstash
systemctl enable kibana

 

• ELK config 파일 설정

vi /etc/elasticsearch/elasticsearch.yml 파일로 진입하여 아래와 같이 주석을 제거하고 binding할 네트워크 대역을 입력 합니다. 저는 0.0.0.0으로 모든 대역으로 설정 했습니다.

systemctl restart elasticsearch

 

vi /etc/kibana/kibana.yml 파일로 진입하여 elasticsearch와 마찬가지로 바인딩할 네트워크 대역을 설정 합니다.

systemctl restart kibana

 

이후 /etc/logstash/conf.d 디렉토리에 test.conf 와 같은 파일을 생성하고 아래 코드를 삽입 합니다.

input {
  beats {
    port => 5044
    host => "0.0.0.0"
  }
}
filter {
  if [fileset][module] == "apache2" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => ["%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \[%{HTTPDATE:[apache2][access][time]}\] \"%{WORD:[apache2][access][method]} %{DATA:[apache2][access][url]} HTTP/%{NUMBER:[apache2][access][http_version]}\" %{NUMBER:[apache2][access][response_code]} %{NUMBER:[apache2][access][body_sent][bytes]}( \"%{DATA:[apache2][access][referrer]}\")?( \"%{DATA:[apache2][access][agent]}\")?",
          "%{IPORHOST:[apache2][access][remote_ip]} - %{DATA:[apache2][access][user_name]} \\[%{HTTPDATE:[apache2][access][time]}\\] \"-\" %{NUMBER:[apache2][access][response_code]} -" ] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[apache2][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[apache2][access][time]"
      }
      useragent {
        source => "[apache2][access][agent]"
        target => "[apache2][access][user_agent]"
        remove_field => "[apache2][access][agent]"
      }
      geoip {
        source => "[apache2][access][remote_ip]"
        target => "[apache2][access][geoip]"
      }
    }
    else if [fileset][name] == "error" {
      grok {
        match => { "message" => ["\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{LOGLEVEL:[apache2][error][level]}\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message]}",
          "\[%{APACHE_TIME:[apache2][error][timestamp]}\] \[%{DATA:[apache2][error][module]}:%{LOGLEVEL:[apache2][error][level]}\] \[pid %{NUMBER:[apache2][error][pid]}(:tid %{NUMBER:[apache2][error][tid]})?\]( \[client %{IPORHOST:[apache2][error][client]}\])? %{GREEDYDATA:[apache2][error][message1]}" ] }
        pattern_definitions => {
          "APACHE_TIME" => "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
        }
        remove_field => "message"
      }
      mutate {
        rename => { "[apache2][error][message1]" => "[apache2][error][message]" }
      }
      date {
        match => [ "[apache2][error][timestamp]", "EEE MMM dd H:m:s YYYY", "EEE MMM dd H:m:s.SSSSSS YYYY" ]
        remove_field => "[apache2][error][timestamp]"
      }
    }
  }
}
output {
  elasticsearch {
    hosts => localhost
    manage_template => false
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

systemctl restart logstash

 

• ELK 설치 확인

브라우저를 통해 http://서버IP:5601으로 접속 시 아래와 같은 화면이 나오면 정상적으로 설치된 것 입니다.