ELK

ELK

八月 13, 2019

Filebeat + ELK日志收集

目录

[隐藏]

  • 1说明
    • [1.1软件版本]
    • [1.2架构图]
  • 2各组件安装过程
    • 2.1ElasticSearch
      • [2.1.1安装依赖]
      • [2.1.2安装软件]
      • [2.1.3配置环境]
      • [2.1.4启动服务]
      • [2.1.5es集群配置]
    • 2.2Logstash
      • [2.2.1安装软件]
      • [2.2.2配置文件]
      • [2.2.3启动服务]
    • 2.3Kibana
      • [2.3.1安装软件]
      • [2.3.2配置文件]
      • [2.3.3启动服务]
    • 2.4Filebeat
      • [2.4.1安装软件]
      • [2.4.2配置文件]
      • [2.4.3启动服务]
  • 3Logstash过滤规则
    • [3.1日志格式规范]
    • 3.2grok过滤
      • [3.2.1grok配置]
      • [3.2.2自定义正则表达式]
      • [3.2.3grok调试器]
    • 3.3geoip过滤
      • [3.3.1geoip库下载]
      • [3.3.2安装geoip插件]
      • [3.3.3geoip配置]
      • [3.3.4kibana可视化地图展示]
    • [3.4multiline过滤]
  • 4kibana汉化
    • [4.1下载汉化包]
    • [4.2汉化方法]
  • 5x-pack启用和破解
    • [5.1x-pack启用]
    • [5.2设置x-pack密码]
    • [5.3配置kibana]
    • [5.4配置logstash]
    • [5.5x-pack破解]
  • [6定期清除索引]

说明[[编辑]

软件版本[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
本文档仅针对CentOS 7.2以上版本
软件版本
elasticsearch: 6.5.4
logstash: 6.5.4
kibana: 6.5.4
filebeat: 6.5.4

官网下载地址:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.rpm
https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-x86_64.rpm
https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.rpm
https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-6.5.4-x86_64.rpm

架构图[[编辑]

Elk.png

各组件安装过程[[编辑]]

ElasticSearch[[编辑]]

安装依赖[[编辑]]

1
yum install java-1.8.0-openjdk

安装软件[[编辑]]

1
rpm -i https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.rpm

配置环境[[编辑]]

1
2
3
4
mkdir -p /data/db/elasticsearch
mkdir -p /data/logs/elasticsearch
chown elasticsearch.elasticsearch /data/db/elasticsearch
chown elasticsearch.elasticsearch /data/logs/elasticsearch

调整系统参数增大vm.max_map_count到至少262144

1
2
3
sudo vim  /etc/sysctl.conf
添加 vm.max_map_count=262144
sudo sysctl -p

增大文件句柄数至少 65536 ulimit -a查看

1
2
3
sudo vim /etc/security/limits.conf
* soft nofile 65536
* hard nofile 65536

替换配置文件 节点名称和服务器IP根据节点实际情况填写

1
2
3
4
5
6
7
8
cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: XXX
node.name: XXX
path.data: /data/db/elasticsearch
path.logs: /data/logs/elasticsearch
network.host: X.X.X.X
http.port: 9200
EOF

启动服务[[编辑]]

1
2
3
systemctl daemon-reload
systemctl start elasticsearch.service
systemctl enable elasticsearch.service

es集群配置[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
集群配置中最重要的两项是node.name与network.host,每个节点都必须互通。其中node.name是节点名称主要是在Elasticsearch自己的日志加以区分每一个节点信息。
dis-covery.zen.ping.unicast.hosts是集群中的节点信息,可以使用IP地址、可以使用主机名(必须可以解析)。
dis-covery.zen.ping.unicast.hosts: ["Node1_IP:9300", "Node2_IP:9300", "Node3_IP:9300"]

vim /etc/elasticsearch
cluster.name: es-cluster # 集群名称
node.name: es01 # 节点名称,仅仅是描述名称,用于在日志中区分

network.host: X.X.X.X # 当前节点的IP地址
http.port: 9200 # 对外提供服务的端口,9300为集群服务的端口

dis-covery.zen.ping.unicast.hosts: ["172.18.68.11", "172.18.68.12","172.18.68.13"]
# 集群个节点IP地址,也可以使用els、els.shuaiguoxia.com等名称,需要各节点能够解析

dis-covery.zen.minimum_master_nodes: 2 # 为了避免脑裂,集群节点数最少为 半数+1

Logstash[[编辑]]

安装软件[[编辑]]

1
rpm -i https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.rpm

配置文件[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
cat > /etc/logstash/conf.d/logstash.conf << EOF
input {
beats {
port => 5044
}
}

filter {
date {
timezone => "Asia/Shanghai"
match => [ "timestamp", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
}

output {
elasticsearch {
hosts => [ "X.X.X.X:9200" ]
index => "logstash-%{+YYYY.MM.dd}"
}
}
EOF

启动服务[[编辑]]

1
2
systemctl enable logstash.service
systemctl start logstash.service

Kibana[[编辑]]

安装软件[[编辑]]

1
rpm -i https://artifacts.elastic.co/downloads/kibana/kibana-6.5.4-x86_64.rpm

配置文件[[编辑]]

1
2
3
4
5
cat > /etc/kibana/kibana.yml << EOF
server.port: 5601
server.host: 0.0.0.0
elasticsearch.url: "http://X.X.X.X:9200"
EOF

启动服务[[编辑]]

1
2
systemctl enable kibana.service
systemctl start kibana.service

Filebeat[[编辑]]

安装软件[[编辑]]

1
rpm -ivh http://dl.qe23.com:60022/packages/filebeat-6.5.4-x86_64.rpm

配置文件[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
cat > /etc/filebeat/filebeat.yml << EOF
filebeat.prospectors:
- input_type: log
paths:
- /data/logs/nginx/*.log
exclude_lines: ["^DBG","^$"]
fields:
type: nginxlog
fields_under_root: true
exclude_files: [".gz$"]

output.logstash:
hosts: ["X.X.X.X:5044"]
EOF

启动服务[[编辑]]

1
2
systemctl start filebeat.service
systemctl enable filebeat.service

Logstash过滤规则[[编辑]]

日志格式规范[[编辑]]

1
2
3
4
5
6
7
8
9
举例一个nginx日志格式
log_format access '$http_x_forwarded_for $remote_addr $remote_user $host [$time_local] "$request" "$request_body" '
'$status $body_bytes_sent $bytes_sent "$http_referer" $request_length $request_time '
'complete:"$request_completion" "$http_user_agent"';

相应的日志样例
106.226.248.84 121.10.141.6 - wvw.8888.com [03/Jun/2019:19:53:39 +0800] "POST /h5/api/tj.php HTTP/1.0" "_server=30051&sign=4b62af2babff9a88a13ba10d8126a592" 200 32 418 "http://www.baidu.com" 694 0.086 complete:"OK" "Dalvik/2.1.0 (Linux; U; Android 6.0.1; OPPO R9s Build/MMB29M)"

logstash有许多的过滤器,其中grok正则匹配把那些看似毫无意义、非结构化的日志数据解析成可查询的结构化数据,是目前 Logstash 解析过滤的最好方式。

grok过滤[[编辑]]

grok配置[[编辑]]

1
2
3
4
5
6
7
8
9
filter {
grok {
patterns_dir => "/etc/logstash/patterns"
match => { "message" => "%{NGINXACCESS}" }
}
}

patterns_dir指定正则表达式的文件目录,不指定话默认读取安装目录的vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns
match可以配置多个messages,多条正则匹配,第一条匹配失败会往下匹配

自定义正则表达式[[编辑]]

1
2
3
4
5
6
7
8
拷贝一份默认正则到自定义目录,方便修改
mkdir -p /etc/logstash/patterns
cp -raf /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-patterns-core-4.1.2/patterns/grok-patterns /etc/logstash/patterns/grok-patterns

新增grok正则配置 /etc/logstash/patterns/grok-patterns (其中IP_XFF为x-forwarded-for的正则表达式,NGINXACCESS为nginx日志的正则表达式)
WZ ([^ ]*)
IP_XFF [\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\,\s]*
NGINXACCESS %{IP_XFF:http_x_forwarded_for} %{IP:proxy_ip} \- %{WZ:domain} \[%{HTTPDATE:timestamp}\] %{QS:request} %{QS:request_body} %{NUMBER:status} %{NUMBER:bytes} %{NUMBER:bytes_sent} %{QS:referer} %{NUMBER:request_length} %{BASE16FLOAT:request_time} %{DATA:complete} %{QS:agent}

grok调试器[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
grok正则表达式,可在kibana页面的grok调试器进行调试,能够解析为json数据即可,以下是在grok调试器解析上面日志样子结构化输出的结果
{
"request": "\"POST /h5/api/tj.php HTTP/1.0\"",
"referer": "\"http://www.baidu.com\"",
"agent": "\"Dalvik/2.1.0 (Linux; U; Android 6.0.1; OPPO R9s Build/MMB29M)\"",
"bytes_sent": "418",
"request_time": "0.086",
"request_body": "\"_server=30051&sign=4b62af2babff9a88a13ba10d8126a592\"",
"request_length": "694",
"bytes": "32",
"domain": "wvw.8888.com",
"http_x_forwarded_for": "106.226.248.84",
"complete": "complete:\"OK\"",
"proxy_ip": "121.10.141.6",
"timestamp": "03/Jun/2019:19:53:39 +0800",
"status": "200"
}

除了grok过滤器外,还有mutate、date、multiline、geoip等过滤器。

geoip过滤[[编辑]]

geoip库下载[[编辑]]

1
2
3
4
5
GeoIP库可以根据IP地址提供对应的 大洲,国家,省市,经纬度等地域信息,GeoLite是基于IP来确定IP所在物理地址的一个数据库,可定期更新。
wget http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz
mkdir /etc/logstash/GeoLite2/
tar zxvf GeoLite2-City.tar.gz -C /etc/logstash/GeoLite2/
mv /etc/logstash/GeoLite2/GeoLite2-City_20190716/GeoLite2-City.mmdb /etc/logstash/GeoLite2/

安装geoip插件[[编辑]]

1
/usr/share/logstash/bin/logstash-plugin install logstash-filter-geoip

geoip配置[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
filter {
geoip {
source => "client_ip"
database => "/etc/logstash/GeoLite2/GeoLite2-City.mmdb"
add_field => ["[geoip][coordinates]","%{[geoip][longitude]}"]
add_field => ["[geoip][coordinates]","%{[geoip][latitude]}"]
}
mutate {
convert => [ "[geoip][coordinates]", "float"]
}
}

kibana可视化地图展示[[编辑]]

1
2
3
4
5
6
7
8
若要支持中文展示即设置高德地图
kibana.yml配置
timelion.enabled: false
tilemap.url: 'http://webrd02.is.autonavi.com/appmaptile?lang=zh_cn&size=1&scale=1&style=7&x={x}&y={y}&z={z}'

kibana页面添加区域图
Y坐标选择Count请求数
X坐标可选择geoip.region_name.keyword(省份)或geoip.country_name.keyword(国家)来分别展示国内分布和全球分布

Map.png

multiline过滤[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
用于合并数组结构的日志,默认数组结构的日志会被拆分为多行,查询日志时就没可读性,所以需要把每个数组都合并为一条日志
/usr/share/logstash/bin/logstash-plugin install logstash-filter-multiline

新增配置
filter {
multiline {
pattern => "^\[\d{4}-|^\d{4}-"
negate => true
what => "previous"
}
}

kibana汉化[[编辑]]

下载汉化包[[编辑]]

1
2
3
wget -O Kibana_Hanization-master.zip https://codeload.github.com/anbai-inc/Kibana_Hanization/zip/master
unzip Kibana_Hanization-master.zip
cd Kibana_Hanization-master

汉化方法[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
### 汉化方法(6.x)
* 1、拷贝此项目中的translations`文件夹`到您的kibana目录下的`src/legacy/core_plugins/kibana/`目录。若您的kibana无此目录,那还是尝试使用此项目old目录下的汉化方法吧。
* 2、修改您的kibana配置文件kibana.yml中的配置项:i18n.locale: "zh-CN"
* 3、重启Kibana,汉化完成
### 汉化方法(7.x)
* 官方自带汉化资源文件(位于您的kibana目录下的`node_modules/x-pack/plugins/translations/translations/`目录。
* 修改您的kibana配置文件kibana.yml中的配置项:`i18n.locale: "zh-CN"`,重启Kibana则汉化完成。

若上面方法不可用,可采用旧方法
cd Kibana_Hanization-master/old
python main.py Kibana安装目录
**注意:此项目适用于Kibana 5.x-6.x的任意版本,汉化过程不可逆,汉化前请注意备份!** 汉化资源会慢慢更新完善,已汉化过的Kibana可以重复使用此项目汉化更新的资源。除一小部分资源外,大部分资源无需重启Kibana,刷新页面即可看到效果。
意见反馈:redfree@anbai.com Windows请自行安装Python2.7

x-pack启用和破解[[编辑]]

x-pack启用[[编辑]]

1
2
3
4
5
修改/etc/elasticsearch/elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true

systemctl restart elasticsearch.service

设置x-pack密码[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
/usr/share/elasticsearch/bin/elasticsearch-setup-passwords auto
Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user.
The passwords will be randomly generated and printed to the console.
Please confirm that you would like to continue [y/N]y

Changed password for user apm_system
PASSWORD apm_system = C22ZHbO1HqwQKNmi663L

Changed password for user kibana
PASSWORD kibana = 01ZGYU0sHpHEMVaotka1

Changed password for user logstash_system
PASSWORD logstash_system = 3CNP0ghxUa1tI0Ph5b3V

Changed password for user beats_system
PASSWORD beats_system = A7IWv4G1YvqirTF3K8fr

Changed password for user remote_monitoring_user
PASSWORD remote_monitoring_user = gwZYAvPicXi79KH64hdM

Changed password for user elastic
PASSWORD elastic = JZYclzSFZO7HJLKM2111

需要记住elastic密码,用于之后登陆kibana

配置kibana[[编辑]]

1
2
3
4
5
6
修改/etc/kibana/kibana.yml 
xpack.security.enabled: true
elasticsearch.username: "elastic"
elasticsearch.password: "JZYclzSFZO7HJLKM2111"

systemctl restart kibana.service

配置logstash[[编辑]]

1
2
3
4
5
6
7
8
9
修改/etc/logstash/conf.d/logstash.conf
elasticsearch {
user => "elastic"
password => "JZYclzSFZO7HJLKM2111"
hosts => [ "X.X.X.X:9200" ]
index => "logstash-%{+YYYY.MM.dd}"
}

systemctl restart logstash.service

x-pack破解[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
重启elk各个组件,之后登录kibana就会多了用户登录和权限管理以及监控部分的功能,但由于免费版本的x-pack只有一个月的试用liscene,下面给破解一下

license破解
破解前先修改 /etc/elasticsearch/elasticsearch.yml
xpack.security.enabled: false

systemctl restart elasticsearch

cd /usr/share/elasticsearch/modules/x-pack-core
上传改动过的jar包以及json,
x-pack-core-6.5.4.jar 具体修改和jar解包打包这里不说明,网上能找到
liscene.json liscene文件可以去官网申请 https://register.elastic.co/marvel_register
将 "type":"basic" 替换为 "type":"platinum" # 基础班变更为铂金版
将 "expiry_date_in_millis":1561420799999替换为 "expiry_date_in_millis":3107746200000# 1年变为50年


更新许可
curl -H "Content-Type: application/json" -XPUT 'X.X.X.X:9200/_xpack/license?acknowledge=true' -d @license.json
也可以在后台许可管理上传,现在可以看到liscene有效期已经到了2050年了。

Your license will expire on August 16, 2050 10:13 PM CST

定期清除索引[[编辑]]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
定期删除过期索引脚本 
crontab -l
0 1 * * * sh /data/scripts/elk_index_clear.sh

#!/bin/bash

###################################
#删除早于七天的ES集群的索引
###################################
function delete_indices() {
comp_date=`date -d "7 day ago" +"%Y-%m-%d"`
date1="$1 00:00:00"
date2="$comp_date 00:00:00"

t1=`date -d "$date1" +%s`
t2=`date -d "$date2" +%s`

if [ $t1 -le $t2 ]; then
echo "$1时间早于$comp_date,进行索引删除"
#转换一下格式,将类似2017-10-01格式转化为2017.10.01
format_date=`echo $1| sed 's/-/\./g'`
curl -s -XDELETE --user elastic:JZYclzSFZO7HJLKM2LVJ http://10.0.16.3:9200/*$format_date
fi
}

curl -s -XGET --user elastic:JZYclzSFZO7HJLKM2LVJ http://10.0.16.3:9200/_cat/indices | awk -F" " '{print $3}' | awk -F"-" '{print $NF}' | egrep "[0-9]*\.[0-9]*\.[0-9]*" | sort | uniq | sed 's/\./-/g' | while read LINE
do
#调用索引删除函数
delete_indices $LINE
done