大数据Logstash

安装logstash+kibana+elasticsearch+redis搭建集中式日志分析平台

作者：聂勇 欢迎转载，请保留作者信息并说明文章来源！

本文是参考logstash官方文档实践的笔记，搭建环境和所需组件如下：

  • Redhat 5.7 64bit / CentOS 5.x
  • JDK 1.6.0_45
  • logstash 1.3.2 (内带kibana)
  • elasticsearch 0.90.10
  • redis 2.8.4

搭建的集中式日志分析平台流程如下：
Logstash架构示意图

elasticsearch

1、下载elasticsearch。

1
wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.10.tar.gz

2、解压后，进入bin目录。执行如下命令，让elasticsearch以前台方式启动：

1
./elasticsearch -f

[2014-01-16 16:21:31,825][INFO ][node ] [Saint Elmo] version[0.90.10], pid[32269], build[0a5781f/2014-01-10T10:18:37Z]
[2014-01-16 16:21:31,826][INFO ][node ] [Saint Elmo] initializing …
[2014-01-16 16:21:31,836][INFO ][plugins ] [Saint Elmo] loaded [], sites []
[2014-01-16 16:21:35,425][INFO ][node ] [Saint Elmo] initialized
[2014-01-16 16:21:35,425][INFO ][node ] [Saint Elmo] starting …
[2014-01-16 16:21:35,578][INFO ][transport ] [Saint Elmo] bound_address {inet[/0.0.0.0:9300]}, publish_address {inet[/10.0.2.15:9300]}

Redis

1、其安装方式可以参考我的另一篇文章Redis编译安装。

2、进入其bin目录，执行如下命令，使之在控制台输出debug信息：

1
./redis-server --loglevel verbose

[32470] 16 Jan 16:45:57.330 * The server is now ready to accept connections on port 6379
[32470] 16 Jan 16:45:57.330 - 0 clients connected (0 slaves), 283536 bytes in use

logstash日志生成器（shipper）

1、新建一个配置文件：shipper.conf，其内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
input {
    stdin {
        type => "example"
    }
}
output {
    stdout {
        codec => rubydebug
    }
    redis {
        host => "127.0.0.1"
        port => 6379
        data_type => "list"
        key => "logstash"
    }
}

2、启动shipper。执行如下命令：

1
java -jar logstash-1.3.2-flatjar.jar agent -f shipper.conf

终端窗口将出现如下提示信息：

Using milestone 2 output plugin ‘redis’. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}

然后在终端窗口直接按回车，将出现如下信息：

{
“message” => “”,
“@version” => “1”,
“@timestamp” => “2014-01-16T08:15:19.400Z”,
“type” => “example”,
“host” => “redhat”
}

这个json信息将发送给redis， 同时redis的终端窗口将出现类似下面的提示信息：

[32470] 16 Jan 17:09:23.604 - Accepted 127.0.0.1:44640
[32470] 16 Jan 17:09:27.127 - DB 0: 1 keys (0 volatile) in 4 slots HT.
[32470] 16 Jan 17:09:27.127 - 1 clients connected (0 slaves), 304752 bytes in use

logstash日志索引器（indexer）

1、新建一个配置文件：indexer.conf，其内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
input {
  redis {
    host => "127.0.0.1"
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"
    # We use the 'json' codec here because we expect to read
    # json events from redis.
    codec => json
  }
}
output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {
    host => "127.0.0.1"
  }
}

2、启动日志索引器。执行如下命令：

1
java -jar logstash-1.3.2-flatjar.jar agent -f indexer.conf

终端窗口将出现如下提示信息：

Using milestone 2 input plugin ‘redis’. This plugin should be stable, but if you see strange behavior, please let us know! For more information on plugin milestones, see http://logstash.net/docs/1.3.2/plugin-milestones {:level=>:warn}
You are using a deprecated config setting “debug_format” set in stdout. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>”debug_format”, :plugin=>, :level=>:warn}

索引器从Redis接收到信息，在终端窗口会显示类似如下的信息：

{“message”:””,”@version”:”1”,”@timestamp”:”2014-01-16T17:10:03.831+08:00”,”type”:”example”,”host”:”redhat”}{“message”:””,”@version”:”1”,”@timestamp”:”2014-01-16T17:13:20.545+08:00”,”type”:”example”,”host”:”redhat”}

logstash WEB界面（kibana）

1、启动kibana。执行如下命令：

1
java -jar logstash-1.3.2-flatjar.jar web

2、打开浏览器（须支持HTML5），输入地址：http://127.0.0.1:9292/index.html#/dashboard/file/logstash.json。 界面效果如下：
kibana

参考资料

分享到： QQ空间 微信 新浪微博 腾讯微博 人人网