Flume 安装

本文将介绍在 Linux 下安装 Flume。Linux 系统为 Ubuntu 12.04,Flume 版本是 1.7.0。

在安装 Flume 1.7.0 之前,先要了解它对系统的要求:

  1. Java 运行环境 Java 1.7 或更高版本
  2. 足够的内存来配置 sources, channels 和 sinks
  3. 足够的硬盘空间来配置 channels 和 sinks
  4. agent 有用到的目录的读/写权限

安装 Java

在线安装 Java 7

1
2
sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update && sudo apt-get install oracle-java7-installer

配置环境变量

编辑 /etc/profile,新增以下几行:

1
2
3
4
5
export JAVA_HOME=/usr/lib/jvm/java-7-oracle
export PATH=$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=$CLASSPATH:.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib
export FLUME_HOME=/home/happy.zhang/workspace/flume/apache-flume-1.7.0-bin
export PATH=$PATH:FLUME_HOME/bin

修改后保存文件,然后执行

1
source profile

使修改生效。

安装 flume

下载

1
wget http://mirrors.hust.edu.cn/apache/flume/1.7.0/apache-flume-1.7.0-bin.tar.gz

下载之后,对文件解压

1
tar -xzvf apache-flume-1.7.0-bin.tar.gz

即可。

简单示例

解压完成以后,配置文件放在 conf 目录中。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
apache-flume-1.7.0-bin# tree -L 1
.
├── bin
├── CHANGELOG
├── conf
├── DEVNOTES
├── doap_Flume.rdf
├── docs
├── lib
├── LICENSE
├── NOTICE
├── README.md
├── RELEASE-NOTES
└── tools

在 conf 中新建 example.conf 文件,内容如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# example.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444
# Describe the sink
a1.sinks.k1.type = logger
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

编辑好配置文件后,在 apache-flume-1.7.0-bin 目录下启动 flume:

1
# ./bin/flume-ng agent --conf conf --conf-file ./conf/example.conf --name a1 -Dflume.root.logger=INFO,console

新开一个终端,执行以下命令:

1
2
3
4
5
# telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
haha

启动 flume 的终端中会输出一下内容:

1
2016-12-19 18:12:53,452 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 61 68 61 0D haha. }

参考:

  1. System Requirements
  2. ubuntu如何在线安装java7?
  3. A simple example