flume+kafka+storm单机部署

flume-1.6.0

kafka0.9.0.0

storm0.9.6
一.部署flume
1.解压
tar -xzvf apache-flume-1.6.0-bin.tar.gz -C ../app/
2.在conf目录中添加自己的配置文件，内容是source、sink、channel的配置
cd ../app/apache-flume-1.6.0-bin/conf
新建配置文件flume-directory-kafka.conf，内容如下
# flume-directory-kafka.conf: A single-node Flume configuration
# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1
# Describe/configure the source
a1.sources.r1.type = spooldir
a1.sources.r1.channels = c1
a1.sources.r1.spoolDir = /home/koushengrui/flumeDemoLog
a1.sources.r1.fileHeader = true
# Describe the sink
a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink
a1.sinks.k1.topic = webService_log
a1.sinks.k1.brokerList = montnets:9092
a1.sinks.k1.requiredAcks = 1
a1.sinks.k1.batchSize = 20
a1.sinks.k1.channel = c1
# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100
# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1
3.启动flume
必须在bin目录以上级别目录启动，在bin目录启动无效，因为log4j.properties文件在跟bin目录同级的conf目录中
./bin/flume-ng agent --conf conf --conf-file ./conf/flume-directory-kafka.conf --name a1 -Dflume.root.logger=INFO,console
二.部署zookeeper并启动
三.部署kafka
1.解压
tar -xzvf kafka_2.11-0.9.0.0.tgz -C ../app/
2.修改conf目录中的server.properties文件
cd ../app/kafka_2.11-0.9.0.0/config/
vi ./server.properties
broker.id=1
zookeeper.connect=montnets:2181
3.在每一台节点上启动broker
cd ../bin
./kafka-server-start.sh ../config/server.properties
4.查看现有topic的状态
./kafka-topics.sh --list --zookeeper montnets:2181
如果之前有topic的话，此时会显示出来
5.新建对应flume的配置文件的topic  webService_log
./kafka-topics.sh --create --zookeeper montnets:2181 --replication-factor 1 --partitions 1 --topic webService_log
6.此时就可以校验flume与kafka是否整合成功了。用一个comsumer从webService_log中读取信息
./kafka-console-consumer.sh --zookeeper montnets:2181 --from-beginning --topic webService_log
在/home/koushengrui/flumeDemoLog目录中添加一个内容如下的名为log1的文件，观察是否在kafka的控制台输出了文件信息。经校验，整合成功。
<EVTCONT>
[{USERID:"用户账号1",MSG:"这里填写xxx帐号离线等详情" }]
</EVTCONT>
四.部署storm
1.解压
tar -xzvf apache-storm-0.9.6.tar.gz -C ../app/
2.修改conf目录下的storm.yaml文件
cd ../app/apache-storm-0.9.6/conf/
vi ./storm.yaml
#所使用的zookeeper集群主机
storm.zookeeper.servers:
- "montnets"
#nimbus所在的主机名
nimbus.host: "montnets"
3.启动storm
分别要启动nimbus及supervisor
在nimbus主机上
./storm nimbus
./storm ui 提供web页面,通过http://montnets:8080/可以访问Storm UI。
在supervisor主机上
./storm supervisor
五.把项目打jar包上传，此处命名jar包为demoTopo，其中入口类(有main方法的类)为com.montnets.test.Entrance
./storm jar ../demoTopo.jar com.montnets.test.Entrance
./storm kill topologyName