flume配置多源sink
標(biāo)簽：
大數(shù)據(jù)
一、配置示意图：
二、Flume参数配置说明：
三、问题记录：
滚动生成新文件说明
1.minBlockReplicas=1 该值设为1
参考：flume中sink到hdfs，文件系统频繁产生文件，文件滚动配置不起作用？
http://blog.csdn.net/simonchi/article/details/43231891
2.rollCount/rollSize/rollInterval最好简单配置，（只设置一个，多个不知道能不能成功，其他设为0）
四、配置列表：
a1.sources = r1 a1.sinks = k1 k2 a1.channels = c1 c2 a1.sources.r1.selector.type = replicating# Describe/configure the sourcea1.sources.r1.type = http a1.sources.r1.port = 5140 a1.sources.r1.handler = org.apache.flume.source.http.JSONHandler a1.sources.r1.channels = c1 c2# Use a channel which buffers events in memorya1.sinks.k1.channel = c1 a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 a1.sinks.k2.channel = c2 a1.channels.c2.type = memory a1.channels.c2.capacity = 1000 a1.channels.c2.transactionCapacity = 100######to kafka# Describe the sink k1a1.sinks.k1.type = org.apache.flume.sink.kafka.KafkaSink a1.sinks.k1.topic = testa1.sinks.k1.brokerList = 192.168.206.10:9092 a1.sinks.k1.requiredAcks = 1 a1.sinks.k1.batchSize = 20#####a1 to hdfs###### Describe the sinka1.sinks.k2.type = hdfs a1.sinks.k2.hdfs.path = hdfs://master:9000/flume/%Y%m%d a1.sinks.k2.hdfs.filePrefix = log_%H_%M a1.sinks.k2.hdfs.fileSuffix = .loga1.sinks.k2.hdfs.useLocalTimeStamp = truea1.sinks.k2.hdfs.writeFormat = Text a1.sinks.k2.hdfs.fileType = DataStream####one hour savea1.sinks.k2.hdfs.round = truea1.sinks.k2.hdfs.roundValue = 1 a1.sinks.k2.hdfs.roundUnit = hour#### write new file file 1Ma1.sinks.k2.hdfs.rollInterval = 0 a1.sinks.k2.hdfs.rollSize=1048576 a1.sinks.k2.hdfs.rollCount=0 a1.sinks.k2.hdfs.batchSize = 100 a1.sinks.k2.hdfs.threadsPoolSize = 10 a1.sinks.k2.hdfs.idleTimeout = 0 a1.sinks.k2.hdfs.minBlockReplicas = 1
作者：玄月府的小妖在debug
链接：https://www.jianshu.com/p/7a73a887e2f3
點擊查看更多內(nèi)容
為 TA 點贊
評論
評論
共同學(xué)習(xí)，寫下你的評論
評論加載中...
作者其他優(yōu)質(zhì)文章
正在加載中
感謝您的支持，我會繼續(xù)努力的～
掃碼打賞，你說多少就多少
贊賞金額會直接到老師賬戶
支付方式
打開微信掃一掃，即可進(jìn)行掃碼打賞哦