According to https://flume.apache.org/
Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.
In this post we will describe how to install Flume and try simple experiments.
Flume is based on java. Please check if java is installed on your system before start installation.
Latest release is available at download page https://flume.apache.org/download.html
Download package apache-flume-1.7.0-bin.tar.gz, uncompress and put it in a folder.
For example path may be C:\flume\apache-flume-1.7.0-bin
After uncompress, open a terminal and cd on flume conf directory, to setup default configuration.
C:\flume\apache-flume-1.7.0-bin\conf>copy flume-conf.properties.template flume-conf.properties C:\flume\apache-flume-1.7.0-bin\conf>copy flume-env.ps1.template flume-env.ps1
Edit log4.properties to enable more log
Open a text editor
# Name the components on this agent a1.sources = r1 a1.sinks = k1 a1.channels = c1 # Describe/configure the source a1.sources.r1.type = netcat a1.sources.r1.bind = localhost a1.sources.r1.port = 44444 # Describe the sink a1.sinks.k1.type = logger # Use a channel which buffers events in memory a1.channels.c1.type = memory a1.channels.c1.capacity = 1000 a1.channels.c1.transactionCapacity = 100 # Bind the source and sink to the channel a1.sources.r1.channels = c1 a1.sinks.k1.channel = c1
Save config file on a path like C:\flume\apache-flume-1.7.0-bin\conf\netcatBasicExample.conf
Make sure JAVA_HOME is set running
if is empty you can set it
set JAVA_HOME=C:\Program Files\Java\jre1.8.0_66
C:\flume\apache-flume-1.7.0-bin>bin\flume-ng agent --conf-file conf\netcatBasicExample.conf -name a1
Open another terminal and run
telnet localhost 44444
Start write on telnet terminal short message and press return, you will find message echo in the other terminal (see screen below)
To kill the agent, press CTRL+C in the terminal, this will automatically shutdown the connection in the telnet terminal.
Enjoy your test