Flume

According to https://flume.apache.org/

Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. It uses a simple extensible data model that allows for online analytic application.

In this post we will describe how to install Flume and try simple experiments.

Install

Flume is based on java. Please check if java is installed on your system before start installation.

Latest release is available at download page https://flume.apache.org/download.html

 

Download package apache-flume-1.7.0-bin.tar.gz, uncompress and put it in a folder.

Windows

For example path may be C:\flume\apache-flume-1.7.0-bin

After uncompress, open a terminal and cd on flume conf directory, to setup default configuration.

C:\flume\apache-flume-1.7.0-bin\conf>copy flume-conf.properties.template flume-conf.properties 
C:\flume\apache-flume-1.7.0-bin\conf>copy flume-env.ps1.template flume-env.ps1

Edit log4.properties to enable more log

flume.root.logger=DEBUG,console
#flume.root.logger=INFO,LOGFILE

Sample Application

Open a text editor

# Name the components on this agent
a1.sources = r1
a1.sinks = k1
a1.channels = c1

# Describe/configure the source
a1.sources.r1.type = netcat
a1.sources.r1.bind = localhost
a1.sources.r1.port = 44444

# Describe the sink
a1.sinks.k1.type = logger

# Use a channel which buffers events in memory
a1.channels.c1.type = memory
a1.channels.c1.capacity = 1000
a1.channels.c1.transactionCapacity = 100

# Bind the source and sink to the channel
a1.sources.r1.channels = c1
a1.sinks.k1.channel = c1

Save config file on a path like C:\flume\apache-flume-1.7.0-bin\conf\netcatBasicExample.conf

Run

Make sure JAVA_HOME is set running

echo %JAVA_HOME%

if is empty you can set it

set JAVA_HOME=C:\Program Files\Java\jre1.8.0_66

Start Agent

C:\flume\apache-flume-1.7.0-bin>bin\flume-ng agent --conf-file conf\netcatBasicExample.conf -name a1

Open another terminal and run

telnet localhost 44444

Start write on telnet terminal short message and press return, you will find message echo in the other terminal (see screen below)

Screenshot 2017-05-28 19.12.36.png

To kill the agent, press CTRL+C in the terminal, this will automatically shutdown the connection in the telnet terminal.

Enjoy your test

References

https://gudisevablog.wordpress.com/2016/12/08/apache-flume-windows-10/

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s