{"id":17637,"date":"2015-02-28T18:05:28","date_gmt":"2015-02-28T12:35:28","guid":{"rendered":"https:\/\/2thenew.online\/blog\/?p=17637"},"modified":"2024-01-02T17:49:28","modified_gmt":"2024-01-02T12:19:28","slug":"apache-flume-setup-best-practices","status":"publish","type":"post","link":"https:\/\/2thenew.online\/blog\/apache-flume-setup-best-practices\/","title":{"rendered":"Apache Flume : Setup & Best Practices"},"content":{"rendered":"

Apache Flume is an open source project aimed at providing a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large volume of data. It is a complex task when moving data in large volume. We try to minimize the latency in transfer; this is achieved by specifically tweaking the configuration of Flume. First, we’ll see how to setup flume.<\/p>\n

Setting Up Flume :<\/strong><\/em><\/p>\n

    \n
  1. Download flume binary from http:\/\/flume.apache.org\/download.html<\/a><\/li>\n
  2. Extract and put the binary folder in globally accessible place. For e.g. \/usr\/local\/flume\u00a0\u00a0 (Use “tar -xvf apache-flume-1.5.2-bin.tar.gz” <\/em>and “mv apache-flume-1.5.2-bin \/usr\/local\/flume”<\/em>)<\/li>\n
  3. Once done, set the global variables in the “.bashrc<\/em>” file of the user accessing Flume.\"Flume<\/a><\/li>\n
  4. Use “source .bashrc” <\/em>to set the new variables in effect. Test by running command “flume-ng version”.<\/em><\/li>\n
  5. Once the command shows the right version, Flume is set to work on the current system. Please note “flume-ng”<\/em> represents, “Flume Next-Gen<\/strong>“<\/li>\n<\/ol>\n

     <\/p>\n

    Flume Agent is the one which takes care of the whole process of taking data from “source<\/em><\/strong>“, putting it on to the “channel<\/strong><\/em>“, and finally dumping it in the “sink<\/strong><\/em>“. “Sink<\/strong><\/em>” usually is HDFS.<\/em><\/p>\n\"Blog\n

    Basic Configuration<\/strong><\/em> :<\/p>\n

    Apache Flume takes a configuration file every time it runs a task. This task is kept alive in order to listen to any change in the source<\/strong>, and must be terminated manually by the user. A basic configuration to read data taking “local file system”<\/strong> as the source, keeping channel as “<\/em>memory<\/strong>“, and “hdfs<\/strong><\/em>” as sink, could be:<\/p>\n

     <\/p><\/blockquote>\n

    \"Flume<\/a><\/p>\n

    Sources could be anything like an Avro Client being used by Log4JAppender as well.<\/p>\n

    It is worth noting, the above given configuration will produce separate files of 10 records each by default, taking timestamp ( an interceptor <\/strong><\/em>) as the point of reference of last update of the source.<\/p>\n

    The file so created be saved as flume.conf, and can be run as :<\/p>\n

    flume-ng agent -f flume.conf -n source_agent<\/strong><\/em><\/p>\n

    Performance Measures, Issues and Comments:<\/strong><\/em><\/p>\n

      \n
    1. Memory leaks in log4jappender<\/strong> when ingesting just 10000 records.\n
        \n
      • Solved by making thread sleep after every 500 records, thus decreasing the load on the channel<\/li>\n<\/ul>\n<\/li>\n
      • \u00a0GC Memory Leak (Flume level)\n
          \n
        • Solved by keeping transactionCapacity of channel low and capacity of channel high enough<\/li>\n<\/ul>\n<\/li>\n
        • Avro’s Optimized data serialization is not expolited when using log4jappender\n
            \n
          • Solved by directly pushing the file onto avro-client (inbuilt in flume-ng)<\/li>\n<\/ul>\n<\/li>\n
          • Due to memory leaks, could not write more than 12000 records from log4jappender\n
              \n
            • Solved in points 1 & 2<\/li>\n<\/ul>\n<\/li>\n
            • Did performance analysis of data ingestion directly from file as well as avro client of 14000 records when only 10 records rolled up per file in HDFS\n
                \n
              • Using Avro Client\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 \u00a0\u00a0 –> Total time – 00:02:48<\/strong><\/li>\n
              • Directly from File System –> Total time – 00:02:44<\/strong><\/li>\n<\/ul>\n<\/li>\n
              • When using avro client, realtime update to the file was not taken into account\n
                  \n
                • Unsolved problem<\/li>\n<\/ul>\n<\/li>\n
                • When ingesting directly from file, realtime updates were automatically registered, on the basis of timestamp for last modification\n
                    \n
                  • Used the following configuration
                    \nsource_agent.sources.test_source.interceptors = itime
                    \n# http:\/\/flume.apache.org\/FlumeUserGuide.html#timestamp-interceptor
                    \nsource_agent.sources.test_source.interceptors.itime.type = timestamp<\/li>\n<\/ul>\n<\/li>\n
                  • Flume could only create files to write at max 10 records in a single file by default. This decreased the ingestion rate, thus increasing the ingestion time\n
                      \n
                    • Solved by changing the rollCount property of the HDFSSink to the desired number of events\/records per file; this increased the ingestion rate, thus decreasing the ingestion time.<\/li>\n
                    • Also made rollSize and rollInterval as 0, so that they are not used.<\/li>\n<\/ul>\n<\/li>\n
                    • While controlling the channel capacity, encountered Memory Leaks\n
                        \n
                      • Solved by changing the rollCount(HDFSSink) and transactionCapacity(channel) so that the channel is cleared for more data<\/li>\n<\/ul>\n<\/li>\n
                      • Did performance analysis of data ingestion directly from file after applying solutions from 8 & 9 of 20000 records\n
                          \n
                        • Total time – 00:00:04 (<\/strong>Previously around 00:02:30)<\/strong><\/li>\n<\/ul>\n<\/li>\n
                        • How to trigger notification on HDFS update from flume\n
                            \n
                          • Can be potentially solved by Oozie Coordinator<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n

                             <\/p>\n

                            This covers the very basics of setting up Flume and mitigating some of the common issues which one encounters while using it.<\/p>\n","protected":false},"excerpt":{"rendered":"

                            Apache Flume is an open source project aimed at providing a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large volume of data. It is a complex task when moving data in large volume. We try to minimize the latency in transfer; this is achieved by specifically tweaking the configuration of Flume. […]<\/p>\n","protected":false},"author":158,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"iawp_total_views":14,"footnotes":""},"categories":[1395],"tags":[],"class_list":["post-17637","post","type-post","status-publish","format-standard","hentry","category-big-data"],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t