Uploaded image for project: 'Flume (READ-ONLY)'
  1. Flume (READ-ONLY)
  2. FLUME-383

E2E agent keeps dying when using batch and gzip decorators.

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: v0.9.2
    • Fix Version/s: v0.9.5
    • Component/s: Node
    • Labels:
      None
    • Environment:
      CentOS 5.4
      Java build 1.6.0_13-b03

      Description

      When running in E2E using batch and gzip decorators, the agent dies.

      Configs:
      exec config 'xxx' 'syslogTcp(5140)' '{delay (500) => {batch(10000,30000) =>

      {gzip => autoE2EChain}

      }}'

      exec config 'xxx' 'autoCollectorSource' '{gunzip =>

      {unbatch => collectorSink("hdfs://dc1-had01-m001.dc01.revsci.net:9000/data/pixlogs","","60000")}

      }'

      Link to Flume user group thread
      https://groups.google.com/a/cloudera.org/group/flume-user/browse_thread/thread/c3195f16324f43c8#

      2010-12-06 15:29:57,669 ERROR
      com.cloudera.flume.handlers.batch.BatchingDecorator: IOException when
      ending batch!
      java.io.IOException: java.lang.NullPointerException
      at com.cloudera.flume.agent.durability.NaiveFileWALDeco
      $1.fireError(NaiveFileWALDeco.java:250)
      at
      com.cloudera.flume.core.connector.DirectDriver.fireError(DirectDriver.java:
      216)
      at com.cloudera.flume.core.connector.DirectDriver
      $PumperThread.run(DirectDriver.java:102)
      Caused by: java.lang.NullPointerException
      at
      com.cloudera.flume.handlers.thrift.ThriftEventSink.append(ThriftEventSink.java:
      76)
      at
      com.cloudera.flume.core.BackOffFailOverSink.append(BackOffFailOverSink.java:
      144)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.InsistentOpenDecorator.append(InsistentOpenDecorator.java:
      163)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.StubbornAppendSink.append(StubbornAppendSink.java:
      84)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at com.cloudera.flume.agent.durability.NaiveFileWALDeco
      $AckChecksumRegisterer.append(NaiveFileWALDeco.java:99)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at
      com.cloudera.flume.handlers.debug.LazyOpenDecorator.append(LazyOpenDecorator.java:
      70)
      at
      com.cloudera.flume.core.EventSinkDecorator.append(EventSinkDecorator.java:
      58)
      at com.cloudera.flume.core.connector.DirectDriver
      $PumperThread.run(DirectDriver.java:93)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              giang Giang Nguyen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated: