Uploaded image for project: 'Flume (READ-ONLY)'
  1. Flume (READ-ONLY)
  2. FLUME-330

Collector outputing to S3 sink fails and doesn't recover when S3 returns consecutive 404 status code

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: v0.9.1u1
    • Fix Version/s: None
    • Component/s: Node
    • Labels:
      None
    • Environment:
      Ubuntu 9.10 x86

      $ flume version
      Flume 0.9.1+29
      Git repository
       rev 1b753ff62e62a05ca9ac9f3ee8782b5eb8c48da7
      Compiled by bruno on Sun Oct 10 15:25:07 PDT 2010

      Description

      My collector is sourced to autoCollectorSource and sinked to S3N. The collector occasionally fails and doesn't recover until I restart the node. Looking at the agents, they all have backed up logged messages. Once the consumer node is restarted, the agents resume normally and the logged messages slowly deplete. This happens at least a couple times a week.

      Below is relevant log output:
      2010-11-08 09:25:47,451 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/logs%2Fredirector%2Finsertdate%3D2010-11-08%2Finserthour%3D00%2Fweblog-domU-12-31-39-01-5D-24.compute-1.internal-log.00000018.20101108-092546510%2B0000.2386949180521782.seq.gz' - Unexpected response code 404, expected 200
      2010-11-08 09:25:47,470 WARN org.jets3t.service.impl.rest.httpclient.RestS3Service: Response '/logs%2Fredirector%2Finsertdate%3D2010-11-08%2Finserthour%3D00%2Fweblog-domU-12-31-39-01-5D-24.compute-1.internal-log.00000018.20101108-092546510%2B0000.2386949180521782.seq.gz_%24folder%24' - Unexpected response code 404, expected 200

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                pcting Patrick Ting
              • Votes:
                3 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: