Uploaded image for project: 'Flume (READ-ONLY)'
  1. Flume (READ-ONLY)
  2. FLUME-252

Update Tail to get rid of races and truncation problems.

          Details

          • Type: Improvement
          • Status: Closed
          • Priority: Critical
          • Resolution: Fixed
          • Affects Version/s: v0.9.0, v0.9.1
          • Fix Version/s: v0.9.2
          • Component/s: Sinks+Sources
          • Labels:
            None

            Description

            The first tail implementation used buffered readers and file readers. This caused problems because the read call was blocking and couldn't shutdown properly.

            The second tail implementation was mroe closely based on gnu tail's C implementation but relies on RandomAccessFile. This version had problems with races (restarting from beginning FLUME-218) and truncation (new test in FLUME-218 patch by Eric Sammer).

            The new approach will likely use NIO and nonblocking IO to act cleanly, or possibly use a JNI based approach to get to unix system calls to get and follow file descriptors or inode numbers.

              Attachments

                Issue Links

                blocks

                New Feature - A new feature of the product, which has yet to be developed. FLUME-258 TailDir should include source file name metadata for sink

                • Critical - Crashes, loss of data, severe memory leak.
                • Closed
                is duplicated by

                Bug - A problem which impairs or prevents the functions of the product. FLUME-518 Flume is consistently using 70%+ of the CPU

                • Minor - Minor loss of function, or other problem where easy workaround is present.
                • Resolved
                relates to

                Bug - A problem which impairs or prevents the functions of the product. FLUME-148 text source and tail source are incompatible with unix named pipes

                • Major - Major loss of function.
                • Open

                Bug - A problem which impairs or prevents the functions of the product. FLUME-218 race condition in tailsource

                • Blocker - Blocks development and/or testing work, production could not run.
                • Closed

                Bug - A problem which impairs or prevents the functions of the product. FLUME-205 TailSource reads lines using a method(readLine) which does character set interpretation and that breaks all my UTF-8 characters

                • Critical - Crashes, loss of data, severe memory leak.
                • Closed

                Bug - A problem which impairs or prevents the functions of the product. FLUME-248 TailSource not closing files

                • Critical - Crashes, loss of data, severe memory leak.
                • Closed

                Bug - A problem which impairs or prevents the functions of the product. FLUME-320 Race condition in new NIO based TailSource

                • Major - Major loss of function.
                • Resolved

                Bug - A problem which impairs or prevents the functions of the product. FLUME-232 TailSource will add newlines in the middle of log lines

                • Major - Major loss of function.
                • Closed

                  Activity

                    People

                    • Assignee:
                      jon Jonathan Hsieh
                      Reporter:
                      jon Jonathan Hsieh
                    • Votes:
                      2 Vote for this issue
                      Watchers:
                      4 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: