Uploaded image for project: 'Flume (READ-ONLY)'
  1. Flume (READ-ONLY)
  2. FLUME-617

ZK backed multimaster 2.0

          Details

          • Type: Epic
          • Status: Patch Available
          • Priority: Major
          • Resolution: Unresolved
          • Affects Version/s: None
          • Fix Version/s: None
          • Component/s: Master
          • Labels:
            None

            Description

            Multi master configurations for high availability and node scalability suffers from several deficiencies. This jira will serve as a coordination point for another implementation of the master's configuration storage mechanism and or a possible re-architecting of is code.

            These are some of the core issues:

            1) Translations for logical nodes (auto*Chain, logicalSource/Sink) does not work in mutli-master.
            2) Flows abstraction is not scalable.
            3) ZK-backend needs to be revisited to be scalable (single znodes vs several znodes).
            4) OOME error occurs due to ack sharing mechanism.
            5) Each master maintains its own command history.

            Fundamentally, this is due to two core issues:

            1) Not all state information is shared.
            2) Current implementation does not have a leader

              Attachments

                Issue Links

                blocks

                Bug - A problem which impairs or prevents the functions of the product. FLUME-675 autoCollectorSource and auto*Chain/Sinks fail when master is restarted.

                • Critical - Crashes, loss of data, severe memory leak.
                • Open

                Epic - Created by JIRA Software - do not edit or delete. Issue type for a big user story that needs to be broken down. FLUME-674 Flows 2.0

                • Major - Major loss of function.
                • Patch Available

                New Feature - A new feature of the product, which has yet to be developed. FLUME-194 masters need to share additional state information for advanced translation features

                • Major - Major loss of function.
                • Open
                depends on

                Bug - A problem which impairs or prevents the functions of the product. FLUME-114 Multi Masters not syncing updated configs properly

                • Major - Major loss of function.
                • Open

                Bug - A problem which impairs or prevents the functions of the product. FLUME-381 Configs from peer masters are not translated

                • Major - Major loss of function.
                • Open
                relates to

                Bug - A problem which impairs or prevents the functions of the product. FLUME-52 Heartbeat code still uses timestamp rather than logical clock to decide if config is up-to-date

                • Major - Major loss of function.
                • Open

                  Activity

                    People

                    • Assignee:
                      Unassigned
                      Reporter:
                      jon Jonathan Hsieh
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      5 Start watching this issue

                      Dates

                      • Created:
                        Updated: