Details
-
Type:
Epic
-
Status: Patch Available
-
Priority:
Major
-
Resolution: Unresolved
-
Affects Version/s: None
-
Fix Version/s: None
-
Component/s: Master
-
Labels:None
Description
Multi master configurations for high availability and node scalability suffers from several deficiencies. This jira will serve as a coordination point for another implementation of the master's configuration storage mechanism and or a possible re-architecting of is code.
These are some of the core issues:
1) Translations for logical nodes (auto*Chain, logicalSource/Sink) does not work in mutli-master.
2) Flows abstraction is not scalable.
3) ZK-backend needs to be revisited to be scalable (single znodes vs several znodes).
4) OOME error occurs due to ack sharing mechanism.
5) Each master maintains its own command history.
Fundamentally, this is due to two core issues:
1) Not all state information is shared.
2) Current implementation does not have a leader
Attachments
Issue Links
- blocks
-
FLUME-675 autoCollectorSource and auto*Chain/Sinks fail when master is restarted.
-
- Open
-
-
FLUME-674 Flows 2.0
-
- Patch Available
-
-
FLUME-194 masters need to share additional state information for advanced translation features
-
- Open
-
- depends on
-
FLUME-114 Multi Masters not syncing updated configs properly
-
- Open
-
-
FLUME-381 Configs from peer masters are not translated
-
- Open
-
- relates to
-
FLUME-52 Heartbeat code still uses timestamp rather than logical clock to decide if config is up-to-date
-
- Open
-