Uploaded image for project: 'Flume (READ-ONLY)'
  1. Flume (READ-ONLY)
  2. FLUME-182

New Decorator to throttle data and shell command to configure it

    Details

    • Type: New Feature
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: v0.9.2
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      This feature adds a (sink) decorator called "choke decorator" which has the ability to throttle the bytes going across it. Each such decorator is instantiated using an id, for example to put a choke decorator for a (logical) node1 between source and sink we'll use the following syntax:

      node1 : source |

      {choke(cID)=>sink}

      Where cID is a "choke ID" on the physical node containing to node1.

      The master can put a throttle limit on cID using a new command called "settlimit", the syntax is :

      settlimit physicalNodeName cID max_limit

      This command puts a limit on the cId, which means that all the choke decorators on the physical-node with choke id equal to cID will have a collective throttling limit equal to max_limit.

      One also has ability to put a total throttle limit on the physical-node using the following command :
      settlimit physicalNodeName max_limit.

      The settlimit command is implemented using a new RPC between the Master and the Nodes.

      There is a whole policy stuff to follow (in a future Jira); how to divide the physical node limit between different logical-nodes.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                vibhor vibhor bhatt
                Reporter:
                vibhor vibhor bhatt
              • Votes:
                0 Vote for this issue
                Watchers:
                0 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: