Uploaded image for project: 'RecordService (READ-ONLY)'
  1. RecordService (READ-ONLY)
  2. RS-10

Fix memory limit exceeded issue for MR jobs

    Details

      Description

      For MapReduce jobs, the fetch size may impact the memory usage for the RS workers.
      For instance, with 8GB worker memory and 5000 fetch size, a MR job for a 50GB data set will fail. However change the fetch size to 500 the job succeeded.
      Currently, to solve this issue the client needs to adjust the fetch size according to the job type, and perhaps different kinds of workloads may require different fetch sizes. We should solve this problem better, perhaps find a way to dynamically adjust the value, or using a fixed size chunk of shared memory between client and server.

        Attachments

          Activity

            People

            • Assignee:
              lilicn Li Li
              Reporter:
              sunchao Chao Sun
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: