Details
-
Type: Improvement
-
Status: Resolved
-
Priority: Critical
-
Resolution: Fixed
-
Affects Version/s: 0.2.0
-
Fix Version/s: 0.3.0
-
Labels:
Description
For MapReduce jobs, the fetch size may impact the memory usage for the RS workers.
For instance, with 8GB worker memory and 5000 fetch size, a MR job for a 50GB data set will fail. However change the fetch size to 500 the job succeeded.
Currently, to solve this issue the client needs to adjust the fetch size according to the job type, and perhaps different kinds of workloads may require different fetch sizes. We should solve this problem better, perhaps find a way to dynamically adjust the value, or using a fixed size chunk of shared memory between client and server.