Details
Description
This Out of Memory happens when you run large number of jobs (using the distributed cache) to a TaskTracker. Seems the basic issue is with the distributedCacheManager (instance of TrackerDistributedCacheManager in TaskTracker.java), this gets created during TaskTracker.initialize(), and it keeps references to TaskDistributedCacheManager for every submitted job via the jobArchives Map, also references to CacheStatus via cachedArchives map. I am not seeing these cleaned up between jobs, so this can cause memory problems when really large number of jobs are submitted. I am suspecting that this is the cause for the particular problem discussed in this thread: https://groups.google.com/a/cloudera.org/group/cdh-user/browse_thread/thread/e678a3e85ef95beb/157aaa94a5a901d8?pli=1#157aaa94a5a901d8
Attachments
Issue Links
- duplicates
-
DISTRO-336 Memory leak in TrackerDistributeCacheManager
-
- Resolved
-