[RS-60] Flakiness in RS VM - Cloudera Open Source

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 0.2.0
Fix Version/s: 0.3.0
Component/s: VM
Labels:
- Flaky
- VM
Environment:
VM

Description

There is some flakiness in the VM where running a MR job can fail due to RS being unable to find any registered workers. Dennis found the issue by doing the following:

[cloudera@quickstart ~]$ sudo groupadd demogroup

[cloudera@quickstart ~]$ sudo usermod -a -G demogroup $USER

[cloudera@quickstart ~]$ sudo -u impala impala-shell

Starting Impala Shell without Kerberos authentication

Connected to quickstart.cloudera:21000

Server version: impalad version 2.3.0-cdh5.5.0 RELEASE (build 0c891d79aa38f297d244855a32f1e17280e2129b)

***********************************************************************************

(Impala Shell v2.3.0-cdh5.5.0 (0c891d7) built on Mon Nov 9 12:18:12 PST 2015)

To see live updates on a query's progress, run 'set LIVE_SUMMARY=1;'.

***********************************************************************************

[quickstart.cloudera:21000] > create role demorole;

Query: create role demorole

ERROR: AnalysisException: Role 'demorole' already exists.

[quickstart.cloudera:21000] > grant role demorole to group demogroup;

Query: grant role demorole to group demogroup

Fetched 0 row(s) in 0.10s

[quickstart.cloudera:21000] > grant select (firstname, lastname, phonenumber) on table rs.employees to role demorole;

Query: grant select (firstname, lastname, phonenumber) on table rs.employees to role demorole

Fetched 0 row(s) in 4.31s

[quickstart.cloudera:21000] > exit;

Goodbye impala

[cloudera@quickstart ~]$ hadoop jar ./recordservice-client-0.1/lib/recordservice-examples-0.1.jar com.cloudera.recordservice.examples.mapreduce.RecordCount "SELECT * from rs.employees" "/tmp/recordcount_output"

15/12/07 11:25:22 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

15/12/07 11:25:23 INFO core.ThriftUtils: Connecting to RecordServicePlanner at localhost:12050, with timeout: 30000ms

15/12/07 11:25:23 INFO core.ThriftUtils: Connected to RecordServicePlanner at localhost:12050

15/12/07 11:25:23 INFO core.RecordServicePlannerClient: Planning request: TPlanRequestParams(client_version:1.0, request_type:Sql, sql_stmt:SELECT * from rs.employees) with attempt 1/3. timeout= 120000ms

15/12/07 11:25:23 INFO core.RecordServicePlannerClient: Closing RecordServicePlanner connection.

15/12/07 11:25:23 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/cloudera/.staging/job_1449259187792_0005

15/12/07 11:25:23 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.io.IOException: com.cloudera.recordservice.core.RecordServiceException: TRecordServiceException(code:INVALID_REQUEST, message:Worker membership is empty. Please ensure all RecordService Worker nodes are running.)

Exception in thread "main" java.io.IOException: com.cloudera.recordservice.core.RecordServiceException: TRecordServiceException(code:INVALID_REQUEST, message:Worker membership is empty. Please ensure all RecordService Worker nodes are running.)

at com.cloudera.recordservice.mr.PlanUtil.getSplits(PlanUtil.java:235)

at com.cloudera.recordservice.mapreduce.RecordServiceInputFormatBase.getSplits(RecordServiceInputFormatBase.java:47)

at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:304)

at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:321)

at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:199)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1307)

at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1304)

at java.security.AccessController.doPrivileged(Native Method)

at javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

at org.apache.hadoop.mapreduce.Job.submit(Job.java:1304)

at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1325)

at com.cloudera.recordservice.examples.mapreduce.RecordCount.run(RecordCount.java:100)

at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)

at com.cloudera.recordservice.examples.mapreduce.RecordCount.main(RecordCount.java:104)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:606)

at org.apache.hadoop.util.RunJar.run(RunJar.java:221)

at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: com.cloudera.recordservice.core.RecordServiceException: TRecordServiceException(code:INVALID_REQUEST, message:Worker membership is empty. Please ensure all RecordService Worker nodes are running.)

at com.cloudera.recordservice.core.RecordServicePlannerClient.planRequest(RecordServicePlannerClient.java:317)

at com.cloudera.recordservice.mr.PlanUtil.getSplits(PlanUtil.java:226)

... 20 more

Caused by: TRecordServiceException(code:INVALID_REQUEST, message:Worker membership is empty. Please ensure all RecordService Worker nodes are running.)

at com.cloudera.recordservice.thrift.RecordServicePlanner$PlanRequest_result$PlanRequest_resultStandardScheme.read(RecordServicePlanner.java:2167)

at com.cloudera.recordservice.thrift.RecordServicePlanner$PlanRequest_result$PlanRequest_resultStandardScheme.read(RecordServicePlanner.java:2144)

at com.cloudera.recordservice.thrift.RecordServicePlanner$PlanRequest_result.read(RecordServicePlanner.java:2083)

at com.cloudera.recordservice.shade.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)

at com.cloudera.recordservice.thrift.RecordServicePlanner$Client.recv_PlanRequest(RecordServicePlanner.java:133)

at com.cloudera.recordservice.thrift.RecordServicePlanner$Client.PlanRequest(RecordServicePlanner.java:120)

at com.cloudera.recordservice.core.RecordServicePlannerClient.planRequest(RecordServicePlannerClient.java:306)

... 21 more

[cloudera@quickstart ~]$ hadoop jar ./recordservice-client-0.1/lib/recordservice-examples-0.1.jar com.cloudera.recordservice.examples.mapreduce.RecordCount "SELECT firstname, lastname, phonenumber from rs.employees" "/tmp/recordcount_output"

15/12/07 11:25:58 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032

15/12/07 11:26:00 INFO core.ThriftUtils: Connecting to RecordServicePlanner at localhost:12050, with timeout: 30000ms

15/12/07 11:26:00 INFO core.ThriftUtils: Connected to RecordServicePlanner at localhost:12050

15/12/07 11:26:00 INFO core.RecordServicePlannerClient: Planning request: TPlanRequestParams(client_version:1.0, request_type:Sql, sql_stmt:SELECT firstname, lastname, phonenumber from rs.employees) with attempt 1/3. timeout= 120000ms

15/12/07 11:26:00 INFO core.RecordServicePlannerClient: Closing RecordServicePlanner connection.

15/12/07 11:26:00 INFO mapreduce.JobSubmitter: Cleaning up the staging area /tmp/hadoop-yarn/staging/cloudera/.staging/job_1449259187792_0006

15/12/07 11:26:00 WARN security.UserGroupInformation: PriviledgedActionException as:cloudera (auth:SIMPLE) cause:java.io.IOException: com.cloudera.recordservice.core.RecordServiceException: TRecordServiceException(code:INVALID_REQUEST, message:Worker membership is empty. Please ensure all RecordService Worker nodes are running.)