Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Incomplete
-
Affects Version/s: 4.1.0
-
Fix Version/s: None
-
Component/s: app.editor, con.spark
-
Labels:
-
Environment:
Hue ignores all the configurations put into the spark submit configuration panel. Looking at the spark_batch.py, I see it is not considering any of those options.
if snippet['type'] == 'jar':
{ 'file': snippet['properties'].get('app_jar'), 'className': snippet['properties'].get('class'), 'args': snippet['properties'].get('arguments'), }
properties =elif snippet['type'] == 'py':
{ 'file': snippet['properties'].get('py_file'), 'args': snippet['properties'].get('argument'), }
properties =else:
{ 'file': snippet['properties'].get('app_jar'), 'className': snippet['properties'].get('class'), 'args': snippet['properties'].get('arguments'), 'pyFiles': snippet['properties'].get('py_file'), 'files': snippet['properties'].get('files'), # driverMemory # driverCores # executorMemory # executorCores # archives }
properties =I have also attached a screenshot to this issue for your review.
Hue ignores all the configurations put into the spark submit configuration panel. Looking at the spark_batch.py, I see it is not considering any of those options. if snippet ['type'] == 'jar': properties = { 'file': snippet['properties'].get('app_jar'), 'className': snippet['properties'].get('class'), 'args': snippet['properties'].get('arguments'), } elif snippet ['type'] == 'py': properties = { 'file': snippet['properties'].get('py_file'), 'args': snippet['properties'].get('argument'), } else: properties = { 'file': snippet['properties'].get('app_jar'), 'className': snippet['properties'].get('class'), 'args': snippet['properties'].get('arguments'), 'pyFiles': snippet['properties'].get('py_file'), 'files': snippet['properties'].get('files'), # driverMemory # driverCores # executorMemory # executorCores # archives } I have also attached a screenshot to this issue for your review.
