Details
Description
I'm trying to create spark session using curl:
curl -X POST --data '
' -H "Content-Type: application/json" localhost:8998/sessions
Output of curl is about session starting:
{"id":0,"appId":null,"owner":null,"proxyUser":null,"state":"starting","kind":"spark","appInfo":{"driverLogUrl":null,"sparkUiUrl":null},"log":[]}
In livy's log (attached) I see messages about spark session creation, but after a while I see a message about spark session successfully created, and immediately it starts shutdown process.
If I check session status using curl command, it's in DEAD state:
$ curl http://localhost:8998/sessions | jq . { "from": 0, "total": 1, "sessions": [ { "id": 0, "appId": "application_1482343367445_0210", "owner": null, "proxyUser": null, "state": "dead", "kind": "spark", "appInfo": { "driverLogUrl": null, "sparkUiUrl": "http://ip-10-200-139-129.ec2.internal:20888/proxy/application_1482343367445_0210/" }, "log": [] } ] }
In yarn tasks list I can see this session in Finished state and there are no errors or exceptions in driver's log or executor's logs (attached).
With the same installation, I can successfully start and use pyspack sessions using the same livy server.