Goals, Unify and aggregate:
- Jobs :YARN, Impala, Spark, Sqoop....
- ATS Flows
- Batch jobs: query export, S3 copies, indexing...
- Scheduled jobs: integrated schedules
Some ideas about general UX:
- Job browser displays only the first 1000 jobs from resource manager start https://github.com/cloudera/hue/issues/451
- pagination with 1000s of jobs
- log pagination as we break
- kill wf of hs2 script, error when stopping hs2 in jb
- pagination, job status filtering is broken after each background refresh
- Cannot kill ‘Accepted’ jobs from jobbrowser as can't ckick on them
- re-think how we can get the job logs in a stable way + testable
- link back to objects, like Hive, Sqoop... queries
- better spark https://issues.apache.org/jira/browse/SPARK-3454
- charting summary of # of tasks, progress bars...
- speed with 10k+ jobs
- aggreate hive, impala, oozie jobs?
- Mappers and Reducers show the same progress
- tabs don't always support back button
- progress label color
- left bar should have hierarchical links (jobs --> task --> ...)
- left bar should have more options/icons
- tasks: put the task in green/orange/red depending on its status maybe
- Job priority is gone on the application lists : http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/HUE-Job-browser-not-displaying-job-priorities/m-p/14454#M262
- Task bottleneck visualization
- Timeline view?
- Smarter filtering http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API
- http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API, queues, tags...
- Spark https://github.com/hammerlab/spree
- We could live progress the tasks, counters... when on their respective tabs
- Not super clear facet selection (succeeded checked) https://dl.dropbox.com/s/8t3dyd1qfl5fiv1/Screenshot%202016-10-11%2019.46.55.png?dl=0
Random errors when killing running jobs, or jobs that just finished
Tried killing the hive job which was 50% complete - it gave error message “There was a problem communicating with the server. Refresh the page”
Logs for running jobs fails with exception