[jb] Revamp v2

Goals, Unify and aggregate:

Browser

• Jobs :YARN, Impala, Spark, Sqoop....
• ATS Flows
• Workflows
• Schedules
• Bundles

App stats: https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_Statistics_API

History

• Batch jobs: query export, S3 copies, indexing...
• Scheduled jobs: integrated schedules

Some ideas about general UX:

1. Job browser displays only the first 1000 jobs from resource manager start https://github.com/cloudera/hue/issues/451
2. pagination with 1000s of jobs
3. log pagination as we break
4. kill wf of hs2 script, error when stopping hs2 in jb
5. pagination, job status filtering is broken after each background refresh
6. Cannot kill ‘Accepted’ jobs from jobbrowser as can't ckick on them
7. re-think how we can get the job logs in a stable way + testable
8. link back to objects, like Hive, Sqoop... queries
9. better spark https://issues.apache.org/jira/browse/SPARK-3454
10. charting summary of # of tasks, progress bars...
11. breadcrumbs
12. speed with 10k+ jobs
13. aggreate hive, impala, oozie jobs?
14. Mappers and Reducers show the same progress
15. tabs don't always support back button
16. progress label color
17. left bar should have hierarchical links (jobs --> task --> ...)
18. left bar should have more options/icons
19. tasks: put the task in green/orange/red depending on its status maybe
20. Job priority is gone on the application lists : http://community.cloudera.com/t5/Web-UI-Hue-Beeswax/HUE-Job-browser-not-displaying-job-priorities/m-p/14454#M262
21. Task bottleneck visualization
22. Timeline view?
23. Smarter filtering http://hadoop.apache.org/docs/r2.7.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API
24. https://github.com/apache/spark/pull/2342
25. http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Application_Statistics_API
26. http://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API, queues, tags...
27. Spark https://github.com/hammerlab/spree
28. We could live progress the tasks, counters... when on their respective tabs
29. Not super clear facet selection (succeeded checked) https://dl.dropbox.com/s/8t3dyd1qfl5fiv1/Screenshot%202016-10-11%2019.46.55.png?dl=0

Large issues
Random errors when killing running jobs, or jobs that just finished
Tried killing the hive job which was 50% complete - it gave error message “There was a problem communicating with the server. Refresh the page”

Logs for running jobs fails with exception
Sometimes only