Uploaded image for project: 'Hue (READ ONLY)'
  1. Hue (READ ONLY)
  2. HUE-2568

[oozie] Validate that we can generate the graph of this workflow

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.6.0
    • Fix Version/s: 3.9.0
    • Component/s: con.oozie
    • Labels:
      None
    • Environment:

      CDH 5.2 HUE 3.6.0

      Description

      Hi, I have failed action materialization of coordinator.
      I goto coordinator page, where I can see the list of actions.
      I do click failed materialzation and get an error.

      P.S. The same wflow did displayed correctly in previous version Hue 2.* from CDH 4.7.

      Hi, here is an error:

      ERROR    Internal Server Error: /oozie/list_oozie_workflow/0000446-150214184607753-oozie-oozi-W/
      Traceback (most recent call last):
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/core/handlers/base.py", line 111, in get_response
          response = callback(request, *callback_args, **callback_kwargs)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/apps/oozie/src/oozie/views/dashboard.py", line 110, in decorate
          return view_func(request, *args, **kwargs)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/apps/oozie/src/oozie/views/dashboard.py", line 233, in list_oozie_workflow
          workflow_graph, full_node_list = Workflow.gen_status_graph_from_xml(request.user, oozie_workflow)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/apps/oozie/src/oozie/models.py", line 554, in gen_status_graph_from_xml
          workflow.delete(skip_trash=True)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/apps/oozie/src/oozie/models.py", line 136, in delete
          self.doc.all().delete()
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/query.py", line 513, in delete
          collector.collect(del_query)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/deletion.py", line 151, in collect
          reverse_dependency=reverse_dependency)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/deletion.py", line 95, in add
          if not objs:
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/query.py", line 130, in __nonzero__
          iter(self).next()
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/query.py", line 118, in _result_iter
          self._fill_cache()
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/query.py", line 892, in _fill_cache
          self._result_cache.append(self._iter.next())
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/query.py", line 291, in iterator
          for row in compiler.results_iter():
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/sql/compiler.py", line 763, in results_iter
          for rows in self.execute_sql(MULTI):
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/models/sql/compiler.py", line 818, in execute_sql
          cursor.execute(sql, params)
        File "/opt/cloudera/parcels/CDH-5.2.0-1.cdh5.2.0.p0.36/lib/hue/build/env/lib/python2.7/site-packages/Django-1.4.5-py2.7.egg/django/db/backends/postgresql_psycopg2/base.py", line 52, in execute
          return self.cursor.execute(query, args)
      

      Here is a wflow-code:

      <workflow-app name="calculate-visits-item_info-matching-wflow" xmlns="uri:oozie:workflow:0.4">
      
          <start to="calculate-visits-item_info"/>
      
          <action name="calculate-visits-item_info">
              <pig>
                  <job-tracker>${jobTracker}</job-tracker>
                  <name-node>${nameNode}</name-node>
                  <prepare>
                      <delete path="${visits_item_info_out_dir}" />
                  </prepare>
                  <configuration>
                      <property>
                          <name>mapred.job.queue.name</name>
                          <value>default</value>
                      </property>
                      <property>
                          <name>mapreduce.map.memory.mb</name>
                          <value>4096</value>
                      </property>
                      <property>
                          <name>pig.exec.reducers.bytes.per.reducer</name>
                          <value>50000000</value>
                      </property>
                      <property>
                          <name>pig.tmpfilecompression</name>
                          <value>true</value>
                      </property>
                      <property>
                          <name>pig.tmpfilecompression.codec</name>
                          <value>gz</value>
                      </property>
                      <property>
                          <name>mapred.compress.map.output</name>
                          <value>true</value>
                      </property>
                      <property>
                          <name>mapred.output.compress</name>
                          <value>true</value>
                      </property>
                      <property>
                          <name>avro.output.codec</name>
                          <value>snappy</value>
                      </property>
      
                      <property>
                          <name>mapred.map.max.attempts</name>
                          <value>2</value>
                      </property>
                      <property>
                          <name>mapred.reduce.max.attempts</name>
                          <value>2</value>
                      </property>
                      <property>
                          <name>mapred.task.timeout</name>
                          <value>3600000</value> <!-- 1 hour -->
                      </property>
                  </configuration>
      
                  <script>pig/match_visit_and_item_info.pig</script>
                  <param>sessionLength=${sessionLength}</param>
      
                  <param>visit=${visit_md_in_dir}</param>
                  <param>item_info=${item_info_in_dir}</param>
                  <param>shops=${shop_in_dir}</param>
                  <param>output=${visits_item_info_out_dir}</param>
                  <param>path_to_schema=${path_to_schema_visits_item_info}</param>
      
                  <param>udf=match_sources.py</param>
                  <param>shop_url_matching_file_name=shop_url_matching_file_name</param>
      
                  <file>pig/udf/match_sources.py#match_sources.py</file>
                  <file>${shop_url_matching_in_dir}/${shop_url_matching_file_name}#shop_url_matching_file_name</file>
              </pig>
              <ok to="add-partition-visits-item_info"/>
              <error to="kill"/>
          </action>
      
          <action name="add-partition-visits-item_info">
              <java>
                  <job-tracker>${jobTracker}</job-tracker>
                  <name-node>${nameNode}</name-node>
                  <configuration>
                      <property>
                          <name>mapred.queue.name</name>
                          <value>default</value>
                      </property>
                  </configuration>
                  <main-class&gt;ru.desert.common.hive.action.AddPartitionAction</main-class&gt;
                  <arg>
                      {
                      "jdbcDriver":   "org.apache.hive.jdbc.HiveDriver",
                      "jdbcUrl"   :   "${jdbcUrl}",
                      "userName"  :   "${userName}",
                      "tableName" :   "${tableName}",
                      "location"  :   "${visits_item_info_out_dir}",
                      "partitionCalculator" :  "${partitionCalculator}"
                      }
                  </arg>
              </java>
              <ok to="end"/>
              <error to="kill"/>
          </action>
      
          <kill name="kill">
              <message>Error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
          </kill>
          <end name="end"/>
      
      </workflow-app>
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                krish Sai K Chirravuri
                Reporter:
                seregasheypak Sergey Sheypak
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: