Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: CDH4.3.0
    • Fix Version/s: None
    • Component/s: MapReduce
    • Labels:
    • Environment:
      1)
      Eclipse Juno with JDK 1.6
      Windows
      Maven 1.6
      2) Unix

      Description

      We needed to run the json twitter parser via PIG code through java UDF we used elephant-bird 4.0 for to write UDF, it was running well under cdh3u5 but when migrated to server with cdh4.3.0 then udf jar does not run fine and pig script complains and uses the old map reduce.

      please see error below 1) error1 and also attached pig log file as error.txt.

      Therefore I re-configured the maven to use new elephant-bird 4.1 files for the UDF project, old pom file attachedcdh3u5.pom.txt), I updated it to new POM file (cdh4.3.0.pom) and even after adding dependencies for new jars of cdh4.3.0 and elephant-bird (4.1), code is complaining same error.

      hadoop-core-2.0.0-mr1-cdh4.3.0.jar
      elephant-bird-core-4.1.jar
      elephant-bird-pig-4.1.jar
      elephant-bird-hadoop-compat-4.1.jar
      pigunit-0.11.0-cdh4.3.0-sources.jar ( Juno eclipse)
      pigunit-0.11.0-cdh4.3.0.jar

      jython-2.5.0.jar ( had added this dependency as I was getting error on eclipse ( java.lang.NoClassDefFoundError: org/antlr/runtime/RecognitionException), complete log when run on eclipse using pigunit-0.11.0-cdh4.3.0 is present at below 2) error2).

      Could you please let us know if there is any bug or any workaround with this.

      1) error1: when udf jar is created on eclipse and run on the linux server..
      2013-09-11 06:59:46,945 [main] WARN org.apache.pig.backend.hadoop23.PigJobControl - falling back to default JobControl (not using hadoop 0.23 ?)
      java.lang.NoSuchFieldException: jobsInProgress
      at java.lang.Class.getDeclaredField(Class.java:1884)
      at org.apache.pig.backend.hadoop23.PigJobControl.<clinit>(PigJobControl.java:58)
      at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.newJobControl(HadoopShims.java:102)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:285)
      at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:177)
      at org.apache.pig.PigServer.launchPlan(PigServer.java:1266)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1251)
      at org.apache.pig.PigServer.execute(PigServer.java:1241)
      at org.apache.pig.PigServer.executeBatch(PigServer.java:335)
      at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:137)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:170)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
      at org.apache.pig.Main.run(Main.java:604)
      at org.apache.pig.Main.main(Main.java:157)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
      2013-09-11 06:59:46,947 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added to the job

      2) error 2:When tried to run on eclipse, after compilation.
      java.lang.NoClassDefFoundError: org/antlr/runtime/RecognitionException
      at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1599)
      at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1547)
      at org.apache.pig.PigServer.registerQuery(PigServer.java:518)
      at org.apache.pig.tools.grunt.GruntParser.processPig(GruntParser.java:991)
      at org.apache.pig.pigunit.pig.GruntParser.processPig(GruntParser.java:61)
      at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:412)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:194)
      at org.apache.pig.pigunit.pig.PigServer.registerScript(PigServer.java:56)
      at org.apache.pig.pigunit.PigTest.registerScript(PigTest.java:171)
      at org.apache.pig.pigunit.PigTest.assertOutput(PigTest.java:248)
      at com.sears.hadoop.pig.piggybank.udf.SitesArrayMapTest.test2JSON(SitesArrayMapTest.java:15)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:589)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

        Attachments

        1. cdh3u5.pom.txt
          3 kB
          Hasan
        2. cdh4.3.0.pom.txt
          4 kB
          Hasan
        3. error.txt.txt
          13 kB
          Hasan

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hasan_huzaibi Hasan
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: