Uploaded image for project: 'Sqoop (READ-ONLY)'
  1. Sqoop (READ-ONLY)
  2. SQOOP-216

Hive overwrite fails even when table is dropped.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: hive
    • Labels:
      None

      Description

      When I import a table using the command:

      sqoop import --connect jdbc:mysql://192.168.3.122:3306/fbapp_monster --table zodiacs -m 1 --hive-import --username anihq --password xxxxxx

      sqoop creates the table and I can access it in hive.

      If I then type "hive" at the command line and enter hive, when I type "show tables" I can see that it imported the table zodiacs.

      If I then type

      drop table zodiacs

      the table is dropped and no longer shows up when I type "show tables".

      However, if I then rerun the exact same sqoop import command above, I get an error telling me the zodiacs table already exists.

      11/06/02 01:57:28 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
      11/06/02 01:57:28 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
      11/06/02 01:57:28 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
      11/06/02 01:57:28 INFO tool.CodeGenTool: Beginning code generation
      11/06/02 01:57:28 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1
      11/06/02 01:57:28 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1
      11/06/02 01:57:28 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
      11/06/02 01:57:28 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-core.jar
      Note: /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.java uses or overrides a deprecated API.
      Note: Recompile with -Xlint:deprecation for details.
      11/06/02 01:57:29 ERROR orm.CompilationManager: Could not rename /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.java to /home/dzwick/./zodiacs.java
      11/06/02 01:57:29 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-dzwick/compile/acd207299e55066ca99f044c59991ad3/zodiacs.jar
      11/06/02 01:57:29 WARN manager.MySQLManager: It looks like you are importing from mysql.
      11/06/02 01:57:29 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
      11/06/02 01:57:29 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
      11/06/02 01:57:29 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
      11/06/02 01:57:29 INFO mapreduce.ImportJobBase: Beginning import of zodiacs
      11/06/02 01:57:29 INFO manager.MySQLManager: Executing SQL statement: SELECT t.* FROM `zodiacs` AS t LIMIT 1
      11/06/02 01:57:30 INFO mapred.JobClient: Cleaning up the staging area hdfs://h14-200.sv3:54310/n1/tmp/mapred/mapred/staging/dzwick/.staging/job_201104142112_9830
      11/06/02 01:57:30 ERROR tool.ImportTool: Encountered IOException running import job: org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory zodiacs already exists
      at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:132)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:872)
      at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:833)
      at java.security.AccessController.doPrivileged(Native Method)
      at javax.security.auth.Subject.doAs(Subject.java:396)
      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1115)
      at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:833)
      at org.apache.hadoop.mapreduce.Job.submit(Job.java:476)
      at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:506)
      at com.cloudera.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:107)
      at com.cloudera.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:166)
      at com.cloudera.sqoop.manager.SqlManager.importTable(SqlManager.java:386)
      at com.cloudera.sqoop.manager.MySQLManager.importTable(MySQLManager.java:125)
      at com.cloudera.sqoop.tool.ImportTool.importTable(ImportTool.java:350)
      at com.cloudera.sqoop.tool.ImportTool.run(ImportTool.java:423)
      at com.cloudera.sqoop.Sqoop.run(Sqoop.java:144)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
      at com.cloudera.sqoop.Sqoop.runSqoop(Sqoop.java:180)
      at com.cloudera.sqoop.Sqoop.runTool(Sqoop.java:218)
      at com.cloudera.sqoop.Sqoop.main(Sqoop.java:228)

      I know a similar bug has been reported before, and a patch has been released, but that patch dealt with the situation where the table being imported already exists. It seems to me that the problem persists even if the original import has been dropped.

      Any idea what's going on and how to fix this?

      Thanks,
      Dylan

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dylanzwick Patrick Dylan Zwick
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: