Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-667

HDFS configuration is not initialized

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.16.0
    • Fix Version/s: 0.17.0
    • Component/s: None
    • Labels:
      None

      Description

      The hdfs-site.xml file is not correctly loaded when using the Datasets API. Using a "hdfs:/" URI with no authority results in the following exception:

      org.kitesdk.data.DatasetIOException: Could not get a FileSystem
      	at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:68)
      	at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:50)
      	at org.kitesdk.data.spi.Registration.lookupDatasetUri(Registration.java:103)
      	at org.kitesdk.data.Datasets.exists(Datasets.java:353)
      	at org.kitesdk.data.Datasets.exists(Datasets.java:373)
      	at org.kitesdk.data.spi.filesystem.TestProvidedConfiguration.test(TestProvidedConfiguration.java:81)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      	at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      	at org.junit.runner.JUnitCore.run(JUnitCore.java:157)
      	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:74)
      	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:211)
      	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:67)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
      Caused by: java.io.IOException: Incomplete HDFS URI, no host: hdfs:/
      	at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:139)
      	at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
      	at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
      	at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
      	at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
      	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
      	at org.kitesdk.data.spi.filesystem.Loader$URIBuilder.getFromOptions(Loader.java:66)
      	... 31 more
      

      The work-around is to call HdfsConfiguration.init() before using the Datasets API, although it looks like this happens automatically for other FS uses. For example, getting the local FS will correctly initialize the HDFS config.

        Attachments

          Activity

            People

            • Assignee:
              blue Ryan Blue
              Reporter:
              blue Ryan Blue
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: