Uploaded image for project: 'Kite SDK'
  1. Kite SDK
  2. KITE-463

FilesystemDataset.deleteAll() on non-partitioned Dataset throws IllegalStateException

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.14.1
    • Fix Version/s: 0.15.0
    • Component/s: Data Module
    • Labels:
      None

      Description

      As I was playing around with some changes related to CDK-450, I noticed that FilesystemDataset.deleteAll() results in IllegalStateException:

      java.lang.IllegalStateException: Attempt to retrieve the partition strategy on a non-partitioned descriptor:DatasetDescriptor{format=Format

      {name=avro}

      , schema={"type":"record","name":"user","fields":[

      {"name":"username","type":"string"}

      ,

      {"name":"email","type":"string"}

      ]}, location=hdfs://localhost:51583/var/folders/9k/wlyxzx3d4mb2kx3jkdk9t9t56bkgvq/T/1402351372217-0, properties={}, partitionStrategy=null}
      at com.google.common.base.Preconditions.checkState(Preconditions.java:172)
      at org.kitesdk.data.DatasetDescriptor.getPartitionStrategy(DatasetDescriptor.java:210)
      at org.kitesdk.data.spi.filesystem.FileSystemView.partitionIterator(FileSystemView.java:148)
      at org.kitesdk.data.spi.filesystem.FileSystemView.deleteAllUnsafe(FileSystemView.java:156)
      at org.kitesdk.data.spi.filesystem.FileSystemDataset.deleteAll(FileSystemDataset.java:130)
      at org.kitesdk.data.spi.filesystem.TestFileSystemDataset.testDeleteAllWithoutPartitions(TestFileSystemDataset.java:506)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      at org.junit.runners.Suite.runChild(Suite.java:128)
      at org.junit.runners.Suite.runChild(Suite.java:24)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
      at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28)
      at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
      at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
      at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390)
      at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197)

      The JavaDoc says that deleteAll() may throw UnsupportedOperationException but IllegalStateException would never be expected. I assume the intention here would be that deleteAll() would be supported for this case and it would result in all of the data files in the Dataset being deleted.

      I have not gone back to look if this ever worked to know how it might have behaved before. It is possible that it may be desired to treat this as unsupported since it would not be an atomic operation?

        Attachments

          Activity

            People

            • Assignee:
              ben.roling Ben Roling
              Reporter:
              ben.roling Ben Roling
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: