Uploaded image for project: 'Kite SDK (READ-ONLY)'
  1. Kite SDK (READ-ONLY)
  2. KITE-1096

unable to load large files due java heap space

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Duplicate
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.1.0
    • Component/s: Command-line Interface
    • Labels:
      None

      Description

      command line interface is failing when i tried to load 1gb file. I tried to increase the memory using command line I am getting fallowing error. can you please advice how i can work around it?

      [root@bda1node03 forecast]# flags="-Xmx2048m" kite-dataset csv-import AccuWeatherForecast11222015.csv ctest1

      Exception in thread "main" java.lang.ClassNotFoundException: -Xmx2048m
      at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:348)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      ----------------------------------------------------------------------------

      Exception in thread "main" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
      at java.util.Arrays.copyOfRange(Arrays.java:3664)
      at java.lang.String.<init>(String.java:201)
      at java.lang.StringBuilder.toString(StringBuilder.java:407)
      at au.com.bytecode.opencsv.CSVParser.parseLine(CSVParser.java:250)
      at au.com.bytecode.opencsv.CSVParser.parseLineMulti(CSVParser.java:174)
      at au.com.bytecode.opencsv.CSVReader.readNext(CSVReader.java:237)
      at org.kitesdk.data.spi.filesystem.CSVFileReader.advance(CSVFileReader.java:174)
      at org.kitesdk.data.spi.filesystem.CSVFileReader.next(CSVFileReader.java:168)
      at org.kitesdk.shaded.com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
      at org.kitesdk.shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
      at org.kitesdk.shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
      at org.kitesdk.data.spi.filesystem.MultiFileDatasetReader.hasNext(MultiFileDatasetReader.java:125)
      at com.google.common.collect.Lists.newArrayList(Lists.java:138)
      at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256)
      at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217)
      at org.apache.crunch.impl.mem.collect.MemCollection.<init>(MemCollection.java:79)
      at org.apache.crunch.impl.mem.MemPipeline.read(MemPipeline.java:166)
      at org.apache.crunch.impl.mem.MemPipeline.read(MemPipeline.java:157)
      at org.kitesdk.tools.TransformTask.run(TransformTask.java:135)
      at org.kitesdk.cli.commands.CSVImportCommand.run(CSVImportCommand.java:186)
      at org.kitesdk.cli.Main.run(Main.java:178)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
      at org.kitesdk.cli.Main.main(Main.java:256)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
      [root@bda1node03 forecast]# vi test6.csv

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                chandrastarbucks chandra s koripella
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: