Details
-
Type:
Bug
-
Status: Resolved
-
Priority:
Blocker
-
Resolution: Duplicate
-
Affects Version/s: 1.1.0
-
Fix Version/s: 1.1.0
-
Component/s: Command-line Interface
-
Labels:None
Description
command line interface is failing when i tried to load 1gb file. I tried to increase the memory using command line I am getting fallowing error. can you please advice how i can work around it?
[root@bda1node03 forecast]# flags="-Xmx2048m" kite-dataset csv-import AccuWeatherForecast11222015.csv ctest1
Exception in thread "main" java.lang.ClassNotFoundException: -Xmx2048m
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.util.RunJar.run(RunJar.java:214)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
----------------------------------------------------------------------------
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects
at java.util.Arrays.copyOfRange(Arrays.java:3664)
at java.lang.String.<init>(String.java:201)
at java.lang.StringBuilder.toString(StringBuilder.java:407)
at au.com.bytecode.opencsv.CSVParser.parseLine(CSVParser.java:250)
at au.com.bytecode.opencsv.CSVParser.parseLineMulti(CSVParser.java:174)
at au.com.bytecode.opencsv.CSVReader.readNext(CSVReader.java:237)
at org.kitesdk.data.spi.filesystem.CSVFileReader.advance(CSVFileReader.java:174)
at org.kitesdk.data.spi.filesystem.CSVFileReader.next(CSVFileReader.java:168)
at org.kitesdk.shaded.com.google.common.collect.Iterators$7.computeNext(Iterators.java:648)
at org.kitesdk.shaded.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at org.kitesdk.shaded.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at org.kitesdk.data.spi.filesystem.MultiFileDatasetReader.hasNext(MultiFileDatasetReader.java:125)
at com.google.common.collect.Lists.newArrayList(Lists.java:138)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:256)
at com.google.common.collect.ImmutableList.copyOf(ImmutableList.java:217)
at org.apache.crunch.impl.mem.collect.MemCollection.<init>(MemCollection.java:79)
at org.apache.crunch.impl.mem.MemPipeline.read(MemPipeline.java:166)
at org.apache.crunch.impl.mem.MemPipeline.read(MemPipeline.java:157)
at org.kitesdk.tools.TransformTask.run(TransformTask.java:135)
at org.kitesdk.cli.commands.CSVImportCommand.run(CSVImportCommand.java:186)
at org.kitesdk.cli.Main.run(Main.java:178)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.kitesdk.cli.Main.main(Main.java:256)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
[root@bda1node03 forecast]# vi test6.csv
Attachments
Issue Links
- duplicates
-
KITE-898 Importing a large local CSV file causes out of memory error
-
- Resolved
-