Vfrdtyky

I am running the below loop in Kotlin and is throwing an out of memory error. I am running this for reading the rows in a csv file. Size of "records" is 6422.

fun readCSVFile(filePath: String): List<String> {

    val reader = FileReader(filePath)

    val records = CSVFormat.DEFAULT.parse(reader)

    val rows = mutableListOf<String>()



    var output = ""

    records.forEach() {

        val size = it.size()

        for (i in 0 until it.size()-1) {

            output = output + it.get(i) + ","

        }

        output.dropLast(1)

        rows.add(output)

    }

    return rows

}

Below is the exception I get.

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded

at java.util.Arrays.copyOf(Arrays.java:3332)

at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)

at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)

at java.lang.StringBuilder.append(StringBuilder.java:136)

at trivago.ti.tools.FileProcessor.readCSVFile(FileProcessor.kt:16)

at trivago.ti.tools.ComparatorMainKt.main(ComparatorMain.kt:25)

I have the same logic executing in Java but it works fine. Below is what I have in Java.

private static List<String> readCSVFile(String filePath) throws IOException {

    Reader in = new FileReader(filePath);

    Iterable<CSVRecord> records = CSVFormat.DEFAULT.parse(in);

    List<String> rows = new ArrayList<>();

    for (CSVRecord record : records) {

        StringBuilder builder = new StringBuilder();

        for (int i = 0; i < record.size(); i++)

            builder.append(record.get(i) + ",");

        builder.deleteCharAt(builder.length() - 1);

        rows.add(builder.toString());

    }

    return rows;

}

Why is Kotlin having a problem with this? Am I doing something wrong with the loop? Any help would be much appreciated as I'm a newbie to Kotlin.

Use a StringBuilder in your kotlin code as well. You are creating a log of String objects in heap. String is immutable and this code:

var output = ""

output = output + ","

is creating two objects in heap, although you have only reference to one of them. So the other one is eligible for GC to remove it. GC is "working" too hard in your case, that's why you get java.lang.OutOfMemoryError: GC overhead limit exceeded.

fun readCSVFile(filePath: String): List<String> {

    val reader = FileReader(filePath)

    val records = CSVFormat.DEFAULT.parse(reader)

    val rows = mutableListOf<String>()



    var output = StringBuilder("")

    records.forEach() {

        output = StringBuilder("")

        val size = it.size()

        for (i in 0 until it.size()-1) {

            output = output.append(it.get(i) + ",")

        }

        output.deleteCharAt(output.length - 1)

        rows.add(output.toString())

    }

    return rows

}

You code will also run a lot faster, as creating a new object is quite costly.

I think there is a bug in your code

records.forEach() {

    output = "" // clear output ;)

    ...

}

Compare it to your java code

for (CSVRecord record : records) {

    StringBuilder builder = new StringBuilder(); // clear builder

    ...

}

You have two problems in your Kotlin code:

1. You are using strings and string concatenation - which is costly operation. You should use StringBuilder as well.


2. You are setting output = "" outside of the foreach loop - for each iteration, you have all previous rows inside output