Avoiding BufferOverflow errors

If the size of the data buffer is too small, you may get BufferOverflow errors when running the graph.

In an unsuccessful run, the Console Tab will show a BufferOverflowException error similar to this example: 
ERROR [WatchDog] - Node DATA_READER0 error details:
java.lang.RuntimeException: The size of data buffer is only 12288. 
Set appropriate parameter in defautProperties file.
    at org.jetel.data.StringDataField.serialize(StringDataField.java:285)
    at org.jetel.data.DataRecord.serialize(DataRecord.java:466)
    at org.jetel.graph.DirectEdge.writeRecord(DirectEdge.java:234)
    at org.jetel.graph.Edge.writeRecord(Edge.java:371)
    at org.jetel.component.DataReader.execute(DataReader.java:264)
    at org.jetel.graph.Node.run(Node.java:425)
    at java.lang.Thread.run(Thread.java:619)
Caused by: java.nio.BufferOverflowException
    at java.nio.Buffer.nextPutIndex(Buffer.java:501)
    at java.nio.DirectByteBuffer.putChar(DirectByteBuffer.java:465)
    at org.jetel.data.StringDataField.serialize(StringDataField.java:282)
    ... 6 more
You can avoid these errors by increasing the buffer settings in the defaultProperties configuration file, copying the file into your LDI project, and then specifying the file to be used in the run configuration of a graph. The defaultProperties configuration file is located in the cloveretl.engine.jar JAR file, whose default location is: 
CloverETL Designer\plugins\com.cloveretl.gui_3.0.1\lib\lib\cloveretl.engine.jar

To modify the defaultProperties configuration file and add it to your LDI project:

  1. Copy the cloveretl.engine.jar JAR file to a temporary location (for example, a temp directory).
  2. Extract the file org\jetel\data\defaultProperties from the JAR file into the temp directory.
  3. Open the defaultProperties in a text editor.
  4. Make these changes to the defaultProperties file:
    1. Increase Record.MAX_RECORD_SIZE to a size such as 65535.
    2. Increase DataParser.FIELD_BUFFER_LENGTH to a size such as 8192.
    3. Set DataFormatter.FIELD_BUFFER_LENGTH to the same size as DataParser.FIELD_BUFFER_LENGTH.
    4. Increase DEFAULT_INTERNAL_IO_BUFFER_SIZE to a size such as 130000.
    Note that these are suggested recommendations. Your final settings depend on the characteristics of your data set, which could mean that you may have to further increase these settings if your ingest operations are still failing due to memory problems.
  5. Place the defaultProperties configuration file in your LDI project folder, by copying it into the Navigator pane.

  6. From the Designer tool bar, choose Run > Run Configurations.
  7. From the left pane of the Run Configurations menu, select a graph to edit and then click the Arguments tab in the run configuration. If the graph you want to edit is not listed, you can either run the graph (so that its name will be listed) or create a new configuration for the graph.
  8. Enter the following text in the Program arguments field: 
    -config defaultProperties
    At this point, the Arguments tab should look like this example:

  9. Click Apply to save your changes.
  10. Click either Run (to run the graph with the modified run configuration) or Close (to close the Run Configurations menu).