Uploaded image for project: 'Sqoop (READ-ONLY)'
  1. Sqoop (READ-ONLY)
  2. SQOOP-217

BigDecimalSplitter triggers OutOfMemoryError when split range/num mappers is a repeating decimal < 1

    Details

    • Type: Bug
    • Status: Open
    • Priority: Critical
    • Resolution: Unresolved
    • Affects Version/s: 1.2.0
    • Fix Version/s: None
    • Component/s: import
    • Labels:
      None

      Description

      Because of a bug in BigDecimalSplitter.tryDivide(), if the range of values (max - min) / numMappers is something like 1/6, the splitSize is returned as 0. When that happens, splitSize is set to MIN_INCREMENT, which is approximately 4.94E-320.

      And then this loop in BigDecimalSplitter.split() will consume all available memory:

      while (curVal.compareTo(maxVal) <= 0)

      { splits.add(curVal); curVal = curVal.add(splitSize); }

      The fix is to modify the exception handling portion of tryDivide() to be something like:

      private BigDecimal tryDivide(BigDecimal numerator, BigDecimal denominator) {
      try

      { return numerator.divide(denominator); }

      catch (ArithmeticException ae)

      { return numerator.divide(denominator, 100, RoundingMode.HALF_UP); }

      }

      This will return 0.666...7 (100 digits) as the result, instead of 0.

      Seems like there should be some way to specify the minimum rounding required to get a valid (non-exception) result from the division, but I didn't see it in the BigDecimal documentation.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kkrugler Ken Krugler
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: