Debugging an assumed pre-runtime exception com.google.inject.internal.util.$ComputationException in an Apache Spark app

by J. Doe   Last Updated July 17, 2017 08:05 AM

How to debug a complex third-party library in Apache Spark app?

  • Unfolding the stack trace would make no sense as stated here (not sure if it possible anyway)
  • While I've got some hints it could be a version conflict between transient dependencies, how can I narrow down how actually the error happens?

It seems to me that the exception is coming before JVM execution enters the main method, somewhere buried in a third-party component

17/07/17 07:23:05 INFO spark.SparkContext: Added JAR /some/path/spark-app.jar at http://172.17.0.2:5681/jars/spark-app.jar with timestamp 1500276185968
Exception in thread "main" **com.google.inject.internal.util.$ComputationException: java.lang.ArrayIndexOutOfBoundsException: 43168**
        at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:553)
        at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:419)
        at com.google.inject.internal.util.$CustomConcurrentHashMap$ComputingImpl.get(CustomConcurrentHashMap.java:2041)
        at com.google.inject.internal.util.$StackTraceElements.forMember(StackTraceElements.java:53)
        at com.google.inject.internal.Errors.formatSource(Errors.java:690)
        at com.google.inject.internal.Errors.formatInjectionPoint(Errors.java:720)
        at com.google.inject.internal.Errors.formatSource(Errors.java:684)
        at com.google.inject.internal.Errors.format(Errors.java:555)
        at com.google.inject.ProvisionException.getMessage(ProvisionException.java:59)
        at my.app.spark.app.main.Main.main(Main.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 43168
        at com.google.inject.internal.asm.$ClassReader.<init>(Unknown Source)
        at com.google.inject.internal.asm.$ClassReader.<init>(Unknown Source)
        at com.google.inject.internal.asm.$ClassReader.<init>(Unknown Source)
        at com.google.inject.internal.util.$LineNumbers.<init>(LineNumbers.java:62)
        at com.google.inject.internal.util.$StackTraceElements$1.apply(StackTraceElements.java:36)
        at com.google.inject.internal.util.$StackTraceElements$1.apply(StackTraceElements.java:33)
        at com.google.inject.internal.util.$MapMaker$StrategyImpl.compute(MapMaker.java:549)
        ... 18 more

(JVM 1.8; Apache Spark 1.6.2 - Scala 2.10.5)

Tags : java spark


Related Questions



Designing clickstream analysis?

Updated June 10, 2017 18:05 PM

spark output back to web page

Updated November 15, 2016 08:02 AM

Exploiting Apache Spark Data

Updated February 17, 2017 12:05 PM