8

I am trying to register a class with array (Spark Java with Kryo activated), log shows a clear message:

Class is not registered: org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]

I have written several combinations, but these do not work:

        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[]")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class")); // ERROR
        kryo.register(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation$Array")); // ERROR
        kryo.register(Class.forName("[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("[Lorg.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation")); // ERROR
        kryo.register(Class.forName("Array[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation]")); // ERROR
        kryo.register(Class.forName("[[org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"));  // ERROR

I also tried to write a registration class without Class.forName but Java cannot resolve the symbol InMemoryFileIndex$SerializableBlockLocation:

kryo.register(org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation[].class);

All other classes work in my KryoRegister.class.

jgp
  • 2,069
  • 1
  • 21
  • 40
MrElephant
  • 302
  • 4
  • 26

2 Answers2

4

I found a similar question here, try:

kryo.register(Array.newInstance(Class.forName("org.apache.spark.sql.execution.datasources.InMemoryFileIndex$SerializableBlockLocation"), 0)
                        .getClass());
xingbin
  • 27,410
  • 9
  • 53
  • 103
0

Found out you can also try:

spark.kryo.classesToRegister [Lorg.apache.spark.sql.execution.datasources.SerializableBlockLocation;

to register "org.apache.spark.sql.execution.datasources.SerializableBlockLocation[]"

Alexander K
  • 151
  • 1
  • 3