deserialize failing with java.lang.OutOfMemoryError: Java heap space

[andyczerwonka]

Search before asking

• I had searched in the issues and found no similar issues.

Version

latest 0.5.0 snapshot

Component(s)

Java/Scals

Minimal reproduce step

Here's the exception we're seeing.

org.apache.fury.exception.DeserializationException: Deserialize failed, read objects are: [PredictorStorageData(null,null,null), ConfigurationMetadata(Perfume SMILES price-process,Default Graph Model generated from data inspection.,null), Some(null), {}]
	at org.apache.fury.Fury.handleReadFailed(Fury.java:784)
	at org.apache.fury.Fury.deserialize(Fury.java:755)
	at org.apache.fury.Fury.deserialize(Fury.java:685)
	at org.apache.fury.ThreadLocalFury.deserialize(ThreadLocalFury.java:99)
	at io.citrine.mithril.modules.predictors.Predictor$.$anonfun$decode$1(Predictor.scala:111)
	at scala.util.Try$.apply(Try.scala:210)
	at io.citrine.mithril.modules.predictors.Predictor$.decode(Predictor.scala:107)
	at io.citrine.mithrilworkbench.Main$.$anonfun$tryToDecodeThisPredictor$1(Main.scala:330)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.scala:18)
	at scribe.LoggerSupport.elapsed(LoggerSupport.scala:67)
	at scribe.LoggerSupport.elapsed$(LoggerSupport.scala:62)
	at scribe.package$.elapsed(package.scala:8)
	at io.citrine.mithrilworkbench.Main$.tryToDecodeThisPredictor(Main.scala:323)
	at io.citrine.mithrilworkbench.Main$.$anonfun$main$5(Main.scala:340)
	at io.citrine.mithrilworkbench.Main$.$anonfun$main$5$adapted(Main.scala:340)
	at mainargs.Invoker$.$anonfun$invoke0$3(Invoker.scala:63)
	at mainargs.Result.flatMap(Result.scala:13)
	at mainargs.Result.flatMap$(Result.scala:12)
	at mainargs.Result$Success.flatMap(Result.scala:23)
	at mainargs.Invoker$.invoke0(Invoker.scala:62)
	at mainargs.Invoker$.invoke(Invoker.scala:72)
	at mainargs.Invoker$.$anonfun$runMains$1(Invoker.scala:92)
	at mainargs.Result.flatMap(Result.scala:13)
	at mainargs.Result.flatMap$(Result.scala:12)
	at mainargs.Result$Success.flatMap(Result.scala:23)
	at mainargs.Invoker$.groupArgs$1(Invoker.scala:92)
	at mainargs.Invoker$.runMains(Invoker.scala:107)
	at mainargs.ParserForMethods.runRaw0(Parser.scala:170)
	at mainargs.ParserForMethods.runEither(Parser.scala:108)
	at mainargs.ParserForMethods.runEither(Parser.scala:151)
	at mainargs.ParserForMethods.runOrExit(Parser.scala:56)
	at io.citrine.mithrilworkbench.Main$.main(Main.scala:341)
	at io.citrine.mithrilworkbench.Main.main(Main.scala)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.reflect.Array.multiNewArray(Native Method)
	at java.base/java.lang.reflect.Array.newInstance(Array.java:114)
	at org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.newArray(ArraySerializers.java:177)
	at org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.read(ArraySerializers.java:126)
	at org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.read(ArraySerializers.java:43)
	at org.apache.fury.Fury.readDataInternal(Fury.java:916)
	at org.apache.fury.Fury.readRef(Fury.java:831)
	at org.apache.fury.serializer.collection.AbstractMapSerializer.generalJavaRead(AbstractMapSerializer.java:568)
	at org.apache.fury.serializer.collection.AbstractMapSerializer.genericJavaRead(AbstractMapSerializer.java:443)
	at org.apache.fury.serializer.collection.AbstractMapSerializer.readElements(AbstractMapSerializer.java:435)
	at org.apache.fury.serializer.collection.MapSerializer.read(MapSerializer.java:46)
	at org.apache.fury.serializer.collection.MapSerializer.read(MapSerializer.java:27)
	at org.apache.fury.Fury.readDataInternal(Fury.java:916)
	at org.apache.fury.Fury.readRef(Fury.java:831)
	at org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438)
	at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382)
	at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316)
	at org.apache.fury.Fury.readDataInternal(Fury.java:916)
	at org.apache.fury.Fury.readRef(Fury.java:831)
	at org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438)
	at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382)
	at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316)
	at org.apache.fury.Fury.readDataInternal(Fury.java:916)
	at org.apache.fury.Fury.readRef(Fury.java:831)
	at org.apache.fury.serializer.ObjectSerializer.readOtherFieldValue(ObjectSerializer.java:438)
	at org.apache.fury.serializer.ObjectSerializer.readAndSetFields(ObjectSerializer.java:382)
	at org.apache.fury.serializer.ObjectSerializer.read(ObjectSerializer.java:316)
	at org.apache.fury.Fury.readDataInternal(Fury.java:916)
	at org.apache.fury.Fury.readRef(Fury.java:818)
	at org.apache.fury.Fury.deserialize(Fury.java:751)
	at org.apache.fury.Fury.deserialize(Fury.java:685)
	at org.apache.fury.ThreadLocalFury.deserialize(ThreadLocalFury.java:99)

What did you expect to see?

We're working with a 1G binary and we have 8G of heap space.

What did you see instead?

Caused by: java.lang.OutOfMemoryError: Java heap space
	at java.base/java.lang.reflect.Array.multiNewArray(Native Method)
	at java.base/java.lang.reflect.Array.newInstance(Array.java:114)
	at org.apache.fury.serializer.ArraySerializers$ObjectArraySerializer.newArray(ArraySerializers.java:177)

Anything Else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[chaokunyang]

Hi @andyczerwonka , can I reproduce this error in my local environment?

[chaokunyang]

And could you provide your Fury creation code here too?

[andyczerwonka]

And could you provide your Fury creation code here too?

private val fury = Fury
    .builder()
    .withLanguage(Language.JAVA)
    .requireClassRegistration(false)
    .withScalaOptimizationEnabled(true)
    .withRefTracking(true)
    .withStringCompressed(true)
    .withLongCompressed(true)
    .withIntCompressed(true)
    .withAsyncCompilation(true)
    .buildThreadSafeFury()

[andyczerwonka]

Hi @andyczerwonka , can I reproduce this error in my local environment?

Unfortunately not, because the code is proprietary and I cannot share it. I was hoping the stack tracewould give us some insight into where it's failing so I could create a sample model.

[chaokunyang]

Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode

[andyczerwonka]

Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode

Assuming you want me to withAsyncCompilation(false), yes?

[chaokunyang]

Could you disable sync compilation and try it again? There are some inconsistentency between compile mode and interpretor mode

Assuming you want me to withAsyncCompilation(false), yes?

Yes, that's what I mean. Does it still fail?