Boxed OneOfs

[damianw]

Let’s implement an alternate form of oneof that doesn’t crash the compiler!

• Runtime API
• Kotlin generator
• Reflection-based adapters
• Java generator
• Redacted support
• Tests & coverage for NameAllocator / name collisions
• JSON support Kotlin
• JSON support Java
• Fix ordering so oneofs are interleaved with non-oneofs in generated code Kotlin generated message's contructor respect fields declaration order #1997
• Fix generator to not double readUnknownFields() calls
• Fix generator to use CONSTANT_CASE for oneof declarations and their keys
• Confirm we don't have circular deps: WIP: Sample of new handling for OneOf's #1856 (comment)

---

@damianw’s original report below:

Wire version: 3.1.0

For example, Perfetto's FtraceEvent generates this Kotlin file. Attempting to compile it (with Kotlin 1.3.70) results in:

e: java.lang.IllegalStateException: Backend Internal error: Exception during file facade code generation
File being compiled: file:///.../build/generated/source/wire/perfetto/protos/FtraceEvent.kt
The root cause java.lang.RuntimeException was thrown at: org.jetbrains.kotlin.codegen.ClassFileFactory$OutputClassFile.asByteArray(ClassFileFactory.java:315)
        at org.jetbrains.kotlin.backend.common.CodegenUtil.reportBackendException(CodegenUtil.kt:247)
        at org.jetbrains.kotlin.codegen.PackageCodegenImpl.generate(PackageCodegenImpl.java:78)
        at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generatePackage(CodegenFactory.kt:88)
        at org.jetbrains.kotlin.codegen.DefaultCodegenFactory.generateModule(CodegenFactory.kt:67)
        ...
Caused by: java.lang.RuntimeException: Error generating class file perfetto/protos/FtraceEvent$Companion$ADAPTER$1.class (compiled from [/.../build/generated/source/wire/perfetto/protos/FtraceEvent.kt]): Method too large: perfetto/protos/FtraceEvent$Companion$ADAPTER$1.redact (Lperfetto/protos/FtraceEvent;)Lperfetto/protos/FtraceEvent;
        at org.jetbrains.kotlin.codegen.ClassFileFactory$OutputClassFile.asByteArray(ClassFileFactory.java:315)
        at org.jetbrains.kotlin.cli.common.output.OutputUtilsKt.writeAll(outputUtils.kt:32)
        at org.jetbrains.kotlin.cli.common.output.OutputUtilsKt.writeAll(outputUtils.kt:42)
        at org.jetbrains.kotlin.cli.jvm.compiler.KotlinToJVMBytecodeCompiler.writeOutput(KotlinToJVMBytecodeCompiler.kt:101)
        ...
Caused by: org.jetbrains.org.objectweb.asm.MethodTooLargeException: Method too large: perfetto/protos/FtraceEvent$Companion$ADAPTER$1.redact (Lperfetto/protos/FtraceEvent;)Lperfetto/protos/FtraceEvent;
        at org.jetbrains.org.objectweb.asm.MethodWriter.computeMethodInfoSize(MethodWriter.java:2089)
        at org.jetbrains.org.objectweb.asm.ClassWriter.toByteArray(ClassWriter.java:458)
        at org.jetbrains.kotlin.codegen.ClassBuilderFactories$2.asBytes(ClassBuilderFactories.java:118)
        at org.jetbrains.kotlin.codegen.DelegatingClassBuilderFactory.asBytes(DelegatingClassBuilderFactory.kt:36)
        ... 46 more

[pavelreiter]

This issue is caused by limitations of JVM, which are enforced by the Jetbrains ASM dependency

Java version of generated proto is avoiding this issue by using Builder.

Also this issue reminds me of the older one with Java and large proto definitions #691

[JakeWharton]

This will affect more than just the redact method so we should create a test message with maybe a 300 fields and then perform partitioning every 100 or so.

For redact we can change

override fun redact(value: FtraceEvent): FtraceEvent = value.copy(..)

with

override fun redact(value: FtraceEvent): FtraceEvent {
  return redact3(redact2(redact1(value)))
}
private fun redact1(value: FtraceEvent): FtraceEvent = value.copy(..)
private fun redact2(value: FtraceEvent): FtraceEvent = value.copy(..)
private fun redact3(value: FtraceEvent): FtraceEvent = value.copy(..)

where each method is responsible only for its batch of fields.

The hardest one to fix will be the decode method.. maybe we can punt on that one for now and just try to keep its body as optimal as possible.

[pavelreiter]

I have two things to share

1. Partitioning "partitioning every 100 or so"
   100 is too high, 50 worked for me (I didn't try to pinpoint it more)
2. high field count also affects constructors of Java generated classes if they are in the oneOf part - basically it is return to the already mentioned Constructor with more than 255 arguments generated #691 with the oneOf spin

[swankjesse]

Declaration: we won’t support simultaneously using compact (ie. reflective adapter) with boxed oneofs. Compact is not particularly common and we don’t want the baggage.

[damianw]

Following up on this, the redact function is particularly problematic because of KT-73789, which results in methods which are significantly longer than necessary.

Has there been any movement on having the generator split up that function? If not, I might need to take a stab at it.

The good news is that this isn't likely to be an issue with the other generated functions and may only really need to be fixed for redact.

edit: implemented in #3214