High CPU Consumption

[ym-corey]

Hi @chuff (maybe)?

We've started using the GPP library in our live environment to get a better idea of how it performs etc., and to ensure it works. While we're not receiving many GPP strings, we are receiving TcfEuV2/TCF2/GDPR strings in large volume. On our servers, we've noticed that CPU consumption for handling a TcfEuV2 (GDPR IAB TCF V2) string processing through the TcfEuV2 model eats up around 8% CPU, while the same strings processing through the older IAB TCF TCString library only consumes around 0.8% CPU.

Looking at the TcfEuV2 model, I see that the entire string is decoded upon creation rather than lazy decoding only the segments that are relevant/used. Would it be feasible to update the decryption/parsing/processing methodology to be lazy?

For performance stats, I used this tool

[travisbeale]

We are having the same problem. This library creates an additional 15-20% CPU usage in our ad server.

[patmmccann]

Was this solved by #34 ?

[ym-corey]

Hi @patmmccann ,

I ran some more tests today. Since we aren't receiving GPP in high volume I substituted the usage of the TCString library with TcfEuV2 on GDPR strings to simulate what the CPU consumption/distribution within our Java process would be if we instead needed to process GPP at the same volume.

Our control traffic shows around 4% CPU (TCString.decode(consentString)), while the test shows around 18% (new TcfEuV2(consentString)). This is on a service where around 80% of inbound traffic requires processing of a consent string.

My understanding is that the TCString library is lazily decoding each section of GDPR, whereas the GPP library is lazily decoding each section of GPP. The key difference being that the GDPR string is not lazily decoded in the GPP library.

[patmmccann]

@ym-corey I recommend you reach out to @Net-burst ; who made substantial contributions to make prebid-server-java performant at decoding gpp strings. I am sure he is much more knowledgable than I on this matter

[ym-corey]

I've reviewed the prebid-server-java code and while there is some code in place to prevent re-encoding of strings (specifically the TcfEuV2 and UsPrivacyV1 sections) for outbound requests it otherwise doesn't resolve or work around the lazy decoding issue present in the GppModel. I suspect prebid-server-java will also notice a large cpu spike when CMPs switch from providing GDPR strings to GPP strings.

[Net-burst]

Hey, @ym-corey. Our team made a decision to wait for IAB fix/workaround and work from there instead of jumping the gun. We will re-evaluate the bottlenecks after we update the library and will open-source our work by making a PR here.
And yeah, we are already seeing an increase in CPU usage, so this will become a priority...

[yuzawa-san]

Please review #56
I was able to cut down on CPU and memory significantly.

[captify-adanilov]

Would be good to have this issue sorted out, as the library may cause a massive increase in CPU load at the moment. I eventually had to revert to using the older TCF library, keeping GPP only as a fallback.

[yuzawa-san]

my PR is now ready for review. please see comment #56 (comment)