Filter logs using attributes does not work starting version 3.7.2

[dvanackere-lpg]

Expected behavior

Adding an attribute to a log and add a filter with this attribute should remove the log in Application Insights.
This was the behavior since at least version 3.4.x until version 3.7.1

Actual behavior

The attribute is added, the log is modified accordingly, but the log is not filtered.

To Reproduce

applicationinsights.json:
{ "sampling": { "percentage": 100, "overrides": [{ "telemetryType": "trace", "attributes": [{ "key": "TODELETE", "value": ".*", "matchType": "regexp" } ], "percentage": 0 } ] }, "preview": { "processors": [{ "type": "log", "body": { "toAttributes": { "rules": [ "^Noisy log (?<TODELETE>.*)" ] } } } ] } }

In the application, add a noisy log:
logger.info { "Noisy log that should not be sent to AppInsights" }

System information

Please provide the following information:

• SDK Version: 3.7.2 to 3.7.5 KO, 3.4.x to 3.7.1 OK
• Using spring-boot: yes

I suspect that starting version 3.7.2, the 'sampling overrides' and the 'attribute processors' are not handled in the same order (processor then sampling before 3.7.2, sampling then processor starting 3.7.2).

[jeanbisutti]

David (@dvanackere-lpg) Please see #4527 (comment)

You could use one of the following solutions to remove the noisy log:

• https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-log
• https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-config#autocollected-logging

[dvanackere-lpg]

Hello Jean Bisutti (@jeanbisutti) , thanks for your response, unfortunately your suggestions would not work for us as the noisy logs come from third party libraries in the framework itself (from Azure, sorry) and disable them completely (or raising the level to ERROR) would remove some important monitoring (in Azure Functions, some log configurations must be at Information level to enable the monitoring).

[jeanbisutti]

David (@dvanackere-lpg)

Alternative solutions:

• You could try to use DCR. A tutorial: https://learn.microsoft.com/azure/azure-monitor/logs/tutorial-workspace-transformations-portal
• You can ask a maintainer of this library a way to disable the noisy logs
• Remove some third party telemetry data by encapsulating a library call in a method: https://learn.microsoft.com/en-us/azure/azure-monitor/app/java-standalone-sampling-overrides#suppress-collecting-telemetry-for-a-java-method

[dvanackere-lpg]

Jean Bisutti (@jeanbisutti)
Thanks for your suggestions, however we prefer to filter unwanted logs at the source (instead of using a DCR), mostly for costing reasons (at some point, the volume of noisy logs were greater than our logs) and it's cleaner.
Encapsulating the call would not work either because some of the logs we filter are in the azure functions worker itself (before our spring application starts). For instance:

• OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify were deprecated in JDK 13 and will likely be removed in a future release.
• Function xxx invoked by Java Worker
  ...

I understand the filter is a preview feature, so it's behaviour is subject to change. However it was nice to have.
I think our best solution at the moment is to stay with version 3.7.1, as long as it's possible.