Add backward compatibility configs for tally to otel migration

[yycptt]

What changed?

• Add backward compatibility configs for tally to otel migration

Why?

• There are some breaking changes when switching from tally to otel metric framework.

1. Unit name will be appended to the metric name
2. _total will be appended to counter metrics
3. Timer metric will be emitted as milliseconds instead of seconds
   This PR adds configs for controlling both behaviors when using otel framework so that the behavior will be the same as tally.

How did you test it?

• Unit tests
• Tested locally with various combination of configurations.

Potential risks

• No risk by default. If new options are enabled, timer metric value may be wrong or not bucketed correctly.

Documentation

Is hotfix candidate?

• No.

[yycptt]

Related to #7211

[bergundy]

LGTM, looks like there's good test coverage.

[yycptt]

Also added config for disabling the addition of _total to Counters

[1nu]

Hi! I might be wrong, but to me it seems like OTEL is missing backwards compatibility for tally users that configured prefix.

[yycptt]

Hi! I might be wrong, but to me it seems like OTEL is missing backwards compatibility for tally users that configured prefix.

Hi @1nu, thanks for reporting!

Yeah, looks like it's the case and Prefix is never supported when framework is set to opentelementry. I will look into how to keep the behavior the same for tally users.

[yycptt]

Support Prefix in OTEL: #7297

[1nu]

I just discovered another potential backward compatibility issue in handling tags keys and values with dashes, reporting here just in case. Tally replaces dashes with underscores in both key and value, opentelemetry only replaces dashes in the key.
For example:

...
    tags:
      temporal-cluster: "order-management"
...

becomes temporal_cluster="order_management" using tally and temporal_cluster="order-management" with opentelemetry.

[yycptt]

I have reverted the change to always use OTEL as the framework, so there will be no impact for Tally users in the next release.

We will look into those issues and do some more testing. Also needs to continue to support statsD users somehow with otel before fully deprecate tally.

Thanks again @1nu for flagging those issues!