JIT/CheckProjects tool isn't used correctly in CI

[jkoritzinsky]

I've noticed that the src/tests/JIT/CheckProjects/CheckProjects.csproj test doesn't actually work in CI, and hasn't even worked locally since repo consolidation in .NET 5.

This test is meant to validate that the project files in the src/tests tree meet some basic invariants established by the JIT team. Mainly that the _d, _do, _r, and _ro suffixed project files set the right DebugType and Optimize flags to be run in their specific configuration no matter the build configuration of the test tree.

The main problem with this tool is that we currently only run it on Helix, where we don't have a clone of the repo. Additionally, it hasn't been updated since repo consolidation and it went unnoticed since the tool silently passes when the file layout is not what it expects.

I think we should do one of the following actions to solve these problems:

1. Delete the test. If we don't feel that we need or want this validation, then we don't need to have the test, and it's just adding a test that doesn't give us anything in our test tree.
2. Move the test to src/tests/Common and add a job in PRs like the formatting jobs that only triggers on changes to the test tree to validate the invariants the tool expects.
3. Move the test to https://github.com/dotnet/jitutils and consume it in a PR job (like step 2, but the JIT team would own the tool in their own repo, like the jitformat tool).

cc: @dotnet/jit-contrib

Tagging subscribers to this area: @hoyosjs
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I've noticed that the src/tests/JIT/CheckProjects/CheckProjects.csproj test doesn't actually work in CI, and hasn't even worked locally since repo consolidation in .NET 5.

This test is meant to validate that the project files in the src/tests tree meet some basic invariants established by the JIT team. Mainly that the _d, _do, _r, and _ro suffixed project files set the right DebugType and Optimize flags to be run in their specific configuration no matter the build configuration of the test tree.

The main problem with this tool is that we currently only run it on Helix, where we don't have a clone of the repo. Additionally, it hasn't been updated since repo consolidation and it went unnoticed since the tool silently passes when the file layout is not what it expects.

I think we should do one of the following actions to solve these problems:

1. Delete the test. If we don't feel that we need or want this validation, then we don't need to have the test, and it's just adding a test that doesn't give us anything in our test tree.
2. Move the test to src/tests/Common and add a job in PRs like the formatting jobs that only triggers on changes to the test tree to validate the invariants the tool expects.
3. Move the test to https://github.com/dotnet/jitutils and consume it in a PR job (like step 2, but the JIT team would own the tool in their own repo, like the jitformat tool).

cc: @dotnet/jit-contrib

Author:	jkoritzinsky
Assignees:	-
Labels:	area-Infrastructure-coreclr
Milestone:	-

[EgorBo]

My vote for 1.
The whole concept of _ro, _o seems to be a legacy

[AndyAyersMS]

I don't think we need this anymore. It was useful back when we had just pulled over thousands of tests from desktop and didn't always get the details right.

The whole concept of _ro, _o seems to be a legacy

The idea here was to make sure we handle all the different variants of IL / optimization flags that can happen. We don't tend to add these variants for new tests which is probably fine, but I don't see any reason to get rid of the variants that already exist.