The `detectTokens` method seems to be complex. We should split this up, extracting logic to separate files, so that it is easier to understand and test.
The
detectTokensmethod seems to be complex. We should split this up, extracting logic to separate files, so that it is easier to understand and test.