Added implementation for Scaled RoPE.

[hazemessamm]

Proposed changes

Reference: https://kaiokendev.github.io/context#a-bigger-problem

Checklist

Put an x in the boxes that apply.

• I have read the CONTRIBUTING document
• I have run pre-commit run --all-files to format my code / installed pre-commit prior to committing changes
• I have added tests that prove my fix is effective or that my feature works
• I have updated the necessary documentation (if needed)

[awni]

Thanks for adding this! I have a question about the use of scale. O/w this is nice and we should merge it 👍

[awni]

When I've seen this parameter before it usually is the denominator in a division rather than multiplied with. So you would set scale = 4.0 and then positions = positions * (1 / scale).

What do you think? Do you have examples where it is used?

[hazemessamm]

I agree with you if the user is passing integers, I assumed that the user will pass the scale fraction so that I can multiply it directly with the positions without the need to do (1 / scale). What do you think? Should I change the scale parameter to expect integers and do (1 / scale) before multiplying or just assume that the user will pass (1 / scale) to the scale parameter directly?

[awni]

I think we can keep it the way it is. Calling it a scale makes it sound like it directly multiplies the positions (rather than divides them), so from a naming perspective the way you have it makes sense.

[awni]

Could you rebase this on main and resolve conflicts, then we can merge it?