allow unicode characters in character literals

[andrewrk]

While solving #2088 I am about to push a change that makes this test pass:

test "unicode escape in character literal" {
    var a: u24 = '\U01f4a9';
    expect(a == 128169);
}

This makes sense since character literals are just comptime_int. There's no footgun here because you can't accidentally misuse it:

test "aoeu" {
    var str = "hello";
    str[1] = '\U01f4a9';
}

/home/andy/dev/zig/build/test.zig:5:14: error: integer value 128169 cannot be implicitly casted to type 'u8'
    str[1] = '\U01f4a9';
             ^

With that in mind, I think it makes sense to allow utf-8 characters in character literals, since we have UTF-8 source encoding. I propose this test should pass:

const std = @import("std");

test "utf8 character literal" {
    const x = '💩';
    std.testing.expect(x == 128169);
}

[emekoi]

wait since we have utf8 source encoding can 💩, or あい be a valid identifier? or was this already discussed in a previous issue?

[shawnl]

@emekoi that is explicit rejected, unless using the C ABI, in which you can use @"any utf-8 string, with arbitrary bytes using \x00 syntax"

[daurnimator]

\U01f4a9

Why did we pick an uppercase U for this rather than lowercase? Most languages use \u for unicode characters.

While on this topic; was there any consideration of \u{01f4a9} with braces?

[shawnl]

While on this topic; was there any consideration of \u{01f4a9} with braces?

It isn't necessary. There is a single quote ' at the end, so it is unambiguous. However there is a big difference. \u covers the Basic Multilingual Plane BMP where the most useful stuff is, while \U is the supplementary planes. (such as Egyptian Hieroglyphics and Emoji) see this chart https://www.unicode.org/roadmaps/bmp/

[daurnimator]

It isn't necessary. There is a single quote ' at the end, so it is unambiguous. However there is a big difference. \u covers the Basic Multilingual Plane BMP where the most useful stuff is, while \U is the supplementary planes. (such as Egyptian Hieroglyphics and Emoji) see this chart https://www.unicode.org/roadmaps/bmp/

I think having two different escapes for this is really confusing. Instead there should just be one form of unicode escape.

The \u{} syntax is used by javascript (since ES6), lua (since 5.3), swift (who swapped from our current syntax!) and seems to be an accepted recent improvement in languages

[shawnl]

Oh, yeah I wasn't seeing that it needs to be consistent with string literal escape syntax. Yeah, I like \u{} is clearer, but \x still has to be supported as is (where it is the same as in C), because \x80 is differn't from \u{80}. \u{} should accept 1, 2, or 3 bytes.

[daurnimator]

@shawnl agreed, \x should be kept, \uXXXX and \UXXXXXX should be replaced with \u{X}.

[shawnl]

@andrewrk can we get an approved on the new character literal syntax, which will also apply for escape sequences inside strings? i'd like to code it up.

[daurnimator]

@shawnl you've got your approved, see #2129 :)