Support for non-standard integer types?

[DrChainsaw]

Sorry if this is naive, but would it be possible to support fixed point representations of non-standard integer sizes? It looks like it would be a simple thing to fix:

Example:

julia> using FixedPointNumbers, BitIntegers

julia> BitIntegers.@define_integers 24;

julia> sizeof(Int24)
3

julia> sizeof(Fixed{Int24, 2}) # I think this is what causes reinterpret below to fail
4

julia> reinterpret(Fixed{Int24, 2}, UInt8.(1:3))
ERROR: ArgumentError: cannot reinterpret an `UInt8` array to `Fixed{Int24, 2}` whose first dimension has size `3`.
The resulting array would have non-integral first dimension.

Stacktrace:
 [1] (::Base.var"#thrownonint#243")(S::Type, T::Type, dim::Int64)
   @ Base ./reinterpretarray.jl:27
 [2] reinterpret(#unused#::Type{Fixed{Int24, 2}}, a::Vector{UInt8})
   @ Base ./reinterpretarray.jl:42
 [3] top-level scope
   @ REPL[8]:1

julia> Base.sizeof(::Type{<:Fixed{Int24}}) = 3

# This stuff below is correct, right?

julia> reinterpret(Fixed{Int24, 0}, UInt8.(1:3))
1-element reinterpret(Fixed{Int24, 0}, ::Vector{UInt8}):
 197121.0Q23f0

julia> reinterpret(Int24, UInt8.(1:3))
1-element reinterpret(Int24, ::Vector{UInt8}):
 197121

julia> reinterpret(Fixed{Int24, 1}, UInt8.(1:3))
1-element reinterpret(Fixed{Int24, 1}, ::Vector{UInt8}):
 98560.5Q22f1

julia> reinterpret(Int24, UInt8.(1:3)) / 2
1-element Vector{Float64}:
 98560.5

[kimikage]

I would like to relax as much as possible the limitations which come from the assumptions of the built-in types. For example, in the future, I would like to support a 14-bit FixedPoint type with 16-bit size including 2-bit padding.

However, padding (or alignment) is a troublesome issue. Personally, I don't think it's a good idea for FixedPointNumbers to interfere with Julia's memory layout policy. In other words, the following should be an invariant property.

julia> Core.sizeof(Fixed{Int24,0}) # not `Base` but `Core`
4

If we need a type of exactly 3 bytes, for example, I think it should have fields "in bytes".

julia> struct UInt24P <: Unsigned 
           data::NTuple{3, UInt8}
       end

julia> sizeof(Normed{UInt24P, 2})
3

Whether Base.sizeof should be overloaded (overridden) is also debatable, but at least the overloading is still not enough.

julia> a = reinterpret(Fixed{Int24, 0}, UInt8.(1:3))
1-element reinterpret(Fixed{Int24, 0}, ::Vector{UInt8}):
 197121.0Q23f0

julia> a[1] = 0
ERROR: Padding of type Fixed{Int24, 0} is not compatible with type UInt8.

I would start by surveying past discussions on the JuliaLang/julia and Discourse.

[DrChainsaw]

Ugh, I should have realized its not that simple.

I guess the padding happens because Fixed wraps the Int24 in a struct, right? Similar as with this struct:

julia> struct Misaligned
       a::Int8
       b::Int16
end

julia> sizeof(Misaligned)
4

It was a bit surprising that I did not encounter the issue when testing my application, but it seems like things work if the reinterpreted array is collected:

julia> aa = reinterpret(Fixed{Int24, 1}, UInt8.(1:3)) |> collect
1-element Array{Q22f1,1} with eltype Fixed{Int24, 1}:
 98560.5Q22f1

julia> aa[1] = 1
1

julia> aa
1-element Array{Q22f1,1} with eltype Fixed{Int24, 1}:
 1.0Q22f1

julia> aa .+= 3 
1-element Array{Q22f1,1} with eltype Fixed{Int24, 1}:
 4.0Q22f1

I have not put it under any further scrutinity than this though and realizing the padding/alignment issue does make me a bit nervous that things will fail unpredictably.

The tuple approach is quite interesting, but won't one end up in the painful situation of having to define almost every operator for UInt24P?

julia> aa = reinterpret(Normed{UInt24P, 1}, UInt8.(1:3))
1-element reinterpret(Normed{UInt24P, 1}, ::Vector{UInt8}):
Error showing value of type Base.ReinterpretArray{Normed{UInt24P, 1}, 1, UInt8, Vector{UInt8}, false}:
ERROR: MethodError: no method matching typemax(::Type{UInt24P})

julia> aa .+ 1
ERROR: MethodError: no method matching typemax(::Type{UInt24P})

[kimikage]

I don't have a definite solution for this issue, but I am thinking of including padding for the FixedPoint types. For example, Fixed{Int24, 1} would be a 32-bit type with 1-byte padding, and Normed{UInt12, 12} would be a 16-bit type.
For bit-packed arrays, it might be a good idea to provide some kind of lazy evaluation interface (in a separate package) as an alternative to ReinterpretArray.

I think the first step to address this issue is to define bitwidth based on typemax instead of the size of the type.

[kimikage]

cf. https://discourse.julialang.org/t/vaporware-rfc-width-qualified-fixedpointnumbers/113774