Repository navigation
@where does not preserve CategoricalArray #157
Description
Julia v0.6, DataFrames, CategoricalArray and Query masters
julia> using DataFrames, Query
julia> df = DataFrame(a = categorical(["a", "b", "a", "c"]))
4×1 DataFrames.DataFrame
│ Row │ a │
├─────┼───┤
│ 1 │ a │
│ 2 │ b │
│ 3 │ a │
│ 4 │ c │
julia> typeof(df[:a])
CategoricalArrays.CategoricalArray{String,1,UInt32,String,Union{}}
julia> dff = df |> @where(_.a != "b") |> DataFrame
3×1 DataFrames.DataFrame
│ Row │ a │
├─────┼───┤
│ 1 │ a │
│ 2 │ a │
│ 3 │ c │
julia> typeof(dff[:a])
Array{CategoricalArrays.CategoricalValue{String,UInt32},1}
IIUC Array{CategoricalValue} is suboptimal in terms of storage. Also CategoricalArrays API like levels(dff[:a]) is not available.
Activity
nalimilan commentedon Oct 12, 2017
Indeed
Array{CategoricalValue}isn't supposed to be used. It's kind of annoying that we don't have a way to specify that a given element type should always be stored in a given container type.alyst commentedon Oct 12, 2017
I have some minimal solution to this, by it still involves moving+extending
Enumerableabstract type definitions fromQueryOperators.jltoTableTraits.jland using them fromIterableTables.jl. So it's pretty invasive.I can open PRs, but I suppose the core JuliaData contributors already have some strategy how to fix this and the other design problems (e.g. IterableTables.jl being dependent on every data-related package).
davidanthoff commentedon Oct 24, 2017
The proper way to implement this would be to make sure that at collection into a
DataFrameany column of typeCategoricalValueuses the proper container type. Essentially just anotherelseifcase here. Having said that, I think anything like that would require a bit more thinking about larger implications. I was a couple of times close to just adding this, but in the end I'm just not yet sure whether I want this in the iterable tables interface or not (main tradeoff is complexity, I'm trying to keep iterable tables as simple as possible, and I'm not sure this would still fall into that category).davidanthoff commentedon Oct 24, 2017
YES, I completely agree. I think it would be absolutely fantastic if there was a standard way in base to say "for element type T, the default array type should be X", and if things like
[a,b,c](wherea,bandcare instances ofT) constructed anX, and andfilletc. all used that. I think a simple function likegetdefaultarraytype(::Type{T}) where T = Array{T}in base would probably do it, where customTs could add a method and the various base array creation methods would rely on it.I suggested something like that in JuliaLang/julia#22630, but probably doomed all chances of success of that proposal by embedding it in a crazy rename idea ;) I think a much less ambitious proposal along the lines of the above paragraph would still be a great idea.
nalimilan commentedon Oct 25, 2017
Maybe we should just override
similar(::Array, ::Type{<:CategoricalValue})?davidanthoff commentedon Nov 16, 2017
Where would that be picked up?
I guess I was so far mostly worrying about
[a,b,c]whena,bandcare of typeCategoricalValue(or in my caseDataValue) and how one can intercept that. I think we can already interceptCategoricalValue[a,b,c], and this line seems key to intercepting all cases.I kind of feel that if someone is asking for an
Arrayspecifically, he/she should get that specific type, right? Not sure...nalimilan commentedon Nov 16, 2017
In this case, the caller wouldn't require an
Array, just an object "similar" to the inputArray, so I'd say it wouldn't be absurd to return aCategoricalArrayif the element type is a categorical value.