Repository navigation
Implement mutate follwoing group_by #331
Description
require(disk.frame)
setup_disk.frame()
options(future.globals.maxSize = Inf)
test_frame<-
tidyr::expand(
data=tibble(),
"Ticker":=LETTERS[1:5],
"date":=Sys.Date()-(1:100)
) %>%
mutate(minute=map(1:n(),~sample(20:120,1)+1:sample(20:120,1))) %>%
unchop(minute) %>%
mutate(px=runif(n())) %>%
as.disk.frame() %>%
hard_group_by(Ticker,date,outdir="df_test")
res1<-
test_frame %>%
group_by(Ticker,date) %>%
mutate(max_m=max(minute)) %>%
summarize(
max_of_max=max(max_m),
max_minute=max(minute)
) %>%
collect()
res2<-
disk.frame("df_test") %>%
group_by(Ticker,date) %>%
mutate(max_m=max(minute)) %>%
summarize(
max_of_max=max(max_m),
max_minute=max(minute)
) %>%
collect()
all(res1$max_of_max==res1$max_minute)
all(res2$max_of_max==res2$max_minute)
Activity
nassuphis commentedon Apr 3, 2021
nassuphis commentedon Apr 3, 2021
it seems the res2 max calculation does not respect the grouping specified by group_by
nassuphis commentedon Apr 3, 2021
max_of_max is always larger than max_minute, and equal to the max across all dates
despite the group_by(Ticker,date)
this only happens if the disk.frame is on a file, if the same calculation is off a pipeline the result is ok.
xiaodaigh commentedon Apr 4, 2021
group_byandmutatecannot be combined this way in adisk.frame. To do ANY summary you must usesummarize. I thinkmutatewill compute the max within each chunk so it's not correct.nassuphis commentedon Apr 5, 2021
doesnt
hard_group_by(Ticker,date,outdir="df_test")ensure that chunks are exactly what is needed to computethe group-wise max?
if chunks exactly match the Ticker,date grouping, then max(minute) is both the max of the group and max of chunk?
what does
hard_group_bydo? does it not create a chunking from the grouping?xiaodaigh commentedon Apr 6, 2021
It doesn't but
disk.framedoesn't know how to handlemutateaftergroup_by. I think in the next minor version I will throw an error isgroup_byis not followed bysummarizationand in the next major version I will handlemutate.The issue is that
mutateis basicallygroup_byand thenjoinback to the original in one step which can be handled indisk.frame[-]disk.frame saved to disk does not match disk.frame that was saved[/-][+]Implement `mutate` follwoing `group_by`[/+]