[Question] Can groupby agg scale in HPAT? #180

Hi,

I am trying to use HPAT to accelerate data science workloads, especially the ETL process.

The data frame I am using contains 21,721,922 rows and 45 columns. All the data entries use float64 dtype. There is no missing data after cleaning.

I was putting the following code into an HPAT decorated function. It simply groups the data frame by ``year" and calculates the average number for each year. I am tracking the execution time of the groupby-agg operator.

    t0 = time.time()
    tmp1 = df.groupby('YEAR')['INCTOT'].mean()
    tt = time.time() - t0

I am using a server with 2 x Intel(R) Xeon(R) CPU E5-2699 v4 CPU, where it has 44 cores in total.

The results look like this:

Baseline is to use Pandas only without HPAT.

We observe that when the number of processes used increases, the time spent on groupby-agg also increases. Since GroupBy-agg is a simple map-reduce parallel pattern which should be able to parallelize, the observation is a bit weird to me as far as I understood.

Second, even we only use one thread, applying HPAT gives slowdown compared to pandas.

The groupby-count results of my dataset. Note that in each year, plenty of data entries exist--- there should have sufficient parallelism.

Am I missing something? Could you give some suggestions on how should I do to accelerate the groupby-agg operation using HPAT?

Thank you so much.

Best regards,
Hongyuan Liu