From 79be257707db40d824da2e29b025ea7435128a84 Mon Sep 17 00:00:00 2001 From: yahoNanJing <90197956+yahoNanJing@users.noreply.github.com> Date: Thu, 15 Sep 2022 11:45:16 +0800 Subject: [PATCH] Update quarterly_roadmap.md --- docs/source/contributor-guide/quarterly_roadmap.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/source/contributor-guide/quarterly_roadmap.md b/docs/source/contributor-guide/quarterly_roadmap.md index c593e859d731b..954638be18a1a 100644 --- a/docs/source/contributor-guide/quarterly_roadmap.md +++ b/docs/source/contributor-guide/quarterly_roadmap.md @@ -49,6 +49,8 @@ A quarterly roadmap will be published to give the DataFusion community visibilit - Make production ready - Shuffle file cleanup + - Periodically + - Add grpc interface for clients to actively call the cleanup for a job or the whole system - Fill functional gaps between DataFusion and Ballista - Improve task scheduling and data exchange efficiency - Better error handling @@ -57,6 +59,13 @@ A quarterly roadmap will be published to give the DataFusion community visibilit - Schedule restart - Improve monitoring and logging - Auto scaling support + - Better configuration management +- All-at-once job task scheduling +- Shuffle improvement + - Shuffle memory control + - Improve shuffle IO to avoid producing too many files + - Support sort-based shuffle + - Support range partition - Support for multi-scheduler deployments. Initially for resiliency and fault tolerance but ultimately to support sharding for scalability and more efficient caching. - Executor deployment grouping based on resource allocation