Design Discussion: Implementing Queueing logic to enable parallel run of index and download logics #64

Problem Statement:

We are aiming to improve the system by running the index and download logics in parallel. To achieve this, we need to implement a queueing mechanism that can effectively manage these two processes.

Proposed Solution - Use Redis Sets as Queues

How it works:

Utilize a Redis set (or two sets, one for workflows and one for actions) to queue items for processing.
Download logic will insert items to these sets after it downloads it.
Index logic will pop items from these sets and index them.

Pros:

Simplicity: Easy to implement and manage.
No Duplicates: Sets prevent duplicate items by default.

Cons:

Ordering: Redis sets do not guarantee order, which could be an issue if ordering is essential.

What do you think?