QueuedTracking setup on multiple servers (new guide)

[mattab]

Here are some notes I wrote earlier and thought it would be useful to put in the FAQ maybe?

How do I setup QueuedTracking on multiple tracking servers?

Say you have

• 4 servers,
• and setup 8 workers in QueuedTracking,
• and disabled the option in the settings to process the queue during the tracking request,

then on each of your 4 frontend servers, you need to:

• setup 2 crontab to run each minute and which execute the command:
  ./console queuedtracking:process --queue-id=X

Where X is the queue ID. Each server handles 2 queues. So the 4 servers handle the 8 queues.

Queue ID starts at 0.

Notes:

• execute the command ./console queuedtracking:monitor to track the state of the queue
• When using multiple workers it might be worth to lower the number of "Number of requests to process" to eg 15 in "Plugin Settings". By default 25 requests are inserted in one step by using transactions. This means different workers might have to wait for each other. By lowering that number each worker will block the DB for less time.
• you can optionally receive an email when the number of requests queued in a single queue reaches a configured threshold. You can configure this in your config/config.ini.php config file using the following configuration:

[QueuedTracking]
notify_queue_threshold_emails[] = example@example.org
notify_queue_threshold_single_queue = 250000 

[okossuth]

Hello
Can multiple workers process the same queueid ? We have a situation were we had originally 4 workers processing 4 queues, but due to slowness in our setup those workers were not fast enough to process the queues and now we have a ton of requests pending to be processed. Can we use multiple workers to process these specific 4 queues somehow? Thanks

[danielsss]

Hello
Can multiple workers process the same queueid ? We have a situation were we had originally 4 workers processing 4 queues, but due to slowness in our setup those workers were not fast enough to process the queues and now we have a ton of requests pending to be processed. Can we use multiple workers to process these specific 4 queues somehow? Thanks

+1

[okossuth]

Forgot to mention we are using matomo 3.14.1

[tsteur]

Hi @okossuth @danielsss
Multiple workers work on the same queue automatically if you don't set the queue-id option. However, they don't work on it at the very same time in parallel. They work on it one after another. It's not possible that multiple workers work on the very same queue in parallel (only one after another) as otherwise the tracked data could end up wrong and random visits with eg 0 actions could be created etc.

[uglyrobot]

Just a suggestion, if we used lpop or better blpop then that would eliminate potential race conditions, allow use of only one shared queue, and unlimited workers processing the same queue with no need for complicated locking. It would also scale to any level. We had our workers stop for a day now we have a 60GB queue size that we are trying to catch up with, but it's taking forever as only one worker can process each queue.

The main downside being that if the processing of the popped data fails then there are no retries. However I don't think that's a big deal, and even if it is can work around that by adding the data back to the beginning of the list, or into a failed queue.

[tsteur]

Thanks @uglyrobot the problem is less around redis but more about Matomo and how it tracks data etc. There's a related issue in core eg matomo-org/matomo#6415 basically if two workers were to work on the same a queue and one worker was processing the second tracking request of a visit slightly faster than another worker does the first tracking request Matomo could store wrong data in its database and sometimes even additionally create multiple visits.

[StevieKay90]

I seem to be having issue with the following command which is stopping me from executing this correctly;

./console queuedtracking:process --queue-id=X

When activating ./console queuedtracking:process --queue-id=0 specifically for queue-id=0, it doesn’t work, i get this error

ERROR [2020-07-06 09:10:58] 4700 Uncaught exception: C:\inetpub\wwwroot\vendor\symfony\console\Symfony\Component\Console\Input\ArgvInput.php(242): The “–queue-id” option requires a value.

It works for fine for “./console queuedtracking:process --queue-id=1”

is this a known issue or am i doing something incorrectly?

[tsteur]

@StevieKay90 could you send us the output of your system check see https://matomo.org/faq/troubleshooting/how-do-i-find-and-copy-the-system-check-in-matomo-on-premise/ ? The output should be anonymised automatically.

[StevieKay90]

Thanks for quick response!

its here;

matomo_system_check.txt

[StevieKay90]

Hi Thomas, i've just found out that if you set ./console queuedtracking:process --queue-id=00 it works, good help from the community!

One thing which is vexing me though is why queue=0 seems to the most full, its not evenly distributing the load. The other queues are just a handful of requests in but queue 0 has over 200
Is there a way to stop this?

[tsteur]

Thanks for this. I still can't reproduce it just yet. @sgiehl any chance you have a windows running with Matomo and can try to reproduce this? I'm wondering if it's maybe windows related.

[sgiehl]

@tsteur don't have a matomo running directly on windows. But I could check if my Windows VM where I had set this up once is still running. But I guess it's already outdated and I would need to set it up again. Let me know if it's important enough to spend time on it.

[StevieKay90]

Hi all @tsteur @sgiehl thanks for taking a look into this

as you can see its quickly is becoming a big problem here for me, i'm going to have to stop queued tracking

This has happened since the upgrade, previously i haven't run into this issue. Any interim advice would be great

[tsteur]

@StevieKay90 could you remove the queue-id parameter? Then the requests in the first queue should get processed

[StevieKay90]

@tsteur I have done, i'm not using command line at all now i'm using the "Process during tracking request option"
It just seems to heave the vast majority of requests into one queue and as its one worker at a time, it ant handle all the requests in the queue