Events Dead Letter Queue alert - #1605
Conversation
There was a problem hiding this comment.
If the workflow is created and the sensor fails to observe the response for some reason (maybe like a timeout) can another submission/retry attempt create a duplicate since each submission uses generateName?
There was a problem hiding this comment.
I think generateName resolves when the workflow gets created by the controller, so even if the same trigger fires twice, it should generate two different names
| (f":rotating_light: *An automated workflow created from trigger {name} has failed*.\n" | ||
| "This is likely due to a misconfigured workflow template name and/or parameters in the trigger template. " |
There was a problem hiding this comment.
Would it be better if the message say submission failed rather than workflow failed? It would also help to include the workflow template and namespace so the failure is easier to locate?
Also should we avoid assuming template/parameter misconfiguration, since it can also be caused due to other factors like timeout or premissions (basically something from our side) ?
There was a problem hiding this comment.
Would it be better if the message say submission failed rather than workflow failed? It would also help to include the workflow template and namespace so the failure is easier to locate?
I think I can add the template name easily enough, but getting the namespace will require a bit more thought - it isn't known at the time this webhook is called, so I'd have to parameterize the whole Slack message. I'll give it a try and implement it if it's fairly trivial, but if not I'll make a ticket to enhance the message.
Also should we avoid assuming template/parameter misconfiguration, since it can also be caused due to other factors like timeout or premissions (basically something from our side) ?
Yeah I think the wording does need some revision, but what I wanted to prevent was us getting lots of issues reported that ultimately boil down to a typo in the template name (which should be more likely than Workflows being down!). This alert will only trigger if the workflow cannot be created on the cluster at all - whether it gets stuck in pending or not doesn't matter for this
There was a problem hiding this comment.
Could we add minimum: 0 to numRetries in both schemas? If someone sets numRetries : -1 it can silently disable retires in the sync hook, which could hide configuration mistake?
There was a problem hiding this comment.
Good point, will add some validation
2 participants
AP-1285
The DLQ in Argo Events allows the implementation of an additional trigger that runs if all the retries of the primary trigger (i.e. the workflow) fail. This PR gives the ability to specify a Slack channel that an alert will be sent to if the workflow fails to trigger. The Slack channel must have the Analysis Platform Alerts bot added for the messages to work, but this can be done in direct messages with the bot to avoid congesting 'actual' channels