Proposal: Deploy packs using StackStorm instead of CircleCI #29

Activity

1. blag

   added

   help wantedExtra attention is neededExtra attention is needed

   community

   proposal

   status:under discussion

   exchange

   on May 23, 2020
2. arm4b commented on May 23, 2020

   

   arm4b

   on May 23, 2020

   Member

   More actions

   Yes, there are problems with the missing locking and synchronization for the index file when building many packs in parallel which could be improved.

   However it's important the way how StackStorm-Exchange was built, open, transparent and self-sufficient based on free CircleCI and requiring no additional infrastructure. Additionally,
   Anyone could just clone our system and enable CircleCI without the need to spin-up a dedicated StackStorm instance for it and avoiding complexity.
3. arm4b commented on May 23, 2020

   

   arm4b

   on May 23, 2020

   Last edited by arm4b

   Member

   More actions

   Talking about the Deployment and index synchronization issue for StackStorm-Exchange,
   looking at the CircleCI config this could be improved so we minimize the problem making races really rare to hit.

   Here is how to do that:

   Minimize the Exchange index conflict

   • this is achievable by making the index update taking minimum possible time
   • We should re-download index on deployment step before re-generating it updating/pushing it. This means the time window when the race/conflict possible is around second or so.
   • pushing outdated index should result in a git conflict and deployment failure, which should be safely restarted, see below

   Make deployment operations idempotent

   • write all the steps in deployment step so they're fully idempotent
     • if deploy failed due to index conflict, - re-starting the deployment step will lead to retry for the failed operation, skipping all the previous steps that were successful
     • re-run the deployment step deploy || deploy to minimize human if deployment failed due to index conflict (race) if other pack already updated the index

   This way we can minize the StackStorm-Exchange pack deployment issues up to state when human interruption is almost not needed. Some of those steps are already in-place in the current CircleCI Deployment, so this should be fairly simple change and a quick win.
4. arm4b commented on May 23, 2020

   

   arm4b

   on May 23, 2020

   Last edited by arm4b

   Member

   More actions

   Alternative conflict-free approach: deploy isolated per-pack index update

   • instead of every pack pushing the change to main index.json, let them push isolated per-pack index meta, there is already dir for that: https://github.com/StackStorm-Exchange/index/tree/master/v1/packs
   • Setup every minute CircleCI cron job that will iterate over the pack dirs with their meta indexes and generate main StackStorm-Exchange index if there is a diff.

   ^^ This will require more work comparing to previous enhancement, but this approach removes the possibility of index conflict at all.

   So there are plenty of options making things simple, working and self-sufficient, instead of relying on external StackStorm or lock-control instance.
5. cognifloyd commented on Apr 7, 2022

   

   cognifloyd

   on Apr 7, 2022

   Member

   More actions

   We have moved Exchange CI to GHA away from CircleCI, so I think we can close this.
6. arm4b commented on Apr 7, 2022

   

   arm4b

   on Apr 7, 2022

   Member

   More actions

   Good call, agreed 💯
7. arm4b

   closed this as completedon Apr 7, 2022
8. arm4b

   removed

   status:under discussion

   on Apr 7, 2022