Repository navigation
[RFC] Use Github Actions for StackStorm-Exchange CI and Maintenance #63
Description
Current State
StackStorm-Exchange is using CircleCI for (1) pack testing and (2) updating the exchange index. Most of the CI logic is centralized in StackStorm-Exchange/ci to simplify exchange-wide CI updates. The CircleCI config (.circleci/config.yml) is a key piece of the CI infrastructure that cannot be centralized; a copy is kept in every pack on the exchange. CircleCI config can be edited by pack authors to add jobs, add docker images or steps to the standard jobs, etc. Also, the schedule for when weekly pack tests occur (Saturday/Sunday) is in the pack-local circle config.
All other administrative tasks across the exchange require manual intervention by oneor more of the project maintainers. There are several versions of shared scripts that the maintainers pass around (in StackStorm-Exchange/ci, StackStorm-Exchange/exchange-tools, and StackStorm-Exchange/exchange-misc) to handle these administrative tasks. Other updates might be handled by ad-hoc scripting.
aside: this proposal has nothing to do with recent discussion around dropping python2.7 testing on the exchange. It is about optimizing our CI infrastructure for the Exchange.
Goals
- Automate as many of the manual administrative tasks as possible. Wherever possible, prefer solutions that happen on-push, or on-merge, or other no-touch triggers.
- Allow individual packs to add additional custom testing workflows.
- Allow individual packs to customize the standard workflows/jobs (eg add docker images, add system deps, setup custom testing harnesses, download extra repos needed for tests, ...)
- Given that customization, minimize the effort required to:
- adjust the weekly test schedule for multiple packs (Saturday/Sunday/etc).
- do exchange-wide pack updates for our standard pack CI workflows: pack tests, index update. An autonomous solution that updates pack CI config in all repos once a "master" copy of the config gets merged would be ideal.
Proposal
I recommend we move the Exchange's CI workflows from Circle CI to Github Actions. This includes writing a variety of new workflows to automate as many administrative tasks as possible.
For at least one of those workflows, use a tool that allows us to merge yaml files so that the standard CI jobs can be extended in ways that do not prevent automated updates of the file. Some form of centralized "variables" could be used to define, for example, the test schedule for various packs.
Other CI provider options
Many projects (including StackStorm) have been abandoning Travis CI for many reasons, so that is not a viable option for the exchange.
Circle CI has served us well. Circle CI uses a single config file for all of the CI workflows. However, a single config file for all CI makes it more difficult to both allow customization, and simplify exchange-wide CI updates.
The first bullet under Goal 4 provides another reason not to use CircleCI. The schedule is another potential merge conflict that makes doing exchange-wide pack updates difficult.
Why Github Actions (GHA)?
Looking at the goals listed above, writing workflows for 1 could probably be accomplished with any CI provider. But GHA might have a slight edge as it is much closer to the Github API, and many community-written-actions alreadfy implement some of the workflows, or steps in those workflows, that we'll need.
Goal 2 is the clearest winner with moving to GHA. This comes free with GHA, as each workflow is a separate file in the repo. So, any custom workflow files will be safe from exchange-wide updates to the standard workflows.
The tension between goals 3 and 4 require additional effort. But, with additional tooling we can define extension points in our standard CI pack test workflow(s) so that it can be updated by both pack authors and by exchange-wide CI updates. There are few yaml templating/merging options available. At least one of them should be runnable within github workflows.
Proposed new workflows
For the first bullet under goal 4, adjust weekly test schedule, we could have a centralized file that assigns packs to CI slots (currently Saturday or Sunday). Then, when that file gets updated (eg w/ a new pack), that can trigger a workflow that pushes out the assigned schedule to packs that need an update.
For the second bullet under goal 4, exchange-wide pack CI updates, we could have a central template file for the standard CI workflow config. When a PR that updates that template is merged, that would trigger a CI workflow that goes through all the exchange's packs, regenerates the affected standard workflows. That regeneration would take into account the schedule, and it would merge any customizations into the standard workflow.
Also, when the files that provide the customizations are updated in one of the packs, that should trigger regenerating the affected workflow file as well.
YAML merge tool options
Here are two options:
modulesync/pdkgflows
I would prefer gflows, assuming it works well for our use cases.
modulesync and/or pdk
Written in ruby.
Quoting @nmaludy: https://stackstorm-community.slack.com/archives/CSBELJ78A/p1612183089198000
The way the Puppet community handles problems like this is to use a tool called
modulesyncand/orpdk(they perform the exact same).Basically they work by having a "template" directory structure with a bunch of ERB (think Jinja) templates. Those templates get rendered based on a config file
.sync.ymlcustomizations that can be unique per-repo (pack in our case).This way if one pack needs to modify the CircleCI config by adding jobs or service containers, steps, whatever... It is very easy to do so and those differences from the "normal" are documented and able to be maintained, even when the template config changes.
These tools also provide a way to mass-render and make PRs across a huge number of repos. Basically, for each repo, make branch, run
modulesync, push branch, make PR.The other difference here is that these tools push the CI scripts to each repo, rather than having a central CI repo that everyone has to clone. Bonus points on this is that each repo has its own Gemfile.lock (think requirements.txt ) that can be used for correctly hashing CI requirements for a repo. I know we've had the problem in the past where a dependency in some cloned repo was causing problems and even though we updated it, the CI cache wasn't changed because we weren't properly aggregating our
requirements.txt.
https://github.com/voxpupuli/modulesync
https://puppet.com/docs/pdk/1.x/pdk_reference.html
gflows
Written in go.
gflows is a tool (and a github action) that uses jsonnet or ytt to template github workflow files.
Being a golang static-binary, it can be simply downloaded/run both in CI, and locally on pack authors' machines. Thus, the pack author will not have to set up any special environment (ruby, python, or otherwise) to regenerate any customized standard workflow file locally when authoring the pack.
https://github.com/jbrunton/gflows
Alternatives
One approach to adding customization within the CircleCI config is to one or more bash scripts throughout the standard workflows so pack authors can add their customization there. This is the approach taken by: StackStorm-Exchange/ci#101
This satisfies only part of goal 3. Other changes, like adding docker images or adding workflows, would also need a yaml merge solution.
Request for Comment
Does anyone have any additional pros/cons or opinions to add about:
- moving StackStorm-Exchange from CircleCI to Github Actions
- automating exchange maintenance with new workflows
gflowsvsmodulesync- using
gflowsto allow explicit customization of the pack CI
Activity
arm4b commentedon Feb 3, 2021
That's a very detailed proposal, thanks for putting it together! 👍
My concern is that the occurrence and severity of the problem highlighted and trying to solve it this way is unproportionally minor comparing to a proposed major change, eg. switching the entire CI platform.
From the other side we all see how GH Actions are gaining popularity, people are getting more fluent with them and might be a good de-facto standard CI for the Exchange in the future. From the disadvantages @nmaludy mentioned one time is that GH Actions don't have SSH debugging that CircleCI has which might be very helpful sometimes.
I'm thinking that we may find more advantages with time about moving to GH Actions.
@nmaludy @blag Whare are the other pain points behind the Exchange with CircleCI, could GH Actions help there?
blag commentedon Feb 4, 2021
Before we change over from CircleCI to GHA, let's take minute and get some data on how many CircleCI configs have been customized, and how invasive those customizations have been. I would guess that not many packs have customized a great deal.
I think a larger issue is that we've had updates to the StackStorm-Exchange/ci sample CircleCI config that haven't been pushed out to all packs, so there's probably a few previous versions of StackStorm-Exchange/ci's CircleCI config in various Exchange packs. It would be nice if we had an automated process to push out changes to StackStorm-Exchange/ci's CircleCI config to all packs. And we can either do that with StackStorm itself, or we can do it with GitHub Actions in the StackStorm-Exchange/ci repository itself (eg: a deploy workflow that runs for each PR that touches
.circle/circle.yml.sample, and grabs the diff of that PR for the.circle/circle.yml.samplefile and tries to apply it to the.circleci/config.ymlfiles of all packs, reporting any failures to Slack so a human can look at it).I don't see the weekly test scheduling as a big deal, except for massive failures (like Python 2.7 tests failing across the board). And even then, it's just a matter of deleting a bunch of email. I don't really know what benefit we get from keeping/controlling that information in a single place, since in the years that I've been working with StackStorm, we haven't ever adjusted that schedule, and I don't really see a need to. The weekend pack CI runs are the exact same workflows that run for every opened PR and every PR merge. And the ideal is that the weekend tests are only an early warning system for StackStorm pack devs (eg: transitive dependency causing issues). In practice, 99% of the weekend pack CI failures are due to GitHub Personal Access Tokens expiring due to disuse.
There's a balance to be struck here. Some packs have a legitimate need to be able to customize the test environment, and I absolutely think that we should support whatever is needed. However, for the sake of keeping our cognitive and maintenance burdens as low as possible, we should strive to keep the test configs as consistent as we reasonably can. Letting pack authors go crazy with their test configs is going to end in frustration and tears for everybody.
To summarize my post here:
MACHINE_USERNAMEandMACHINE_PASSWORDat all, since the PATs that expire all the time are only used to clone pack repositories that are already public.Sorry to pour cold water on this idea. Having maintained StackStorm Exchange for a few years now, I'd like to think I have a pretty good perception of what our priorities should be.
arm4b commentedon Feb 5, 2021
Some of the StackStorm-Exchange problems are highlighted in https://github.com/orgs/StackStorm-Exchange/projects/1
cognifloyd commentedon Feb 10, 2021
I looked into what it would take to switch just the
deploystep to GHA, leaving the tests in CircleCI (at least for the time being).What the deploy step does
The deploy step does two related things. First it pushes new tags into the pack repo when the version changes in pack.yaml. Then it updates the index.
The first part would be very easy in GHA as select workflows can have automatic write access to the repo. By default PR workflows do not have write access or access to secrets. Schedule and other event triggers can however.
The second part pushes updated metadata to the
indexrepo. The token github provides only provides access to the current repo, however. So, we would need to create a ssh key pair, put the public key in thecirepo, and keep the private key in a secret that is made available during the workflow through an action like:webfactory/ssh-agentHow to move
deployto GHA without moving tests.Looking at GHA workflow triggers, we could use either
check_suiteorcheck_runto allow triggeringdeployonly after CircleCI has finished its work.Such a workflow would need one of these triggers:
But, we can only trigger once the check_suite/check_run completes, and only on the repo's default branch.
The workflow that runs would need to investigate the
payloadto determine if it can proceed to actually run the rest of the deploy step. The interesting bits of each payload are:It would need to make sure any relevant checks (like
app.slug=circleci-checks) haveconclusion=successbefore continuing with the workflow.cognifloyd commentedon Feb 10, 2021
Re customized CircleCI config:
Before last week's exchange-wide pack CI updates, some packs had reformatted the CircleCI config slightly. Now, the CI config has been standardized across all packs with these exceptions:
arm4b commentedon Oct 29, 2021
Looking back, following the Security discussions https://github.com/StackStorm/private-discussions/issues/5 SSH access to CI system for Exchange Packs looks like a downside/risk these days.
Nowadays GH Actions stabilized and gained its trusted reputation as a de-facto CI standard in Github that works flawlessly.
+1 for migration from CircleCI to GH Actions for a native & seamless integration to fix the current pain points of StackStorm Exchange.
cognifloyd commentedon Oct 30, 2021
I no longer believe a shared CircleCI + GHA is a possibility because so much of the exchange infra is broken with CircleCI changes that force using an ssh deploy key. That in turn breaks our deploy workflow because it conflicts with how we're using the PAT to clone and modify repos in CircleCI.
This is an overview what I think we need to do to move the current CI to GHA. Some of my other comments above explain additional workflows and future improvements, but ignore the bits about combining CircleCI with GHA.
We have a common set of CI workflows for all StackStorm-Exchange workflows:
I would start with converting the build_and_test_python36 job to GHA before worrying about the deploy job. The deploy job will be more involved since it requires changes across repositories.
For CircleCI, we had to copy the .circleci/config.yml from a master copy to each pack repo.
We use the same steps for so many repos, I would love a way to not have copies of the jobs config in each repo. I believe we can use Composite Actions to centralize (pieces of?) these jobs.
https://docs.github.com/en/actions/creating-actions/creating-a-composite-action
The master copy of the exchange CircleCI config is here:
https://github.com/StackStorm-Exchange/ci/blob/master/.circle/circle.yml.sample
That ci repo would probably be a good place to put the composite actions, assuming we can put more than one composite action in the same repo (edit: we can). Then, in each pack repo we would have a much lighter weight GHA workflow that specifies the cron schedule for weekly tests and uses those composite actions.
One gotcha in all of this, is there are a couple of repos that had to modify the main workflow: the vault and zabbix packs:
Here are the two changes are in the vault pack. We might need a way to add a few pack-specific commands to allow packs to inject changes like these:
https://github.com/StackStorm-Exchange/stackstorm-vault/blob/master/.circleci/config.yml#L33-L35
https://github.com/StackStorm-Exchange/stackstorm-vault/blob/master/.circleci/config.yml#L45-L46
The zabbix pack has added some jobs, but luckily the shared jobs are the unmodified, so those new jobs can become a separate workflow after we switch the exchange-wide jobs to GHA.
arm4b commentedon Nov 1, 2021
Good stuff. Nice idea with the GH composite actions, similar to CircleCI orbs 👍
And yeah, there should be a basic way for pack maintainers to do something outside of the default CI pipeline.
Perhaps having a new PoC stackstorm-exchange pack would be a good way to experiment with all the machinery & show it.
cognifloyd commentedon Nov 12, 2021
OK I stubbed together some composite actions. They will not work yet, but hopefully they're a good starting point.
https://github.com/StackStorm-Exchange/ci/tree/gha/gha
To use these, I imagine a workflow that looks something like this:
We will still need to manage the cron schedule in each pack repo.
In circle, each of the job steps were shell scripts that were painful to update. Using the composite-actions should alleviate that because we won't have to synchronize any shell scripts across all of the pack repos. Instead each of the common steps is just a reference to our composite actions with input vars to allow per-pack customization if required.
cognifloyd commentedon Nov 12, 2021
For the deploy workflow, we have a variety of problems:
repoaccess, then that gives you access to all repos that user can accessSerializing index updates
A persistent service would make serializing updates more natural, but then we have to deal with a persistent service (If we do go with a persistent service, GCP's free-tier offers 1 free e2-micro VM instance per month).
But, maybe we can get away with creating a semi-persistent "service" using a github actions workflow.
Based on the Github Usage Limits, workflows are limited as follows:
If we had a workflow running for the max of 72-hours, splitting that into 6-hour jobs would mean a workflow with 12 jobs where the job concurrency is limited to 1. But I'm not sure how to start a workflow every 72-hours. Using a cron scheduled workflow, we could easily do a workflow that runs every day (eg
0 12 * * *). A 24-hour workflow would have 4 serial 6-hour jobs (maybe a matrix of 4 jobs, concurrency limited to 1).I don't think there's a good way to receive webhook events from github within a github action workflow. But we can poll Github's events API for StackStorm-Exchange org events. The index is meant to be eventually consistent since we have to serialize index updates, so this Events API caveat should not be a problem:
Index update workflow
So, the index update workflow/job would do something like this:
checkout index repo
Pack Deploy workflow
Each pack's deploy workflow, then would only have to:
Sane Github credentials management
Doing index updates this way fits within how Github currently manages tokens for workflows (read/write for the current repo; read-only for everything else), so we would not need any PATs or the persistent credentials that come with a Github app or an oauth app.
cognifloyd commentedon Nov 12, 2021
OK. I've spent a lot of time figuring this out. I don't know when I'll have time to pick it up again. If someone else can please pick this up and work on any of these pieces, I would appreciate the help.
arm4b commentedon Nov 25, 2021
I like the way how the pack CI logic would be hidden via abstraction behind the GHA composite actions in the [RFC] Use Github Actions for StackStorm-Exchange CI and Maintenance #63 (comment). The example is nice! 👍
I don't think long-running always-polling CI is a good idea. Hard to manage, hard to debug, the logic would be surprising (bad) and it's not what GHA was designed for. Also, it's easy to miss an event there if the workflow was interrupted for whatever reason.
Using an external service and storage/state is not an option. There were ideas to rely on external st2, but the point is that users should be able to create their own pack index without external tools and services, like the old (current) Exchange was (is). Some notes in here: Proposal: Deploy packs using StackStorm instead of CircleCI #29
I understand you want to rework the entire way we build an index. Avoid
pushfrom each pack to the Index at all and do onlypullfrom the GHA Index repo? The advantage, as I understand is - we won't need PAT and similar secrets anymore for every repo which would improve the security model. @cognifloyd I definitely like the idea and the advantages behind the model 👍, despite being cautious about the redesign.With what you propose, we can avoid a long-running workflow and run the Index Update Workflow once every 5 mins by cron, which will store in git some state/checksum or similar to continue where we left off. Worst case, if events API won't work, can use https://docs.github.com/en/rest/reference/repos#list-organization-repositories list-org-repos API checking the
updated_ator something similar API for each repo and doing the needful :)Eventual consistency is fine,
5minswould be good enough for the index rebuild.11 remaining items
cognifloyd commentedon Dec 9, 2021
OK. This is ready for more eyes. Thanks go to @ym-dubler for helping to test, invalidate a bunch of my assumptions, and push this forward! And thanks to LogicMonitor for dedicating resources to this issue!
We need to merge changes across multiple repos in this order:
ghutility to access github API StackStorm-Exchange/exchange-tools#2index/v1/exclude_packs.txtfile are used.I already merged the
ghabranch on the test pack into its default branch (aaainstead ofmaster). Here are the workflows we added:These will need to be updated to replace
@ghawith@masteronce we've got the above changes merged.After that I can start working on pushing these workflows to all the packs. I will need a senior maintainer to help disable CircleCI as we switch each pack over to GHA.
cognifloyd commentedon Dec 9, 2021
So, just to clarify: the tag release workflow adds the tag on push to master (or whatever is the default branch) if the latest tag doesn't match the current version in pack.yaml.
lm-ydubler commentedon Dec 9, 2021
Happy to have helped and glad to have worked with you for a solution.
cognifloyd commentedon Dec 17, 2021
@winem @lm-ydubler and I just had a meeting, we talked about doing this to roll the GHA updates out:
@winem approves PRs
Merge Add Github Actions workflows and composite actions for packs to use StackStorm-Exchange/ci#121
Merge Do index updates via GitHub actions StackStorm-Exchange/index#24
Update all pack repos:
echo CircleCI is disabled on StackStorm-ExchangeA senior maintainer deletes PATs/secrets from CircleCI for all pack repos
cognifloyd commentedon Dec 29, 2021
Should we add the gha workflows to https://github.com/StackStorm-Exchange/exchange-template ?
arm4b commentedon Dec 29, 2021
It doesn't contain any CI workflows, so probably good as is without adding another dependency to the repo.
cognifloyd commentedon Dec 29, 2021
OK. afaict, we have excised CircleCI from packs on the exchange.
The zabbix pack still has some custom integration test jobs that run on CircleCI in addition to our standard GHA-based tests.
Can a senior maintainer (an org admin) please:
cognifloyd commentedon Dec 29, 2021
I created some skeleton workflows to show the outline of GHA workflows to bootstrap a pack repo and add maintainers to it.
StackStorm-Exchange/exchange-incubator#172
If you have some time, please pick one or more of the tasks in those workflows and implement them. It's on the
ghabranch of the exchange-incubator, so anyone on the TSC can push to that branch.cognifloyd commentedon Dec 30, 2021
I would like to use that repo as a template repo to bootstrap new pack repos.
https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/creating-a-template-repository
This would simplify maintenance for the misc things we find that need to go in new packs as mentioned in StackStorm-Exchange/exchange-incubator#152 (comment) and StackStorm-Exchange/exchange-incubator#153.
I think https://github.com/StackStorm-Exchange/exchange-template is designed to be used by pack authors. But does that repo have much utility for pack authors? I suspect something like https://github.com/EncoreTechnologies/cookiecutter-stackstorm would be more useful to pack authors. So, would it be a ok to repurpose it for setting up new packs?
cognifloyd commentedon May 19, 2022
Progress report on bootstrapping packs via GHA:
So, once that last PR is merged, we can bootstrap packs from incubator PRs with a
!bootstrap packcomment.Possible future workflows: