Feature: Can we have distributed configuration for each services?

[twinkle-simple]

Description

Hi team,

First of all, thank you for building this great tool — it's very helpful for GitOps workflows with APISIX.

❓ Problem

Currently, adc.yaml stores all configurations (services, routes, upstream, etc.) in a single YAML file. As the system scales with many services and teams, maintaining a monolithic file becomes hard to manage, review, and track changes.

🚀 Feature Request

Please support loading adc.yaml configuration from multiple files, structured by service or resource type. For example:

adc/
├── services/
│ ├── mockbin.yaml (having routes and import upstream in this file)
│ └── httpbin.yaml (having routes and import upstream in this file)
├── upstream/
│ ├── jane.yaml
│ └── john.yaml

Thank you so much !!

[bzp2010]

No doubt, this is meaningful. We have long been aware of it, so we have provided a brief solution for it.

For monorepo

The files can be placed according to the structure you mentioned above.

Next, you can use glob expressions to reference multiple files in multiple directories.

Ref: https://github.com/api7/adc/releases/tag/v0.17.0

For polyrepo

In the workflow, you need to either clone multiple external repositories or use them as Git submodules in your local repository. This allows you to download all configurations from both the local and external repositories using a recursive clone.

The subsequent process remains consistent and still uses glob expressions.

In terms of workflow, you may wish to consider configuring some webhooks for your organisation's repository. This will enable your developers to trigger configuration validation or synchronisation in the main repository whenever external repositories update gateway configurations.

---

BTW, you need to keep your developers in line and make a rule for naming resources. Otherwise, if resource names are the same, they might accidentally overwrite each other.

[twinkle-simple]

Hi @bzp2010, this is my current configuration after splitting:

ll services/
total 40
drwxr-xr-x 2 root root 4096 Jun 13 13:27 ./
drwxr-xr-x 5 root root 4096 Jun 13 13:24 ../
-rw-r--r-- 1 root root  532 Jun 13 13:36 catch-all-service.yaml
-rw-r--r-- 1 root root  607 Jun 13 13:36 coroot-service.yaml
-rw-r--r-- 1 root root  896 Jun 13 13:35 httpbin-service.yaml
-rw-r--r-- 1 root root  783 Jun 13 13:36 kafka-service.yaml
-rw-r--r-- 1 root root  535 Jun 13 13:35 soap-service.yaml
-rw-r--r-- 1 root root  373 Jun 13 13:35 sse-service.yaml
-rw-r--r-- 1 root root  585 Jun 13 13:36 tcp-check-service.yaml
-rw-r--r-- 1 root root  631 Jun 13 13:36 test-service.yaml

and after using the statement for applying configuration:

adc sync -f /root/spec/services/* --backend apisix --server <server_ip> --token <token>

it just apply for only 1 catch-all-service.yaml:

This is each service configuration:
catch-all-service.yaml

services:
  - name: Catch All Service
    upstream:
      name: default
      scheme: https
      type: roundrobin
      nodes:
        - host: apps.learnwebservices.com
          port: 443
          weight: 1
          priority: 0
      retry_timeout: 0
      pass_host: pass
    routes:
      - name: all
        uris:
          - /*
        methods:
          - GET
          - POST
          - PUT
          - DELETE
          - PATCH
          - HEAD
          - OPTIONS
          - CONNECT
          - TRACE
          - PURGE

coroot-service.yaml

services:
  - name: coroot
    upstream:
      name: default
      scheme: https
      type: roundrobin
      nodes:
        - host: apps.learnwebservices.com
          port: 443
          weight: 1
          priority: 0
      retry_timeout: 0
      pass_host: pass
    routes:
    - name: coroot
      uris:
        - /*
      methods:
        - GET
        - POST
        - PUT
        - DELETE
        - PATCH
        - HEAD
        - OPTIONS
        - CONNECT
        - TRACE
        - PURGE
      hosts:
        - lt-cooroot.com
      plugins:
        file-logger:
          path: logs/error.log

httpbin-service.yaml

services:
  - name: httpbin Service
    description: "This is the httpbin Services proxy service"
    labels:
      app: httpbin
      custom_key: custom_value
    upstream:
      name: httpbin-upstream
      scheme: http
      type: roundrobin
      hash_on: vars
      nodes:
        - host: httpbin.org
          port: 80
          weight: 1
          priority: 0
      retry_timeout: 0
      pass_host: pass
    routes:
      - name: Generate UUID
        methods:
          - GET
        uris:
          - /uuid
      - name: Anything
        methods:
          - GET
        uris:
          - /anything
      - name: Rate Limited IP
        methods:
          - GET
        uris:
          - /ip
        plugins:
          limit-count:
            _meta:
              disable: false
            count: 2
            time_window: 10
            rejected_code: 429
            policy: local

kafka-service.yaml

services:
  - name: Kafka Service
    upstream:
      name: kafka-loadtest
      nodes:
        - host: 10.60.45.50
          port: 9095
          weight: 1
        - host: 10.60.45.51
          port: 9095
          weight: 1
        - host: 10.60.45.52
          port: 9095
          weight: 1
      timeout:
        connect: 60
        send: 60
        read: 60
      type: roundrobin
      scheme: kafka
      pass_host: pass
      keepalive_pool:
        idle_timeout: 60
        requests: 1000
        size: 320
    routes:
      - name: kafka-test
        uris:
          - /kafka
        methods:
          - GET
          - POST
          - PUT
          - DELETE
          - PATCH
          - HEAD
          - OPTIONS
          - CONNECT
          - TRACE
          - PURGE

[bzp2010]

You may not have noticed this:

It now supports such syntax:
adc sync -f 'spec/*.yaml'
adc sync -f 'spec/**/*.yaml'

Note that the glob expressions need to be quote-wrapped because we don't want the glob implementation in the shell to intervene for a uniform user experience.

https://github.com/api7/adc/releases/tag/v0.17.0

---

Technically, the ADC does not support the following commands:

adc sync -f a.yaml b.yaml [1]

Instead, it requires:

adc sync -f a.yaml -f b.yaml

The adc sync -f 'spec/**/*.yaml' mentioned above is a special case where the glob expression is passed to the ADC itself and evaluated by it.

If you don't add the quotes, bash or zsh will evaluate it and end up generating [1]-style commands, which the ADC doesn't allow.

[twinkle-simple]

Hi @bzp2010

First of all, Thank you so much for your supports.
I already applied it since it can be synced for multiple files successfully.
Now it just has 1 problem that I want to create another specific upstream files which is included in multiple service files.
How should we handle that? Can you guy give me a solution for this?

I really appreciate your support til now!!!!!!

Now the following format is that my corporation want to have:

├── services
          ├──catch-all-service.yaml (including upstreams.yaml)
          ├──httpbin-service.yaml (including upstreams.yaml)
├── upstream
          ├──upstreams.yaml

If you have any solution for this, please don't hesitate for reaching out the solution!!!!
Many thanks again!