A secure, stateless, clustered headless server built for high-performance multi-tenancy, real-time collaboration, dynamic process orchestration, and distributed mesh execution. This module serves as the primary orchestrator of the CoCreate ecosystem.
For a complete guide, deployment examples, and API reference, visit the CoCreate Server Documentation.
- Architectural Overview & Core Design
- Core Server Modules
- Component Mesh Architecture
- Programmatic API Reference
- Ingress & Connection Controls
- Process & OS Lifecycle Management
- IPC Transport Engine
- Reactive Cluster Storage Manager
- Distributed Cluster Orchestration
- Targeting Matrix & Routing Directives
- Self-Healing & Jittered Reconnection Logic
- Dynamic Tenant Eviction & Cascading Teardown Sequence
- Security-Hardened Design
- Installation
- Quick Start
- Announcements
- Roadmap
- How to Contribute
- About
- License
@cocreate/cluster is engineered to run stateless, highly available workloads across multi-core systems and distributed cloud infrastructure.
The server operates a single Primary process that forks and manages worker processes across CPU cores.
The Primary process is responsible for:
- Cluster orchestration
- Worker lifecycle management
- Fallback HTTP/HTTPS listeners
- Inter-process communication (IPC)
Each Worker runs an identical application stack handling:
- HTTP requests
- WebSocket connections
- API routing
- Database operations
- Static asset delivery
The server minimizes startup time, memory usage, and supply-chain risk by maintaining a lightweight dependency footprint while composing functionality through CoCreate modules.
When every worker enters draining or cordoned mode, the Primary process automatically serves HTTP 307 Temporary Redirect responses so incoming requests can immediately reconnect through external load balancers without downtime.
| Module | Purpose |
|---|---|
index.js |
Primary server orchestrator |
ipc.js |
Inter-process communication engine |
storage.js |
Cluster-wide reactive storage |
getInfrastructure.js |
Infrastructure detection |
getIp.js |
Public and private IP resolution |
The server composes multiple standalone CoCreate modules into a unified runtime.
| Module | Responsibility |
|---|---|
@cocreate/certificates |
TLS certificate management |
@cocreate/socket-server |
WebSocket server |
cluster/src/mesh |
Inter-server communication |
@cocreate/crud-server |
Database routing |
@cocreate/autoscaler |
Automatic scaling |
@cocreate/telemetry |
Monitoring and metrics |
@cocreate/api |
Dynamic API execution |
@cocreate/file/server |
Static file serving |
@cocreate/lazy-loader |
Lazy loading and webhooks |
Cluster initializes server.manager after CRUD. Its persisted fields are
organization_id, host and apiKey; a null organization ID means unmanaged.
The existing primary inventory read hydrates those fields, and fork configuration
passes them to workers without another database lookup. The runtime send
function is non-enumerable and is not copied into inventory records.
if (server.manager.organization_id) {
await server.manager.send(existingEnvelope);
}send POSTs the envelope to https://<manager.host>/manager, with the credential
in the Authorization Bearer header and caller scope removed from JSON. Manager.http delegates provider requests to API.http, which authorizes before
API.send. Manager resolves parent scope and authorizes server/cluster inventory
object.update requests before dispatching to CRUD. The manager handler does
not invoke state.save, preventing mirrored-state loops. Existing API/workflow HTTP
routes remain available independently.
State's existing persistence helper mirrors successful state.save envelopes once after the local write. Each worker sends its existing worker section; the primary sends its existing master/node section. Mirror delivery has a three-second timeout; failure is logged without failing or repeating local persistence. There is no delivery queue or replay, and direct CRUD writes outside this path are not captured. Telemetry is not mirrored.
AutoScaler uses manager.send at its existing send points and preserves its drain, capacity and recovery sequence. Unmanaged nodes retain server.api.send. Initial server creation persists the manager credential in both inventory copies; future scale-out reservations copy its connection fields. Cluster-key permissions allow instance endpoints and server/cluster updates. Cluster/provider AST predicates, metadata preparation, bootstrap handling and initial provider-result synchronization remain separate work; this service does not imply those integrations are complete.
Cluster routes /api and its /api/… subpaths to server.api.http(req, res) and
POST /workflows/<name> to server.workflow.http(req, res), following the existing
server.api.webhook(req, res) entrance pattern. /manager and /manager/… subpaths
similarly reach server.manager.http. Prefixes respect path segment boundaries,
so /apis and /managerial retain normal routing. The API HTTP handler still
requires endpoint in its JSON envelope; path-based endpoint extraction is a
separate future change. Each module resolves organization
scope from the hostname and authorizes before execution. The former
src/http-operations.js adapter is removed. Denial, authorization errors or a
missing authorization service return 403. The shared server.authorize.request
helper checks permission, preserves the sanitized envelope, rejects changes to
scope/operation targets, and strips the transport API key. Default/key permission merging is
unchanged, with no extra key-record lookup.
Send Content-Type: application/json and an Authorization: Bearer <key> header,
or supply the existing envelope's apikey field. Without a key, authorization
evaluates its normal default permissions. Conflicting header/body keys are
rejected. Caller-supplied socket/user identity and organization/host scope are
not trusted. The authorized, sanitized envelope is used for execution, with the
transport API key removed. Responses use Cache-Control: no-store; bodies are
limited to 1 MiB and only POST is supported.
Example body for /api:
{
"api": "configured-provider",
"endpoint": "configured-operation",
"method": "post",
"data": { "cluster_id": "cluster-document-id" }
}The adapter requires api, endpoint and method; the direct url shortcut is
not exposed. Authorization evaluates the existing envelope method, such as
post, rather than a new api.send permission. Configure operation/data
restrictions instead of granting an unrestricted HTTP verb for managed scaling.
Example body for /workflows/example:
{ "example": [{ "input": "value" }] }The adapter sets method: "workflow.example"; an explicit method must match the
path. No permissions are granted by these routes. /webhooks/* retains its AST
handler, and trusted custom server.routes retain their existing precedence.
Managed credentials and parent dispatch remain separate integration work.
After CRUD initializes, the primary runs src/apikey.js to read one
upload-capable API key (type: "key", admin: true, including the supported
string/number boolean forms) from its configured organization's keys array.
Selection is ordered by record ID. It does not infer another organization from
the cluster name or fetch GitHub credentials.
The handler uses Config.prompt({ apikey: { value } }) to update memory,
the process environment, and ~/CoCreate.config.js while preserving the other
global settings. Workers inherit the environment rather than repeating the
lookup or writing the file. No key is generated when none exists. Read or
validation failures produce a generic boot diagnostic and do not prevent the
server from starting; credentials are never logged by the handler.
Cloud startup data contains node identity, host, and bootstrap storage, but no CoCreate API key. The runtime obtains that key after storage is available.
Cluster's src/routing.js loads routing configuration after CRUD, workflows,
and the node identity are initialized. The primary invokes configure-cluster-routing after
startup readiness (all workers listening in clustered mode; HTTP and HTTPS
listening in single-process mode). Workers do not register independently.
That initialization workflow registers the node and returns a prepared removal
request, retained in memory by routing.js. The existing coordinated shutdown
path sends that request directly through server.api.send before stopping
workers or closing Mesh. There is no removal workflow. This includes the
existing SIGINT, SIGTERM, and
SIGBREAK handling and application restart/reboot/shutdown paths. Routing does
not install another set of process signal handlers. Workflow failures are
logged without preventing the existing lifecycle from continuing.
The initialization workflow name can be set in CoCreate configuration:
{
"routing": {
"enabled": true,
"workflow": "configure-cluster-routing"
}
}Matching servers_config records with type: "routing" override this system
configuration using the existing server/cluster scope rules. Their config
object accepts enabled and workflow; set
enabled: false in that object to disable the hooks.
Each invocation uses the node's configured organization and host and the normal workflow envelope, for example:
{
method: 'workflow.configure-cluster-routing',
organization_id: server.organization_id,
host: server.host,
'configure-cluster-routing': [{
server_id: server.id,
cluster_id: server.cluster_id,
ip: server.ip,
host: server.host,
infrastructure: server.infrastructure,
location: server.location
}]
}The workflow returns configure-cluster-routing[0].remove, an API request containing
api, endpoint, and data, including provider registration IDs. A custom
initialization workflow returns the same remove field under its own input
key. Shutdown adds a reason, such as SIGTERM, and uses the node's configured
organization and host for API execution. If no removal request was prepared,
shutdown performs no new workflow or inventory lookup.
The hooks do not create workflow definitions or contain Bunny API logic. The initialization workflow owns provider selection and request preparation. It must return nonsecret cleanup data, not API credentials. The existing API dispatcher still resolves the provider configuration and endpoint by name; this change does not introduce a shared API cache.
Registration is shared across repeated readiness notifications, and withdrawal waits for an in-flight registration before running. Cleanup requires a graceful shutdown; abrupt process or host failure remains a routing fallback concern.
Every server instance extends Node.js EventEmitter.
| Property | Description |
|---|---|
id |
Unique server identifier |
isPrimary |
Indicates Primary process |
workerId |
Worker identifier |
totalWorkers |
Number of workers |
state |
Current server state |
storage / store |
Reactive cluster storage |
send() |
Server-side IPC and mesh message dispatcher |
server.on("status_change", ({ oldStatus, newStatus }) => {
console.log(`State changed from ${oldStatus} to ${newStatus}`);
});Enable or disable maintenance mode.
await server.state.cordon(true);
server.emit("cordon", {
value: true
});
await server.state.cordon(false);Drain selected organizations.
await server.state.shed([
"org_abc123",
"org_xyz789"
]);
await server.state.shed(false);Gracefully disconnect active clients.
await server.state.drain(true);
await server.state.drain(false);server.state.restart();
server.state.restart({
isPrimary: true
});Worker restart:
- Drains connections
- Exits gracefully
- Primary automatically forks replacement
Primary restart:
- Closes cluster services
- Exits cleanly
- System service restarts application
server.state.reboot();Gracefully reboots the operating system after draining connections.
server.state.shutdown();Gracefully powers off the machine after draining connections.
The IPC subsystem synchronizes Primary and Worker processes and routes messages
to peer servers through the server mesh. server.send() is the server-side
transport. A browser does not call this function directly: it calls
CoCreate.socket.send() with the same envelope, and the receiving socket server
passes that envelope into server.send() when server routing is required.
Browser to server/worker:
await CoCreate.socket.send({
method: "cordon",
server: "server_b_id",
worker: 2,
value: true
});The message follows this path:
CoCreate.socket.send(data)
-> socket server
-> server.send(data)
-> Primary IPC / server mesh
-> targeted worker event
Inside a server process, use server.send() directly.
Broadcast:
server.send({
method: "custom_worker_event",
broadcast: true,
data: {
status: "ok"
}
});Target worker:
server.send({
method: "restart",
worker: 2
});Multiple workers:
server.send({
method: "drain",
worker: [1, 3]
});Worker control messages use the same envelope:
// Allow or prevent new traffic on one worker.
server.send({
method: "cordon",
server: "server_b_id",
worker: 2,
value: true
});
// Drain a worker's active connections and cordon it.
server.send({
method: "drain",
server: "server_b_id",
worker: 2,
value: true
});
// Shed selected organizations from a worker.
server.send({
method: "shed",
server: "server_b_id",
worker: 2,
organizations: ["org_abc123"]
});
// Stop shedding and allow the worker to accept traffic again.
server.send({
method: "shed",
server: "server_b_id",
worker: 2,
organizations: false
});Primary process and host lifecycle controls target master: true:
server.send({
method: "restart",
server: "server_b_id",
master: true
});
server.send({
method: "reboot",
server: "server_b_id",
master: true
});
server.send({
method: "shutdown",
server: "server_b_id",
master: true
});restart restarts the server application. reboot and shutdown operate on
the host and therefore should be exposed only through an authorized control
surface with explicit confirmation.
server, cluster, and worker accept either one value or an array of values.
Broadcasting is opt-in: set broadcast: true only when a message should reach
all eligible targets. An explicit worker target takes precedence over
broadcast worker selection.
Messages carrying organization_id are routed only to workers currently serving
that organization and through mesh sockets connected on that organization's URL
path. The field is a routing scope, not a request to broadcast. IPC owns the local
organization-to-worker map. The originating workerId and serverId are retained
as lifecycle metadata so messages can be routed without duplicate execution.
Cluster-wide synchronized key/value storage.
await server.storage.set(
"node_capacity",
{
availableSlots: 45
}
);
const capacity =
await server.storage.get(
"node_capacity"
);
const totalKeys =
await server.storage.size();
await server.storage.delete(
"node_capacity"
);
await server.storage.clear();Reactive listeners:
server.storage.on(
"change:node_capacity",
(newValue) => {
console.log(newValue);
}
);
server.storage.on(
"delete:node_capacity",
() => {
console.log("Deleted");
}
);Cluster owns its inter-server mesh implementation in src/mesh.
server.mesh.send({
method: "mesh.custom_command",
server: "server_b_id",
data: {
key: "value"
}
});| Target | Example |
|---|---|
| Single server | { server: "server_b_id" } |
| Multiple servers | { server: ["server_b_id","server_c_id"] } |
| Worker | { server: "server_b_id", worker: 2 } |
| Broadcast | { broadcast: true } |
Peer reconnections use exponential backoff with randomized jitter.
\min \left( 30000, 1000\times1.5^{attempt} \right) + random(500,4500) $$
This prevents reconnect storms after temporary outages.
Tenant lifecycle is driven by active socket connections.
Once the final client disconnects:
- A 60-second timer begins.
- Workers release tenant resources.
- Peer servers receive notification.
- Database pools, telemetry buffers, and mesh tunnels are released.
- Mutual TLS encrypted cluster communication
- Dedicated internal mesh network
- Automatic UUID generation for distributed tracing
- Ephemeral TLS certificates managed by
@cocreate/certificates
npm install @cocreate/clusteryarn add @cocreate/clusterimport CoCreateServer from "@cocreate/cluster";
async function startServer() {
const server =
await CoCreateServer.init();
server.on(
"status_change",
({ oldStatus, newStatus }) => {
console.log(
`${oldStatus} -> ${newStatus}`
);
}
);
}
startServer();Release notes include new features, performance improvements, security updates, and compatibility changes. Follow the repository to stay informed about the latest releases.
Upcoming improvements include:
- Enhanced cluster orchestration
- Improved process supervision
- Adaptive worker allocation
- Expanded telemetry integration
- Additional deployment adapters
Community contributions are welcome.
Please review:
- Documentation
- CONTRIBUTING.md
- GitHub Issues
- GitHub Discussions
Bug reports, feature requests, documentation improvements, and pull requests are appreciated.
@cocreate/cluster is designed, built, and maintained by the CoCreate Developer Experience Team.
It serves as the core runtime responsible for process orchestration, clustering, networking, module composition, and lifecycle management across the CoCreate platform.
This software is dual-licensed under the GNU Affero General Public License version 3 (AGPLv3) and a commercial license.
For open-source projects and non-commercial use, this software is available under the AGPLv3. See the LICENSE file for complete license terms.
Organizations requiring proprietary use may obtain a commercial license from CoCreate.
New signup and repository startup default to main, so the first upload targets the primary tenant hostname. Explicit branch choices remain supported; dev/test environment flows can be selected later.
The cluster document owns a repositories array. URL is each entry's identity;
no generated repository entry ID is used. The primary queries its own cluster
once during repository initialization, before workers fork. Global repository
config and servers_config type repository records are no longer loaded. The old
App repository server-config seed is removed; existing records of that type are
ignored. Organization git metadata remains descriptive and is not a startup
configuration fallback.
repositories: [{
url: 'git@github.com:owner/project.git',
api: '<tenant repository credential record ID>',
branch: 'main',
remote: 'origin',
localFsRoot: '../',
directory: 'project',
onStartup: 'clone',
onShutdown: 'commit-and-push',
startupTimeoutMs: 60000,
shutdownTimeoutMs: 10000,
initialUpload: { enabled: true, completed: false }
}]Relative localFsRoot resolves against server.projectRoot or the running project's working directory; absolute roots are also supported. The checkout is directly /, normally a sibling of the running server project. If directory is omitted, it derives from the repository URL name. Duplicate URLs or checkout destinations fail before cloning, and the running project itself cannot be a checkout destination. There is no organization/sites nesting.
Each configured checkout is cloned if missing. Existing checkouts are reused without pulling/resetting; startup branch switches require a clean checkout and a matching remote. Set enabled:false, onStartup:false or onShutdown:false on an entry to control its lifecycle. Workers share the primary's prepared checkouts.
The runtime service remains server.repository, with entries available in server.repository.repositories. The first enabled prepared checkout is the default AI workspace. Workers receive its path and platform organization through SERVER_REPOSITORY_ROOT and SERVER_REPOSITORY_ORGANIZATION_ID. AI uses that actual checkout rather than creating another organization/sites directory. Its platform access restrictions remain in place. Other consumers can select another entry's checkout.workspace; authenticated Git environments can be requested by URL with server.repository.environment(url).
Shutdown attempts each enabled repository's commit-and-push once, preserving its current branch. Each operation is bounded by its entry timeout and the remaining node shutdown budget; a failed repository does not silently skip all later ones. Errors are reported without preventing node shutdown. Credentials still come from the node organization's enabled apis records, with repository URL matching and strict SSH host verification. Keys/cache/staging live outside checkouts under /.cocreate/repositories/, never in Git.
After checkout and node listener readiness, the primary calls repository.publish. Entries with initialUpload.enabled:true and completed:false run coc upload from their checkout; disabled or completed entries are skipped. The CLI must be available on PATH, with runtime/global CoCreate credentials and a reachable target from CoCreate.config.js. No credentials are inserted into Git or CLI arguments.
A repository_upload_reservations record keyed by cluster+URL prevents concurrent nodes from running the initial upload. The primary memoizes publish so repeated worker-ready events do not launch it twice; workers never call it. Publication rereads the URL entry and sets initialUpload.completed and completedAt only after the CLI exits successfully. The cluster patch is mirrored through the manager when configured; the tenant cluster remains the local publication authority. Failed uploads leave completion pending and release their reservation for a later startup. If the process crashes, or upload succeeds but completion persistence is uncertain, the reservation is retained for explicit reconciliation; no lease expiry blindly launches a duplicate. Shutdown aborts an in-flight upload and bounds the wait before attempting Git checkpoints. Later AI/change uploads are not gated by this initial flag.
Existing clusters require their repositories array and runtime key permissions to be populated; no automatic migration from old repository configs or organization git metadata is performed.
Manager exposes non-enumerable send(envelope), createServer(cluster, deployment), terminateServer(cluster, deployment), http and config properties; only parent connection fields are serialized. Initialization loads the platform organizationConnection settings after CRUD/API/workflows. Its enabled flag controls Mesh's optional creation hook; tenant signup does not copy that platform config.
On activation, Mesh reads the organization-type cluster by manager.for (tenant organization ID) and manager.organization_id (platform ID), then its running servers in platform scope. This query uses cluster_id and master.status running, without peer discovery's smaller-ID filter. If no running server is returned, Mesh invokes manager.createServer(cluster). Nonrunning inventory, including failed records, does not block creation or get deleted by this decision.
Manager executes the command without inventory eligibility queries, reservations or creation deduplication. It uses the supplied cluster.initialDeployment, provider profile, storage reference, hostname and runtime settings, resolving the tenant bootstrap source using cluster.manager.for for script preparation. Cluster.name is an editable label; cluster.manager stores organization_id, host and for in both database copies. initialDeployment and repositories remain top-level cluster configuration. Signup prepares those documents and repository registration, then awaits manager.createServer(cluster) before reporting success. Later Mesh activation retains its running-server creation decision. Manager creates platform and tenant inventory before dispatching the existing create-organization-server workflow, mirrors the provider patch, and retains the existing startup watchdog. Concurrent creation commands are allowed; ordinary AutoScaler policy handles excess running capacity. ensureServer is removed.
Managed scale-out passes its chosen deployment and server_id to the same createServer path after API.http authorization. Omitting deployment uses cluster.initialDeployment; passing it leaves that static anchor unchanged. Parent inventory, configuration and provider operations use platform identity. Startup data uses cluster.manager.for, and both inventories receive the same server/cluster IDs and provider patch before success is returned. The requested cloud must match the authorized API.
Managed scale-in includes cluster_id and dispatches terminateServer. It mirrors terminating before the parent-scoped provider request and failed on rejection; success retains terminating and schedules the existing cleanup. AutoScaler still owns draining, transport retry and final shutdown. deleteServer only removes inventory. Scale-in accepts the persisted infrastructure.instanceId as its provider VM identifier.
Outbound discovery already calls connectPeer for every returned server with a usable address, preserving the existing smaller-ID query and connection ordering. Cluster grouping and orchestrator-to-tenant links are unchanged.
Outgoing sockets retry from their close handler. Normal close code 1000 now suppresses retries; explicit node shutdown and organization closure send that code. Restarts/reboots and uncertain signal exits use 1012, while network loss uses the transport's abnormal closure path and remains retryable. The existing ten-retry limit is unchanged. Failed-peer provider/inventory reconciliation remains a separate TODO; no automatic deletion or new diagnostic protocol is introduced.
Manager observes first-node creation and managed create-instance requests for two minutes after a successful instance-ID response. A matching authorized server update must persist successfully to confirm receipt. An early update arriving before the provider response is retained; an update handled by another managing worker/node is checked through persisted server status at timeout. Missing confirmation logs the server ID and leaves a TODO for platform notification and at most one optional replacement. No restart, deletion, retry or notification service is implemented by this watchdog. Timers are process-local and unreferenced; they do not survive a managing-process restart.
After the drain/capacity sequence, AutoScaler commits master.status terminating before the provider request. Success retains it; a termination error overrides it with failed. It then dispatches State shutdown to the selected server. State.save skips failed/terminating records while shutdown cleanup proceeds. Server inventory is retained briefly rather than deleted in the termination request.
AutoScaler stores removal outcomes in master.status. Capacity/drain cancellations before provider dispatch restore running; termination failures remain failed. The separate decommission status object is no longer used; decommissionStrategy remains the policy configuration name.
After persisting a managed server terminating update, Manager schedules cleanup once after one minute. It resolves the tenant scope from the stored cluster.manager.for and calls AutoScaler's shared deleteServer helper to delete the tenant and platform copies, without rechecking server status or calling the cloud provider. Duplicate reports do not postpone cleanup. A successfully persisted failed update cancels the pending timer directly. AutoScaler also sweeps its own cluster every ten minutes, starting ten minutes after initialization; managed sweeps request platform deletion through the authorized Manager object.delete servers operation. Already-missing records are successful deletions. Failed records are not selected. The management permission reference includes server-only deletion; existing issued keys require that grant for tenant-initiated platform cleanup. These process-local timers do not survive restarts.
Mesh emits organization.active from activation and organization-socket registration, and organization.inactive from its existing close/removal callbacks. Organization close sends the existing mesh.orgDeleted frame before closing its sockets. AutoScaler consumes these confirmed signals for its own organization: minClusterSize zero permits the existing scaleIn lifecycle, while CPU/memory alone cannot remove the final node. Socket Server's existing grace period and IPC worker aggregation remain responsible for local activity tracking; the own organization can report idle without unloading services needed for shutdown.