Antipattern audit: goroutine leaks, race conditions, unbounded growth, missing recovery

[intel352]

Audit Results

Comprehensive audit of the workflow engine identified 24 antipatterns across goroutine management, synchronization, resource lifecycle, and error handling. 2 critical, 12 high, 8 medium, 2 low.

Critical (2)

1. EventProcessor.periodicCleanup() goroutine never stops — module/event_processor.go:84 — Start() spawns goroutine, Stop() is empty, no stop channel or context
2. CronScheduler data race on running and jobs — module/scheduler.go:56-104 — plain bool/slice accessed from main and background goroutines without synchronization

High — Missing Panic Recovery (3)

3. foreach step goroutines — module/pipeline_step_foreach.go:219 — no recover() (parallel step already has it)
4. Scheduler job goroutines — module/scheduler.go:77 — panic in a job crashes the process
5. workflow_call async goroutine — module/pipeline_step_workflow_call.go:137 — fire-and-forget with no recovery

High — Goroutine Lifecycle (2)

6. CronScheduler.Stop() blocks forever — module/scheduler.go:103 — unbuffered channel send when goroutine isn't in select
7. RuntimeManager watcher goroutines untracked — module/runtime_manager.go:194,334 — no WaitGroup, no recovery

High — Unbounded Growth (5)

8. RateLimitStep.buckets — module/pipeline_step_rate_limit.go:119 — per-IP buckets never evicted
9. APIGateway gatewayRateLimiter.buckets — module/api_gateway.go:84 — same pattern
10. WebhookSender.deadLetter — module/webhook_sender.go:111 — never evicted
11. globalOAuthCache — module/pipeline_step_http_call.go:24 — entries never removed
12. InMemoryLock.locks — scale/distributed_lock.go:50 — lock entries never removed

Medium (8)

13-17. Missing panic recovery in: handlers/state_machine.go:401, module/api_workflow_handler.go:47, middleware/apikey.go:75, module/kafka_broker.go:246, module/log_collector.go:199
18. CronScheduler unbuffered stop channel (dupes #6)
19. EventProcessor fragile nested lock ordering — module/event_processor.go:140
20. DBQueryCachedStep cache only evicts on re-access
21. StateMachine maxInstances/TTL not actively enforced
22. observability/reporter.go silently drops registration errors

Low (2)

23. RuntimeManager DB errors discarded
24. dynamic/resource_limits.go goroutine outlives timeout

---

Related: #384 (HTTP listener crash — fixed)

[intel352]

Fixed in v0.4.4. All critical and high severity items addressed:

Critical (2):

• EventProcessor.periodicCleanup() goroutine leak — added stop channel
• CronScheduler data race — atomic.Bool for running, mutex for jobs, close(stopCh) for shutdown

High — Panic Recovery (3+6 medium):

• Added defer recover() to 9 goroutine sites: foreach step, scheduler jobs, workflow_call async, state machine recovery, API workflow handler, API key middleware, Kafka consumer, log collector, runtime manager watchers

High — Unbounded Growth (5):

• RateLimitStep.buckets — TTL eviction (10min idle)
• APIGateway gatewayRateLimiter.buckets — same
• WebhookSender.deadLetter — size cap + TTL eviction (24h)
• globalOAuthCache — periodic cleanup of expired entries
• InMemoryLock.locks — TTL eviction (30min idle)

All eviction goroutines have stop mechanisms and panic recovery. All tests pass with -race.