feat: delay in next node #588

In graph config allow user to specify delay, next state would be queued after specified delay.

Motivation

1. Workflow pacing – Many workflows need delays between steps (e.g., rate limiting, waiting for external systems, cooling-off periods).
2. Time-based orchestration – Enable workflows with built-in timing without requiring nodes to raise ReQueueAfterSignal.
3. Declarative delays – Configure delays at graph definition time rather than in node code, improving separation of concerns.
4. Batch processing – Add delays between batch operations to avoid overwhelming downstream systems.

Current State

What Exists

Current Limitations

• No way to specify delays in graph configuration
• Must use ReQueueAfterSignal in node code to add delays
• Delays are not visible in graph definition
• Cannot configure different delays for different next nodes from the same node

Proposed Solution

Delay Configuration per Next Node

Allow specifying a delay for each next node connection in the graph configuration. When a state completes, the next states will be created with enqueue_after set to current time + delay.

Key Features

1. Per-edge delays – Configure delay for each next_nodes entry
2. Flexible delay format – Support milliseconds (integer) or human-readable strings (e.g., "5m", "1h", "30s")
3. Backward compatibility – If no delay specified, next states are enqueued immediately (current behavior)
4. Multiple next nodes – Each next node can have its own delay

Proposed Design

Data Model

Graph Node Configuration:

Current structure:

{
  "node_name": "DataLoader",
  "identifier": "loader",
  "next_nodes": ["processor", "validator"]
}

Proposed structure:

{
  "node_name": "DataLoader",
  "identifier": "loader",
  "next_nodes": ["processor", "validator"],
  "next_node_delays": {
    "processor": 5000,
    "validator": 10000
  }
}

Alternative: Delay as part of next_nodes mapping

{
  "node_name": "DataLoader",
  "identifier": "loader",
  "next_nodes": {
    "processor": {"delay": 5000},
    "validator": {"delay": 10000}
  }
}

Recommended: Separate field for backward compatibility

Control Flow

flowchart TD
    A["State completes successfully"] --> B["create_next_states() called"]
    B --> C["Get next_node_identifiers from template"]
    C --> D["For each next_node"]
    D --> E{"Delay configured?"}
    E -->|yes| F["Calculate enqueue_after = now + delay"]
    E -->|no| G["enqueue_after = now (immediate)"]
    F --> H["Create State with enqueue_after"]
    G --> H
    H --> I["State will be enqueued at enqueue_after time"]

Acceptance Criteria

• GraphNodeModel supports next_node_delays field (optional dict)
• NodeTemplate supports next_node_delays field (optional dict)
• Validation: delay values must be non-negative integers
• Validation: delay keys must match valid next_node identifiers
• create_next_states applies delays when creating next states
• States with delays have correct enqueue_after timestamp
• Backward compatibility: graphs without delays work as before
• Unit tests for delay application logic
• Integration tests for end-to-end delay flow
• Documentation updated in docs/exosphere/graph-components.md

Engineering Tasks

1. Python SDK:

   • Add next_node_delays field to GraphNodeModel in exospherehost/models.py
   • Add validation for delay values (non-negative integers)
   • Add validation that delay keys match next_nodes identifiers

2. State Manager:

   • Add next_node_delays field to NodeTemplate in app/models/node_template_model.py
   • Update create_next_states to apply delays when creating states
   • Validate delay keys exist in next_nodes during graph validation

3. Testing:

   • Unit tests for delay validation
   • Integration tests for delayed state creation
   • Test backward compatibility (no delays specified)
   • Test multiple next nodes with different delays

4. Documentation:

   • Update docs/exosphere/graph-components.md with delay configuration
   • Add examples showing delay usage
   • Document delay behavior and timing

Risks & Mitigations

Open Questions

1. Delay format: Should we support human-readable strings (e.g., "5m", "1h") in addition to milliseconds?
2. Maximum delay: Should there be a maximum delay cap to prevent accidental long-term scheduling?
3. Delay validation: Should delays be validated at graph creation time or state creation time?
4. Dashboard UI: How should delays be displayed in the graph visualization?
5. Absolute vs relative: Should we support absolute timestamps (like RequeueAtSignal) in addition to relative delays?

Additional Notes

• This feature complements ReQueueAfterSignal - use graph-level delays for declarative timing, signal-based delays for dynamic timing
• Delays are applied at state creation time, not at node execution time
• Consider future enhancement: delay expressions based on node outputs (e.g., delay based on data size)
• Aligns with feature-plan feat: RequeueAtSignal #587 (RequeueAtSignal) - both provide timing control but at different levels