PipeStream: A Recursive Entity Streaming Protocol for Distributed Processing over QUIC

This document specifies PipeStream, a recursive entity streaming protocol for hierarchical task decomposition and distributed processing over QUIC transport. PipeStream enables the decomposition ("dehydration") of complex inputs into constituent entities, their transmission across distributed processing nodes, and subsequent rehydration (reassembly) at destination endpoints. The protocol's primary motivating use case is distributed document processing, but its recursive scatter-gather design is applicable to any domain requiring hierarchical decomposition of work units. The protocol employs a dual-stream architecture consisting of a data stream for entity payload transmission and a control stream for tracking entity completion status and maintaining consistency. PipeStream defines four hierarchical data layers for entity representation: BlobBag for raw binary data, SemanticLayer for annotated content with metadata, ParsedData for structured extracted information, and CustomEntity for application-specific extensions. PipeStream is organized into three protocol layers: Layer 0 (Core) provides basic streaming with dehydrate/rehydrate semantics; Layer 1 (Recursive) adds hierarchical scoping and digest propagation; Layer 2 (Resilience) adds yield/resume, claim checks, and completion policies. All implementations support Layer 0; Layers 1 and 2 are optional and negotiated at connection time. To ensure consistency across distributed processing pipelines, PipeStream implements checkpoint blocking, whereby processing nodes synchronize at defined points before proceeding. This mechanism guarantees that all constituent parts of a dehydrated entity are successfully processed before rehydration operations commence.