Query Pipelines
The query pipeline is a fundamental building block of Syncraft’s query engine. Every query is a query pipeline, which can branch into multiple pipelines. Depending on your query logic, you can imagine this as a tree that grows, shrinks, and branches. The query pipeline's purpose is to provide the data, execute some computing of the data, or, in some cases, both.
Query pipeline Types
QueryAndSelect
Pipeline used for executing queries against the data sources. QueryAnd select pipeline can refer to one or join multiple data sources and branch into numerous query pipelines. At its core, it is the base building block in the data retrieval process. Learn More
ReMap
Pipeline used for remapping data and organizing pipeline state. ReMaps is created as a toolbox for data consolidation and transformation steps since it is a pure selection pipeline, and they cannot retrieve data from sources but only manipulate pipeline input state. Learn More
Filter
Pipeline used to guard specific parts of the pipeline. This pipeline contains no data retrieval, consolidation, or transformation logic. It is a predicate function as part of the query pipeline flow control. Learn More
If-Else
Pipeline used for conditional branching. It is used when pipeline logic needs to branch differently based on a predicate test. Fundamental parts of this pipeline are Learn More
- Condition - predicate query executed against pipeline input state
- Then pipeline - query pipelines that run if the condition has matched
- Or pipeline - query pipeline that runs if the condition hasn’t matched
Loop
Pipeline used to execute looping query logic. This can be treated as programmatic “While” loops that will run the provided pipeline until the break condition is matched. The purpose of this timeline is to hold the rest of the pipeline execution until the condition is satisfied. Whether we are talking about awaiting some status change in some database or if we are ingesting large files, having a way to execute certain logic until the condition is matched is a compelling concept that fortifies data logic isolation from the caller. Learn More
Higher Order Pipelines
Higher-order pipelines in Syncraft's query engine provide a powerful way to group individual pipeline blocks and define the execution flow of pipelines. By combining multiple pipeline blocks, developers can create complex query logic and control the order in which each block is executed.
Serial
The Serial pipeline is a higher-order pipeline that executes its child pipelines sequentially. Each child pipeline is completed one after another, and the output of one pipeline becomes the input for the following pipeline in the sequence. This allows for precise control over the data flow and enables developers to perform sequential operations on the data. Learn More
Parallel
The Parallel pipeline, on the other hand, executes its child pipelines concurrently. Each child pipeline is run independently, allowing for parallel processing and improving query performance. This is particularly useful when dealing with large datasets or computationally intensive operations. Learn More
Race
The Race pipeline introduces a different concept by executing its child pipelines simultaneously and returning the result of the first pipeline to complete. This can be useful in scenarios where the fastest response is desired, such as real-time applications or when dealing with time-sensitive data. Learn More
By utilizing higher-order pipelines, developers can build complex and flexible query pipelines that meet the specific requirements of their applications. Whether defining the execution order, performing parallel processing, or prioritizing speed, they offer a versatile approach to designing efficient and customizable query flows.
Pipeline State
Once the pipeline receives input, that input becomes the initial state of the pipeline's inner state, and that internal state is a starting point for the pipeline. Now, depending on the pipeline type, the state is transformed differently. Through pipeline execution, that inner state changes and evolves before it is passed to the next pipeline state. It is crucial to mention that the Filter Pipeline cannot modify the state, for the filter pipeline output is the same as the input if it passes the filter condition.
Further Reading
For further reading and to explore Syncraft's capabilities, we recommend the following pages:
Visit these pages to understand Syncraft and start building your data applications comprehensively.