Race Pipeline
The Race Pipeline in Syncraft introduces a competitive execution model where multiple pipelines are launched simultaneously, but only the result of the fastest completing pipeline is returned. This is particularly helpful in real-time or time-sensitive applications where the fastest response is desired over others. Here's a step-by-step tutorial to understand and implement a Race Pipeline in Syncraft:
Core Concepts:
- Simultaneous Execution: All pipelines under a Race Pipeline are triggered to execute at the same time.
- Fastest Response Retrieval: Only the result of the first pipeline to complete its execution is returned.
- Performance Efficiency: By focusing on the quickest result, the Race Pipeline enhances performance efficiency.
{
"race": {
"footwear": {
"query": {
"masterCategory": {
"includes": "Footwear"
}
},
"root": "Product",
"selection": {
"*": true
}
},
"apparel": {
"query": {
"masterCategory": {
"includes": "Apparel"
}
},
"root": "Product",
"selection": {
"*": true
}
}
},
"remap": {
"selection": {
"products": {
"dataPath": [],
"transformer": "if is_null(.footwear) { . = .apparel } else { . = .footwear } "
}
}
}
}
Steps:
1. Define the Pipelines:
Create the pipelines which will compete against each other. In this example, we have two pipelines querying same roots (Products and Products) based on different criteria.
2. Trigger the Race:
Wrap the pipelines within a race block. This initiates the race, triggering all pipelines to execute simultaneously.
3. Collect the Fastest Result:
In the remap block, specify how you want to capture the result of the fastest completing pipeline. Here, we collect the fastest result under the field products.
Usage Scenarios:
- Time-Sensitive Data Processing: When the speed of data retrieval is critical, Race Pipeline ensures the fastest possible response.
- Competitive Execution: When there are multiple pipelines capable of providing the required data, Race Pipeline helps in selecting the fastest among them.
- Optimized Query Performance: Helps in optimizing the query performance by focusing on obtaining the quickest result.
The Race Pipeline is an essential tool in Syncraft's arsenal for ensuring speed and efficiency in data processing. It's engineered to meet the demanding performance requirements, especially in real-time applications or any scenario where time is of the essence. Through a competitive execution environment among child pipelines, it guarantees that the fastest response is always at your disposal, enhancing the overall performance and timeliness of your data delivery operations.