Parallel Pipeline

The Parallel Pipeline in Syncraft empowers developers to execute multiple pipelines concurrently. Unlike the Serial Pipeline, where each step waits for the previous one to complete, steps in a Parallel Pipeline run at the same time. This is particularly useful when there are independent data retrieval or transformation tasks that can be performed in parallel to optimize the overall execution time.

Core Components:

Collections of Pipelines: Organized sets of pipelines that can be executed concurrently.
Concurrent Execution: Pipelines within a collection are executed in parallel.
Data Collection: The results from each parallel pipeline are collected together for further processing or output.

Try In Playground


{
    "collect": {
        "documents": [
            {
                "query": {
                    "category": {
                        "includes": "Footwear"
                    }
                },
                "root": "Product",
                "selection": {
                    "*": true
                }
            }
        ],
        "sessions": [
            {
                "query": {
                    "id": {
                        "eq": 2
                    }
                },
                "root": "Users",
                "selection": {
                    "id": true,
                    "name": true
                }
            },
            {
                "query": {
                    "Timestamp": {
                        "gt": "2023-09-13"
                    },
                    "UserID": {
                        "eq": {
                            "@ref": [
                                "id"
                            ]
                        }
                    }
                },
                "root": "SessionRecord",
                "selection": {
                    "ActionType": true,
                    "Category": true,
                    "ProductID": true,
                    "SubCategory": true,
                    "Timestamp": true,
                    "User": {
                        "@ref": [
                            "name"
                        ]
                    }
                }
            }
        ]
    },
    "remap": {
        "selection": {
            "actions": {
                "@ref": [
                    "sessions"
                ]
            },
            "documents": {
                "@ref": [
                    "documents"
                ]
            }
        }
    }
}

Workflow:

Document Retrieval: The first collection of parallel pipelines queries the Product root for documents related to "Footwear".
Session Data Retrieval: The second collection retrieves user information from the Users root and session data from the SessionRecord root concurrently.
Data Remapping: Once the parallel executions are complete, data is remapped to organize the output, combining the results from both collections.

Real-world Scenarios:

Parallel Pipeline is indispensable in scenarios like:

Optimizing Performance: When there are independent data tasks that can be run concurrently to improve performance.
Complex Data Aggregation: Aggregating data from multiple sources or roots simultaneously.

The Parallel Pipeline in Syncraft allows developers to optimize query execution by running independent pipelines concurrently, making it a robust tool for handling complex, multi-source data retrieval, and transformation tasks in an efficient manner.

Learn More

Looping Pipelines

Continue Your Journey

Building Race Pipelines