ReMap Pipeline

The ReMap pipeline in Syncraft is a potent mechanism for reorganizing and transforming data within your pipelines. Unlike other pipelines that can fetch data from external sources, ReMap solely operates on the existing data within the pipeline's input state. This makes it a critical tool for data consolidation and reshaping, which is especially beneficial in pure selection pipelines.

Let's delve into a practical example to better grasp the ReMap pipeline's capabilities:

Example:

Below is a JSON representation of a query that incorporates a ReMap Pipeline:

Try In Playground

[
  {
    "query": {
      "id": {
        "eq": 2
      }
    },
    "selection": {
      "id": true,
      "name": true
    },
    "root": "Users"
  },
  {
    "query": {
      "Timestamp": {
        "gt": "2023-09-10"
      },
      "UserID": {
        "eq": {
          "@ref": [
            "id"
          ]
        }
      }
    },
    "selection": {
      "User": {
        "@ref": [
          "name"
        ]
      },
      "ActionSummary": {
        "dataPath": [],
        "transformer": ". = join!([\"User\", .User, \"has done action \", .ActionType, \"on\", .Timestamp, \"on a product from category\", .Category, \"and sub category\", .SubCategory], \" \")",
        "own": true
      },
      "ProductID": true,
      "Category": true,
      "ActionType": true,
      "SubCategory": true,
      "Timestamp": true
    },
    "root": "SessionRecord"
  },
  {
    "selection": {
      "actions": {
        "@ref": [
          "ActionSummary"
        ]
      }
    }
  }
]

Output:

The output from the above query will yield the following data:


{
    "queryId": "f67bba2c-c7ad-47d7-bccb-65e638bf51c1",
    "data": {
        "actions": [
            "User Bob has done action PURCHASE on 2023-09-10 02:11:31 UTC on a product from category Personal Care and sub category Skin Care",
            // ...
        ]
    },
    "analysis": null
}

Breakdown:

Initial Query:

The first block queries the Users model, filtering for a user with an ID of 2.

Session Record Query:

The second block queries the SessionRecord model, filtering sessions based on the UserID and Timestamp.

Transformation with ReMap:

Within the second block's selection, a transformer is applied to generate a summary of the user's actions.
The transformer uses the Processing ReMap Language, which is rooted in the Vector ReMap Language, to concatenate a string summarizing each session action.

Selection of Transformed Data:

The third block in the query pipeline selects the transformed ActionSummary field to be included in the final output under the key actions.

The ReMap pipeline is a key player in your data transformation toolkit within Syncraft. By understanding and utilizing its capabilities, you can efficiently reshape and consolidate your data, ensuring it's structured precisely as needed for your application's requirements.

Continue Your Journey

Query & Select

Continue Your Journey

Filters and Data Guards