crp-order-quantity-kpi-service - Details for Query 27

Submitted Time: 2026/04/20 06:14:35
Duration: 17 s
Succeeded Jobs: 30 31 32 33 34 35 36 37 38 39

Plan Visualization

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [id="node0" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; 1 [id="node1" labelType="html" label="CollectLimit shuffle records written: 35 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 1 local bytes read: 868.0 B merged fetch fallback count: 0 local blocks read: 14 remote merged chunks fetched: 0 remote blocks read: 0 local merged bytes read: 0.0 B remote reqs duration: 0 ms local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 16 ms (0 ms, 0 ms, 2 ms (stage 82.0: task 615)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time: 0 ms remote bytes read: 0.0 B remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 2.1 KiB (62.0 B, 62.0 B, 62.0 B (stage 82.0: task 618))" tooltip="CollectLimit 1"]; 2 [id="node2" labelType="html" label="InMemoryTableScan number of output rows: 9,999" tooltip="InMemoryTableScan"]; 3 [id="node3" labelType="html" label=" AdaptiveSparkPlan " tooltip="AdaptiveSparkPlan isFinalPlan=true"]; subgraph cluster4 { isCluster="true"; id="cluster4"; label="WholeStageCodegen (12)\n \nduration: total (min, med, max (stageId: taskId))\n18.3 s (2 ms, 344 ms, 1.4 s (stage 74.0: task 594))"; tooltip="WholeStageCodegen (12)"; 5 [id="node5" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 3.8 s (0 ms, 75 ms, 701 ms (stage 74.0: task 594)) peak memory total (min, med, max (stageId: taskId)) 9.3 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 74.0: task 573)) number of output rows: 9,999 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[product#1127, plant#1128], functions=[])"]; 6 [id="node6" labelType="html" label="HashAggregate spill size: 0.0 B time in aggregation build total (min, med, max (stageId: taskId)) 3.7 s (0 ms, 73 ms, 699 ms (stage 74.0: task 594)) peak memory total (min, med, max (stageId: taskId)) 9.3 MiB (256.0 KiB, 256.0 KiB, 256.0 KiB (stage 74.0: task 573)) number of output rows: 9,999 number of sort fallback tasks: 0 avg hash probes per key: 0" tooltip="HashAggregate(keys=[product#1127, plant#1128], functions=[])"]; 7 [id="node7" labelType="html" label=" Project " tooltip="Project [product#1127, plant#1128]"]; 8 [id="node8" labelType="html" label="Filter number of output rows: 9,999" tooltip="Filter (((size(demandTimeBuckets#1577, true) = 0) OR isnull(demandTimeBuckets#1577)) AND ((size(demandPointInTime#1580, true) = 0) OR isnull(demandPointInTime#1580)))"]; 9 [id="node9" labelType="html" label="BroadcastHashJoin number of output rows: 9,999" tooltip="BroadcastHashJoin [coalesce(plant#1128, ), isnull(plant#1128), coalesce(product#1127, ), isnull(product#1127)], [coalesce(plant#1573, ), isnull(plant#1573), coalesce(product#1572, ), isnull(product#1572)], LeftOuter, BuildRight, false"]; 10 [id="node10" labelType="html" label=" Project " tooltip="Project [product#1127, plant#1128]"]; 11 [id="node11" labelType="html" label="Generate number of output rows: 9,999" tooltip="Generate explode(_extract_quantityDenominator#1607), [product#1127, plant#1128], true, [quantityDenominator#1216]"]; 12 [id="node12" labelType="html" label=" Project " tooltip="Project [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205.quantityDenominator AS _extract_quantityDenominator#1607]"]; 13 [id="node13" labelType="html" label="Generate number of output rows: 9,999" tooltip="Generate explode(filteredUnitOfMeasureConversions#1205.quantityNumerator), [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205], true, [quantityNumerator#1213]"]; } 14 [id="node14" labelType="html" label=" Project " tooltip="Project [product#1127, plant#1128, filter(unitOfMeasureConversions#1200, lambdafunction((((lambda u#1206.measurementUnit1.code = _extract_code#1686) AND (lambda u#1206.measurementUnit1.internalRefUUID = _extract_internalRefUUID#1687)) AND ((lambda u#1206.measurementUnit2.code = baseUnitOfMeasure#1198.code) AND (lambda u#1206.measurementUnit2.internalRefUUID = baseUnitOfMeasure#1198.internalRefUUID))), lambda u#1206, false)) AS filteredUnitOfMeasureConversions#1205]"]; subgraph cluster15 { isCluster="true"; id="cluster15"; label="WholeStageCodegen (11)\n \nduration: total (min, med, max (stageId: taskId))\n11.0 s (4 ms, 254 ms, 703 ms (stage 74.0: task 594))"; tooltip="WholeStageCodegen (11)"; 16 [id="node16" labelType="html" label="SortMergeJoin number of output rows: 9,999 spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 74.0: task 573))" tooltip="SortMergeJoin [product#1127], [internalUUID#1194], LeftOuter"]; } subgraph cluster17 { isCluster="true"; id="cluster17"; label="WholeStageCodegen (9)\n \nduration: total (min, med, max (stageId: taskId))\n13.9 s (23 ms, 370 ms, 1.0 s (stage 74.0: task 576))"; tooltip="WholeStageCodegen (9)"; 18 [id="node18" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 55 ms (0 ms, 0 ms, 55 ms (stage 74.0: task 605)) peak memory total (min, med, max (stageId: taskId)) 72.3 MiB (64.0 KiB, 2.1 MiB, 2.1 MiB (stage 74.0: task 573)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 74.0: task 573))" tooltip="Sort [product#1127 ASC NULLS FIRST], false, 0"]; } 19 [id="node19" labelType="html" label="Exchange shuffle records written: 9,999 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 9,999 local bytes read total (min, med, max (stageId: taskId)) 315.6 KiB (3.3 KiB, 8.2 KiB, 19.3 KiB (stage 74.0: task 590)) merged fetch fallback count: 0 local blocks read: 641 remote merged chunks fetched: 0 remote blocks read: 632 data size total (min, med, max (stageId: taskId)) 1406.1 KiB (32.5 KiB, 38.1 KiB, 43.9 KiB (stage 66.0: task 556)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 98 ms (1 ms, 1 ms, 29 ms (stage 66.0: task 549)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 1.7 s (0 ms, 0 ms, 673 ms (stage 74.0: task 594)) remote bytes read total (min, med, max (stageId: taskId)) 309.0 KiB (3.9 KiB, 8.3 KiB, 24.3 KiB (stage 74.0: task 576)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 2.0 s (4 ms, 6 ms, 688 ms (stage 74.0: task 594)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 624.6 KiB (15.4 KiB, 16.8 KiB, 18.4 KiB (stage 66.0: task 556))" tooltip="Exchange hashpartitioning(product#1127, 37), ENSURE_REQUIREMENTS, [plan_id=1895]"]; subgraph cluster20 { isCluster="true"; id="cluster20"; label="WholeStageCodegen (8)\n \nduration: total (min, med, max (stageId: taskId))\n34.9 s (616 ms, 882 ms, 1.4 s (stage 66.0: task 542))"; tooltip="WholeStageCodegen (8)"; 21 [id="node21" labelType="html" label=" Project " tooltip="Project [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]"]; 22 [id="node22" labelType="html" label="SortMergeJoin number of output rows: 9,999 spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 66.0: task 539))" tooltip="SortMergeJoin [coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128)], [coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689)], LeftOuter"]; } subgraph cluster23 { isCluster="true"; id="cluster23"; label="WholeStageCodegen (6)\n \nduration: total (min, med, max (stageId: taskId))\n40.9 s (622 ms, 896 ms, 2.1 s (stage 66.0: task 542))"; tooltip="WholeStageCodegen (6)"; 24 [id="node24" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 20 ms (0 ms, 0 ms, 3 ms (stage 66.0: task 542)) peak memory total (min, med, max (stageId: taskId)) 76.3 MiB (2.1 MiB, 2.1 MiB, 2.1 MiB (stage 66.0: task 539)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 66.0: task 539))" tooltip="Sort [coalesce(product#1127, ) ASC NULLS FIRST, isnull(product#1127) ASC NULLS FIRST, coalesce(plant#1128, ) ASC NULLS FIRST, isnull(plant#1128) ASC NULLS FIRST], false, 0"]; } 25 [id="node25" labelType="html" label="Exchange shuffle records written: 9,999 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 9,999 local bytes read total (min, med, max (stageId: taskId)) 353.4 KiB (8.5 KiB, 10.2 KiB, 11.9 KiB (stage 66.0: task 563)) merged fetch fallback count: 0 local blocks read: 640 remote merged chunks fetched: 0 remote blocks read: 729 data size total (min, med, max (stageId: taskId)) 859.3 KiB (20.4 KiB, 23.5 KiB, 27.2 KiB (stage 57.0: task 434)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 92 ms (1 ms, 2 ms, 3 ms (stage 57.0: task 424)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 861 ms (0 ms, 0 ms, 165 ms (stage 66.0: task 567)) remote bytes read total (min, med, max (stageId: taskId)) 395.2 KiB (7.4 KiB, 10.2 KiB, 20.4 KiB (stage 66.0: task 568)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 1.2 s (4 ms, 9 ms, 176 ms (stage 66.0: task 567)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 748.6 KiB (18.1 KiB, 20.3 KiB, 22.9 KiB (stage 57.0: task 434))" tooltip="Exchange hashpartitioning(coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128), 37), ENSURE_REQUIREMENTS, [plan_id=1225]"]; 26 [id="node26" labelType="html" label="Exchange shuffle records written: 9,999 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 9,999 local bytes read total (min, med, max (stageId: taskId)) 209.5 KiB (7.1 KiB, 7.5 KiB, 8.0 KiB (stage 57.0: task 434)) merged fetch fallback count: 0 local blocks read: 28 remote merged chunks fetched: 0 remote blocks read: 9 data size total (min, med, max (stageId: taskId)) 859.3 KiB (20.4 KiB, 23.5 KiB, 27.2 KiB (stage 52.0: task 340)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 34 ms (0 ms, 0 ms, 1 ms (stage 52.0: task 348)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 15 ms (0 ms, 0 ms, 5 ms (stage 57.0: task 487)) remote bytes read total (min, med, max (stageId: taskId)) 68.0 KiB (7.3 KiB, 7.5 KiB, 7.8 KiB (stage 57.0: task 493)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 28 ms (1 ms, 2 ms, 7 ms (stage 57.0: task 489)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 277.5 KiB (7.1 KiB, 7.5 KiB, 8.0 KiB (stage 52.0: task 340))" tooltip="Exchange hashpartitioning(product#1127, plant#1128, 37), REPARTITION_BY_NUM, [plan_id=1218]"]; subgraph cluster27 { isCluster="true"; id="cluster27"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n6.8 s (75 ms, 114 ms, 523 ms (stage 52.0: task 346))"; tooltip="WholeStageCodegen (1)"; 28 [id="node28" labelType="html" label="ColumnarToRow number of output rows: 9,999 number of input batches: 37" tooltip="ColumnarToRow"]; } 29 [id="node29" labelType="html" label="Scan parquet number of files read: 37 scan time total (min, med, max (stageId: taskId)) 5.9 s (71 ms, 111 ms, 419 ms (stage 52.0: task 352)) metadata time: 2 ms size of files read: 305.6 KiB number of output rows: 9,999" tooltip="FileScan parquet [product#1127,plant#1128] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<product:string,plant:string>"]; subgraph cluster30 { isCluster="true"; id="cluster30"; label="WholeStageCodegen (7)\n \nduration: 0 ms"; tooltip="WholeStageCodegen (7)"; 31 [id="node31" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 13.2 s (247 ms, 328 ms, 488 ms (stage 66.0: task 560)) peak memory total (min, med, max (stageId: taskId)) 2.8 GiB (78.0 MiB, 78.0 MiB, 78.0 MiB (stage 66.0: task 539)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 66.0: task 539))" tooltip="Sort [coalesce(_extract_internalRefUUID#1688, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1688) ASC NULLS FIRST, coalesce(_extract_internalRefUUID#1689, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1689) ASC NULLS FIRST], false, 0"]; } 32 [id="node32" labelType="html" label="Exchange shuffle records written: 15,976,839 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 15,976,839 local bytes read total (min, med, max (stageId: taskId)) 238.3 MiB (9.3 MiB, 11.0 MiB, 11.0 MiB (stage 66.0: task 562)) merged fetch fallback count: 0 local blocks read: 434 remote merged chunks fetched: 0 remote blocks read: 935 data size total (min, med, max (stageId: taskId)) 2.1 GiB (58.9 MiB, 58.9 MiB, 58.9 MiB (stage 61.0: task 522)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1.5 s (28 ms, 31 ms, 88 ms (stage 61.0: task 503)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 39 ms (0 ms, 0 ms, 35 ms (stage 66.0: task 549)) remote bytes read total (min, med, max (stageId: taskId)) 513.4 MiB (9.3 MiB, 9.4 MiB, 20.4 MiB (stage 66.0: task 566)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 6.6 s (19 ms, 72 ms, 573 ms (stage 66.0: task 552)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 751.7 MiB (20.3 MiB, 20.3 MiB, 20.3 MiB (stage 61.0: task 498))" tooltip="Exchange hashpartitioning(coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689), 37), ENSURE_REQUIREMENTS, [plan_id=1786]"]; subgraph cluster33 { isCluster="true"; id="cluster33"; label="WholeStageCodegen (5)\n \nduration: total (min, med, max (stageId: taskId))\n21.1 s (387 ms, 541 ms, 969 ms (stage 61.0: task 501))"; tooltip="WholeStageCodegen (5)"; 34 [id="node34" labelType="html" label=" Project " tooltip="Project [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]"]; } 35 [id="node35" labelType="html" label="Exchange shuffle records written: 15,976,839 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 15,976,839 local bytes read total (min, med, max (stageId: taskId)) 335.6 MiB (6.7 MiB, 6.7 MiB, 11.9 MiB (stage 61.0: task 525)) merged fetch fallback count: 0 local blocks read: 145 remote merged chunks fetched: 0 remote blocks read: 188 data size total (min, med, max (stageId: taskId)) 2.1 GiB (203.9 MiB, 246.8 MiB, 246.8 MiB (stage 53.0: task 361)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 1.2 s (93 ms, 121 ms, 205 ms (stage 53.0: task 361)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 61.0: task 497)) remote bytes read total (min, med, max (stageId: taskId)) 438.8 MiB (9.1 MiB, 14.2 MiB, 14.2 MiB (stage 61.0: task 527)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 4.5 s (29 ms, 112 ms, 317 ms (stage 61.0: task 509)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 774.4 MiB (72.5 MiB, 87.7 MiB, 87.7 MiB (stage 53.0: task 390))" tooltip="Exchange RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1234]"]; subgraph cluster36 { isCluster="true"; id="cluster36"; label="WholeStageCodegen (2)\n \nduration: total (min, med, max (stageId: taskId))\n24.0 s (60 ms, 85 ms, 2.7 s (stage 53.0: task 369))"; tooltip="WholeStageCodegen (2)"; 37 [id="node37" labelType="html" label=" Project " tooltip="Project [plant#1160.internalRefUUID AS _extract_internalRefUUID#1689, product#1161.internalRefUUID AS _extract_internalRefUUID#1688, productionAspect#1163.productMovementPlants.unitOfIssue.code AS _extract_code#1686, productionAspect#1163.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#1687]"]; 38 [id="node38" labelType="html" label="ColumnarToRow number of output rows: 15,976,839 number of input batches: 3,902" tooltip="ColumnarToRow"]; } 39 [id="node39" labelType="html" label="Scan parquet number of files read: 1 scan time total (min, med, max (stageId: taskId)) 15.8 s (60 ms, 85 ms, 1.9 s (stage 53.0: task 412)) dynamic partition pruning time: 0 ms metadata time: 2 ms size of files read: 1123.8 MiB number of output rows: 15,976,839 number of partitions read: 1" tooltip="FileScan parquet [plant#1160,product#1161,productionAspect#1163,systemId#1158] Batched: true, DataFilters: [], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20...., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<plant:struct<internalRefUUID:string>,product:struct<internalRefUUID:string>,productionAspe..."]; subgraph cluster40 { isCluster="true"; id="cluster40"; label="WholeStageCodegen (10)\n \nduration: 0 ms"; tooltip="WholeStageCodegen (10)"; 41 [id="node41" labelType="html" label="Sort sort time total (min, med, max (stageId: taskId)) 256 ms (0 ms, 1 ms, 70 ms (stage 74.0: task 590)) peak memory total (min, med, max (stageId: taskId)) 104.4 MiB (2.1 MiB, 2.1 MiB, 4.1 MiB (stage 74.0: task 573)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 74.0: task 573))" tooltip="Sort [internalUUID#1194 ASC NULLS FIRST], false, 0"]; } 42 [id="node42" labelType="html" label="Exchange shuffle records written: 154,509 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 146,015 local bytes read total (min, med, max (stageId: taskId)) 5.9 MiB (215.2 KiB, 222.8 KiB, 231.2 KiB (stage 74.0: task 588)) merged fetch fallback count: 0 local blocks read: 999 remote merged chunks fetched: 0 remote blocks read: 370 data size total (min, med, max (stageId: taskId)) 73.4 MiB (2028.3 KiB, 2030.0 KiB, 2032.0 KiB (stage 59.0: task 478)) local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 93 ms (2 ms, 2 ms, 2 ms (stage 59.0: task 451)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 2 ms (0 ms, 0 ms, 2 ms (stage 74.0: task 593)) remote bytes read total (min, med, max (stageId: taskId)) 2.2 MiB (219.8 KiB, 223.4 KiB, 231.7 KiB (stage 74.0: task 576)) number of partitions: 37 remote reqs duration total (min, med, max (stageId: taskId)) 1.8 s (8 ms, 10 ms, 688 ms (stage 74.0: task 594)) remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 8.1 MiB (220.6 KiB, 223.0 KiB, 226.2 KiB (stage 59.0: task 478))" tooltip="Exchange hashpartitioning(internalUUID#1194, 37), ENSURE_REQUIREMENTS, [plan_id=1276]"]; 43 [id="node43" labelType="html" label="Exchange shuffle records written: 154,509 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 154,509 local bytes read total (min, med, max (stageId: taskId)) 7.8 MiB (214.1 KiB, 216.7 KiB, 218.7 KiB (stage 59.0: task 479)) merged fetch fallback count: 0 local blocks read: 37 remote merged chunks fetched: 0 remote blocks read: 0 data size: 73.4 MiB local merged bytes read: 0.0 B local merged chunks fetched: 0 shuffle write time: 15 ms remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 fetch wait time total (min, med, max (stageId: taskId)) 0 ms (0 ms, 0 ms, 0 ms (stage 59.0: task 450)) remote bytes read: 0.0 B number of partitions: 37 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written: 7.8 MiB" tooltip="Exchange RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1266]"]; subgraph cluster44 { isCluster="true"; id="cluster44"; label="WholeStageCodegen (3)\n \nduration: total (min, med, max (stageId: taskId))\n971 ms (160 ms, 811 ms, 811 ms (stage 54.0: task 356))"; tooltip="WholeStageCodegen (3)"; 45 [id="node45" labelType="html" label=" Project " tooltip="Project [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]"]; 46 [id="node46" labelType="html" label="Filter number of output rows: 154,509" tooltip="Filter isnotnull(internalUUID#1194)"]; 47 [id="node47" labelType="html" label="ColumnarToRow number of output rows: 154,509 number of input batches: 38" tooltip="ColumnarToRow"]; } 48 [id="node48" labelType="html" label="Scan parquet number of files read: 1 scan time total (min, med, max (stageId: taskId)) 728 ms (159 ms, 569 ms, 569 ms (stage 54.0: task 356)) dynamic partition pruning time: 0 ms metadata time: 0 ms size of files read: 7.3 MiB number of output rows: 154,509 number of partitions read: 1" tooltip="FileScan parquet [internalUUID#1194,baseUnitOfMeasure#1198,unitOfMeasureConversions#1200,systemId#1193] Batched: true, DataFilters: [isnotnull(internalUUID#1194)], Format: Parquet, Location: PreparedDeltaFileIndex(1 paths)[hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20...., PartitionFilters: [], PushedFilters: [IsNotNull(internalUUID)], ReadSchema: struct<internalUUID:string,baseUnitOfMeasure:struct<internalRefUUID:string,code:string>,unitOfMea..."]; 49 [id="node49" labelType="html" label="BroadcastExchange time to broadcast: 12 ms time to build: 53 ms time to collect: 557 ms number of output rows: 9,999 data size: 4.5 MiB" tooltip="BroadcastExchange HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=1291]"]; subgraph cluster50 { isCluster="true"; id="cluster50"; label="WholeStageCodegen (4)\n \nduration: total (min, med, max (stageId: taskId))\n782 ms (86 ms, 112 ms, 196 ms (stage 55.0: task 364))"; tooltip="WholeStageCodegen (4)"; 51 [id="node51" labelType="html" label="ColumnarToRow number of output rows: 9,999 number of input batches: 6" tooltip="ColumnarToRow"]; } 52 [id="node52" labelType="html" label="Scan parquet number of files read: 6 scan time total (min, med, max (stageId: taskId)) 745 ms (84 ms, 100 ms, 193 ms (stage 55.0: task 364)) metadata time: 0 ms size of files read: 54.2 KiB number of output rows: 9,999" tooltip="FileScan parquet [product#1572,plant#1573,demandTimeBuckets#1577,demandPointInTime#1580] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanac..., PartitionFilters: [], PushedFilters: [], ReadSchema: struct<product:string,plant:string,demandTimeBuckets:array<struct<demandStartDateTime:timestamp,d..."]; 1->0; 2->1; 3->2; 5->3; 6->5; 7->6; 8->7; 9->8; 10->9; 11->10; 12->11; 13->12; 14->13; 16->14; 18->16; 19->18; 21->19; 22->21; 24->22; 25->24; 26->25; 28->26; 29->28; 31->22; 32->31; 34->32; 35->34; 37->35; 38->37; 39->38; 41->16; 42->41; 43->42; 45->43; 46->45; 47->46; 48->47; 49->9; 51->49; 52->51; }

Details

== Physical Plan ==
AdaptiveSparkPlan (83)
+- == Final Plan ==
   ResultQueryStage (81), Statistics(sizeInBytes=8.0 EiB)
   +- CollectLimit (80)
      +- TableCacheQueryStage (79), Statistics(sizeInBytes=703.1 KiB, rowCount=1.00E+4)
         +- InMemoryTableScan (1)
               +- InMemoryRelation (2)
                     +- AdaptiveSparkPlan (78)
                     +- == Final Plan ==
                        ResultQueryStage (48)
                        +- * HashAggregate (47)
                           +- * HashAggregate (46)
                              +- * Project (45)
                                 +- * Filter (44)
                                    +- * BroadcastHashJoin LeftOuter BuildRight (43)
                                       :- * Project (38)
                                       :  +- * Generate (37)
                                       :     +- * Project (36)
                                       :        +- * Generate (35)
                                       :           +- Project (34)
                                       :              +- * SortMergeJoin LeftOuter (33)
                                       :                 :- * Sort (23)
                                       :                 :  +- ShuffleQueryStage (22), Statistics(sizeInBytes=1406.1 KiB, rowCount=1.00E+4)
                                       :                 :     +- Exchange (21)
                                       :                 :        +- * Project (20)
                                       :                 :           +- * SortMergeJoin LeftOuter (19)
                                       :                 :              :- * Sort (9)
                                       :                 :              :  +- ShuffleQueryStage (8), Statistics(sizeInBytes=859.3 KiB, rowCount=1.00E+4)
                                       :                 :              :     +- Exchange (7)
                                       :                 :              :        +- ShuffleQueryStage (6), Statistics(sizeInBytes=859.3 KiB, rowCount=1.00E+4)
                                       :                 :              :           +- Exchange (5)
                                       :                 :              :              +- * ColumnarToRow (4)
                                       :                 :              :                 +- Scan parquet  (3)
                                       :                 :              +- * Sort (18)
                                       :                 :                 +- ShuffleQueryStage (17), Statistics(sizeInBytes=2.1 GiB, rowCount=1.60E+7)
                                       :                 :                    +- Exchange (16)
                                       :                 :                       +- * Project (15)
                                       :                 :                          +- ShuffleQueryStage (14), Statistics(sizeInBytes=2.1 GiB, rowCount=1.60E+7)
                                       :                 :                             +- Exchange (13)
                                       :                 :                                +- * Project (12)
                                       :                 :                                   +- * ColumnarToRow (11)
                                       :                 :                                      +- Scan parquet  (10)
                                       :                 +- * Sort (32)
                                       :                    +- ShuffleQueryStage (31), Statistics(sizeInBytes=73.4 MiB, rowCount=1.55E+5)
                                       :                       +- Exchange (30)
                                       :                          +- ShuffleQueryStage (29), Statistics(sizeInBytes=73.4 MiB, rowCount=1.55E+5)
                                       :                             +- Exchange (28)
                                       :                                +- * Project (27)
                                       :                                   +- * Filter (26)
                                       :                                      +- * ColumnarToRow (25)
                                       :                                         +- Scan parquet  (24)
                                       +- BroadcastQueryStage (42), Statistics(sizeInBytes=4.5 MiB, rowCount=1.00E+4)
                                          +- BroadcastExchange (41)
                                             +- * ColumnarToRow (40)
                                                +- Scan parquet  (39)
                     +- == Initial Plan ==
                        HashAggregate (77)
                        +- HashAggregate (76)
                           +- Project (75)
                              +- Filter (74)
                                 +- BroadcastHashJoin LeftOuter BuildRight (73)
                                    :- Project (71)
                                    :  +- Generate (70)
                                    :     +- Project (69)
                                    :        +- Generate (68)
                                    :           +- Project (67)
                                    :              +- SortMergeJoin LeftOuter (66)
                                    :                 :- Sort (60)
                                    :                 :  +- Exchange (59)
                                    :                 :     +- Project (58)
                                    :                 :        +- SortMergeJoin LeftOuter (57)
                                    :                 :           :- Sort (51)
                                    :                 :           :  +- Exchange (50)
                                    :                 :           :     +- Exchange (49)
                                    :                 :           :        +- Scan parquet  (3)
                                    :                 :           +- Sort (56)
                                    :                 :              +- Exchange (55)
                                    :                 :                 +- Project (54)
                                    :                 :                    +- Exchange (53)
                                    :                 :                       +- Project (52)
                                    :                 :                          +- Scan parquet  (10)
                                    :                 +- Sort (65)
                                    :                    +- Exchange (64)
                                    :                       +- Exchange (63)
                                    :                          +- Project (62)
                                    :                             +- Filter (61)
                                    :                                +- Scan parquet  (24)
                                    +- BroadcastExchange (72)
                                       +- Scan parquet  (39)
+- == Initial Plan ==
   CollectLimit (82)
   +- InMemoryTableScan (1)
         +- InMemoryRelation (2)
               +- AdaptiveSparkPlan (78)
               +- == Final Plan ==
                  ResultQueryStage (48)
                  +- * HashAggregate (47)
                     +- * HashAggregate (46)
                        +- * Project (45)
                           +- * Filter (44)
                              +- * BroadcastHashJoin LeftOuter BuildRight (43)
                                 :- * Project (38)
                                 :  +- * Generate (37)
                                 :     +- * Project (36)
                                 :        +- * Generate (35)
                                 :           +- Project (34)
                                 :              +- * SortMergeJoin LeftOuter (33)
                                 :                 :- * Sort (23)
                                 :                 :  +- ShuffleQueryStage (22), Statistics(sizeInBytes=1406.1 KiB, rowCount=1.00E+4)
                                 :                 :     +- Exchange (21)
                                 :                 :        +- * Project (20)
                                 :                 :           +- * SortMergeJoin LeftOuter (19)
                                 :                 :              :- * Sort (9)
                                 :                 :              :  +- ShuffleQueryStage (8), Statistics(sizeInBytes=859.3 KiB, rowCount=1.00E+4)
                                 :                 :              :     +- Exchange (7)
                                 :                 :              :        +- ShuffleQueryStage (6), Statistics(sizeInBytes=859.3 KiB, rowCount=1.00E+4)
                                 :                 :              :           +- Exchange (5)
                                 :                 :              :              +- * ColumnarToRow (4)
                                 :                 :              :                 +- Scan parquet  (3)
                                 :                 :              +- * Sort (18)
                                 :                 :                 +- ShuffleQueryStage (17), Statistics(sizeInBytes=2.1 GiB, rowCount=1.60E+7)
                                 :                 :                    +- Exchange (16)
                                 :                 :                       +- * Project (15)
                                 :                 :                          +- ShuffleQueryStage (14), Statistics(sizeInBytes=2.1 GiB, rowCount=1.60E+7)
                                 :                 :                             +- Exchange (13)
                                 :                 :                                +- * Project (12)
                                 :                 :                                   +- * ColumnarToRow (11)
                                 :                 :                                      +- Scan parquet  (10)
                                 :                 +- * Sort (32)
                                 :                    +- ShuffleQueryStage (31), Statistics(sizeInBytes=73.4 MiB, rowCount=1.55E+5)
                                 :                       +- Exchange (30)
                                 :                          +- ShuffleQueryStage (29), Statistics(sizeInBytes=73.4 MiB, rowCount=1.55E+5)
                                 :                             +- Exchange (28)
                                 :                                +- * Project (27)
                                 :                                   +- * Filter (26)
                                 :                                      +- * ColumnarToRow (25)
                                 :                                         +- Scan parquet  (24)
                                 +- BroadcastQueryStage (42), Statistics(sizeInBytes=4.5 MiB, rowCount=1.00E+4)
                                    +- BroadcastExchange (41)
                                       +- * ColumnarToRow (40)
                                          +- Scan parquet  (39)
               +- == Initial Plan ==
                  HashAggregate (77)
                  +- HashAggregate (76)
                     +- Project (75)
                        +- Filter (74)
                           +- BroadcastHashJoin LeftOuter BuildRight (73)
                              :- Project (71)
                              :  +- Generate (70)
                              :     +- Project (69)
                              :        +- Generate (68)
                              :           +- Project (67)
                              :              +- SortMergeJoin LeftOuter (66)
                              :                 :- Sort (60)
                              :                 :  +- Exchange (59)
                              :                 :     +- Project (58)
                              :                 :        +- SortMergeJoin LeftOuter (57)
                              :                 :           :- Sort (51)
                              :                 :           :  +- Exchange (50)
                              :                 :           :     +- Exchange (49)
                              :                 :           :        +- Scan parquet  (3)
                              :                 :           +- Sort (56)
                              :                 :              +- Exchange (55)
                              :                 :                 +- Project (54)
                              :                 :                    +- Exchange (53)
                              :                 :                       +- Project (52)
                              :                 :                          +- Scan parquet  (10)
                              :                 +- Sort (65)
                              :                    +- Exchange (64)
                              :                       +- Exchange (63)
                              :                          +- Project (62)
                              :                             +- Filter (61)
                              :                                +- Scan parquet  (24)
                              +- BroadcastExchange (72)
                                 +- Scan parquet  (39)


(1) InMemoryTableScan
Output: []

(2) InMemoryRelation
Arguments: [product#1127, plant#1128], StorageLevel(disk, memory, 2 replicas)

(3) Scan parquet 
Output [2]: [product#1127, plant#1128]
Batched: true
Location: InMemoryFileIndex [hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-order-qty-opt-service/out/product-plant-list/10000000004/shardId=0_1_10000000004]
ReadSchema: struct<product:string,plant:string>

(4) ColumnarToRow [codegen id : 1]
Input [2]: [product#1127, plant#1128]

(5) Exchange
Input [2]: [product#1127, plant#1128]
Arguments: hashpartitioning(product#1127, plant#1128, 37), REPARTITION_BY_NUM, [plan_id=1218]

(6) ShuffleQueryStage
Output [2]: [product#1127, plant#1128]
Arguments: 0

(7) Exchange
Input [2]: [product#1127, plant#1128]
Arguments: hashpartitioning(coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128), 37), ENSURE_REQUIREMENTS, [plan_id=1225]

(8) ShuffleQueryStage
Output [2]: [product#1127, plant#1128]
Arguments: 4

(9) Sort [codegen id : 6]
Input [2]: [product#1127, plant#1128]
Arguments: [coalesce(product#1127, ) ASC NULLS FIRST, isnull(product#1127) ASC NULLS FIRST, coalesce(plant#1128, ) ASC NULLS FIRST, isnull(plant#1128) ASC NULLS FIRST], false, 0

(10) Scan parquet 
Output [4]: [plant#1160, product#1161, productionAspect#1163, systemId#1158]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-productplant]
ReadSchema: struct<plant:struct<internalRefUUID:string>,product:struct<internalRefUUID:string>,productionAspect:struct<productMovementPlants:struct<unitOfIssue:struct<internalRefUUID:string,code:string>>>>

(11) ColumnarToRow [codegen id : 2]
Input [4]: [plant#1160, product#1161, productionAspect#1163, systemId#1158]

(12) Project [codegen id : 2]
Output [4]: [plant#1160.internalRefUUID AS _extract_internalRefUUID#1689, product#1161.internalRefUUID AS _extract_internalRefUUID#1688, productionAspect#1163.productMovementPlants.unitOfIssue.code AS _extract_code#1686, productionAspect#1163.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#1687]
Input [4]: [plant#1160, product#1161, productionAspect#1163, systemId#1158]

(13) Exchange
Input [4]: [_extract_internalRefUUID#1689, _extract_internalRefUUID#1688, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1234]

(14) ShuffleQueryStage
Output [4]: [_extract_internalRefUUID#1689, _extract_internalRefUUID#1688, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: 1

(15) Project [codegen id : 5]
Output [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Input [4]: [_extract_internalRefUUID#1689, _extract_internalRefUUID#1688, _extract_code#1686, _extract_internalRefUUID#1687]

(16) Exchange
Input [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Arguments: hashpartitioning(coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689), 37), ENSURE_REQUIREMENTS, [plan_id=1786]

(17) ShuffleQueryStage
Output [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Arguments: 6

(18) Sort [codegen id : 7]
Input [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Arguments: [coalesce(_extract_internalRefUUID#1688, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1688) ASC NULLS FIRST, coalesce(_extract_internalRefUUID#1689, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1689) ASC NULLS FIRST], false, 0

(19) SortMergeJoin [codegen id : 8]
Left keys [4]: [coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128)]
Right keys [4]: [coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689)]
Join type: LeftOuter
Join condition: None

(20) Project [codegen id : 8]
Output [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Input [6]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]

(21) Exchange
Input [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: hashpartitioning(product#1127, 37), ENSURE_REQUIREMENTS, [plan_id=1895]

(22) ShuffleQueryStage
Output [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: 7

(23) Sort [codegen id : 9]
Input [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: [product#1127 ASC NULLS FIRST], false, 0

(24) Scan parquet 
Output [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]
Batched: true
Location: PreparedDeltaFileIndex [hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-dl-stream-service/cornerstone/sap-cic-product-product-v2]
PushedFilters: [IsNotNull(internalUUID)]
ReadSchema: struct<internalUUID:string,baseUnitOfMeasure:struct<internalRefUUID:string,code:string>,unitOfMeasureConversions:array<struct<measurementUnit1:struct<internalRefUUID:string,code:string>,measurementUnit2:struct<internalRefUUID:string,code:string>,quantityNumerator:int,quantityDenominator:int>>>

(25) ColumnarToRow [codegen id : 3]
Input [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]

(26) Filter [codegen id : 3]
Input [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]
Condition : isnotnull(internalUUID#1194)

(27) Project [codegen id : 3]
Output [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Input [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]

(28) Exchange
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1266]

(29) ShuffleQueryStage
Output [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: 2

(30) Exchange
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: hashpartitioning(internalUUID#1194, 37), ENSURE_REQUIREMENTS, [plan_id=1276]

(31) ShuffleQueryStage
Output [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: 5

(32) Sort [codegen id : 10]
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: [internalUUID#1194 ASC NULLS FIRST], false, 0

(33) SortMergeJoin [codegen id : 11]
Left keys [1]: [product#1127]
Right keys [1]: [internalUUID#1194]
Join type: LeftOuter
Join condition: None

(34) Project
Output [3]: [product#1127, plant#1128, filter(unitOfMeasureConversions#1200, lambdafunction((((lambda u#1206.measurementUnit1.code = _extract_code#1686) AND (lambda u#1206.measurementUnit1.internalRefUUID = _extract_internalRefUUID#1687)) AND ((lambda u#1206.measurementUnit2.code = baseUnitOfMeasure#1198.code) AND (lambda u#1206.measurementUnit2.internalRefUUID = baseUnitOfMeasure#1198.internalRefUUID))), lambda u#1206, false)) AS filteredUnitOfMeasureConversions#1205]
Input [7]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687, internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]

(35) Generate [codegen id : 12]
Input [3]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205]
Arguments: explode(filteredUnitOfMeasureConversions#1205.quantityNumerator), [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205], true, [quantityNumerator#1213]

(36) Project [codegen id : 12]
Output [3]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205.quantityDenominator AS _extract_quantityDenominator#1607]
Input [4]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205, quantityNumerator#1213]

(37) Generate [codegen id : 12]
Input [3]: [product#1127, plant#1128, _extract_quantityDenominator#1607]
Arguments: explode(_extract_quantityDenominator#1607), [product#1127, plant#1128], true, [quantityDenominator#1216]

(38) Project [codegen id : 12]
Output [2]: [product#1127, plant#1128]
Input [3]: [product#1127, plant#1128, quantityDenominator#1216]

(39) Scan parquet 
Output [4]: [product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Batched: true
Location: InMemoryFileIndex [hdlfs://7da5304c-1f57-47fa-b75a-0f0b4349c280.files.hdl.prod-eu20.hanacloud.ondemand.com:443/crp-demand-service/out/demand/10000000004/0_1_10000000004]
ReadSchema: struct<product:string,plant:string,demandTimeBuckets:array<struct<demandStartDateTime:timestamp,demandQuantity:double,demandVariance:double>>,demandPointInTime:array<struct<demandDateTime:timestamp,demandQuantity:double,demandVariance:double>>>

(40) ColumnarToRow [codegen id : 4]
Input [4]: [product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]

(41) BroadcastExchange
Input [4]: [product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=1291]

(42) BroadcastQueryStage
Output [4]: [product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Arguments: 3

(43) BroadcastHashJoin [codegen id : 12]
Left keys [4]: [coalesce(plant#1128, ), isnull(plant#1128), coalesce(product#1127, ), isnull(product#1127)]
Right keys [4]: [coalesce(plant#1573, ), isnull(plant#1573), coalesce(product#1572, ), isnull(product#1572)]
Join type: LeftOuter
Join condition: None

(44) Filter [codegen id : 12]
Input [6]: [product#1127, plant#1128, product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Condition : (((size(demandTimeBuckets#1577, true) = 0) OR isnull(demandTimeBuckets#1577)) AND ((size(demandPointInTime#1580, true) = 0) OR isnull(demandPointInTime#1580)))

(45) Project [codegen id : 12]
Output [2]: [product#1127, plant#1128]
Input [6]: [product#1127, plant#1128, product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]

(46) HashAggregate [codegen id : 12]
Input [2]: [product#1127, plant#1128]
Keys [2]: [product#1127, plant#1128]
Functions: []
Aggregate Attributes: []
Results [2]: [product#1127, plant#1128]

(47) HashAggregate [codegen id : 12]
Input [2]: [product#1127, plant#1128]
Keys [2]: [product#1127, plant#1128]
Functions: []
Aggregate Attributes: []
Results [2]: [product#1127, plant#1128]

(48) ResultQueryStage
Output [2]: [product#1127, plant#1128]
Arguments: 8

(49) Exchange
Input [2]: [product#1127, plant#1128]
Arguments: hashpartitioning(product#1127, plant#1128, 37), REPARTITION_BY_NUM, [plan_id=1131]

(50) Exchange
Input [2]: [product#1127, plant#1128]
Arguments: hashpartitioning(coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128), 37), ENSURE_REQUIREMENTS, [plan_id=1156]

(51) Sort
Input [2]: [product#1127, plant#1128]
Arguments: [coalesce(product#1127, ) ASC NULLS FIRST, isnull(product#1127) ASC NULLS FIRST, coalesce(plant#1128, ) ASC NULLS FIRST, isnull(plant#1128) ASC NULLS FIRST], false, 0

(52) Project
Output [4]: [plant#1160.internalRefUUID AS _extract_internalRefUUID#1689, product#1161.internalRefUUID AS _extract_internalRefUUID#1688, productionAspect#1163.productMovementPlants.unitOfIssue.code AS _extract_code#1686, productionAspect#1163.productMovementPlants.unitOfIssue.internalRefUUID AS _extract_internalRefUUID#1687]
Input [4]: [plant#1160, product#1161, productionAspect#1163, systemId#1158]

(53) Exchange
Input [4]: [_extract_internalRefUUID#1689, _extract_internalRefUUID#1688, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1134]

(54) Project
Output [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Input [4]: [_extract_internalRefUUID#1689, _extract_internalRefUUID#1688, _extract_code#1686, _extract_internalRefUUID#1687]

(55) Exchange
Input [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Arguments: hashpartitioning(coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689), 37), ENSURE_REQUIREMENTS, [plan_id=1157]

(56) Sort
Input [4]: [_extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]
Arguments: [coalesce(_extract_internalRefUUID#1688, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1688) ASC NULLS FIRST, coalesce(_extract_internalRefUUID#1689, ) ASC NULLS FIRST, isnull(_extract_internalRefUUID#1689) ASC NULLS FIRST], false, 0

(57) SortMergeJoin
Left keys [4]: [coalesce(product#1127, ), isnull(product#1127), coalesce(plant#1128, ), isnull(plant#1128)]
Right keys [4]: [coalesce(_extract_internalRefUUID#1688, ), isnull(_extract_internalRefUUID#1688), coalesce(_extract_internalRefUUID#1689, ), isnull(_extract_internalRefUUID#1689)]
Join type: LeftOuter
Join condition: None

(58) Project
Output [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Input [6]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687, _extract_internalRefUUID#1688, _extract_internalRefUUID#1689]

(59) Exchange
Input [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: hashpartitioning(product#1127, 37), ENSURE_REQUIREMENTS, [plan_id=1164]

(60) Sort
Input [4]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687]
Arguments: [product#1127 ASC NULLS FIRST], false, 0

(61) Filter
Input [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]
Condition : isnotnull(internalUUID#1194)

(62) Project
Output [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Input [4]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200, systemId#1193]

(63) Exchange
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: RoundRobinPartitioning(37), REPARTITION_BY_NUM, [plan_id=1141]

(64) Exchange
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: hashpartitioning(internalUUID#1194, 37), ENSURE_REQUIREMENTS, [plan_id=1165]

(65) Sort
Input [3]: [internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]
Arguments: [internalUUID#1194 ASC NULLS FIRST], false, 0

(66) SortMergeJoin
Left keys [1]: [product#1127]
Right keys [1]: [internalUUID#1194]
Join type: LeftOuter
Join condition: None

(67) Project
Output [3]: [product#1127, plant#1128, filter(unitOfMeasureConversions#1200, lambdafunction((((lambda u#1206.measurementUnit1.code = _extract_code#1686) AND (lambda u#1206.measurementUnit1.internalRefUUID = _extract_internalRefUUID#1687)) AND ((lambda u#1206.measurementUnit2.code = baseUnitOfMeasure#1198.code) AND (lambda u#1206.measurementUnit2.internalRefUUID = baseUnitOfMeasure#1198.internalRefUUID))), lambda u#1206, false)) AS filteredUnitOfMeasureConversions#1205]
Input [7]: [product#1127, plant#1128, _extract_code#1686, _extract_internalRefUUID#1687, internalUUID#1194, baseUnitOfMeasure#1198, unitOfMeasureConversions#1200]

(68) Generate
Input [3]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205]
Arguments: explode(filteredUnitOfMeasureConversions#1205.quantityNumerator), [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205], true, [quantityNumerator#1213]

(69) Project
Output [3]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205.quantityDenominator AS _extract_quantityDenominator#1607]
Input [4]: [product#1127, plant#1128, filteredUnitOfMeasureConversions#1205, quantityNumerator#1213]

(70) Generate
Input [3]: [product#1127, plant#1128, _extract_quantityDenominator#1607]
Arguments: explode(_extract_quantityDenominator#1607), [product#1127, plant#1128], true, [quantityDenominator#1216]

(71) Project
Output [2]: [product#1127, plant#1128]
Input [3]: [product#1127, plant#1128, quantityDenominator#1216]

(72) BroadcastExchange
Input [4]: [product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Arguments: HashedRelationBroadcastMode(List(coalesce(input[1, string, true], ), isnull(input[1, string, true]), coalesce(input[0, string, true], ), isnull(input[0, string, true])),false), [plan_id=1175]

(73) BroadcastHashJoin
Left keys [4]: [coalesce(plant#1128, ), isnull(plant#1128), coalesce(product#1127, ), isnull(product#1127)]
Right keys [4]: [coalesce(plant#1573, ), isnull(plant#1573), coalesce(product#1572, ), isnull(product#1572)]
Join type: LeftOuter
Join condition: None

(74) Filter
Input [6]: [product#1127, plant#1128, product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]
Condition : (((size(demandTimeBuckets#1577, true) = 0) OR isnull(demandTimeBuckets#1577)) AND ((size(demandPointInTime#1580, true) = 0) OR isnull(demandPointInTime#1580)))

(75) Project
Output [2]: [product#1127, plant#1128]
Input [6]: [product#1127, plant#1128, product#1572, plant#1573, demandTimeBuckets#1577, demandPointInTime#1580]

(76) HashAggregate
Input [2]: [product#1127, plant#1128]
Keys [2]: [product#1127, plant#1128]
Functions: []
Aggregate Attributes: []
Results [2]: [product#1127, plant#1128]

(77) HashAggregate
Input [2]: [product#1127, plant#1128]
Keys [2]: [product#1127, plant#1128]
Functions: []
Aggregate Attributes: []
Results [2]: [product#1127, plant#1128]

(78) AdaptiveSparkPlan
Output [2]: [product#1127, plant#1128]
Arguments: isFinalPlan=true

(79) TableCacheQueryStage
Output: []
Arguments: 0

(80) CollectLimit
Input: []
Arguments: 1

(81) ResultQueryStage
Output: []
Arguments: 1

(82) CollectLimit
Input: []
Arguments: 1

(83) AdaptiveSparkPlan
Output: []
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.activity.hdlfsEnabled	true
spark.activity.isolateSession	true
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true
temporal.activity.id	36e0914c-7e4e-3a52-bc93-e306e07299ba
temporal.activity.type	BufferDataSnapShot