SAP Fiori Training - SAP BTP Cloud Application Programming Course
                     How to Optimize PySpark Performance 
in Azure
Practical techniques for faster, more reliable Spark workloads
Focus: Azure Databricks, Spark execution, and data layout
Goal: reduce wasted compute while meeting processing SLAs
Approach: measure first, tune the biggest bottleneck, then retest
www..viisuallpath +91  
..iin 7032290546
1. Measure Before You Tune
. Use the Spark UI to inspect stages, task duration, and shuffle read/write
. Check for skew: a few tasks running much longer than the rest
. Review executor CPU, memory, garbage collection, and spill metrics
. Compare runtime and cost against a baseline using the same input
. Change one setting at a time so the impact is clear
www..viisuallpath +91  
..iin 7032290546
2. Read Less Data, Earlier
. Select only the columns needed for the output; avoid SELECT *
. Filter rows as early as possible to reduce downstream work
. Use Parquet or Delta for columnar reads and predicate pushdown
. Partition data by commonly filtered fields when it suits query patterns
. Avoid tiny files; compact them when file-open overhead becomes significant
www..viisuallpath +91  
..iin 7032290546
3. Choose Partitions and Parallelism Wisely
. Use enough partitions to keep workers busy without creating thousands of tiny 
tasks
. Repartition when redistributing data is necessary; it triggers a shuffle
. Use coalesce to reduce partitions when a full shuffle is unnecessary
. Tune shuffle partitions based on data volume and cluster capacity
. Inspect task-size differences instead of relying on a fixed partition count
www..viisuallpath +91  
..iin 7032290546
4. Reduce Shuffle and Join Costs
. Filter and project inputs before joining to reduce data exchanged
. Broadcast a genuinely small table to avoid shuffling the large side
. Use Spark SQL functions and built-in expressions instead of Python UDFs where 
possible
. Check join keys for nulls, inconsistent types, and skewed values
. Use Adaptive Query Execution where supported to improve plans at runtime
www..viisuallpath +91  
..iin 7032290546
5. Make Delta Lake Work for You
. Use Delta tables for reliable writes, schema controls, and efficient reads
. Compact small files to improve scan performance
. Use OPTIMIZE and data-skipping features where available and appropriate
. Consider liquid clustering or partitioning based on query patterns and platform 
support
. Avoid over-partitioning by high-cardinality fields such as unique IDs
www..viisuallpath +91  
..iin 7032290546
6. Configure Azure Databricks Compute
. Choose a cluster or job compute size that fits the workload, not the largest option
. Use autoscaling for variable workloads and stable sizing for predictable jobs
. Prefer job compute for scheduled production workloads where appropriate
. Review worker utilization, memory pressure, and spill before scaling up
. Track cost per successful run as well as elapsed time
www..viisuallpath +91  
..iin 7032290546
7. Make Pipelines Reliable and Repeatable
. Cache only reused datasets that fit the available memory; unpersist when finished
. Avoid repeated actions that recompute the same expensive transformation
. Write outputs incrementally when the workload supports it
. Use checkpoints selectively for long lineage or iterative workloads
. Make retries safe with deterministic logic and idempotent writes
www..viisuallpath +91  
..iin 7032290546
8. A Practical Optimization Workflow
. Capture baseline: input size, runtime, cost, and key Spark UI metrics
. Identify the dominant issue: scan, shuffle, skew, memory, or file layout
. Apply one targeted change and rerun with representative data
. Validate row counts and business results before comparing performance
. Document the winning configuration and monitor it after deployment
www..viisuallpath +91  
..iin 7032290546
For More Information About 
SAP BTP CAP Fiori
Address:- Flat no: 205, 2nd Floor,
   Nilagiri Block, Aditya Enclave,  
Ameerpet, Hyderabad-16
Ph. No: +91-7032290546
www.visualpath.in
 [email protected]
www..viisuallpath +91  
..iin 7032290546
Thank you 
Scan  me
Scan Me
www..viisuallpath +91  
..iin 7032290546