← All tutorials

Skill Zuno · skillzunos.com · Python Data Science

PySpark tutorial

Learn PySpark on Skill Zuno. This PySpark tutorial covers New, Easy, Medium, Hard, and Diff levels so you can study practical PySpark problems on skillzunos.com.

Open PySpark on Skill Zuno

Also see the PySpark interview questions.

PySpark is the Python API for Apache Spark, the distributed engine for processing big data across a cluster. It provides DataFrames and Spark SQL, Structured Streaming and MLlib, and an optimiser that plans work across many machines. Interviews cover the Spark architecture, lazy transformations and actions, partitions and shuffles, joins and skew, caching, UDF performance and tuning jobs.

PySpark topics covered

  • Spark architecture — The driver, executors, cluster managers and the SparkSession.
  • DataFrames and Spark SQL — Reading data, transformations, and SQL queries over the same data.
  • Transformations and actions — Lazy evaluation, the DAG, and narrow versus wide transformations.
  • Partitions and shuffles — repartition, coalesce and why shuffles are expensive.
  • Joins and performance — Broadcast joins, handling data skew, caching and adaptive query execution.
  • UDFs, streaming and MLlib — Pandas UDFs, Structured Streaming and machine learning pipelines.

What you can learn

  • PySpark tutorial for beginners and experienced developers
  • Level-based practice: New, Easy, Medium, Hard, and Diff
  • Python Data Science topics alongside PySpark on skillzunos.com

More Python Data Science tutorials

All tutorials