Skip to main content

08. Databricks Notebooks

Notebooks provide an interactive environment for developing SQL, Python, Scala, and other supported workloads.

Typical lifecycle​

Explore → Develop → Test → Version → Deploy → Schedule → Monitor

SQL vs PySpark​

Use SQL when the transformation is naturally relational and readable as SQL.

Use PySpark when you need programmatic control, complex transformations, reusable functions, DataFrame APIs, or broader Spark functionality.

Notebook best practices​

Keep cells focused​

One logical operation per cell improves readability and debugging.

Parameterize​

Avoid hard-coded environment-specific values.

Separate configuration​

Keep paths, catalog names, dates, and environment settings outside core transformation logic.

Avoid hidden state​

A production notebook should not depend on variables created accidentally by a previous cell execution.

Version control​

Production notebooks should participate in Git-based development and deployment workflows.

Keep business logic testable​

Complex logic should be structured so it can be unit-tested rather than existing only as interactive cells.

Notebook is not the architecture​

Notebook
  │
  ├── Exploration
  ├── Debugging
  └── Thin production entry point
             │
             ▼
      Reusable code / pipeline
             │
             ▼
        Production job

Common mistakes​

  • Hard-coded credentials.
  • Hard-coded storage paths.
  • Huge notebooks.
  • Repeated full-table processing.
  • No source control.
  • No tests.
  • Ignoring Spark execution plans.

Next​

Continue with the next topic in 01. Databricks Foundations.