Databricks interview questions and answers

What is Databricks?

Databricks is a cloud data and AI platform built around Apache Spark, available on AWS, Azure and Google Cloud. It gives notebooks, managed Spark compute, Delta Lake tables, SQL warehouses, pipelines, job scheduling, governance through Unity Catalog, and tools for machine learning and AI apps, all on data stored in your cloud storage.

What is a lakehouse?

A lakehouse keeps data in open file formats on cheap cloud object storage, like a data lake, but adds warehouse features on top: ACID transactions, schemas, indexing-like performance features, governance and fast SQL. One copy of the data then serves BI reports, data engineering and machine learning.

What are the control plane and the compute plane?

The control plane is run by Databricks and holds the web app, notebooks, job scheduler and cluster manager. The compute plane is where clusters actually process data: in classic compute it runs in your own cloud account, and with serverless compute it runs in Databricks-managed infrastructure. Your table data stays in cloud storage.

What is the difference between all-purpose compute, job compute and SQL warehouses?

All-purpose clusters are shared and interactive, for notebooks and exploration. Job clusters start for one scheduled job and end when it finishes, at a lower price per unit. SQL warehouses are compute tuned for SQL queries and BI tools. Serverless versions start in seconds and need no cluster setup.

What is a DBU?

A Databricks Unit is the unit Databricks bills by: a measure of processing capability per hour. The DBU rate depends on the compute type and tier, such as jobs, all-purpose or SQL, and with classic compute you also pay your cloud provider for the virtual machines.

What do the driver and the workers do in a Databricks cluster?

The driver runs your notebook or job code, builds the execution plan and schedules tasks. Workers run executors that process partitions of data in parallel. Collecting a big result to the driver, for example with collect() or toPandas(), can run it out of memory.

What is Delta Lake?

Delta Lake is an open-source table format and storage layer. Data is stored as Parquet files, and a transaction log records every change, which adds ACID transactions, schema enforcement, time travel, MERGE, and fast metadata handling. It is the default table format on Databricks.

How does the Delta transaction log work?

The _delta_log folder holds one JSON file per commit listing the files added and removed, plus periodic Parquet checkpoint files that summarise the state. Readers rebuild the current table by reading the latest checkpoint and later commits, so they never see half-written data.

How does Delta Lake handle two jobs writing to the same table?

It uses optimistic concurrency control. Each writer reads a version, writes new files, then tries to commit the next version. If another commit landed first, Delta checks whether the two changes conflict, for example both touched the same files; if not, it retries the commit, otherwise one job fails with a concurrent modification error.

What is time travel in Delta Lake?

Every commit creates a new table version, and you can query an older version by number or timestamp, compare versions, or restore a table after a bad write. DESCRIBE HISTORY lists the versions with who changed what.

SELECT * FROM sales VERSION AS OF 12;
SELECT * FROM sales TIMESTAMP AS OF '2026-11-01';
RESTORE TABLE sales TO VERSION AS OF 12;
DESCRIBE HISTORY sales;

What is the difference between schema enforcement and schema evolution?

Schema enforcement rejects writes whose columns or types do not match the table, so bad data cannot slip in. Schema evolution allows planned changes, such as new columns, when you enable it with the mergeSchema option on a write or automatic schema evolution for MERGE.

df.write.format("delta").mode("append").option("mergeSchema", "true").saveAsTable("events")

How does MERGE INTO work?

MERGE matches source rows to target rows on a condition, then updates or deletes matched rows and inserts unmatched ones, all in one atomic transaction. It is the standard way to apply upserts and change data. The source must not have several rows matching one target row.

MERGE INTO customers AS t
USING updates AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET *
WHEN NOT MATCHED THEN INSERT *;

What do OPTIMIZE and Z-ORDER do?

OPTIMIZE compacts many small files into fewer large ones, which speeds reads. ZORDER BY sorts data within those files on chosen columns so rows with similar values sit together, letting queries that filter on those columns skip more files using file statistics.

OPTIMIZE events ZORDER BY (customer_id);

What is liquid clustering?

Liquid clustering, set with CLUSTER BY, replaces fixed partitioning and Z-ordering for new tables. Data is clustered on the chosen keys incrementally, you can change the keys later without rewriting the table, and it handles skewed or high-cardinality columns better. Databricks recommends it for new tables.

CREATE TABLE events (event_id BIGINT, customer_id BIGINT, event_date DATE)
CLUSTER BY (customer_id, event_date);

What does VACUUM do on a Delta table?

VACUUM deletes data files that are no longer referenced by the table and are older than the retention period, seven days by default. It saves storage, but afterwards you cannot time travel to versions that needed those files, so do not shorten retention while long jobs or streams may still read old versions.

VACUUM events;            -- default 7-day retention
VACUUM events RETAIN 240 HOURS;

What is the Change Data Feed?

When enabled on a table, the Change Data Feed records row-level inserts, updates and deletes for each version, with a _change_type column. Downstream jobs can read only what changed since the last version they processed instead of re-reading the whole table.

ALTER TABLE customers SET TBLPROPERTIES (delta.enableChangeDataFeed = true);
SELECT * FROM table_changes('customers', 5);

What is lazy evaluation in Spark?

Transformations such as select, filter and join only build a plan; nothing runs until an action such as count, write or display asks for a result. Spark can then optimise the whole plan, for example by pushing filters down to the files, before running it.

What is a shuffle and why is it expensive?

A shuffle moves data between executors so rows with the same key end up together, which wide operations such as groupBy, join and distinct need. It writes and reads data over disk and network, so it is usually the slowest part of a job. Narrow operations such as filter and select need no shuffle.

What is a broadcast join and when should you use it?

Spark sends a full copy of the small table to every executor, so the large table is joined locally without being shuffled. It suits joining a big fact table with a small dimension table. Spark broadcasts automatically below a size threshold, and you can hint it.

from pyspark.sql.functions import broadcast

result = orders.join(broadcast(countries), "country_code")

What does Adaptive Query Execution (AQE) do?

AQE re-optimises a query while it runs, using real statistics from finished stages. It merges small shuffle partitions, switches a sort-merge join to a broadcast join when one side turns out small, and splits skewed partitions. It is on by default in current Spark and Databricks runtimes.

How do you handle data skew?

Skew means a few keys hold most rows, so a few tasks run far longer than the rest. Check task times in the Spark UI, let AQE split skewed partitions, broadcast the smaller side if possible, filter out junk keys such as nulls, or salt the hot keys by adding a random suffix and aggregating in two steps.

How do partition pruning and data skipping make queries faster?

Partition pruning skips whole folders when a query filters on the partition column. Data skipping uses the minimum and maximum values Delta stores for columns in each file to skip files that cannot match the filter. Clustering data on commonly filtered columns makes skipping much more effective.

What is the difference between repartition and coalesce?

repartition(n) does a full shuffle to create n evenly sized partitions and can increase or decrease the count, optionally by a column. coalesce(n) only merges existing partitions without a full shuffle, so it is cheaper but can only reduce the count and may leave uneven partitions.

When should you cache a DataFrame?

Cache when the same DataFrame is reused by several actions, such as in iterative work or when one cleaned dataset feeds several outputs. Do not cache data used once, and unpersist when finished, because cached data takes executor memory. The Databricks disk cache also speeds repeated reads of the same files automatically.

What is Photon?

Photon is Databricks' native vectorised query engine written in C++. It runs Spark SQL and DataFrame operations much faster for many workloads, especially scans, joins and aggregations, without code changes. It costs more per DBU, so check whether the speed-up lowers the total cost.

Why should you avoid Python UDFs where possible?

A normal Python UDF sends rows from the JVM to a Python process and back, row by row, and Spark cannot optimise inside it. Built-in functions run inside the engine and are much faster. If you need Python, a pandas UDF processes data in vectorised batches with Arrow and is far quicker than a row UDF.

What is Auto Loader?

Auto Loader incrementally ingests new files from cloud storage as they arrive, using the cloudFiles source in Structured Streaming. It tracks processed files in a checkpoint, so each file is loaded once, infers and evolves the schema, and puts unexpected fields in a rescued-data column instead of failing.

df = (spark.readStream.format("cloudFiles")
      .option("cloudFiles.format", "json")
      .option("cloudFiles.schemaLocation", "/Volumes/main/raw/_schemas/orders")
      .load("/Volumes/main/raw/orders"))

(df.writeStream
   .option("checkpointLocation", "/Volumes/main/raw/_checkpoints/orders")
   .trigger(availableNow=True)
   .toTable("main.bronze.orders"))

When would you use COPY INTO instead of Auto Loader?

COPY INTO is a SQL command that loads files into a Delta table and skips files it already loaded, which suits simple batch loads of up to thousands of files. Auto Loader scales better to millions of files, supports continuous streaming and handles schema evolution more fully.

What is the medallion architecture?

Data moves through layers. Bronze holds raw data as ingested, so you can always reprocess. Silver holds cleaned, deduplicated and joined data with proper types. Gold holds business-level tables, such as daily revenue by region, ready for reports and models.

What are Lakeflow Declarative Pipelines (formerly Delta Live Tables)?

You declare the tables you want and the queries that produce them, in SQL or Python, and Databricks works out the order, runs the pipeline, manages compute, retries failures and tracks lineage. Streaming tables handle incremental ingestion, and materialized views handle transformations that are refreshed.

What are expectations in a declarative pipeline?

Expectations are data quality rules attached to a table, such as "amount must be positive". For each rule you choose to keep bad rows and record the violation, drop them, or fail the update. Results show up in the pipeline's quality metrics.

CREATE OR REFRESH STREAMING TABLE silver_orders (
  CONSTRAINT positive_amount EXPECT (amount > 0) ON VIOLATION DROP ROW
) AS SELECT * FROM STREAM(bronze_orders);

What does the checkpoint location do in Structured Streaming?

The checkpoint stores which source data has been processed, such as file names or offsets, and the state of aggregations. If the stream stops or the cluster fails, it restarts from the checkpoint with no data lost or duplicated in Delta sinks. Each stream needs its own checkpoint, and deleting it reprocesses everything.

Which trigger modes does Structured Streaming offer?

The default runs micro-batches one after another. processingTime runs a micro-batch at a fixed interval. availableNow processes all data available now, in batches, then stops, which turns a streaming job into a cheap scheduled incremental batch. Continuous processing exists for very low latency but supports fewer operations.

What is a watermark in streaming?

A watermark tells Spark how late data may arrive, such as 10 minutes behind the latest event time seen. Spark keeps aggregation state only within that window and drops older state, so memory stays bounded. Events arriving later than the watermark may be ignored.

from pyspark.sql.functions import window

counts = (events
  .withWatermark("event_time", "10 minutes")
  .groupBy(window("event_time", "5 minutes"), "page")
  .count())

How do you run MERGE inside a streaming job?

Use foreachBatch: Spark hands each micro-batch to your function as a normal DataFrame, and inside it you deduplicate the batch and run a Delta MERGE into the target. With a checkpoint, this gives an incremental upsert pipeline.

def upsert(batch_df, batch_id):
    batch_df.dropDuplicates(["order_id"]).createOrReplaceTempView("updates")
    batch_df.sparkSession.sql("""
        MERGE INTO silver.orders t USING updates s ON t.order_id = s.order_id
        WHEN MATCHED THEN UPDATE SET * WHEN NOT MATCHED THEN INSERT *
    """)

stream.writeStream.foreachBatch(upsert).option("checkpointLocation", ckpt).start()

What is Unity Catalog?

Unity Catalog is Databricks' central governance layer for data and AI assets across workspaces. It manages tables, views, volumes for files, functions and models, with one permission model, auditing, automatic lineage and data discovery. It replaces the per-workspace Hive metastore.

What is the three-level namespace?

Objects are named catalog.schema.table, for example main.sales.orders. A catalog often maps to an environment or business unit, a schema groups related tables, and permissions can be granted at any level and are inherited downwards.

What is the difference between managed and external tables?

For a managed table, Unity Catalog controls the storage location and the files: dropping the table deletes the data after a short recovery window, and Databricks can optimise it automatically. An external table points at a path you manage; dropping it removes only the metadata and leaves the files.

How do you grant access in Unity Catalog?

Grant privileges to groups rather than individuals, using SQL. A user needs USE CATALOG on the catalog and USE SCHEMA on the schema, plus SELECT to read or MODIFY to change a table.

GRANT USE CATALOG ON CATALOG main TO `analysts`;
GRANT USE SCHEMA ON SCHEMA main.sales TO `analysts`;
GRANT SELECT ON TABLE main.sales.orders TO `analysts`;

How do you hide rows or columns from some users?

Unity Catalog supports row filters and column masks: SQL functions attached to a table that decide which rows a user sees, for example only their region, and whether a column such as phone number is shown or masked. Dynamic views are the older way to do the same.

How do you handle passwords and keys in Databricks code?

Store them in a secret scope, backed by Databricks or a cloud key vault, and read them at run time with dbutils.secrets.get. Values are redacted if printed in notebook output. Never hard-code credentials in notebooks or commit them to Git.

password = dbutils.secrets.get(scope="prod-db", key="password")

What does Unity Catalog lineage give you?

Lineage is captured automatically down to column level for queries run on Unity Catalog tables, showing which tables, notebooks, jobs and dashboards read or write each table. It helps with impact analysis before a change, debugging bad data and audits. System tables expose lineage, audit logs and billing for queries.

How do you schedule and orchestrate work in Databricks?

Lakeflow Jobs, formerly Workflows, run a DAG of tasks such as notebooks, Python scripts, SQL, pipelines or other jobs, with schedules or file-arrival triggers, dependencies, parameters, retries, alerts and job clusters. A failed run can be repaired by rerunning only the failed tasks.

How do you do CI/CD for Databricks?

Keep code in Git through Git folders, define jobs, pipelines and settings as code with Databricks Asset Bundles, and deploy the same bundle to dev, staging and production targets from a CI pipeline after tests pass. This avoids hand-edited jobs that drift between environments.

How is MLflow used on Databricks?

MLflow tracks experiments, recording parameters, metrics and artifacts for each training run, packages models in a standard format, and registers model versions, which on Databricks live in Unity Catalog with permissions and lineage. It also traces and evaluates LLM and agent applications.

How would you build a RAG app on Databricks?

Ingest and chunk documents into a Delta table, create a vector search index that syncs from that table and stores embeddings, serve the chosen language model through a model serving endpoint, and connect them in an app that retrieves chunks and generates answers. Unity Catalog governs the table, index and model.

How do you implement an SCD Type 2 dimension on Delta Lake?

Keep start date, end date and an is_current flag. In one MERGE, close the current row when tracked columns change and insert the new version; a common trick unions the changed rows a second time with a null merge key so they fall into the insert branch. Declarative pipelines can do this with AUTO CDC (formerly APPLY CHANGES) and SCD Type 2.

MERGE INTO dim_customer t
USING staged s  -- changed rows twice: once with merge_key, once with NULL
ON t.customer_id = s.merge_key AND t.is_current
WHEN MATCHED AND t.address <> s.address THEN
  UPDATE SET is_current = false, end_date = s.effective_date
WHEN NOT MATCHED THEN
  INSERT (customer_id, address, start_date, end_date, is_current)
  VALUES (s.customer_id, s.address, s.effective_date, NULL, true);

A table has millions of small files. How do you fix it?

Run OPTIMIZE to compact existing files, then stop creating new small files: enable optimized writes and auto compaction, avoid partitioning on high-cardinality columns, use liquid clustering, and make streaming jobs write less often, for example with availableNow runs. Predictive optimization can run maintenance automatically on managed tables.

A Spark job suddenly runs much slower. How do you investigate?

Open the Spark UI and find the slowest stage. Look for skew (a few tasks far slower), spill to disk (too little memory per task), a huge shuffle, a join that should have been broadcast, or a scan reading far more files than expected. Then check what changed: data volume, code, cluster size or runtime version.

How do you delete a user's data for a privacy request?

Run DELETE on every table that holds their data, which writes new versions without those rows, then run VACUUM after the retention period so the old files that still contain the rows are physically removed. Remember downstream copies, and that time travel can still read old versions until VACUUM runs.

How would you reduce Databricks cost?

Run scheduled work on job compute, not all-purpose clusters; set auto-termination and autoscaling limits; use spot instances for workers; right-size clusters from metrics; turn incremental jobs into availableNow runs instead of always-on streams; compact files and cluster data so queries scan less; and watch billing system tables per job.

How do you make a pipeline safe to rerun?

Make every write idempotent: use MERGE on business keys instead of blind appends, overwrite whole partitions or use replaceWhere for date-based reloads, keep streaming checkpoints, and deduplicate input on a unique ID. Then rerunning after a failure gives the same result instead of duplicates.