EPRM - Exam Questions

All 10 exam-style questions for Engineering Production-Ready ML/AI Systems, one set per note. Each answer opens with the key points a grader looks for, then a full model answer. Write or say your answer before revealing. The short drilling cards are on the flashcards page.

Back to the AI Systems home.

L02 Distributed Data Processing

Exam questions

Your dataset is 3 GB and fits in memory. Why not use Spark anyway?

Key points: essential versus accidental complexity; Spark adds a cluster, scheduling, lazy plans and new failure modes; pandas or Polars handle the case; scale only when data size and latency demand it.

The solution should only be as complex as the problem. A 3 GB dataset that fits in memory is small data: pandas handles complex operations on it and Polars can use every core of the machine for simple ones. Spark would add accidental complexity: a session and cluster to configure, lazy plans to debug, partitions and shuffles to tune, and more things that can fail, while solving nothing pandas cannot. The course’s rule is to avoid concurrency, distributed computing and GPU computing as long as you can, and move along the line from single core to multiprocessing to distributed computing only when data size and latency requirements push you there.

How would you compute the mean of a dataset spread across several machines, and why not average the machines' means?

Key points: split the data into chunks; each worker computes a local sum and count; the coordinator divides the total sum by the total count; averaging means is only correct for equal-size chunks; only two numbers per worker cross the network.

Split the data into chunks and give each worker one. Each worker computes the sum $S_k$ and count $n_k$ of its chunk and sends only those two numbers back. The coordinator computes ${\sum }_k{S}_k/{\sum }_k{n}_k$. Averaging the workers’ means gives the right answer only when every chunk has the same size: with chunks [1, 2, 3] and [10] the means are 2 and 10, their average is 6, but the true mean is 4. Sending sums and counts is also cheap, because the raw data never moves. In Spark terms, this is why a global average is cheap: partial results are computed locally per partition and only those are combined.

Why does your pipeline store its data as partitioned Parquet instead of CSV?

Key points: columnar layout reads only needed columns; per-column encoding compresses well; footer metadata and statistics let readers skip row groups; partition folders let readers skip whole days; types and schema are stored.

Analytical and ML jobs read a few columns of many rows, the OLAP pattern. CSV is a row format: to use one column you parse every byte of every row, and nothing says what type a field is. Parquet stores each column of a row group as a separate chunk, so a reader fetches only the columns it uses. Values of one column are similar, so they encode and compress well, which cuts disk and network traffic. The footer holds the schema, the location of each column chunk and statistics such as minimum and maximum values, so a filter can skip row groups that cannot match. Partitioning the folder by day (chck_date_day=...) lets a reader skip whole days, which also makes reading only new data cheap.

What is the difference between OLTP and OLAP, and why should an ML training job not query the app's backend database directly?

Key points: OLTP serves the backend: few records by key, low-latency writes, wide results; OLAP serves analytics: aggregates over many records, bulk loads, tall results; backend stores are optimised for OLTP; big queries cause latency spikes for users.

OLTP (online transaction processing) is the backend’s workload: fetch or update a few records by key, such as one user’s profile, with low-latency writes from user input, returning wide results (many columns, few rows). OLAP (online analytics processing) is the analytics workload: aggregate over huge numbers of records, loaded in bulk by ETL, returning tall results (many rows, few columns), on TBs to PBs of data. A backend database is laid out and tuned for OLTP, typically row-wise. A training job’s full-history scan is slow there and competes with the app’s own queries, causing latency spikes for real users, whose requests take priority. So the data is copied out to storage built for analytics, such as Parquet files, and the heavy reads happen there.

Which of your Spark operations are narrow and which are wide, and why does it matter?

Key points: narrow: each output partition needs one input partition, no data movement (arithmetic, filter, sample); wide: needs rows from many partitions (groupBy, join, sort); wide triggers a shuffle through disk and network; shuffles dominate cost; global count/avg only move partial results.

In a narrow operation every partition can be processed on its own executor without seeing other data: column arithmetic like the z-score withColumn, filters, sampling. In a wide operation an output depends on rows spread over many partitions: a groupBy needs all data for one key together (Spark first reduces each partition to partial aggregates per key), a join needs matching keys together, a sort needs global order. Spark meets that with a shuffle: rows are written out, sent over the network to the executor that owns their key, and read back. That is usually the most expensive part of a job, so you filter and select before grouping or joining, to shuffle less data. A join with a small table, such as per-group statistics, avoids shuffling the large side, because Spark broadcasts the small table to every executor. A whole-dataset count or average is cheap despite combining all partitions, because each partition sends only its partial count or sum.

When does Spark actually execute your code, and what does it gain from waiting?

Key points: transformations only build a plan; actions (head, collect, count, show, write) trigger execution; seeing the whole plan lets Spark optimise; filter reordering, column pruning and predicate pushdown into Parquet; explain() shows the plan.

Spark is lazy. Transformations such as select, filter, withColumn and join return immediately and only add steps to a plan. Execution starts when an action needs a result: head, collect, count, show or write. Because Spark sees the whole plan first, it can optimise it. In the demo the code doubled amount and then filtered on it, but the physical plan filtered first, read only the amount column from Parquet (column pruning), and pushed the amount > 100 condition into the Parquet scan (predicate pushdown), where row group statistics can skip data. explain("formatted") shows that plan without running anything. A practical consequence: a runtime error surfaces only at the action, so the line that fails is not always the line with the bug, although a misspelt column name is caught at once.

") and has 367 tasks to run. What happens, and what would change on a cluster?

Key points: local[2] means everything on one machine with two cores; at most two tasks run at once, the rest queue; roughly one task per partition; driver plans and schedules, executors run tasks; a cluster adds executors on worker nodes via the cluster manager.

local[2] runs everything on your own machine with two cores, so at most two tasks run in parallel and the other tasks wait in a queue. Spark creates roughly one task per partition, plus tasks for work such as listing files and reading the schema. On a cluster, the driver still builds the plan and splits it into tasks, but a cluster manager gives it executors on several worker nodes, each running several tasks in parallel and caching data, and exchanging data with each other in shuffles. More executor cores means more tasks at once; more tasks than cores just means tasks queue. Very many tiny partitions waste time on scheduling, which is why Spark packs small files together when reading.

Your daily scoring job did not run for four days. How does your design recover without code changes?

Key points: do not hard-code “process yesterday”; read the delta between source and sink; keep state of the latest processed timestamp; independent of the scheduler; cold start and backfills handled; cost is a long catch-up run.

A job that reads only the previous day would need code changes and manual runs for each missing day. Replaying the scheduler’s missed triggers works most of the time, but it cannot handle a cold start (data from before the first release, for which no trigger exists) and makes migrations painful. My job instead reads the delta: everything in the source that has no result in the sink yet, found using a saved state with the latest processed timestamp. Whenever it starts, it processes what is missing: one day normally, five days after the outage, all history on the first run. The trade-off is that a catch-up run can take a long time. Because a second run finds an empty delta, the job is also idempotent: rerunning it does not duplicate predictions.

Your job ran for two days and then crashed on a bug. How would you design it so you do not lose two days?

Key points: a single long task is all-or-nothing; use checkpoints that store each operation’s result; or split into tasks that save and read intermediate results; rerun only the failed part; intermediate storage must be shared and durable, not the local disk; costs complexity.

As a single task with no intermediate data, the job either completely succeeds or completely fails, which is fine for short pipelines and painful for long ones. Two fixes: checkpoints, where each operation’s result is stored automatically and a replay resumes from the last one; or splitting the pipeline into tasks (preprocessing, features, prediction) that each write their output to intermediate storage and read their input from the previous task’s output, so only the failed task reruns. The intermediate storage must not be the local hard disk: the crashed machine may be gone and other workers cannot read it. The price is storage and a more complex design. Spark itself recomputes lost partitions when an executor dies, but that does not survive the application crashing.

The run was green but half the predictions are missing from the sink. How did that happen and how do you prevent it?

Key points: connection broke mid-write; a fire-and-forget write never learns of the failure; use an ACID sink so a broken write rolls back; mark success only on ACK, retry on NACK; latency cost is acceptable for batch.

The processing succeeded but the connection broke during the write. With a connectionless, fire-and-forget method, the writer never learns which writes failed, so the run reports success. Two fixes: write to a database with ACID properties, so a transaction either commits completely or is rolled back (atomicity) and a committed one survives failures (durability); and make the run’s success depend on acknowledgements, so a NACK or missing ACK triggers a retry or fails the run. Then a green run means the results reached the target. Waiting for acknowledgements adds latency, which a batch job can afford.

Link to original