EPRM - Flashcards
All 139 flashcards for Engineering Production-Ready ML/AI Systems, one set per note. Each card asks one thing and has a short answer. Say the answer before you reveal it, and only press Got it if you had all of it. Study everything at once, a single lecture, or tick any mix. Progress lives only in this browser tab and disappears when you close it. The long-form questions are on the exam questions page.
Back to the AI Systems home.
L01 Data Roles, ML Lifecycle, Clean Code and Unit Testing
Flashcards
According to Yesilbek, what are the two main reasons ML/AI projects fail?
Roles and teams could not collaborate (with each other or stakeholders), and the models could not be deployed.
What does the 87% figure in Davenport and Zhang's MIT Sloan article actually measure?
A VentureBeat analysis suggesting 87% of AI models are never put into production.
Why does deployment matter so much for return on AI, according to Davenport and Zhang?
Without production deployment there is no economic value.
What is the difference between a brilliant model and a brilliant outcome?
The model is output; the outcome is the change in user or business behaviour it actually causes.
Why are all ML roles considered data roles?
Without data in a usable state there is no ML, so every ML role depends on and works with data.
What four properties should data engineers guarantee for the data they serve?
High quality, right format, available on time, useful for the purpose.
In the ride-hailing example, what format was the timestamp in?
The hexadecimal form of a UTC Unix epoch.
Why can't an analyst just run large queries on the backend database?
It is optimised for backend operations; big queries cause latency spikes, and core operations take priority.
Name three data-quality issues in the ride-hailing event data.
A near-zero-length ride, a second ride start before the first ended, rides that never ended.
What does "shift-left" mean for an ML engineer's relationship to data engineering?
Data engineering work moves onto you: expect to do your own ETL for your ML system.
In the EV example, what is the data and what is the information?
Data: battery percentage. Information: estimated remaining range in kilometres.
What is a data analyst's core task, according to Yesilbek?
Understanding the business, question and data, and turning data into information for decision makers.
What is the difference between ML as a function of a product and ML as a product?
Function: ML adds a feature (YouTube Shorts recommendations). Product: the ML system is the product (Google Translate, ChatGPT).
Why must a data scientist's features be computable in the live environment?
A feature that cannot be computed in production, within latency and cost limits, is useless to the deployed model.
What is the ultimate goal of a data scientist?
Solving the business problem through ML; models and code are only the means.
What two things does an ML engineer focus on?
Productionization of ML solutions and operations of ML systems.
What three activities make up ML operations?
Maintenance, monitoring, firefighting.
Why did handing models to backend engineers go badly?
Data scientists lacked software engineering skills and the engineers lacked ML knowledge, so nobody owned the whole system.
What two benefits does a platform team bring to application teams?
Application teams get faster; infrastructure gets more reliable.
Which two roles does a full-stack ML engineer combine?
Data scientist and ML engineer.
Why does Yesilbek favour full-stack ML engineers over specialised handovers?
One person owns problem and solution end to end; splits and handovers repeatedly caused project failures.
What makes production different from development and acceptance environments?
Only production is live, and it is protected; changes reach it after passing the earlier stages.
Why should all environments run the same tech stack?
So that what passes testing in acceptance behaves the same in production.
Give the steps of the DevOps loop.
Plan, code, build, test, release, deploy, operate, monitor.
Give the steps of the model development loop.
Collect, curate, transform, validate (data); explore, train, evaluate, formulate (ML).
In the MLOps eternal knot, how do the Ops steps lead back to the data side?
Monitoring feeds planning, planning leads to formulating the problem, and formulating leads to collecting new data.
What equation summarises MLOps in Gupta's lifecycle diagrams?
MLOps = DataML + DevOps.
Why should file reading be kept out of a preprocessing function?
The function then does one thing and can be unit tested on a small DataFrame without files on disk.
What is a shallow abstraction?
A wrapper that adds nothing, like a function that only calls pd.read_csv; it just adds indirection.
What does the expression x - mean / std compute, and why is that a dangerous bug?
x minus (mean/std); it runs without error, so wrong features silently reach the model.
Why are flag arguments a code smell?
A boolean parameter shows the function does two things, often with two return types.
What should a code comment explain?
Why the code does something, or a warning of consequences; the code itself shows what.
What should you do with commented-out code?
Delete it; version control keeps the history.
What are the three steps of a unit test?
Setup, collect, compare.
Why mock an external dependency in a unit test?
Real dependencies make tests slow, unrepeatable or destructive; a mock isolates the unit.
Link to originalWhat does FIRST stand for in Yesilbek's version?
Fast, Isolated (independent), Repeatable, Self-validating, Thorough.
L02 Distributed Data Processing
Flashcards
Why do extra threads not speed up CPU-bound Python code in standard CPython?
The global interpreter lock lets one thread run Python bytecode at a time.
What limits each step of single core, multiprocessing, distributed computing?
Clock speed caps a core; core count caps a machine; distributed computing is uncapped.
What is the difference between vertical and horizontal scaling?
Vertical: a bigger machine. Horizontal: more machines.
Which tool fits small data with complex operations, in Yesilbek's grid?
A data processing library such as pandas.
Which tool fits large data with simple operations, in Yesilbek's grid?
Multi-core libraries such as Polars, or concurrency.
Which tool fits large data with complex operations, in Yesilbek's grid?
Distributed computing, such as Spark.
What is accidental complexity?
Complexity your solution adds beyond what the problem itself requires.
Which three things should you avoid as long as you can, according to Yesilbek?
Concurrency, distributed computing, GPU computing.
What is the main read pattern of OLTP?
A small number of records per query, fetched by key.
What is the main write pattern of OLAP?
Bulk import, ETL.
Is a typical OLTP result wide or tall?
Wide: many columns, few rows.
Why should analytics not run heavy queries on an OLTP store?
It is optimised for the backend; big queries cause latency spikes for users.
In the 1970s archive example, how should you shelve documents requested in bulk by topic?
By topic, so one request is one trip to one shelf: the columnar idea.
Why does a columnar layout suit analytical queries?
A query reads only the few columns it needs and skips the rest.
Why do columnar files compress better than row files?
Each chunk holds one column’s values: same type, often repetitive.
What are the main parts of a Parquet file, from outer to inner?
Row groups, column chunks, pages; plus a footer with metadata.
Why is Parquet's metadata in a footer at the end of the file?
A reader reads it first and then jumps straight to the column chunks it needs.
What lets a Parquet reader skip row groups for a filter like amount > 100?
Minimum and maximum statistics per column chunk in the footer metadata.
What does a folder name like chck_date_day=2019-01-01 do in a Parquet dataset?
Partitions by day; a filter on that column skips whole folders.
In a distributed mean, what does each worker send back?
Its chunk’s sum and count.
When does averaging the workers' means give the correct overall mean?
Only when all chunks have the same size.
What is the difference between PySpark and SparkSQL?
PySpark is Spark’s Python DataFrame API; SparkSQL writes the same query as SQL.
What does F.stddev return for a group with a single row?
NULL, because it is the sample standard deviation.
What is a narrow transformation in Spark?
One where each output partition depends on one input partition: no data movement.
Give three wide operations in Spark.
Group by, aggregate, join.
What is a shuffle?
Redistributing rows across executors by key, through disk and network, for a wide operation.
Why is a global count or average cheap even though it combines all partitions?
Each partition sends only a partial count or sum.
How does a broadcast join avoid shuffling the large table?
Spark copies the small table to every executor, so the large side stays put.
When does Spark execute a chain of transformations?
Only when an action needs a result, such as head, collect, count or write.
Why is Spark's evaluation lazy?
Seeing the whole job first lets it optimise the plan before running anything.
What is predicate pushdown?
Applying a filter inside the file scan, so data can be skipped at read time.
What does ReadSchema with only one column show in a Spark plan over Parquet?
Column pruning: Spark reads only the columns the query uses.
Why can a bug in a Spark transformation crash a later line?
Runtime errors surface only when an action runs the plan.
What does the Spark driver do?
Runs your program, builds the plan, splits it into tasks and schedules them.
What runs on a Spark worker node?
An executor that runs tasks and caches data.
What is the relationship between Spark tasks and partitions?
Roughly one task per partition, plus tasks for work like listing files.
"), how many tasks run in parallel?
Two.
Why does spark.read.parquet start jobs before any action?
It lists the files and reads the schema straight away; only data processing waits.
What are the four points where a batch ML pipeline can fail?
Trigger, read, process, write.
What is the good design for a missed trigger?
Read the delta between source and sink, keeping state of the latest processed timestamp.
Why does replaying scheduler triggers fail at cold start?
Data from before the first release never had a trigger to replay.
What does "the delta between source and sink" mean?
Everything in the source that has no result in the sink yet.
What does it mean for a pipeline run to be idempotent?
Running it twice gives the same result as running it once.
Why should a timestamp state track arrival time rather than event time?
Late records with early event times would fall behind the state and be skipped.
What is the downside of running a long pipeline as one task without checkpoints?
All succeeds or all fails: a crash loses all progress.
Why must intermediate results not live on the local hard disk?
The crashed machine may be gone, and other workers cannot read it.
Why can a write failure leave a green run?
A connectionless write never learns that the write failed.
What does atomicity guarantee for a database write?
The transaction completes fully, or the database reverts to its state before it.
What does durability guarantee?
A successful transaction survives a later system failure.
Link to originalWhat should a run's success be based on when writing results?
ACK/NACK from the receiver: success only when every write is acknowledged.
Book Ch20: Data Science and Software Engineering Process Models
Flashcards
Why is data science model development hard to plan in advance?
It is exploratory: it is unknown whether a target accuracy is reachable or how long it will take.
What did the four-hour digit-recognition experiment show about model progress?
Accuracy improved in irregular, unpredictable jumps, and final accuracies differed widely between participants.
What is CRISP-DM?
A cyclic data mining process: business understanding, data understanding, data preparation, modelling, evaluation, deployment.
Why do notebooks suit exploration?
Quick feedback, incremental re-execution, easy copying of alternatives, inline plots, code and results shared together.
What should happen to notebook code when a model moves to production?
Clean it up and migrate it into a modular, tested, versioned pipeline.
What is a data-science trajectory?
The particular path a project takes through the common data science steps.
Why does software engineering favour planning before coding?
Defects in requirements and design cost far more to fix the later they are found.
What does the spiral model add to the waterfall model?
Repeated iterations with growing prototypes, tackling the riskiest parts first.
Why do agile methods iterate?
Requirements are rarely stable or fully known, so teams replan after short sprints with customers.
How does data science iteration differ from agile iteration?
Agile iterates to clarify changing requirements; ML iterates to discover whether a model can meet a fairly stable goal.
Why does the spiral model's "risky part first" strategy fit ML poorly?
Model development rarely splits into risky and safe parts; feasibility of an accuracy target is unclear until tried.
According to Kästner, are copy-paste and global state in exploratory notebooks a problem?
In exploration they are convenient and sufficient; the code just should not be treated as production-ready.
What do untrained software engineers typically do when their first model is too weak?
Collect more data or try deep learning, instead of exploring data and engineering features.
What is model-first development?
Building the model first and designing a system around it once its feasibility is established.
What is the main risk of model-first development?
The model may not fit system needs, such as latency, trust, explainability or safety, forcing a rebuild.
When is system-first development advisable?
When adding ML to an existing system, and in regulated or safety-critical domains.
Link to originalWhat do data scientists often complain about in system-first projects?
Unrealistic model requirements, especially after promises made to clients.
Book Ch21: Interdisciplinary Teams
Flashcards
What is a "unicorn" in discussions of ML teams?
A rare developer combining statistics, engineering and domain expertise in one person.
Give the two reasons Kästner says teams are needed even if you hire a unicorn.
Division of labour (systems are too big for one person) and division of expertise.
How many communication links does a group of n people have?
n(n-1)/2.
Seven people split into teams of 3 and 4, linked by one representative each: how many communication links?
10 (3 + 6 + 1), down from 21 in one group.
What is information hiding?
Others use a component only through its interface, without knowing its implementation, so teams coordinate only on the interface.
What does Conway's law (socio-technical congruence) say?
System structure tends to mirror team structure, so components should align with team boundaries.
Why are ML component interfaces rarely stable?
Data science is exploratory: what the model can do is only discovered as the project evolves.
Why can't fairness be assigned to the model team alone?
Fairness emerges from how all components interact in their environment, so no single component can assess it.
Why does Kästner recommend an interdisciplinary team per component over teams sorted by specialty?
Expertise does not line up with modules: a model team needs engineering skills, a UI team ML knowledge.
What is awareness as a coordination strategy?
Actively broadcasting changes and decisions to everyone who might be affected.
Why must awareness mechanisms be targeted?
Broadcasting everything causes information overload, and important news gets lost in the noise.
What is a T-shaped person?
Deep expertise in one area plus broad basic knowledge of the others, enough to talk to their specialists.
How do performance reviews deepen goal conflicts in ML teams?
Each role is measured on its own metric (features, downtime, accuracy) instead of the project’s goals.
What is groupthink?
Minimising conflict by not exploring alternatives, suppressing dissent and isolating the group from outside views.
Name five interventions against groupthink.
Diverse teams, outside experts, high-status members speaking last, devil’s advocate, exploring a second solution.
What is the difference between a matrix and a project organisation?
Matrix: experts belong to departments and are lent to projects. Project: people belong to one project only.
What did software security teach about spreading rare expertise across teams?
Light literacy, so developers recognise when to call an expert, works better than making everyone an expert.
How does DevOps turn developers and operators from rivals into collaborators?
Joint goal of frequent delivery with rapid feedback, shared vocabulary, and joint tools such as containers and monitoring.
Link to originalWhat do data scientists gain from MLOps in exchange for learning model packaging?
The ability to experiment rapidly in production.
Book Ch22: Technical Debt
Flashcards
What is technical debt?
Shortcuts that are expedient now but make future changes more costly or impossible.
Why is the technical debt metaphor useful with managers?
It explains in familiar terms why maintenance and infrastructure work now avoids larger costs later.
What are the two dimensions of Fowler's technical debt quadrants?
Deliberate versus inadvertent, and reckless versus prudent.
What makes technical debt inadvertent?
Not understanding the consequences of a decision or not knowing better practices.
Give an example of prudent technical debt for delivery robots.
Deploying early with weak navigation models, paying for remote human operators, to win a competitive market.
Give an example of reckless technical debt for delivery robots.
Skipping safety testing of perception models, risking accidents, liability and lost licences.
Which quadrant is ideal for technical debt?
Deliberate and prudent.
Why is using ML where it is not needed a form of technical debt?
Production ML is expensive to engineer and maintain, unlike heuristics or established non-ML solutions.
What is data debt?
Unchecked dependencies on external and internal data whose quality can change unnoticed without schemas, monitoring and provenance.
What is infrastructure debt in ML?
Manual, brittle model deployment, making updates rare and leaving machines on inconsistent model versions.
What does versioning debt prevent you from doing?
Reproducing results and tracing what changed in data or training code when performance drops.
Why does observability debt slow down fixing production problems?
Without production monitoring of model quality, the team guesses at fixes and spends days watching robots by hand.
Name ML-specific sources of code quality debt.
Dead experimental code, glued-together scripts, duplicate feature extraction, poorly named variables.
Why does chaining models create architectural and test debt?
Without model and integration tests you cannot tell which model causes a new system behaviour.
How can skipped requirements analysis create debt, in the delivery robot example?
A speed-optimised robot drives aggressively, pedestrians yield, speed is rewarded: a harmful feedback loop.
Which is harder to repay: requirements and design debt, or code quality and automation debt?
Requirements and design debt: it can leave a fundamentally flawed design, while monitoring or automation can be added later.
Link to originalWhat is the minimum for managing deliberately taken debt?
Track the deferred work in an issue tracker, backlog or roadmap.