Interview Rounds

Fermi problems. Nobody expects the right number — they expect a decomposition they can follow and assumptions they can push on.

1. How much storage would you need to keep one year of logs for a service handling 5,000 requests per second?Decompose: 5,000 rps × 86,400 s ≈ 430M requests a day. At roughly 1 KB per log line that is ~430 GB a day, ~155 TB a year raw. Then apply what a real system does to that number: compression at 5–10× brings it to 15–30 TB, and a tiered retention policy — 7 days hot, 90 days warm, the rest cold or sampled — reduces the expensive part to under a terabyte. Give the raw figure, then the engineered figure, because the second is the answer that shows you have done this.
2. How many GPUs would you need to serve an 8B parameter model to 1,000 concurrent users?Start with memory: 8B at bf16 is ~16 GB of weights, plus KV cache which is what actually scales with concurrency. Roughly 0.5–1 MB per 1K tokens of context per sequence for a model this size, so 1,000 concurrent sequences at a few thousand tokens each is tens of gigabytes — the cache dominates. One 80 GB card fits the weights comfortably but not that concurrency, so you are at 2–4 cards with paged KV cache and continuous batching, and more if the context is long. Say explicitly that the throughput calculation, not the weight calculation, decides the answer.
3. Estimate the daily cost of running an LLM feature for 100,000 users.Build the unit first: requests per user per day × tokens per request × price per token. Say 3 requests, 1,500 input and 300 output tokens. That is 100k × 3 = 300k requests, ~450M input and 90M output tokens a day. Multiply by whatever per-million rate you assume — state it — and you land in the low thousands of dollars daily. Then note the two levers that matter more than the model choice: caching repeated prefixes, and routing easy requests to a smaller model.
4. How many labelled examples do you need to fine-tune a classifier to production quality?Reject the premise gently: it depends on the number of classes, how separable they are, and what quality bar you are being held to. Give the working answer anyway — for a fine-tune on a strong pretrained model, a few hundred well-chosen examples per class usually gets you most of the way, and the return on more data flattens fast. Then say the thing that matters: label quality and coverage of edge cases beat volume, and the way to find out is to plot a learning curve at 50, 100, 500 and extrapolate.
5. How much would it cost to relabel a dataset of 200,000 documents?Time per document × cost per hour × any multiple review. At 2 minutes each that is ~6,700 hours; at a plausible annotation rate that is a large six-figure number, and double it if you need two annotators per item for agreement. Which is exactly why the real answer is not to relabel everything: label a stratified sample, measure model-versus-human agreement, and relabel only the slices where they diverge.
6. Estimate how long it would take to train a model on 10 TB of data.The bound is usually input, not compute. Work out read throughput first — 10 TB from object storage at 1 GB/s is ~3 hours per epoch just to move the bytes, before any GPU work. Then compare to the compute time per epoch. Naming which side is the bottleneck is the answer; candidates who only compute FLOPs miss that most large training jobs are starved by the data loader.
7. How many engineers would you need to rebuild this system in six months?Decompose by component rather than guessing a headcount: list the four or five pieces, estimate each in engineer-months, add 30–50% for integration and unknowns, then divide by six. Say out loud that the division is the weak step — adding people does not divide the timeline, and above a certain size coordination cost eats the gain.
8. Estimate the number of daily active users a feature needs for an A/B test to detect a 2% lift.This one has real arithmetic behind it. Required sample size scales with the baseline variance and inversely with the square of the effect size, so halving the effect you want to detect quadruples the sample. For a conversion metric around 10% baseline and a 2% relative lift, you are into the hundreds of thousands per arm. The point to make: if the traffic does not exist, the honest answer is that the experiment cannot be run at that sensitivity — not that you run it anyway and report a null.

The interviewer plays the system and answers your questions. Bisecting beats guessing, and the first question you ask is most of the score.

1. The model's accuracy dropped 10% overnight. What do you check first?Ask what changed, because something did. In order: was there a deploy of the model, the feature pipeline, or an upstream producer; did the input distribution shift; did the label source change; is the metric itself broken. Check the metric last only if the drop is implausibly sharp — a clean overnight step is far more likely a pipeline or deployment change than genuine model degradation, which drifts rather than steps.
2. Latency went up 3× but CPU and memory look normal. Where do you look?If the machine is not busy, it is waiting. Look for external calls — a slow dependency, a database query gone unindexed, DNS, connection pool exhaustion, lock contention. Compare p50 and p99: a p99-only change points at a subset of requests or a queue, whereas the whole distribution shifting points at a dependency common to all of them.
3. A pipeline succeeded but produced no rows. How do you find out why?Success with no output means a filter matched nothing or an upstream input was empty. Work backwards through the stages, checking row counts at each boundary until you find where it goes to zero. Then note the real defect: a job that can produce zero rows and still report success is missing an assertion, and the fix is that check, not just this incident.
4. Users report the search results got worse, but every metric is flat. What now?Trust the users and distrust the metric. Either the metric averages over a segment where it did get worse, or it measures something users do not care about. Slice by query type, language, device, and new-versus-returning, and go and read actual bad queries. Aggregate metrics hiding a broken slice is the single most common version of this.
5. The same query is fast in staging and slow in production. Why might that be?Data volume and distribution first — a plan that works on a small table can flip to a sequential scan on a large one. Then: stale statistics, missing index in one environment, cold cache, concurrent load, different hardware. Ask for the query plan from both; the answer is usually visible there rather than in the query text.
6. Your model works in a notebook and fails in the service. What is the likely cause?Almost always a difference in preprocessing between the two paths — different library version, different default, a column order, a fitted transformer that was refit instead of loaded. This is training-serving skew, and the structural fix is sharing one code path rather than fixing the specific mismatch.
7. An LLM feature started hallucinating more this week, with no code change.If your code did not change, something else did: the provider's model version, the retrieved context, the data being retrieved over, or the input distribution. Check whether retrieval quality dropped first, since a RAG system with degraded retrieval looks exactly like a model that started hallucinating. Then check whether you are pinned to a model version — an unpinned endpoint is a dependency that updates without telling you.
8. One customer reports errors that you cannot reproduce. How do you proceed?Get the specific request — id, timestamp, payload shape — and trace it rather than trying to reproduce it. Ask what is different about them: volume, region, data size, an unusual character set, an old client version. Cases that only affect one customer are usually about their data, not your logic.
9. Throughput drops sharply above a certain load rather than degrading gradually. What does that suggest?A cliff means a limit being hit, not a resource being consumed: a connection pool, a thread pool, a rate limit, memory tipping into swap or GC pressure, or a queue growing until timeouts cascade. Gradual degradation is saturation; a cliff is a bound. Naming that distinction is the answer.
10. How would you tell whether a problem is in your code or in a dependency?Bisect the boundary. Call the dependency directly with the same input outside your code; if it is slow or wrong there, it is theirs. If not, the difference is in how you are calling it — arguments, concurrency, connection reuse. Doing this before escalating is what makes the escalation credible.

Open-ended "why did this change" and "how would you find out" questions. Structure first, then a recommendation — exploring forever is the failure mode.

1. Sign-ups fell 15% last week. How do you investigate?First establish it is real: compare against seasonality and the same week last year, and check the tracking did not break — a logging change is a common cause of a sudden clean drop. Then decompose the funnel and find which step moved, and slice by channel, platform, region, and new-versus-returning. Isolating the drop to one segment usually names the cause. Close with what you would do about it.
2. Two teams report different numbers for the same metric. How do you resolve it?Do not adjudicate — reconcile. Compare the definitions first: time window, timezone, which events count, deduplication, whether bots and internal traffic are excluded, late-arriving data. It is nearly always a definitional difference rather than a bug, and the durable outcome is one agreed definition in one place rather than a corrected number.
3. A feature launched and retention improved. Did the feature cause it?Not established. Anything else shipping in that window is a confounder, and the users who adopt a new feature are self-selected — engaged users both adopt features and retain. If there was an experiment, use it. If not, the honest answer is a difference-in-differences against a comparable unexposed group, stated with its limitations, plus a proposal to run the experiment properly.
4. How would you measure whether a recommendation system is any good?Offline and online, and be explicit that they disagree. Offline: ranking metrics on held-out interactions, plus coverage and diversity so you catch a model that only recommends the popular items. Online: the metric the business actually wants, guardrails for long-term health, and an awareness of the feedback loop — the model shapes the data it is next trained on.
5. Your A/B test shows no significant difference. What do you report?Report it as no detected effect and give the confidence interval, not "no effect". Then say what the test could have detected: if the interval spans everything the team cares about, the experiment was underpowered and the result is uninformative rather than negative. Distinguishing those two is the whole answer.
6. How would you decide whether to build a model or a set of rules?Rules if the logic is known, stable, and needs to be explainable or audited; a model if the pattern is complex, shifting, or you cannot articulate it. Then the pragmatic point: start with rules as the baseline regardless, because you need something to beat, and a surprising number of projects find the rules are sufficient.
7. A stakeholder wants a dashboard. How do you decide what goes on it?Start from the decision, not the data. Ask what action changes based on this number and how often they will look — then build the smallest thing that supports that decision. Dashboards built by listing available metrics get looked at twice and abandoned.
8. Half your data has a field missing. What do you do?Find out why it is missing before deciding how to handle it — missing at random is a different problem from missing because of the outcome. If a field is absent for a particular segment or period, that pattern is information and often a feature in itself. Only then choose: drop, impute with an explicit indicator column, or model without it. Imputing before diagnosing the mechanism can bake in a bias.
9. How would you detect fraud in a dataset with no fraud labels?Unsupervised first: anomaly detection on behaviour, velocity checks, graph structure over shared attributes such as device or address. Then bootstrap labels — investigate the top anomalies with a human, use those confirmations as seed labels, and move to supervised learning as the labels accumulate. Say that precision at the top of the ranking is what matters, because investigator time is the constraint.
10. You have one week to give an answer that properly needs a month. What do you do?Deliver the decision-grade version: the crudest analysis that could change the decision, with its uncertainty stated plainly and the assumptions listed. Then say what you would do with the remaining three weeks and what could flip. Interviewers are testing whether you can be useful under a constraint without pretending the constraint did not exist.

No right answer exists. The score is whether you name the axis you are trading along and commit to a position.

1. Faster and slightly wrong, or slower and correct?Depends entirely on the cost of being wrong, which is the thing to say first. For a recommendation, fast and approximate. For a payment or a clinical decision, correct. The useful follow-up is that you can often have both by tiering — fast path for the common case, slow path when the fast path is uncertain.
2. When would you accept a less accurate model?When it is explainable and the domain requires it, when it fits the latency or hardware budget and the better one does not, when it is cheaper to retrain against drift, or when it fails in a more recoverable way. Accuracy is one axis among several, and treating it as the only one is what the question is checking for.
3. Buy or build?Buy unless it is your differentiator. The parts worth building are the ones your product is uniquely good at; everything else is undifferentiated work you will then own forever. The counterweight is lock-in and per-unit cost at scale, so ask what it costs at ten times the current volume.
4. How do you decide between fixing technical debt and shipping features?Make the debt concrete: which debt, costing what, how often. Debt that slows every change is worth fixing now; debt in a stable corner nobody touches can wait indefinitely. Fixing debt because it is unpleasant rather than because it is expensive is the trap.
5. The simplest solution does not scale. Do you build for today or for the projected load?Build for roughly ten times current load, not a hundred. Ten times is usually cheap and buys real runway; a hundred times means designing against a projection you do not have evidence for, and the requirements will have changed before you get there. What matters is keeping the interface stable so the internals can be replaced.
6. Would you rather ship late or ship with a known limitation?Ship with a known, documented, contained limitation — provided the people affected know about it and it is not a safety or correctness issue. The word that matters is known: a limitation you have chosen and communicated is a decision, and the same limitation undisclosed is a defect.
7. How much should you invest in monitoring before launch?Enough to answer "is it working" and "is it getting worse" on day one. That is a small, fixed cost. The detailed dashboards can wait for the first real incident, which will tell you what you actually needed to see — building them in advance usually produces panels nobody looks at.
8. You disagree with the metric your team is optimising. What do you do?Show the failure case rather than arguing in the abstract: find the change that would improve the metric and harm users, and put it in front of people. Then propose a guardrail alongside the existing metric rather than trying to replace it, because adding a constraint is a much easier sell than relitigating the goal.

Less common than they used to be, but still asked — usually to see whether you reason aloud rather than freeze.

1. You have two ropes that each burn for an hour but not uniformly. Measure 45 minutes.Light rope A at both ends and rope B at one end simultaneously. A is consumed in 30 minutes regardless of how unevenly it burns, because the two flames together always consume the whole rope in half the time. At that moment light B's other end; B has 30 minutes of rope left burning from both ends, so it takes 15. Total 45. The generalisable idea worth naming: burning from both ends halves the remaining time whatever the density.
2. How would you test whether a coin is fair?Flip it many times and test the observed proportion against 0.5 — a binomial test, or a normal approximation for large n. Decide the sample size in advance from the deviation you want to detect, because stopping when the result looks interesting is how you manufacture significance. And say that you cannot prove fairness, only fail to detect unfairness at some power.
3. A test is 99% accurate for a disease affecting 1 in 10,000. You test positive. What is the chance you have it?Under 1%. In a million people, 100 have it and ~99 test positive; the 999,900 healthy people produce ~9,999 false positives at a 1% false positive rate. So roughly 99 out of 10,098, just under 1%. The base rate dominates, and this is the intuition behind why a rare-event classifier with excellent accuracy can still produce mostly false alarms.
4. You are one of 100 people in a queue and want to know your chance of a particular outcome. How do you approach it?Solve the small case first — 2 people, then 3 — and look for the recurrence. Most queue and sequence puzzles have a pattern that is invisible at n=100 and obvious at n=3. Say you are doing this rather than trying to intuit the general answer.
5. How many trials do you need to be 95% confident an event with probability p occurs at least once?The probability of never seeing it in n trials is (1−p)ⁿ, so solve (1−p)ⁿ ≤ 0.05, giving n ≥ log(0.05)/log(1−p). For p = 0.01 that is about 300. The practical version worth adding: this is why a bug that reproduces 1% of the time needs hundreds of runs before absence of failure means anything.
6. Given a biased coin, how do you generate a fair bit?Von Neumann's trick: flip twice. HT means 0, TH means 1, and HH or TT means discard and repeat. Both mixed outcomes have identical probability p(1−p) whatever the bias, so the output is fair. It wastes flips, which is the trade you are making for not needing to know p.
7. You can ask one yes-or-no question to halve a search space. What makes a good question?One whose answer is closest to equally likely either way, because that maximises the information gained. This is the same principle behind binary search and behind choosing which check to run first when debugging — pick what eliminates the most, not what confirms your favourite hypothesis.
8. How would you estimate something with genuinely no data available?Bound it. Find a lower bound and an upper bound you are confident about, then narrow with any proxy or analogous case. An answer of "between 10,000 and 100,000, and here is why it cannot be outside that" is far more useful than a single invented number, and it is honest about what you know.

Where most ML interviews are actually decided, before any modelling is discussed.

1. A product manager asks you to "add AI" to a feature. What do you ask?What decision changes as a result. If nothing downstream acts differently, the model has no value however accurate it is. Then: what does the output look like, who consumes it, what does being wrong cost in each direction, and is there historical data where the right answer is known. Those four questions turn a vague ask into a specifiable problem, or reveal that it is not one.
2. How do you turn a business problem into an ML problem?Name the decision, then the prediction that would improve it, then the target variable that stands in for that prediction, then the label source. The gap between the thing you care about and the thing you can label is where projects fail — you care about fraud, you can label chargebacks, and those differ in ways that matter.
3. When is the right answer "do not use ML"?When the rule is known and stable, when you have no labels and no path to them, when the cost of a wrong answer exceeds the value of a right one, when you cannot get the features at prediction time, or when a lookup table would do. Saying this unprompted is a seniority signal, because it shows you are solving the problem rather than deploying your skill set.
4. What is the difference between the metric you optimise and the metric you care about?The loss function is what the model minimises; the business metric is what success means; they are almost never the same object. Cross-entropy is not revenue. The job is choosing a proxy close enough that improving it improves the real thing, and monitoring for the divergence — because optimisation pressure finds the gap.
5. How would you decide what the prediction horizon should be?By working backwards from the intervention. If the action takes two days to have an effect, predicting one day ahead is useless however accurate. Shorter horizons are easier and often worthless; the right horizon is the shortest one that still leaves time to act.
6. Where do your labels come from, and why does that question matter so much?Because label provenance determines everything downstream. Human-annotated labels carry annotator bias and disagreement; implicit labels from user behaviour carry selection bias; labels from a downstream system carry that system's errors. And any label delayed relative to prediction creates a gap you must handle in training.
7. How do you handle a problem where the label only exists for cases you acted on?This is selection bias from the existing policy, and it is very common — you only know whether a declined transaction was fraudulent for the ones you approved. The clean fix is a small randomised holdout that gets approved regardless, giving unbiased data. Failing that, propensity weighting, while being honest that it corrects only for what you have measured.
8. What would make you say a project is not worth pursuing?No path to labels, no baseline worth beating, no consumer for the output, or a required accuracy that the available signal cannot support. Saying so early is far cheaper than proving it after six months, and the willingness to say it is part of what is being assessed.

Raising leakage before being asked is one of the strongest signals available in this round.

1. What is data leakage and how does it show up?Information in the training features that would not be available at prediction time. It shows up as a validation score that is too good, a single feature with implausible importance, and a model that collapses in production. Classic sources: a field populated after the outcome, an id encoding the target, and preprocessing fitted before the split.
2. How do you prevent it structurally rather than by inspection?Point-in-time correctness — build each training row from only what was known at that timestamp — plus fitting all preprocessing inside the training fold. A feature store with as-of joins does this for you. Inspection catches leakage you thought to look for; structure catches the rest.
3. Your model has one feature with 90% of the importance. What do you do?Suspect leakage before celebrating. Check when that field is populated relative to the prediction point, and whether it is derived from the outcome. If it is legitimate, ask whether it will still be available and still mean the same thing in production — a dominant feature is a single point of failure regardless.
4. How do you split data with repeated entities?Group by the entity so all rows for one user or patient stay in the same fold. Otherwise the model memorises individuals and validation measures recall of the training set. Combine with a time-based split when the data is also temporal — grouping alone does not stop you training on the future.
5. When is a random train-test split wrong?Whenever the data is temporal, grouped, or spatially correlated. For anything that will be deployed forward in time, split by time, because that is the only split that reflects how the model will actually be used. Random splits on time-series data are the most common way a good-looking result turns out to be meaningless.
6. How do you handle severe class imbalance?First stop using accuracy and switch to PR AUC or recall at a fixed precision. Then class weighting rather than resampling as the default, since weighting keeps the data distribution intact. Undersampling the majority is acceptable if you correct the calibration afterwards; SMOTE is popular and frequently makes things worse on high-dimensional data. Say that the threshold, not the model, is where the imbalance is finally handled.
7. How do you know if your labels are any good?Measure inter-annotator agreement on a sample. If two humans disagree 20% of the time, that is your ceiling and no model will exceed it. Also look at label distribution over time for drift in the annotation guidelines, which is quiet and common.
8. You have 500 labelled examples and 5 million unlabelled. What do you do?Use a pretrained model and fine-tune, which is what makes 500 sufficient. Then active learning: score the unlabelled pool, label where the model is least certain or where the disagreement between models is highest, and iterate. Blindly labelling more at random is the least efficient use of the annotation budget.

Choose a metric and defend it against the cost structure. Accuracy is almost never the answer.

1. Why is accuracy usually the wrong metric?Because it weights both error types equally and is dominated by the majority class. With 1% positives, predicting everything negative scores 99%. The right metric follows from the asymmetry between a false positive and a false negative in that specific application.
2. Precision or recall — how do you choose?By what a mistake costs. Recall matters when a miss is expensive and a false alarm is cheap: disease screening, fraud triage. Precision matters when acting on a wrong positive is costly or annoying: auto-blocking, notifications. Usually you fix one at an acceptable level and optimise the other, rather than balancing them with F1 for its own sake.
3. What does ROC AUC actually measure, and when is PR AUC better?ROC AUC is the probability the model ranks a random positive above a random negative. It uses the false positive rate, which has a huge denominator when negatives dominate — so it stays flatteringly high on imbalanced data. PR AUC uses precision, which reacts to the false positives that actually reach a person, and is the more honest choice when positives are rare.
4. What is calibration and when do you need it?A calibrated model's predicted probabilities match observed frequencies — of everything scored 0.7, about 70% are positive. You need it whenever the number is consumed as a probability rather than a ranking: expected-value decisions, thresholds set by cost, or a human reading the score. Check with a reliability diagram, fix with Platt scaling or isotonic regression on held-out data.
5. How do you choose the operating threshold?Not from the model. From capacity and cost: how many alerts can be handled per day, or what expected value each decision carries. The threshold is a product decision, and it should be revisited when volumes change even if the model does not.
6. Offline metrics improved but the online metric did not. What happened?Common causes, in rough order: distribution mismatch between the offline set and live traffic; the offline metric not being the thing users respond to; a feedback loop where the old model shaped the data you evaluated on; latency added by the new model outweighing its accuracy; and a bug in serving that means the deployed model is not the one you evaluated. Check the last one first — it is more common than people expect.
7. How would you evaluate a model with no ground truth?Proxies and humans. Agreement between independent models, consistency under perturbation, and human review of a stratified sample to build a small trusted evaluation set. For generative output, pairwise human preference or a well-validated model-as-judge — validated meaning you have checked it agrees with humans on the cases you do have.
8. What is a guardrail metric?A secondary metric watched during a change to catch harm the primary metric would hide. Optimise click-through and you can wreck long-term retention; optimise fraud recall and you can double customer friction. The guardrail is what makes it safe to optimise the primary aggressively.

Know the failure modes of whatever you name. Proposing a method you cannot critique is worse than proposing a simpler one.

1. Why start with a baseline, and what should it be?Because a number means nothing without something to compare it to, and a surprising number of problems are solved by the baseline. Use the simplest thing that could work: majority class, a business rule already in use, logistic regression, or last-value-carried-forward for time series. It also surfaces data problems fast, before you have spent a week on architecture.
2. When would you use gradient boosting over a neural network?Tabular data — which is most business data. Boosting handles mixed types and missing values natively, needs far less data and tuning, trains in minutes, and routinely wins. Reach for neural networks when the input is unstructured — text, images, audio — or when you need representation learning that transfers.
3. How do you decide between fine-tuning and prompting a pretrained model?Start with prompting, because it is free to iterate and often sufficient. Fine-tune when you need a consistent output format the prompt cannot enforce, when you have domain behaviour the base model lacks, when latency or cost demands a smaller model, or when the prompt has grown so long it is expensive. With few examples, retrieval plus prompting usually beats fine-tuning.
4. What is the bias-variance trade-off, in practice rather than in theory?High bias is underfitting — the model is too simple and training error is already high; adding data will not help, so add capacity or better features. High variance is overfitting — training error is low and validation error much higher; more data, regularisation, or less capacity helps. The practical value is that it tells you which lever to pull, and the diagnostic is comparing the two errors.
5. How do you handle high-cardinality categorical features?One-hot explodes, so: target encoding computed inside the fold to avoid leakage, hashing when the vocabulary is unbounded, learned embeddings when there is enough data, or grouping the long tail into "other". CatBoost handles it natively with ordered statistics. The leakage risk in target encoding is the part interviewers listen for.
6. Your model overfits. What do you try, in what order?More data first if it is obtainable, because it is the only fix with no downside. Then stronger regularisation and early stopping, then reducing capacity, then augmentation if the domain allows. Check for leakage-in-reverse too — a validation set that is too similar to training makes a model look fine until deployment.
7. How do you approach hyperparameter tuning?Random or Bayesian search rather than grid, because grid wastes most of its budget on parameters that do not matter. Tune the few that do — learning rate above all, then capacity and regularisation. Use a validation set separate from test, and stop when the gain is smaller than the noise between runs, which is sooner than most people stop.
8. How would you make a model explainable if the domain requires it?Prefer an inherently interpretable model when the accuracy cost is small — a regularised linear model or a shallow tree is often within a point or two, and the argument is easier. Otherwise SHAP for per-prediction attribution, with the caveat that an explanation of a model is not an explanation of the world: SHAP tells you what the model used, not what causes the outcome.

"No baseline, no error analysis, no idea what the model gets wrong" is the fastest way to fail this round.

1. How do you actually do error analysis?Take a sample of errors and read them, one at a time, categorising as you go. A hundred examples usually reveals three or four recurring categories, and those categories tell you what to fix. Skipping this and tuning hyperparameters instead is the most common waste of time in applied ML.
2. Your model is 85% accurate. What do you do next?Find out what the 15% is made of before doing anything. If most errors come from one segment or one input type, that is a targeted fix worth more than any model change. Also establish the ceiling: if humans disagree 10% of the time, 85% may already be close to the limit.
3. How do you know whether to invest in more data, better features, or a better model?The learning curve tells you. If validation error is still falling as you add data, more data helps. If it has plateaued well above the achievable error, more data will not — the problem is capacity or features. Compare training and validation error to distinguish the two.
4. What is a slice-based evaluation and why does it matter?Reporting performance separately for meaningful subgroups — device, region, language, customer size, demographic — rather than one aggregate. Aggregates hide a segment where the model is useless, and that segment is usually the one someone complains about. It is also how you find fairness problems before someone else does.
5. The model does well on average and badly for new users. How do you approach that?It is a cold-start problem: the features that carry the signal do not exist yet for them. Options are a separate model using only the features available at signup, sensible defaults from population priors, or content-based rather than behavioural features until history accumulates. Say which segment you are optimising for, because the two models will disagree.
6. How do you decide when a model is good enough to ship?Against the decision it supports, not a round number. It is good enough when it beats what is there today by a margin that survives the confidence interval, when its failure mode is acceptable to whoever is affected, and when you can detect degradation after launch. Shipping something mediocre with monitoring beats waiting for something excellent without it.
7. What do you do when the model is confidently wrong?Confidence and correctness coming apart is a calibration problem and often a distribution problem — the input is unlike anything in training. The mitigations are calibration, an out-of-distribution check, and an abstention path where the system says "I don't know" rather than guessing. Deciding what happens on abstention is a product question, not a modelling one.
8. How would you set up a feedback loop from production back into training?Log inputs, predictions, and eventual outcomes, joined by an id. Then guard against the loop poisoning itself: if you only observe outcomes for cases you acted on, retraining on that data compounds the existing policy's bias. A small random holdout that bypasses the model is what keeps the training data honest.

Training-only answers read as academic. Carry every answer through to serving.

1. What is the difference between data drift and concept drift?Data drift is the input distribution changing — new user mix, new device. Concept drift is the relationship between inputs and outcome changing, so a once-accurate model becomes wrong even on familiar inputs. Data drift you can detect without labels; concept drift you generally cannot, which is why it is the more dangerous one.
2. How would you monitor a model in production?Four layers: the service is up and fast; the inputs look like training data; the prediction distribution is stable; and, where labels arrive, performance by slice. Alert on the input and prediction layers, because they are available immediately, while true performance may be days or weeks behind.
3. How often should you retrain?On a trigger, not a calendar, wherever possible — retrain when drift or performance monitoring says to. A schedule is a reasonable fallback, set from how fast the domain moves. Then say the harder part: every retrain needs automated validation against the incumbent and a rollback path, or automated retraining is just an automated way to ship a worse model.
4. What is training-serving skew and how do you avoid it?The features computed at training time differing from those computed at serving time — different code, different library version, different aggregation window. Avoid it by sharing one transformation path, or by a feature store that serves both. Detect it by logging serving features and comparing their distribution against training.
5. How would you evaluate a RAG system?Evaluate retrieval and generation separately, because they fail differently. Retrieval: recall of the relevant document in the top-k — if it is not retrieved, no prompt will save you. Generation: faithfulness to the retrieved context, and whether it abstains when the context does not contain the answer. Most RAG quality problems are retrieval problems being blamed on the model.
6. How do you reduce hallucination in an LLM feature?Ground it in retrieved context and require citations that can be checked against the source. Constrain the output format so unverifiable claims have nowhere to go. Give it an explicit abstention option and make abstention acceptable in the product. Then verify — a second pass checking each claim against the context catches a lot. The one thing that does not work is instructing it not to hallucinate.
7. What does A/B testing an LLM feature look like, given the output is not a number?Randomise at user level and measure the downstream behaviour you care about — task completion, edit rate, escalation rate, retention — rather than output quality directly. Alongside that, run an offline judge on a fixed prompt set for regression detection. And log everything, because with generative output the qualitative review of the tail is where you find the problems.
8. How would you optimise inference cost and latency?In order of return: use a smaller model where it suffices and route only hard cases to the large one; cache repeated prefixes and repeated requests; quantise; batch for throughput while watching the latency cost; and shorten the prompt, which is the one people forget despite it being pure saving. Measure p95 end to end, after warm-up, including pre- and post-processing.

Pulls from: consolidating multiple ML repositories into one inference pipeline at Infiswift; the RFI document validation agent.

1. Tell me about a time you were handed something poorly defined and had to decide what it actually was.At Infiswift I was asked to "reduce duplication" across our document extraction work. That was the entire brief. When I went and read the code, the real problem turned out to be a different one: several repositories had each grown their own LLM extraction path — its own prompts, its own parsing, its own error handling. Nobody owned the interface, so when a business rule changed someone had to find and update every copy, and they usually missed one. So I reframed it. Not "there is duplicate code" but "there is no single place where a document becomes structured data." I proposed a modular extraction API to be that place. The reframing is what made the work worth doing — deduplicating the code alone would have left us in the same position a few months later.
2. You consolidated several ML repositories into one pipeline. Who decided that needed doing — you or someone else?Both, and I would rather be precise about that than overclaim. The maintainability pain was felt across the engineering teams; people knew it was bad. What I brought was the specific diagnosis and the proposal. I had been working across more than one of those repositories and hit the same thing twice — the same document type classified differently depending on which path happened to process it. I wrote up what I had found with real examples showing the divergence, and proposed consolidating into a single inference pipeline. So the frustration was shared; the framing and the plan were mine.
3. How did you work out what the duplicate LLM workflows actually had in common?I stopped reading the code and started comparing inputs and outputs. At the code level the implementations looked unrelated — different prompt structures, different post-processing, different libraries. But the contract was identical: a document goes in, structured fields and a file type classification come out. Once I lined the actual inputs and outputs up side by side, the shared shape was obvious and all the variation was in the middle. I then checked whether that variation was deliberate by asking the people who wrote them, and in a good number of cases nobody could remember a reason — it was an accident of when that one had been written, not a decision.
4. What did you have to leave alone, and how did you decide?I left alone anything with a genuinely different output contract. One path produced a different downstream artefact, and folding it in would have meant either compromising the shared interface or building a special case into it on day one — neither of which is worth it for a single caller. I also left alone consumers I could not migrate on my own timeline, where the owning team had a freeze or a dependency I did not control. My rule was blast radius rather than elegance: if pulling something in meant I might break what I could not test, it stayed out. Consolidating a few things properly beats consolidating more of them halfway.
5. Whose code were you replacing, and how did that conversation go?It was [the teams], and the conversations went well — mostly because of how I opened them. I did not go in saying their code was duplicated. I went in with the migration already built: a compatibility layer so their existing calls kept working, and their own tests passing against the new pipeline before I asked them for anything. At that point the ask was just "switch when it suits you." The one real point of friction was not about code at all, it was ownership — someone reasonably wanted to know who would be on the hook when a shared pipeline broke at 2am. We settled it by agreeing I owned the pipeline and each team owned its adapter.
6. How did you know when the consolidation was finished rather than merely working?Working is not finished. The new pipeline worked within a few weeks, but at that point we had one more extraction path than we started with — I had added to the problem, not removed it. I treated it as finished when the old paths were deleted and there were no callers left. That is a stricter bar and it took considerably longer than the build did, because the last few consumers are always the awkward ones. I tracked it as a plain list of remaining call sites and worked it down. The day I deleted the last old module was the day I called it done, and I would defend that definition — a consolidation that leaves both versions running has added a system rather than removed one.
7. What would you do differently if you started that consolidation again tomorrow?I would migrate one consumer end to end before designing the shared interface. What I did was read all the existing implementations and design something that covered all of them, which sounds sensible and produced an interface that was slightly wrong for everybody. It carried options nobody used, and it was missing something the very first real migration needed immediately. If I had taken the messiest consumer, moved it properly, and only then generalised, I would have got there faster with a smaller surface area. I have applied that since: build for one real caller, then widen.
8. Describe a time you pushed back on a request because the underlying problem was different from the one stated.The document validation work. The request was to improve classification accuracy — the system was getting file types wrong and downstream processing was failing because of it. Before touching the model I spent time going through the actual failures, and most of them were not classification errors in the way people assumed. Files were being categorised on upload, by filename or by whatever the uploader selected, and the content frequently did not match the label. The model was faithfully classifying documents that had already been mislabelled upstream. So I pushed back on "make the model better" and proposed content-aware validation at ingest instead — an agent using Gemini through LangChain and LangGraph that reads what the document actually is rather than trusting what it claims to be. That removed a whole class of downstream failure that no amount of prompt tuning would have touched.

Pulls from: the self-learning regex framework; choosing LightGBM and Isolation Forest for fraud; ONNX and TensorRT for edge.

1. Walk me through a technical decision you made that you knew would be contested.Promoting stable LLM outputs into deterministic regex rules. It looks like a step backwards — replacing a capable model with pattern matching — so the case has to be made on cost, latency, and reproducibility rather than on capability.
2. You built a framework that generates regex rules from LLM output. Why not simply keep calling the model?Three reasons, in order: a rule is free and instant where a model call costs money and hundreds of milliseconds; a rule gives the same answer every time, which matters for a classification that downstream systems depend on; and a rule can be reviewed and version-controlled. The model stays in the loop for anything the rules do not cover.
3. How did you convince anyone that a generated rule was safe to promote to production?By making promotion a gated pipeline rather than a decision — the rule had to be generated, validated, deduplicated, and versioned before it moved through Development, Preview, and Production. [the validation threshold you actually used]. Safety came from the process, not from trusting the generator.
4. Fraud detection with LightGBM rather than a deep model — how did you argue that, and to whom?Tabular transaction data with strong categorical features is where gradient boosting wins, and it wins with less tuning and less data. Add that it trains in minutes, which matters when the fraud pattern shifts and you need to retrain. Isolation Forest covered the unsupervised side for patterns with no labels yet.
5. Who disagreed with you on a model choice, and what happened?[the actual disagreement and how it resolved]. The structure that lands: what they argued, what you agreed with in their argument, what evidence settled it, and — if they were right — say so.
6. Tell me about a decision you made under time pressure that you were not fully confident in.[the decision]. Say what you did to limit the downside rather than claiming you were secretly confident: the smaller reversible version, the flag you could turn off, the thing you agreed to revisit.
7. When did you choose the boring option over the interesting one, and was it right?The regex promotion is exactly this, and so is LightGBM over a transformer for tabular fraud. Both were the duller choice and both were right. The honest framing is that the interesting option is usually the one that costs someone else maintenance later.
8. What is a technical decision from your past that you now think was wrong?[a real one]. Pick something with a consequence you can describe, explain what you were optimising for at the time, and say what you would need to have known. Avoid decisions that turned out fine anyway.

Pulls from: working with Risk and Compliance at JPMorgan; clinician adoption at Cognizant.

1. Tell me about a time you disagreed with someone more senior than you.[the disagreement]. Show that you separated the disagreement from the person, made the case once with evidence, and then either changed their mind or committed to their decision properly. Interviewers are listening for whether you can lose gracefully.
2. Compliance blocked or slowed something you had built — what happened next?Frame compliance as a requirement you had not gathered yet rather than an obstacle. On the GenAI work that meant bias monitoring, output validation, and prompt injection safeguards became part of the design — which is what made deployment possible at all.
3. How do you explain a model's limitation to someone whose job is to distrust it?In their terms and with numbers. For a risk function that means false positive rate at the operating threshold, what the model cannot see, and what happens when it is wrong. Volunteering the limitation before being asked buys more credibility than defending the model does.
4. Describe a time a stakeholder wanted something you thought was a bad idea.[the request]. The strong version: you built the smallest thing that tested their assumption, and let the result settle it rather than the argument.
5. A clinician told you the model was wrong about a patient. What did you do?Treat it as signal, not user error — the clinician usually has context the model does not. Go and look at that specific case, and often it reveals a data problem rather than a model problem. [what you found when this happened].
6. How do you handle it when a team refuses to adopt something you shipped?Find out why before pushing. Non-adoption is usually cost of switching, a missing feature in their workflow, or lack of trust in the output. Each has a different fix, and only the third is about the model.
7. Tell me about a working relationship that started badly and improved.[the relationship and the turning point]. What lands is a specific change you made in how you worked with them, not a change in how you felt about them.
8. When have you had to say no to a request from a business partner?[the request]. Say no with an alternative and a reason they care about — timeline, risk, or something else it would displace. A no with no alternative reads as unwillingness rather than judgement.

Pulls from: false positives in fraud; misclassified documents; drift monitoring at Cognizant.

1. Tell me about something you shipped that did not work.[the failure]. Structure: what you expected, what happened, how you found out, what you did in the hour after, and the process change that came from it. The last part is what is actually being assessed.
2. What is the worst production incident you have been responsible for?[the incident]. Own your part explicitly and early in the answer. Describing the detection and recovery in detail is more impressive than the incident being small.
3. A misclassified document caused a downstream failure. Walk me through the day it happened.Tell it chronologically and concretely — how it surfaced, what you checked first, what the actual cause turned out to be, what you shipped that day versus that week. [the specifics]. Chronology is what makes it sound real rather than rehearsed.
4. When did a model of yours degrade in production before anyone noticed?This is why the drift and data quality monitoring at Cognizant existed. The honest version names how long it went unnoticed and what monitoring you added afterwards — undetected degradation is normal, having no way to detect it is the failure.
5. What did you change about how you work as a result of a specific failure?[the change]. The test frameworks built with PyTest and Moto are a good anchor if the answer is about validating cloud integrations before they reach production rather than after.
6. Tell me about a time you missed a deadline. What did you tell the people waiting?[the occasion]. The question is really about when and how you communicated it. Raising it early with a revised estimate is the good answer; discovering it on the day is the bad one.
7. Describe an occasion where your testing did not catch something it should have.Usually the gap is between unit tests and reality — mocked cloud services that behave unlike the real ones, or a case absent from the fixtures. [what slipped through]. Then say what class of test you added, not just the one case.
8. What is a mistake you have made more than once?Answer it honestly; the question exists to see whether you will. A common and credible one is under-scoping migration work because the new system was finished and the old one was still running. Say what you now do to catch yourself.

Pulls from: cross-team consolidation; SHAP dashboards raising physician adoption; responsible AI controls with three teams.

1. Tell me about a time you got people to change how they worked without being their manager.The consolidation again, from the other side. Adoption came from making the new path cheaper to use than the old one, not from a mandate. [what you did to lower the switching cost].
2. Physician adoption increased after you built the explainability dashboards. What actually drove that — the dashboards, or something else?Be careful here — the dashboards helped, but adoption usually turns on the risk factors being clinically plausible, on the score arriving inside an existing workflow, and on a respected clinician using it first. Naming the parts you did not control reads as more credible, not less.
3. How did you get Risk, Compliance, and Data Science to agree on one set of controls?By writing down what each team actually needed and finding the controls that satisfied more than one — bias monitoring and output validation served both governance and quality. [how the agreement was reached]. Agreement usually comes from a document, not a meeting.
4. Describe a time you had to bring a sceptical team along with you.[the team and the concern]. Scepticism is usually specific; find the specific objection and address that rather than making a general case.
5. What did you do when a team ignored a standard you had introduced?Check first whether the standard was worth following. If it was, make it the default — in a template, in CI, in the scaffold — rather than in a document. Standards that require remembering do not hold.
6. Tell me about mentoring someone, and what you learned from it.[who and what]. The detail that makes this answer good is what you learned about your own explanations from watching where they got stuck.
7. When have you deliberately given away credit?[the occasion]. Keep it brief and specific. This question is answered as much by how you talk about colleagues throughout the whole interview as by the story.
8. How do you make a case for work that is invisible — testing, monitoring, refactoring?Attach it to a cost that has already been paid. Testing frameworks are easy to justify after a regression; monitoring is easy after an incident. Where there is no incident yet, quantify the risk in time rather than in principle.

Pulls from: LLM evaluation and A/B testing across production projects; the readmission A/B test.

1. Tell me about a time an experiment told you that you were wrong.[the experiment]. The LLM evaluation and A/B testing work is the natural source. What makes the answer strong is that you had committed to a view publicly beforehand.
2. You ran A/B tests on LLM behaviour. What was the most surprising result?[the actual result]. A genuinely common finding worth checking against your own: prompt changes that improved average quality also increased variance, and the variance mattered more downstream than the average did.
3. When did evaluation change your mind about an approach you had already built?Say what the evaluation measured and why the result was not what you expected. Analysing model agreement and reasoning quality — rather than only final accuracy — is what tends to reveal that a model was right for the wrong reason.
4. Describe a strongly held technical opinion you have since abandoned.[the opinion]. Give the evidence that moved you, not just the fact that you moved. Opinions abandoned without a reason read as having had no basis.
5. What is something you believed about ML two years ago that you no longer believe?Something defensible and specific — for instance, that a capable enough model removes the need for deterministic rules. The regex promotion framework exists because that turned out to be false on cost, latency, and reproducibility grounds.
6. Tell me about a time you were convinced by a junior colleague.[the occasion]. Say what they saw that you had stopped seeing. This question quietly tests whether you listen downwards.
7. How do you tell the difference between being persuaded and being worn down?Persuasion changes what you would predict; attrition only changes what you will argue about. A useful test is whether you can now make their case better than they did — if you cannot, you have conceded rather than been convinced.

Pulls from: sub-50 ms feature serving; low-latency edge inference for robotic arm control; HIPAA-compliant retraining.

1. Tell me about a project where the constraint, not the model, was the hard part.The wake-word detection work. An MLP on Mel-spectrogram features is not the interesting part; making it run at low latency on edge hardware for robotic arm control, via ONNX and TensorRT, is where the difficulty sat.
2. Sub-50 ms feature serving — what did you have to give up to hit that?Freshness and flexibility. Anything requiring a large aggregation window has to be precomputed and served from the online store, so the feature set is constrained to what can be maintained in advance. [the specific trade-off you made].
3. How did the edge deployment change what you were willing to build?It moved the decision from accuracy to what survives quantisation and export. An architecture that gains a point of accuracy but has an operation the runtime does not support is worse than a simpler one, because unsupported operations fall back and destroy the latency budget.
4. Describe working under a compliance constraint that shaped the design.HIPAA on the clinical work shaped where data could live, what could be logged, and how retraining had to be automated so patient data never moved through a manual step. The constraint changed the architecture, not just the paperwork.
5. When did you have to ship something you knew was not the best version?[the occasion]. Say what you protected — the interface, the ability to swap the model later, the monitoring — so the shortcut was contained rather than structural.
6. Tell me about a time you cut scope. Who did you tell, and how?[what was cut]. Name who you told and when. Cutting scope silently is the failure mode this question is looking for.
7. What is the tightest deadline you have worked to, and what broke because of it?Answer the second half honestly — something always breaks, usually tests, documentation, or a migration left half-done. [what it was in your case] and whether you went back for it.

Pulls from: building test frameworks with PyTest and Moto; the multi-agent side projects.

1. Tell me about something you built that nobody asked you to build.The testing and evaluation frameworks are the strongest example, and the multi-agent side projects the second. Say what problem you kept hitting that made you build it — unrequested work is only impressive if it was solving something real.
2. You wrote testing frameworks for AI workflows — was that assigned, or did you decide it was needed?[assigned or not]. If it grew out of a regression you had already been bitten by, say that; it explains the motivation better than any claim about quality culture.
3. What did you fix that was not your responsibility?[the fix]. Keep it proportionate — the good version is a small fix with an outsized effect, not a rewrite of someone else's system.
4. Describe a time you inherited something badly built and had to live with it.Say what you stabilised first and what you left alone. Anyone can list the problems; the signal is in choosing which one actually mattered and resisting the rewrite.
5. Your side projects use ADK and the Claude Agent SDK. What made you start them?[the actual motivation]. The credible answer is usually wanting to understand a failure mode properly — evaluation, hallucination rates, and workflow reliability are the parts you cannot learn from a demo.
6. What did the side projects teach you that your job did not?Owning every layer, including the parts a team would normally hide from you — the evaluation harness, the CI, the security scanning. Building the trace-driven evaluation yourself teaches you what the numbers actually mean.
7. Tell me about a time you kept going on something after the interest had worn off.[the project]. The finishing is the point. Say what the last 10% consisted of, since that is the part people abandon.

Every quantified claim on a CV is an invitation. These are asked in a behavioural tone but are really testing whether you understand your own results — and honesty about attribution scores far better than a confident overclaim.

1. Fraud losses fell 22%. How much of that was your model, and how much was everything else the bank changed that year?The honest answer is that you measured the system, not the model in isolation — the model changed, and so did rules, thresholds, and analyst capacity. Say what you would need to isolate it properly: a holdout population scored by the old system over the same period. Interviewers rate this answer highly precisely because most candidates claim the whole 22%.
2. How was that 22% measured, and over what period?[the measurement window and comparison basis]. Be ready to say whether it was year on year, before-and-after deployment, or against a control — and to name the confound that comparison does not remove.
3. Investigation time dropped 40%. Who measured it, and against what baseline?[who measured it and the baseline]. If it came from case handling times before and after the assistant launched, say so, and note that analysts also got faster with practice over the same period.
4. Detection accuracy up 15% with false positives below 2% — what was the trade-off you had to argue for?That pair is the trade-off. Every point of recall costs precision, and in fraud a false positive is a declined transaction for a real customer. The 2% was a business constraint that fixed the operating threshold; accuracy improvements had to be found inside it.
5. What does "5M+ daily transactions" mean for how the system was actually built?Roughly 60 per second average with peaks well above it, so scoring has to be streaming rather than batch, features must be precomputed and served from an online store, and every component needs to degrade rather than queue. The volume dictates the architecture more than the model does.
6. Sub-50 ms — is that median or p99, and does the difference matter here?Say which, and say why it matters: a median under 50 ms with a long tail still fails the transactions that matter most, because slow requests correlate with unusual ones. [which you measured]. If it was median, say so — being caught rounding a percentile is worse than the weaker number.
7. 3× inference throughput after quantisation — what accuracy did that cost?Post-training quantisation to int8 typically costs a small amount of accuracy, and the number only means something measured on your own held-out set rather than a published benchmark. [the accuracy delta you measured]. If you did not measure it, say that too.
8. 0.87 AUC-ROC on readmission. Explain what that number means to a non-technical stakeholder.Take one patient who was readmitted and one who was not, at random: the model gives the readmitted one a higher risk score 87 times out of 100. It is a ranking quality, not an accuracy — the model is not right 87% of the time, and saying it that way is the most common mistake.
9. Feature generation time fell 60%. What was slow before, and what actually changed?[the actual bottleneck]. Typical causes are wide shuffles on skewed keys, recomputing history every run instead of incrementally, and row-by-row work that could be vectorised. Naming which one it was is what separates a real answer from a rehearsed one.
10. Structured clinical coverage improved 35%. How did you verify that the extracted data was correct, not just present?Coverage measures presence, not correctness, so the two have to be checked separately — a sample manually reviewed against the notes, and negation and family-history handling tested explicitly, since those produce confidently wrong extractions rather than missing ones.
11. Readmissions fell 18% in the pilot. How do you separate the model's effect from the care team's?You cannot, fully. The model flags and the care team acts, so what was measured was the combined intervention. Say that plainly, then say what would isolate it: randomising at patient level with the same intervention available to both arms, which raises its own ethical question worth acknowledging.
12. Release cycles went from two weeks to three days. What was the real bottleneck?[the actual bottleneck]. It is rarely the training run — usually manual approval steps, environment drift between staging and production, or the absence of automated validation that made every release a judgement call.
13. Which of your numbers are you least confident in, and why?Answer it. Picking one and explaining the weakness in its measurement is a strong signal, and refusing to pick reads as either not having examined them or not being willing to say so.
14. Which of these results would you expect to hold up if the project were audited?Split them: the ones with a clean measurement — AUC on a held-out set, latency percentiles, throughput — hold up. The ones combining a model with a human process — the 22%, the 40%, the 18% — are directional and depend on how attribution was defined.
15. Pick one number here and tell me what would have made it better.[your choice]. A good instinct is to pick a measurement rather than a result: the number that would have been most improved by a cleaner baseline or a proper control group.