Interview Day

S — At mtl.ai I designed and deployed real-time computer-vision models (detection + segmentation) for live soccer broadcasts — virtual ad replacement that ran during UEFA Euro 2024 and the Champions League. Model validation had become the bottleneck: every new model sat in a slow validation pipeline before it could ship.

T — I took on finding out exactly why, because deployment cadence depended on it.

A — Instead of guessing, I profiled the pipeline end to end and identified the key bottlenecks — [name the top one or two precisely: what was serialized, redundant, or starved?] — then fixed them [what you actually changed] and measured before/after on the same workload.

R — 5× faster model validation → materially faster deployment cycles. [Ship cadence before vs after, if you have it.]

Expect James to probe

• "How did you profile it — what did the trace actually show?"
• "What was the single biggest win of the 5×?"
• "How did you verify the speedup didn't change validation results?"
• "What would the next 5× take?"

S — We were building a production assistant app for online marketplace sellers while the LLM market shifted monthly — OpenAI, Google, and Anthropic all changing models, capabilities, and prices.

T — I led development of the LLM integration layer. Goal: model choice should be a config decision, not an engineering project.

A — Designed a single abstraction unifying all three providers' APIs — normalizing [what was hard: prompting differences? streaming? function calling? error/retry semantics?] — so the app code never touched provider SDKs.

R — Model swaps went from rework to [config change / minutes?]; we optimized cost by routing tasks to the cheapest capable model [any % or $ saved]; the team built features on top without knowing which provider was underneath.

Expect James to probe

• "Where did the abstraction leak?"
• "How did you evaluate quality when swapping models?"
• "How did you handle capabilities one provider had and another didn't?"

S — Estimai lets construction estimators generate bids by ingesting architectural plans. Identifying structural elements — column icons — across multi-page plan sets was manual and error-prone.

T — Nobody owned it. I proposed a PoC and delivered it single-handedly.

A — Key insight: don't rasterize and throw a vision model at pixels — the PDF already contains vector primitives. I extracted them and built detection + cross-page association of column icons directly on the vector data: simpler, cheaper, more precise.

R — Working PoC delivered [timeframe]; [what happened next — adopted? shaped the product direction?].

Expect James to probe

• "Why vector primitives instead of a vision model? What did that trade away?"
• "How did you measure accuracy, and what were the failure modes?"
• "What would it take to production-harden?"

S/T — Math professor 2015–2022: calculus, linear algebra, statistics, probability to thousands of students across seven years.

A — Developed course materials that were adopted department-wide — [what problem drove that: where were students failing, and what did you change?]. Served on the Teachers' Union executive council and the Science Program Committee — [one concrete example of representing colleagues or holding a hard line].

R — The materials outlived my tenure; [pass rates, feedback, anything measurable].

S — My career is serial reinvention: professor → CV/ML engineer → AI full-stack → AI-safety researcher. Latest instance: essentially zero AWS experience when this process began.

A — For the online assessment I built my own interactive study tool — condensed lessons, flashcards, timed practice exams — and drilled daily. (Honest and current: "I build with AI tools daily; I built my study site with an AI coding agent.")

R — Passed the SA assessment about a week after starting from zero.

The #1 Amazon LP — near-certain to come up. The signal: you changed course because of what the user actually needed, at some cost to your own plan. Two candidates from your history:

• Teaching: a course/topic students kept failing → you went to where they actually were, rebuilt how you taught it [which topic? what did you change? what happened to outcomes?]
• Estimai: understanding how estimators really work before building — [did you watch real estimators / iterate the PoC on their feedback?]

Pick ONE, make it STAR, say the moment you realized your original plan was wrong. Use the "Gap-story workshop" voice prompt to build it.

The shape that scores: disagreed respectfully + argued with data + the decision went the other way + you committed genuinely anyway. The commit half matters as much as the backbone half. Candidates:

• Union executive council / Science Program Committee — a position you argued against the department or admin [what, and how it resolved]
• A technical/architecture disagreement at mtl.ai or Explorai [which decision?]
• A methodology disagreement with co-authors on the research paper [what did the data say?]

Amazon asks this in nearly every loop: "Tell me about a time you failed / missed a deadline / got it wrong." The scoring is about honesty and changed behavior, not the failure itself.

• Pick a real one — a project that shipped late, a technical bet that was wrong, a class that went badly. "I work too hard" is an anti-signal.
• Structure: what happened → my share of it, no blame-shifting → the concrete lesson → what I do differently now, with an example of the new behavior working.
• Keep it 60 seconds. Don't wallow, don't spin.

James's background (DevOps/SRE, ML doctorate) means resume claims get probed. For each project be ready for three "why/how" levels:

5× validation speedup

• L1: what the pipeline did, why it was slow overall
• L2: how you found the bottleneck (profiling method, what the data showed)
• L3: the mechanics of the fix, how you proved correctness didn't regress

LLM abstraction layer

• L1: why unify providers (churn, pricing, lock-in)
• L2: interface design — what got normalized, where the abstraction leaked
• L3: evaluation across model swaps; cost-routing logic

Floor-plan PoC

• L1: business problem (estimators, bids, plan ingestion)
• L2: vector primitives vs. raster vision — the tradeoff you took
• L3: cross-page association logic, accuracy measurement, failure modes

Rule at every level: the moment you hit your edge, say so — "beyond that I'd have to check." That answer scores higher with a Principal than a fluent guess.

"I'd start at the user and work in. Route 53 for DNS, pointing at CloudFront so static assets — the front end, images — come from edge locations, served out of S3. Dynamic requests hit an Application Load Balancer spreading traffic across an EC2 Auto Scaling group that spans multiple Availability Zones — that's both my scaling and my fault tolerance. If the workload is spiky or event-driven I'd consider Lambda behind API Gateway instead — pay per request, scales to zero. Data tier: RDS Multi-AZ for relational with automatic failover, or DynamoDB if access patterns are key-value at scale. In front of the database, ElastiCache for hot reads. Anything slow or bursty — image processing, emails — I'd decouple through SQS so the web tier never blocks. Security wraps it: IAM roles not keys, security groups per tier, secrets in Secrets Manager. And CloudWatch everywhere, because you can't scale what you can't see."

Likely follow-ups

• "Traffic 10×'s tomorrow — what breaks first?" → usually the database: read replicas, caching, or rethink with DynamoDB; ASG handles the web tier if health checks and scaling policies are right.
• "EC2 or Lambda — how do you choose?" → steady predictable load favors EC2/containers; spiky, event-driven, or low-ops favors Lambda (mind the 15-min cap and cold starts).
• "How does it survive an AZ going down?" → everything is multi-AZ: ALB targets in ≥2 AZs, RDS standby fails over automatically, statelessness in the web tier is what makes that painless.

The team's world: membership nonprofits — professional associations, alumni orgs, museums. Their pain: small staff, big member-services load, tight budgets. The flagship pattern:

"Say a membership org drowns in routine member questions — benefits, renewals, event details. I'd build a member-services assistant with RAG on Bedrock. Their documents — policies, FAQs, program guides — land in S3; a Bedrock Knowledge Base chunks and embeds them into a vector store like OpenSearch Serverless. At question time we retrieve the relevant passages and ground a foundation model — say Claude on Bedrock — so answers cite the org's actual policies instead of hallucinating. Guardrails keep it on-topic and out of, say, medical or legal advice. Members who call instead of chat: Amazon Connect is the cloud contact center — same knowledge base can power agent assist, and Contact Lens gives sentiment and call analytics. Leadership gets QuickSight dashboards over membership and engagement data. The reason this works for a nonprofit is the cost shape: everything here is serverless, pay-per-use — no idle servers, and no ML team to hire."

Cost levers to name (Frugality!)

• Right-size the model per task — small model for routing/classification, big model only where quality demands it. "I did exactly this with the abstraction layer."
• Serverless everywhere → cost tracks usage, which for a nonprofit tracks mission activity.
• AWS Imagine Grant (annual nonprofit grant program — cash + credits) and nonprofit AWS credits via TechSoup exist — knowing these signals you get the customer.

Likely follow-ups

• "How do you stop it giving wrong answers to members?" → grounding + citations from the Knowledge Base, Guardrails, human handoff threshold in Connect, and an eval set built from real member questions before launch.
• "Why RAG and not fine-tuning?" → the org's content changes weekly (events, policies); RAG updates by re-syncing documents, no retraining, cheaper and auditable.
• "Data privacy?" → member PII stays in their account; Bedrock doesn't train on customer data; IAM + encryption at rest (KMS) by default.

Six pillars, one line each. Drop the framework's name once, naturally ("through a Well-Architected lens…"):

• Operational Excellence — run and improve with automation; small reversible changes.
• Security — least privilege, encrypt everywhere, trace everything.
• Reliability — design for failure; multi-AZ; test recovery.
• Performance Efficiency — right service, right size; measure, don't assume.
• Cost Optimization — pay for what you use; watch and adjust. (THE nonprofit pillar.)
• Sustainability — minimize footprint; serverless and managed services help.

Full lessons, flashcards, and the four practice exams from assessment prep are archived at /assessment.html.