51 questions - Intermediate to experienced
Building an AI demo is easy; running it for real users is what these rounds test. Expect containers and CI/CD for AI apps, the classic MLOps loop (tracking, registry, drift), how you serve models and autoscale GPUs, the managed AI services on AWS, Azure and Google Cloud and when to use each, and above all how you keep cost per request low with routing, caching, batching and the right hardware. Cloud product names changed in 2024 to 2026, so the answers give both the new and the old names.
Last updated:
DevOps automates building, testing, deploying and running software. MLOps adds what models need: versioned data and models, training pipelines, a model registry and monitoring for drift. LLMOps applies the same ideas to apps built on large language models, where the work shifts to prompts, retrieval, evaluation of open-ended answers, token cost and provider limits.
Use a slim base image that matches your CUDA version if you need a GPU, install pinned dependencies in an early layer so rebuilds are fast, run as a non-root user, and add a health endpoint. Large model weights are usually downloaded from object storage at startup or mounted, not baked into the image, to keep images small.
FROM python:3.12-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
USER 1000
CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8080"]On every change: lint, unit tests, build the image and scan it. Then run an evaluation suite against a fixed dataset and block the release if quality, safety or cost per request gets worse. Deploy to staging, run smoke tests, then roll out to production gradually with automatic rollback on errors.
Tools such as Terraform, OpenTofu, AWS CloudFormation, Bicep or Pulumi describe GPUs, endpoints, networks, permissions and budgets in versioned files. Every environment is then built the same way, changes are reviewed like code, and you can recreate the stack in another region or account.
Shadow deployment sends a copy of real traffic to the new model without showing users its answers, so you can compare. Canary sends a small share of users to it and widens if metrics hold. Blue-green keeps the old version running for instant switch-back. For AI apps, watch answer quality and cost, not just errors.
Training, fine-tuning and serving larger open models need GPUs or AI accelerators. Calling a hosted model API, small classical models, embeddings at low volume and the app layer around the model usually run fine on CPUs. Start without GPUs where you can, because they are the most expensive line on the bill.
Experiment tracking records each training run's code version, data version, parameters, metrics and output files, so results can be compared and reproduced. A model registry stores approved model versions with stages or aliases such as staging and production, so deployments pull a named version instead of a loose file. MLflow is the common open-source choice.
Version the code in Git, version the data with a tool such as DVC or table snapshots, pin library versions, fix random seeds, record the hardware and container image, and log everything with the run. Anyone should be able to retrain the same model from the run record.
Skew happens when features are computed one way during training and another way in production, so the live model sees different inputs. Avoid it by sharing one feature code path, often through a feature store, logging live inputs and comparing their distributions with the training data.
Data drift means the inputs change, for example users start asking about a new product. Concept drift means the relationship between inputs and the right answer changes, for example fraud patterns shift. Monitor input distributions for the first and labelled outcomes or quality scores for the second.
On a schedule when data changes steadily, or when monitoring shows drift or a quality drop beyond a threshold. The retraining pipeline should run automatically, evaluate the new model against the current one on the same test set, and promote it only if it is better.
Define steps such as data validation, feature building, training, evaluation and registration as a pipeline with clear inputs and outputs, then run it on an orchestrator such as Kubeflow Pipelines, Airflow, Prefect or the managed pipelines in each cloud. Each step should be cacheable and rerunnable on its own.
Real-time inference answers each request within seconds while the user waits, so it needs warm capacity. Asynchronous inference queues requests and returns results later, which suits long jobs such as transcribing an hour of audio. Batch inference processes a whole dataset on a schedule and is the cheapest per item.
Hosted APIs need no GPUs or ops work, give the strongest models and charge per token, so they win at low or spiky volume. Self-hosting an open model gives control over data, latency and fine-tuning, and can be cheaper at high, steady volume, but you pay for GPUs whether or not they are busy and must run the serving stack.
Inference servers such as vLLM, Hugging Face Text Generation Inference, NVIDIA Triton with TensorRT-LLM and SGLang use continuous batching, efficient attention caches and quantized weights to serve many users per GPU. They expose an HTTP API, so the app code does not change when you swap the model.
CPU usage is a poor signal for GPU work. Scale on requests in flight per replica, queue length, or GPU utilisation, using a tool such as KEDA on Kubernetes or the managed autoscaler of your platform. Keep a minimum number of warm replicas if cold starts are too slow for users.
A new replica must start a large container, load gigabytes of weights into GPU memory and warm up, which can take minutes for big models. Reduce it with smaller or quantized models, weights cached on fast local disk, pre-pulled images, and a few always-warm replicas for latency-critical traffic.
An AI gateway sits between your apps and model providers. It holds the provider keys, enforces per-team rate limits and token budgets, routes requests to the right model, retries or fails over to another provider, caches responses and logs usage and cost in one place.
Create GPU node pools, install the NVIDIA device plugin or GPU operator so pods can request nvidia.com/gpu, and use taints so only GPU workloads land there. Autoscale nodes with the cluster autoscaler, share GPUs with time-slicing or MIG for small models, and set requests and limits so pods fit.
Amazon Bedrock gives managed access to many foundation models through one API, with no servers. Amazon SageMaker AI hosts your own or open models on managed endpoints. Containers on Amazon ECS or EKS run the app layer and self-managed model servers, and AWS Lambda handles light glue code.
Bedrock adds Knowledge Bases for managed RAG, Guardrails for content and PII filtering, agents and the AgentCore services for running agents in production, model evaluation, prompt caching, batch inference, and provisioned throughput when you need reserved capacity. Data stays in your AWS account boundary under IAM control.
Real-time endpoints keep instances running for low-latency requests. Serverless inference scales to zero for spiky traffic but does not support GPUs. Asynchronous inference queues large or long requests and can scale to zero when idle. Batch transform runs offline predictions over a dataset in S3.
No. Fargate and Lambda do not offer GPUs, and Lambda also has a 15-minute time limit. Use them for the API layer, retrieval and orchestration, and run GPU models on SageMaker AI, EC2 GPU instances, ECS on EC2 or EKS GPU node groups, or call Bedrock.
They are AWS-designed machine learning chips: Inferentia for inference and Trainium for training, offered as Inf2 and Trn instances. They can cost less per token or per training step than comparable GPUs, but models must be compiled with the AWS Neuron SDK, so check that your model and framework are supported.
Documents land in S3, an ingestion job chunks and embeds them into a vector store such as Bedrock Knowledge Bases, OpenSearch Serverless or Aurora PostgreSQL with pgvector. An API on API Gateway with Lambda or ECS retrieves chunks and calls a Bedrock model. CloudWatch holds logs and metrics, and IAM roles grant each piece only what it needs.
Give each service an IAM role with least-privilege permissions instead of access keys, keep secrets in Secrets Manager, call Bedrock and S3 through VPC endpoints so traffic stays off the public internet, encrypt data with KMS, and log API calls with CloudTrail.
Microsoft Foundry, formerly Azure AI Foundry, hosts models from many providers behind managed deployments and adds agents, evaluation and safety tools. Azure Machine Learning hosts your own models on managed online endpoints. Azure Container Apps, App Service and AKS run the app layer and self-hosted model servers.
Standard deployments charge per token and share capacity, with global and data-zone variants that trade data residency for availability. Provisioned deployments reserve throughput units for steady, latency-sensitive traffic at a fixed price. Batch deployments process large jobs within about a day at a discount.
Azure Container Apps offers serverless GPUs: you deploy a container, request a GPU profile, and the app scales out with traffic and down to zero when idle, billed per second. For full control of drivers, node types and scheduling, use AKS with GPU node pools instead.
Use managed identities so services authenticate with Microsoft Entra ID instead of API keys, keep any remaining secrets in Key Vault, put Foundry, storage and search behind private endpoints, and apply role-based access control per resource. Content filters in Foundry add safety checks on prompts and answers.
API Management can act as an AI gateway in front of model deployments: it enforces token-per-minute limits per team, balances load across several deployments or regions, retries on throttling, caches similar prompts and emits token usage metrics for chargeback.
Documents sit in Blob Storage, an indexer chunks and embeds them into Azure AI Search, which offers vector, keyword and hybrid search with semantic ranking. The app runs on Container Apps or App Service, retrieves from AI Search and calls a model deployed in Foundry, with Application Insights for traces and managed identity between all services.
Gemini Enterprise Agent Platform, the April 2026 successor to Vertex AI, gives managed access to Gemini and partner models, hosts your own models on endpoints, and adds agent, evaluation and pipeline tools. Cloud Run runs containers serverlessly, including on GPUs, and GKE gives full Kubernetes control with GPUs and TPUs.
Cloud Run can attach an NVIDIA L4 GPU to a container, bills per second and scales to zero when there are no requests, so a small open model or an embedding service costs nothing while idle. Each instance gets one L4 and needs at least 4 CPUs and 16 GiB of memory, and GPU regions are limited.
Choose GKE when you need several or larger GPUs per model, TPUs, custom schedulers, long-running training jobs, fine control of networking, or many models sharing a cluster. Cloud Run is simpler and cheaper for request-driven services that fit on one GPU and can tolerate cold starts.
Documents sit in Cloud Storage, a pipeline chunks and embeds them into AlloyDB or Cloud SQL for PostgreSQL with pgvector, or into the platform's managed vector search. The API runs on Cloud Run, retrieves chunks and calls a Gemini or open model through the Agent Platform, with Cloud Logging and Cloud Trace for monitoring.
Run each service as its own service account with least-privilege IAM roles, use Workload Identity on GKE instead of key files, store secrets in Secret Manager, restrict data movement with VPC Service Controls, and keep audit logs on for model and data access.
For hosted models it is mostly tokens: input tokens (system prompt, chat history, retrieved context) and output tokens, which usually cost several times more each. For self-hosted models it is GPU hours, including idle time. Then come embeddings, vector storage, logging, data transfer and engineers' time.
Most requests are easy. A router, either rules or a small classifier, sends simple requests such as greetings, short extractions or classification to a small cheap model and only hard ones to a large model. A cascade tries the small model first and escalates when its confidence or a check fails.
Prompt caching lets providers reuse the processed form of a repeated prompt prefix, such as a long system prompt, at a lower price, so keep the stable part first. Exact response caching returns stored answers for identical requests, and semantic caching does so for very similar questions. All three also cut latency.
Use it for work that can wait hours, such as tagging documents, nightly summaries, evaluations or backfills. AWS, Azure and Google Cloud all offer batch modes for many models at a large discount, often about half the on-demand price, in exchange for completion within about a day.
Shorten the system prompt, trim or summarise old chat history, send fewer and smaller retrieved chunks after reranking, ask for concise answers and set a maximum output length, return structured output instead of long prose, and avoid sending the same documents on every turn.
Measure your real traffic in tokens per day, estimate the throughput one GPU achieves for your model, and compare the GPU cost per million tokens at realistic utilisation with the API price. Add engineering and on-call time. Self-hosting usually wins only with high, steady volume.
def self_hosted_cost_per_million_tokens(gpu_hourly_usd, tokens_per_second, utilisation=0.5):
tokens_per_hour = tokens_per_second * 3600 * utilisation
return gpu_hourly_usd / tokens_per_hour * 1_000_000
# a $1.20/hour GPU serving 400 tokens/s at 50% utilisation
print(self_hosted_cost_per_million_tokens(1.20, 400)) # about $1.67Use an inference server with continuous batching, quantize weights to 8 or 4 bits where quality allows, pick the smallest GPU the model fits on, serve many LoRA adapters on one base model instead of many full models, and keep utilisation high by sharing GPUs across services.
Spot or preemptible capacity is heavily discounted but can be reclaimed at short notice, so use it for training with checkpoints, batch jobs and stateless replicas behind a queue. For steady baseline load, savings plans, reservations or committed-use discounts trade a one- or three-year commitment for a lower rate.
Tag every resource and request with team, feature and environment, log tokens and cost per request, and build a dashboard of cost per feature and per user. Set budgets and alerts in the cloud console, per-team token limits in the gateway, and review the most expensive prompts every month.
Latency at p50 and p95, time to first token and tokens per second for streaming, error and throttling rates, tokens and cost per request, cache hit rate, GPU utilisation and memory if self-hosted, and quality signals such as user feedback, evaluation scores on sampled traffic and guardrail triggers.
Retry throttled calls with exponential backoff and jitter, queue non-urgent work, spread load across regions or deployments, and configure a fallback model or provider for critical paths. Request higher quotas or reserved throughput before launches, and test the fallback regularly.
import random, time
def call_with_backoff(fn, attempts=5):
for attempt in range(attempts):
try:
return fn()
except RateLimitError:
time.sleep(min(30, 2 ** attempt) + random.random())
return fallback_model_call()Keep a versioned evaluation set of real and edge-case inputs. Run it in CI for every prompt, model or retrieval change, compare scores, latency and cost with the current production version, and block regressions. After release, score a sample of live traffic so quality drift shows up early.
Detect and mask personal data before it reaches the model or the logs, choose regions and deployment types that meet data-residency rules, check that the provider does not train on your data, set log retention limits, and restrict who can read stored prompts.
Guardrails are checks around the model: input filters for prompt injection, abuse and personal data, output checks for unsafe content, schema validation and grounding against sources, and limits on what tools an agent may call. Managed options exist on all three clouds, and you can add your own rules.
CrackInterviewAI listens to the live interview and gives a structured answer on screen for coding, system design, HR and project questions. Download for Windows.