SageMaker deployments: how Hugging Face + AWS agent skills eliminate health-check guesswork

Why “Failed to pass health check” is usually a facts problem, not a philosophy problem

Hit deploy on SageMaker and you’ll sometimes see the classic, inscrutable “Failed to pass health check.” Usually it’s a small, concrete mismatch: the wrong serving container family, a stale image tag for your AWS Region, an incompatible Python wheel, an execution role that can’t read ECR, or no teardown plan for a continuously‑billing GPU endpoint. Any one of those can make a deployment look mysterious. The fix is mostly factual and repeatable.

How Hugging Face + AWS agent “skills” turn guesswork into repeatable deployments

Hugging Face and AWS published six open-source agent skills that encode those deployment facts so coding agents stop guessing. Instead of improvising which AMI, image tag, or Python runtime to use, the skills pull answers from canonical sources (the AWS Deep Learning Containers catalog, regional ECR registries, and account metadata) and apply sensible production defaults such as autoscaling and CloudWatch alarms.

The demo used Kiro as the coding agent and installed the skills from the Hugging Face Skills repo (https://github.com/huggingface/skills.git), pinned in the demo to commit f3186efbbc322121eb5d0f31e8a1d669ee961159. The example deployed Qwen/Qwen3-0.6B (pinned in the demo to commit c1899de289a04d12100db370d81485cdf75e47ca) to an ml.g5.xlarge real‑time endpoint in us‑east‑1. The demo noted a per‑instance on‑demand cost of $1.408/hr for ml.g5.xlarge (us‑east‑1 example; check current SageMaker pricing at aws.amazon.com/sagemaker/pricing/); that continuous billing is why teardown and autoscaling matter.

“Deploying a Hugging Face model to production means making a dozen decisions: choosing the right serving container for the model’s architecture, confirming the current image tag for your AWS Region, and matching an instance type to the model’s memory footprint.”, Hugging Face & AWS

The six skills doing the heavy lifting

  • hf-cloud-sagemaker-deployment-planner, orchestrates the flow and prompts only for what’s necessary.
  • hf-cloud-aws-context-discovery, discovers profile, Region, account, and caller identity.
  • hf-cloud-python-env-setup, creates an isolated Python environment (supports 3.10, 3.11, 3.12; Python 3.13 not supported in the demo due to wheel availability).
  • hf-cloud-sagemaker-iam-preflight, finds or creates a SageMaker execution role (searches patterns such as AmazonSageMaker-ExecutionRole-* and ranks candidates by last‑used date).
  • hf-cloud-serving-image-selection, chooses the serving container family and resolves a current image URI from the AWS DLC/ECR catalog, with permission-aware fallbacks.
  • hf-cloud-sagemaker-production-defaults, applies autoscaling and CloudWatch alarms, standard naming, and includes a teardown script that verifies deletion.

Example outputs from the demo (snapshot)

These values were resolved during the demo’s serving-image-selection step and are shown here as an illustrative snapshot, image URIs, AMIs, and tags change over time and by region.

  • Image URI (demo): 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.28.0-transformers5.15.0-gpu-py312-cu130-ubuntu24.04
  • Inference AMI (demo): al2-ami-sagemaker-inference-gpu-3-1
  • Environment variables written into the model container (demo):
    • SM_VLLM_MODEL = Qwen/Qwen3-0.6B
    • SM_VLLM_HOST = 0.0.0.0
    • SM_VLLM_TRUST_REMOTE_CODE = false
    • SM_VLLM_MAX_MODEL_LEN = 8192

Important operational note: the demo set SM_VLLM_HOST=0.0.0.0 so the vLLM server binds to all interfaces. If the container binds only to localhost (127.0.0.1), SageMaker’s health check will typically fail in the same opaque way you see for a wrong image or stale tag.

How the workflow actually runs, and what can go wrong at each step

  1. Discover AWS context, profile, Region, account, caller identity.

    (Failure mode: wrong Region or profile leads to resolving an image URI in the wrong ECR registry.)

  2. Bootstrap an isolated Python environment, select Python 3.10-3.12 and a current boto3.

    (Failure mode: missing binary wheels for newer Python releases causes runtime import errors.)

  3. IAM preflight, search for existing SageMaker execution roles or create one if permitted; created roles default to AmazonSageMakerFullAccess in the demo.

    (Failure mode: overly permissive roles are a governance risk; lacking ECR permissions prevents image discovery.)

  4. Serving image resolution, select the correct container family (vLLM, TGI, etc.) and resolve a current image URI from the AWS DLC/ECR catalog.

    (Failure mode: wrong family or stale tag -> container fails health checks. The skills fall back to a known‑good tag when DescribeImages is denied and log the reason.)

  5. Create model, endpoint configuration, endpoint, attach autoscaling (min 1, max 4 in the demo) and three CloudWatch alarms (Invocation5XXErrors, ModelLatencyP99, OverheadLatencyP99).

    (Failure mode: leaving min=1 on a GPU endpoint can produce an unexpected baseline bill; alarms without clear thresholds are noisy unless tuned to SLOs.)

  6. Smoke test and teardown, run a smoke call against the live endpoint and provide a teardown.py that verifies deletion.

    (Failure mode: no teardown equals ongoing charges; smoke tests catch basic token truncation or binding issues but don’t validate model correctness against complex SLOs.)

Business impact: why teams should care

  • Faster, auditable deployments. Agents armed with curated facts produce reproducible plan files and logs, useful for approvals and post‑mortems.
  • Cost control. Production defaults include autoscaling and a teardown script, critical because GPU-backed real‑time endpoints bill continuously until removed.
  • Better operational safety. The skills surface permission gaps (e.g., inability to call ecr-public:DescribeImages) and fall back safely while logging reasons, reducing blind failures during deploys.

Known limitations and governance questions to plan for

  • Exotic runtimes and native libraries. The skills focus on common serving families (vLLM, TGI, HF Inference Toolkit). Models that need custom native binaries, special drivers, or unusual hardware (MoE variants, custom acceleration) may still require manual engineering.
  • Least‑privilege automation is not included. The demo may create a role with AmazonSageMakerFullAccess; teams should harden that role before production. A practical least‑privilege baseline could include scoped ECR/S3 read permissions and sagemaker:CreateModel / sagemaker:InvokeEndpoint limited to specific resource ARNs.
  • Networking and KMS. VPC endpoints, private ECR registries, and KMS usage aren’t auto‑injected. Integrate those constraints into your planner or CI pipeline when you have private networking policies.
  • Multi‑account and CI/CD at scale. The skills run per account via AWS CLI/Boto3. For centralized governance you’ll need cross‑account roles, pipelines, and an approval gate.
  • Serverless and scale‑to‑zero tradeoffs. The skills support various inference modes (real‑time default, scale‑to‑zero variants, serverless, async, batch transform). Note that serverless SageMaker historically had limitations (for example, no GPU support in many configurations), so validate the mode that meets your latency, cost, and hardware needs.
  • Fallback governance. When image resolution is blocked, the skill uses a shipped fallback tag. That is pragmatic, but fallback tags can age; add an automated validation or rotation cadence to avoid drifting to stale or insecure images.

How to adopt these skills without getting burned

Adoption is straightforward technically, but governance and a few guardrails make the pattern safe for production.

  • Pin skill commits in CI. The demo pinned to commit f3186ef… so behavior is reproducible; pin the repo snapshot you review and test (https://github.com/huggingface/skills.git).
  • Require role hardening before production traffic. Treat any newly created SageMaker execution role as provisional; replace AmazonSageMakerFullAccess with a role scoped to only the necessary ECR/S3/SageMaker actions and resource ARNs.
  • Gate deployments in CI/CD. Capture the planner’s generated plan file as the deployment artifact and require a human approval or automated policy check before invoking the endpoint creation step.
  • Add billing guards. Enforce teardown in pipelines, add CloudWatch billing alerts, or automate scale‑to‑zero for non‑production environments to avoid surprise GPU bills.
  • Automate smoke tests and token‑limit checks. The demo noticed truncated responses due to max‑token limits; include programmatic assertions that validate model length, response shape, and basic correctness against representative inputs.
  • Govern fallback images. Track when the skill uses a fallback image and require an owner to validate or rotate that tag on a regular cadence.

Two things to do in the first hour

  • Install the pinned skill set into a sandbox account and run a dry run that writes the planner’s output file. Inspect the resolved image URI, AMI, and proposed role before proceeding.
  • Add a CI approval step that refuses to apply the planner output unless a human verifies the execution role policy and any VPC/KMS requirements are satisfied.

Quick technical notes for engineers

  • Kiro supports workspace (.kiro/skills/) and global (~/.kiro/skills/) skill scopes so organizations can centralize skills and allow local overrides.
  • The skills call AWS via Boto3 and AWS CLI by default; the SageMaker Python SDK is supported as an alternative for higher‑level operations.
  • Python interpreters supported in the demo: 3.10, 3.11, 3.12. Python 3.13 was not supported in the demo due to missing binary wheels.

Key takeaways, questions you should be asking

  • Can a coding agent reliably deploy Hugging Face models to SageMaker?

    Yes, when the agent is provided with reproducible, curated facts. The six Hugging Face skills encode image selection, IAM preflight, environment setup, and production defaults so agents like Kiro can produce reliable deployments, logs, and plan files instead of brittle guesses.

  • Which serving image did the demo resolve for Qwen3‑0.6B in us‑east‑1?

    The demo resolved 763104351884.dkr.ecr.us-east-1.amazonaws.com/huggingface-vllm:0.28.0-transformers5.15.0-gpu-py312-cu130-ubuntu24.04 and the inference AMI al2-ami-sagemaker-inference-gpu-3-1. Treat this as a snapshot, image URIs and AMIs vary by region and over time.

  • How are production concerns like autoscaling and monitoring handled?

    The hf-cloud-sagemaker-production-defaults skill applies autoscaling (demo defaults: min 1, max 4) and three CloudWatch alarms (Invocation5XXErrors, ModelLatencyP99, OverheadLatencyP99), and includes a teardown script that verifies resource deletion. Tune thresholds to match your SLOs.

  • What happens if the skill cannot query ECR because of permissions?

    The skill falls back to a known‑good image tag shipped with the skill and logs why it used the fallback. That prevents silent deployment failures, but teams should remediate permissions and track fallback usage so the fallback stays current and vetted.

  • Will this solve every deployment need for enterprise models?

    No. The skills address routine, factual failure modes and provide a strong base for agent‑driven deployments. Custom networking, strict least‑privilege policy automation, multi‑account orchestration, and exotic runtime dependencies still need additional engineering and governance.

Shipping models to production is mostly operations hygiene: canonicalize the facts, automate the routine checks, and require human review for policy decisions. These Hugging Face skills encode many of the facts that agents historically get wrong, turning coding agents into reliable deployment assistants, so long as you pair them with role hardening, network/KMS integration, and billing guards. That combination buys fewer midnight pagers and more predictable, auditable rollouts.

0