Deploying Kimi K3 on AWS: minimum infra, top risks, and a 2-week pilot plan

Deploying Kimi K3 on AWS: the minimums, the risks, and a practical pilot plan

Moonshot AI’s Kimi K3 (released July 27, 2026) is being marketed as “the first open-weight system to reach the 3 trillion parameter class.” The published weights and model card list the total parameter count as 2.8 trillion (Hugging Face model id: moonshotai/Kimi-K3). That gap matters for messaging. For engineering, what matters is that this is a multi-trillion-parameter Mixture-of-Experts (MoE) model with non-trivial GPU, software, and orchestration requirements.

Executive summary

  • Feasible, but not cheap. You can run Kimi K3 on AWS today, but it needs specialized GPU instances (p6-b300 / ml.p6-b300.48xlarge) and a vLLM day-0 serving image. Treat this as a program-level effort, not a quick experiment.
  • Minimum infra footprint. Plan for pods that request 8 NVIDIA B300 GPUs and tensor-parallel sizing that maps to those 8 GPUs. Use the vllm/vllm-openai:kimi-k3 image (or an identical image you build and validate) to load MXFP4 shards and handle MoE routing.
  • Top operational risks. Cost and capacity, cold-start latency, quantization quality (MXFP4), and security when loading remote model code. Resolve these before sending production traffic.

What Kimi K3 actually is, facts and reconciliation

  • Model name / id: Kimi K3, Hugging Face: moonshotai/Kimi-K3
  • Published parameter count: 2.8 trillion parameters (model card)
  • MoE configuration: 896 experts, 16 experts activated per token (vendor specification)
  • Context window: 1, 000, 000 tokens (vendor claim)
  • Modality: native multimodal (text + vision)
  • Weight format: MXFP4 (Microscaling FP4)
  • Serving engine: vLLM day-0 container (vllm/vllm-openai:kimi-k3)

One arithmetic detail to clear up: some materials list “active parameters per token” as about 104 billion. A simple share-of-experts calculation, 2.8T * (16 / 896), gives roughly 50 billion parameters active per token. The larger 104B figure is a vendor metric and likely includes other active components (shared dense layers, embeddings, attention heads, or an “effective active parameters” accounting). Treat the 104B number as a Moonshot claim to validate on your workloads. For capacity planning, use the 8-GPU hardware guidance below and run real throughput and latency tests to measure effective per-token cost.

“the first open-weight system to reach the 3 trillion parameter class.”, Moonshot AI release text

Serving stack: vLLM day-0 image (and a security caveat)

AWS and Moonshot recommend vLLM as the inference engine. The example container tag is vllm/vllm-openai:kimi-k3, a day-0 image that includes K3-specific MoE routing and MXFP4 support. The AWS walkthrough says these changes should be merged into mainline vLLM in upcoming releases. For now, treat the day-0 image as the practical requirement.

Security warning: the common example flag –trust-remote-code lets model-side code in remote repositories execute at load time. That is a high-risk setting for enterprise environments. Do not use –trust-remote-code in production without strict mitigations: pin image digests, use a private container registry, scan images for vulnerabilities, review any remote model code, and ideally bundle the model-serving runtime in your CI/CD pipeline.

Critical vLLM flags you’ll see in examples (focus on these during initial runs):

  • –model moonshotai/Kimi-K3
  • –load-format fastsafetensors
  • –tensor-parallel-size 8
  • –served-model-name Kimi-K3
  • –moe-backend auto

Other flags in the examples (prefix caching, tool parsers, reasoning parsers) help performance and agent features but do not replace the security and image-pinning controls above.

Hardware requirement: p6-b300 / ml.p6-b300.48xlarge

Do not try to host Kimi K3 on a single-GPU consumer card or on general-purpose GPU instances. The validated instance family is p6-b300 (AWS ml.p6-b300.48xlarge), which provides 8 NVIDIA B300 (Blackwell Ultra) GPUs and the interconnect bandwidth needed for large tensor-parallel sharding. The example Kubernetes manifests request nvidia.com/gpu: 8 and set tensor-parallel-size to 8.

Because these instances are specialized and capacity-constrained, AWS suggests two reservation mechanisms:

  • SageMaker Flexible Training Plans (for HyperPod deployments)
  • EC2 Capacity Blocks (for EKS/self-managed deployments)

Deployment patterns on AWS, pick your operational boundary

AWS documents two validated paths. Choose based on how much operational surface you want to own.

SageMaker HyperPod + Inference Operator (managed)

What it gives you: a managed orchestration layer that automates multi-GPU endpoint lifecycle, model download, and health checks. Example manifests live in aws-samples/sagemaker-genai-hosting-examples (SageMakerHyperpod/kimi-k3/kimi-k3.yaml).

When to pick it: you want less Kubernetes plumbing, faster time to first endpoint, and integrated SageMaker lifecycle automation. You still need to manage model updates, network and VPC controls, and CI/CD around the serving image.

Self-managed EKS + EC2 Capacity Blocks

What it gives you: full control. AWS provides an AI on EKS recipe (awslabs/ai-on-eks, infra/solutions/inference-ready-cluster/recipes/kimi-k3) with Helm charts and Terraform examples to reproduce the walkthrough.

When to pick it: you need custom autoscaling, complex networking, or enterprise controls that HyperPod can’t meet. Expect to install and maintain vLLM images, shard download logic, prewarming, and autoscaler integration yourself.

Quick operational checklist (do these before you try to serve real traffic)

  • Confirm ml.p6-b300.48xlarge availability in your target region and secure Capacity Blocks or Flexible Training Plans.
  • Use the vllm/vllm-openai:kimi-k3 image for initial testing, or build and validate an identical, pinned image in your private registry.
  • Set tensor-parallel-size to 8 and request 8 GPUs per pod (nvidia.com/gpu: 8) as shown in the examples.
  • Disable –trust-remote-code in production; if you must enable it during early experiments, pin commits and perform code review.
  • Run MXFP4 quality validation against an FP16/FP32 baseline on representative datasets before user-facing rollout.
  • Design shard prewarming and prefix-caching to reduce cold-start penalties (the example flags include –enable-prefix-caching).
  • Integrate endpoint access with IAM, VPC, and secrets management for inference keys and telemetry.

How to validate a deployment, a short test plan

Don’t rely on a single successful request. Use this minimal validation suite to collect actionable metrics:

  1. Warmup run: Send 100 sequential short prompts to warm caches and load shards. Confirm pods report shards loaded in logs.
  2. Cold-start distribution: Stop pods, then start a new replica and record cold-start time until readiness probe passes. Repeat 5 times to get a distribution.
  3. Latency profile: For batch sizes {1, 8, 16}, measure p50/p95/p99 latency for short replies (≤64 tokens) and long replies (≥512 tokens).
  4. Throughput: Run a 10-minute sustained test at your target concurrency and record tokens/sec, GPU utilization, and memory pressure per GPU.
  5. Quality/A/B: Compare MXFP4 outputs against a higher-precision baseline (FP16/FP32) on a representative test set. Capture exact deltas on metrics that matter to your product (accuracy, F1, BLEU, hallucination rate, etc.).
  6. Security checks: Verify container image digest, run a CVE scan, and ensure no –trust-remote-code is present in production manifests.

Suggested starting hypotheses for pilots, treat these as targets to validate not guarantees: aim for under 1 second p50 for short replies in steady state, and collect tokens/sec per GPU for your workload to compute cost per 1k tokens. If a target isn’t met, iterate on batching, prefix caching, and tensor-parallel placement.

Operational gaps you must plan for

  • Cost and procurement: The walkthrough points to Flexible Training Plans and Capacity Blocks but does not list prices. Treat this as a procurement exercise: reserve capacity early and budget for a sustained multi‑$K/month baseline depending on run hours and replicas.
  • Autoscaling and HA: Examples in the repos show single-replica manifests. Production needs multi-replica strategies, shard preloading, warm pools, and regional failover planning.
  • Quantization tradeoffs: MXFP4 reduces memory and makes the model runnable on available hardware, but task-level quality impacts are not published. Run rigorous A/B validation before customer exposure.
  • Upgrade path: the vLLM K3 commits are currently in a day-0 image. AWS notes they “expect these to be merged to the main vllm container in the upcoming releases.” Define a testing and rollout plan for upstream vLLM merges and image rotation.
  • Licensing and acceptable use: Kimi K3 weights are public on Hugging Face (moonshotai/Kimi-K3). Verify the model card license and have legal review any commercial usage or redistribution constraints.

Minimal pilot plan with owners and timeline

  • Duration: 2 weeks (pilot).
  • Ops/Cloud: secure Capacity Blocks/Flexible Training Plans and provision one ml.p6-b300.48xlarge nodegroup (1 week).
  • ML Engineering: build or pull the vllm day-0 image into a private registry, run the validation suite, and produce tokens/sec and latency profiles (2 sprints overlapping with Ops).
  • Security: review the use of –trust-remote-code, scan images, and approve network and IAM policies (parallel with engineering).
  • Product/PM: define the representative dataset and success metrics for MXFP4 vs baseline tests (start immediately).

Where the examples and artifacts live

The walkthrough provides concrete manifests and recipes you can clone and adapt:

  • HyperPod example: aws-samples/sagemaker-genai-hosting-examples, path: SageMakerHyperpod/kimi-k3/kimi-k3.yaml
  • EKS recipe: awslabs/ai-on-eks, path: infra/solutions/inference-ready-cluster/recipes/kimi-k3
  • Model weights and card: Hugging Face, moonshotai/Kimi-K3 (contains weight format and license details)
  • vLLM docs and day-0 image: vLLM project (use the kimi-k3 tagged image for initial runs)

“We expect these to be merged to the main vllm container in the upcoming releases.”, note from the AWS walkthrough describing the vLLM K3 commits

Practical advice for engineering and product leaders

  • Treat this as a small program: budget, procurement lead time, security signoff, and an ops sprint for shard preloading and autoscaling.
  • Validate MXFP4 on your critical tasks early; do not assume parity with higher precision.
  • Lock down image digests and avoid –trust-remote-code in production. If a vendor claims faster time to market with remote code, weigh that against your organization’s risk tolerance.
  • Measure cost per 1k tokens after you have steady-state tokens/sec numbers, that will drive pricing or subsidy decisions for any product built on K3.

Key takeaways, questions you should ask (and short answers)

  • What hardware and GPU allocation do I need to host Kimi K3 on AWS?

    The recommended instance is ml.p6-b300.48xlarge (p6-b300) with 8 NVIDIA B300 GPUs; example manifests request nvidia.com/gpu: 8 and set tensor-parallel-size to 8.

  • Which serving software and container image are required?

    vLLM is the recommended inference engine. Use the day-0 image vllm/vllm-openai:kimi-k3 for MXFP4 and MoE support unless you build and validate an equivalent pinned image in your registry.

  • Is Kimi K3 “3 trillion” or 2.8 trillion parameters?

    Moonshot markets K3 as 3-trillion-class, while the published weights total 2.8 trillion. Treat the 2.8T figure as the precise published parameter count; the “3T-class” label is a marketing rounding.

  • How many parameters are active per token?

    A simple experts fraction (2.8T × 16/896) gives ≈50 billion parameters active per token. Moonshot’s materials list ~104B as a vendor metric, that likely counts additional active components. Validate with workload tests rather than assuming either number is the final billing unit.

  • Which AWS deployment patterns are validated?

    AWS documents two paths: SageMaker HyperPod with Flexible Training Plans (managed) and self-managed EKS using EC2 Capacity Blocks (more control). Example manifests are in the aws-samples and awslabs repos noted above.

  • What major gaps must I fill before production?

    Cost estimates, latency and throughput benchmarks, autoscaling and warm-pool strategies, MXFP4 quality validation, secure image handling (avoid –trust-remote-code), upgrade testing for vLLM, and license/legal review.

Running Kimi K3 on AWS is practical for organizations that want a large open-weight model under their control, but it is an engineering program. Use the AWS examples to get started, secure capacity early, validate MXFP4 quality on your data, and lock down security before you route any production traffic.

Authors of the AWS walkthrough: Andrew Smith (Sr. Cloud Support Engineer, SageMaker), Erez Zarum (Senior Startups Solutions Architect), and Vivek Gangasani (Worldwide Leader for Solutions Architecture, SageMaker Inference). Moonshot AI publishes the model on Hugging Face under moonshotai/Kimi-K3.

0