A 35.4 GB deep learning image does not train anything while it downloads. In Amazon Web Services’ own measurements, a standard pull of an image that size took 7 minutes 17 seconds before the container was ready. The GPU was already on the bill. The model had not taken a step.
AWS has turned on Seekable OCI indexing for Amazon SageMaker AI training and processing jobs. OCI is the Open Container Initiative image format. SOCI is an open-source way to lazy-load a container. It is not a new training algorithm. SageMaker can start the container after fetching the files required to launch it, then stream the rest while the job runs.
Those minutes are overhead. You pay for them, and they do not move the model. A long pretraining run can hide the tax. A hyperparameter sweep, a stack of experiments, or a run of short processing jobs cannot. Every time a container starts, the unpacking cost starts with it.
The suitcase problem
SageMaker training and processing jobs run from container images in Amazon Elastic Container Registry. Before your code executes, the service normally downloads the full image and decompresses every layer. Modern deep learning images routinely exceed 10 GB. Frameworks, CUDA libraries, and sometimes model weights all come along for the ride.
SOCI does not rewrite that image. It attaches a separate index in ECR, an OCI artifact that maps every file inside. SageMaker looks for the index on its own. A usable index means lazy loading. No index, an unreadable index, or a failed lazy load means a normal full download.
That fallback covers launch. A missing or bad index is not supposed to stop the container from starting. It does not mean the job cannot fail afterward. Scripts that expect a Docker runtime can still break, which is a separate problem from the pull.
Because the index sits beside the image, you are not rebuilding the layers to get this behavior. The original image digest does not change just because an index is attached. Signing and admission tools can still notice the extra artifact. Confirm that with the policy setup you already run. Do not treat “signatures stay valid” as a blanket pass.
What AWS measured
The times below are AWS’s reported figures, medians of repeated runs. They are not a customer study. They are not a forecast. Pull tests used eight images from 1.8 GB to 35.4 GB compressed, on GPU and CPU families including ml.g4dn, ml.g5, ml.g6, ml.g6e, ml.g7, ml.g7e, ml.p4d, ml.p5, and ml.m7i, from xlarge to 48xlarge, in us-west-2 and us-east-1. Download time ran from the start of the image pull until the container was ready. AWS did not publish the number of repeats, the variance, or a dollar cost.
- 1.8 GB: 48 seconds on a standard pull, versus 3-11 seconds with lazy loading. The absolute saving is well under a minute.
- 13.2 GB: 3 minutes 42 seconds versus about 6 seconds.
- 35.4 GB: 7 minutes 17 seconds versus 3-11 seconds.
AWS labeled those rows about 8 times, 34 times, and 38 times faster. The lazy-load side of the first and third rows is a range. A single multiplier is the wrong thing to put in a plan. Other speedup labels in the same announcement, from roughly 10 times to more than 35 times and as high as 46 times, do not reconcile with this table. Plan from the times.
On the 13.2 GB image, across 13 instance type and size combinations, standard pulls fell between 3 minutes 16 seconds and 4 minutes 20 seconds. Lazy loading held at 6-8 seconds. Buying a larger instance did not buy you out of the full download. AWS also says these benchmarks cut GPU idle time during image pulls by up to 90 percent. That is a ceiling on idle time during the pull in AWS’s tests, not a measured result from production accounts, and not a cut to the whole job.
Faster launch is not faster training
AWS then checked whether the rest of the job clawed the time back. One test was ResNet-50 on Imagenette, 9, 469 images, using a 13.2 GB PyTorch container and a 9.9 GB TensorFlow container. The instances were ml.p5.48xlarge, ml.p4d.24xlarge, ml.g7e.48xlarge, ml.g7.48xlarge, ml.g6e.12xlarge, ml.g6e.24xlarge, and ml.g6e.48xlarge. The other was full-parameter supervised fine-tuning of Qwen2.5-0.5B on alpaca-cleaned, 51, 760 examples, with the Hugging Face Trainer and a 9.4 GB Hugging Face PyTorch image, on ml.p5.48xlarge, ml.p4d.24xlarge, and ml.g6.48xlarge.
From the second epoch on, epoch time matched within ordinary run-to-run variation. The first epoch took 3-16 seconds longer under lazy loading, and that penalty did not depend on instance size. Newly touched files are fetched when startup code actually opens them. After subtracting that one-time cost, those jobs still finished 1-6 minutes sooner.
Heavier startup paths did not wipe out the gain in the cases AWS published. torch.compile and a JIT-built CUDA extension added 7-15 seconds, with 3 minutes 25 seconds to 3 minutes 56 seconds still saved. A 6 GB model embedded in the image added 2-3.5 minutes at load time, and the job still finished 2-2.5 minutes sooner.
The mechanism sets the limit. A job that scans, checksums, or preloads most of the image will request those bytes immediately. Lazy loading cannot outrun a workload that insists on touching everything before it does useful work. AWS is clear that the first-epoch penalty depends on the image and on which files startup code opens. Two tested training workloads are not a certificate for every custom runtime. Processing jobs do not have epochs, so those epoch numbers do not transfer. The pull-time numbers are the ones that apply to a processing container.
Two ways to turn it on
The entry point, the training code, and the data channels can stay put. What you change is the image the job points at. That is a job-definition change, and it is the whole switch.
Path one is a pre-indexed AWS Deep Learning Container, marked with a -soci tag suffix. AWS says variants are available for PyTorch Training, TensorFlow Training, vLLM, and SGLang. One PyTorch example published with the announcement is 763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.9-gpu-py312-cu130-ubuntu22.04-sagemaker-v1-soci. Catalog tags move. Confirm the current suffix in ECR before you paste an example URI into a production job.
Path two is your own image. The index describes one exact image. If a Dockerfile starts from a SOCI-tagged Deep Learning Container and then adds a layer, the result is a new image, and the base index does not come with it. Index the image you actually push, and store that index in the same private ECR repository. AWS documents a manual build and an automated path that runs after an image push. The install steps belong in those docs.
To confirm a tag is indexed, look for a child manifest whose artifact type begins with application/vnd.amazon.soci.index. Remove the index and the image goes back to ordinary pulls.
Where the shortcut does not apply
The lazy-load path in this announcement is for SageMaker AI training and processing jobs. vLLM and SGLang appear as pre-indexed image families, not as workloads in the training tests above. Do not read those tags as a measured result for inference endpoints.
Eligibility is narrower than “any container SageMaker can run.” The image has to live in a private Amazon ECR repository. Images from other registries are out. So are images pulled through a private Docker registry when TrainingRepositoryAccessMode is set to Vpc.
AWS states that the capability is available across AWS Regions. That is the announcement’s claim, not an independent check. GovCloud, China, and opt-in regions often diverge from a blanket availability line, so confirm the region you actually run in.
The announcement did not publish a separate price or a pricing change. You still pay for the instance time the job consumes. The bet is that less of that time is spent waiting on a pull.
There is a runtime catch worth a look before you retag anything clever. SOCI-indexed images on SageMaker run on a containerd-based runtime. Scripts that call the Docker CLI, read /var/run/docker.sock, or derive IDs from cgroup paths need a review. SageMaker mounts a /.dockerenv marker so older environment checks still see a container. That file is a compatibility shim. New code should not grow a dependency on it.
A sketch of the bill, not a promise
AWS published no dollar figure. The time arithmetic is still worth doing on your own invoice, and it should stay labeled as a sketch. It was not part of the published training tests.
Use one comparison, the 13.2 GB row: 3 minutes 42 seconds for a full pull, about 6 seconds to a lazy start. The gap is a bit more than three and a half minutes per pull. Count pulls, not jobs in the console. A multi-node job can pull the image on every node, so the invoice moves with instance-pulls.
Two hundred pulls at that gap is on the order of 12 hours of instance time not spent unpacking an image. Three hundred is on the order of 18 hours. Multiply by the price of the instance you actually run. A three-day pretraining job will barely notice. A short processing job, an evaluation pass, or a small fine-tune will, because the download is a much larger share of the job.
Questions worth settling before you retag
-
Will a SOCI index make training itself faster?
Not on the evidence AWS published. After the first epoch, epoch time matched ordinary variation. The saving is earlier startup, minus a 3-16 second first-epoch penalty in those tests, and the jobs still finished 1-6 minutes sooner. A workload that touches most of the image immediately can eat the gain.
-
Do I have to change my training script?
The model code, entry point, and data channels can stay. The job definition changes in one place: the image reference. Point it at a -soci tag, or at your own indexed image.
-
What about scripts that talk to the Docker socket?
Review them before you retag. These images run on a containerd-based runtime. Calls to the Docker CLI, reads of /var/run/docker.sock, and IDs built from cgroup paths are the ones that break. The /.dockerenv file is a compatibility marker, not something new logic should depend on.
-
What if the index is missing or lazy loading fails?
SageMaker falls back to a full download, so a missing or bad index is not supposed to stop the container from launching. That is launch behavior only. It does not insure the rest of the job against a runtime mismatch.
-
Does a custom image pick this up from an AWS base image?
No. Derived images do not inherit the index. The final image you push has to be indexed itself.
-
Does a distributed job pull the image once?
Do not assume that. Each instance that starts the container can pull. The time saved scales with instance-pulls, not with the count of jobs in the console.
-
Is the 90 percent figure a guarantee?
No. It is AWS’s benchmark claim for GPU idle time during image pulls, stated as up to 90 percent. A job that preloads most of its image may see little net benefit, and a 1.8 GB image only saves well under a minute.
-
Which registries qualify, and does this cover inference?
Private Amazon ECR only. Other registries are excluded, as are pulls through a private Docker registry when training repository access mode is Vpc. The lazy-load path described here is for training and processing jobs, not a measured result for inference endpoints.
A lot of the training bill is logistics. SOCI does not make the model smarter. It stops a multi-gigabyte suitcase from being unpacked in full before anyone is allowed to find the keys. For teams whose week is a pile of short jobs, that is the kind of boring speed that shows up on the invoice.