SageMaker Endpoint Alternatives: Baseten Wins at $1.21/Hour

If you are paying for a SageMaker endpoint that sits idle every night, move it to Baseten and keep one replica warm during working hours. Its A10G costs less than SageMaker's on-demand rate even when it never scales down, and it lists SOC 2 Type II and HIPAA on its free self-serve tier. Use Modal for batch and asynchronous jobs, RunPod when cost matters more than paperwork, and stay on SageMaker if you hold a Savings Plan or your security team requires the model to run inside your own VPC. Replicate loses this comparison.

The trap in every one of these migrations is the same. Scale-to-zero cuts the bill only during gaps longer than the platform's idle window plus its startup time, and the first request after each of those gaps is a cold start that one of your users waits through. For a large model that wait is measured in minutes.

All prices below were checked on each vendor's live pricing page on October 30, 2026.

	SageMaker real-time	Baseten	Modal	RunPod Serverless	Replicate
GPU for a 7-8B model	ml.g5.2xlarge (A10G 24 GB)	A10G 24 GB	A10 24 GB	L4 / A5000 / 3090 tier (24 GB)	L40S 48 GB (no 24 GB option listed)
Warm price per hour	$1.52	$1.21	$1.55 (GPU + 4 cores + 32 GiB)	$0.75	$3.51
24/7 for a month (730 h)	$1,110	$881	$1,129	$548	$2,562
Business hours only (198 h)	$1,110 (stays on)	$239	$306	$149	$695
Default idle window	none (always on)	15 minutes	60 seconds	5 seconds	not documented
Startup time billed	n/a	yes, per minute	per second while the container runs	yes, until handler ready	yes (setup and idle)
SOC 2 / HIPAA	HIPAA-eligible service	SOC 2 Type II + HIPAA on Basic	SOC 2 all plans; HIPAA Enterprise only	Trust center lists both	not stated in deployment docs
Private networking	VPC subnets + PrivateLink	Your VPC / self-host on Enterprise	Static egress IPs only	Not documented	"Private endpoints"
Verdict	Stay if you have a commit or a VPC mandate	Default pick	Batch and async	Cost-first, unregulated	Avoid for this job

---

The Workload We Priced

The comparison uses one shape: a 7-8B parameter model in 16-bit weights on a single 24 GB GPU, which is the configuration AWS itself prices in its own example, a Llama 2 7B served from an ml.g5.2xlarge at $1.52 per hour in US East. Every vendor here sells a GPU that fits it except one.

The line items, each from the vendor's own page:

• SageMaker's ml.g5.2xlarge is $1.52 per hour on demand, which is $1,109.60 for a 730-hour month.
• Baseten's A10G is $0.02012 per minute, or $1.21 per hour with the VM's CPU and memory included. That is the 4 vCPU, 16 GiB instance, half a g5.2xlarge's CPU and memory; the 8 vCPU, 32 GiB A10G is $0.02424 per minute, or $1.45 per hour.
• Modal's A10 is $0.000306 per second ($1.10 per hour), plus CPU at $0.0000131 per physical core per second and memory at $0.00000222 per GiB per second. Sized like a g5.2xlarge (4 physical cores, 32 GiB), that adds $0.44, for $1.55 per hour.
• RunPod's 24 GB serverless tier (L4, A5000, 3090) is $0.75 per hour. None of those is an A10G, so benchmark token throughput on your own model before you treat the hourly rates as equivalent.
• Replicate's pricing page lists a T4 (16 GB, too small for a 7B model in 16-bit weights) and then jumps to an L40S at $3.51 per hour.

Two Modal multipliers belong in the budget. Pinning a function to a region costs 1.15x for a broad region such as us and 1.75x for a narrow one, and most compliance reviews will ask you to pin. At 1.15x the warm rate is $1.78 per hour, or about $1,298 a month.

Is Scale-to-Zero Cheaper Than a Warm SageMaker Endpoint?

Only for the hours nobody calls the model, and only when the gaps are long. At warm-to-warm parity, Modal is within 2% of SageMaker's on-demand price, Baseten is 21% cheaper, and RunPod is about half. The large savings in the table come from not billing nights and weekends.

Two traffic patterns show where that holds and where it breaks.

Pattern A is an internal tool used 9 hours a day on 22 weekdays, 198 busy hours a month. Every platform that scales to zero wins easily. Baseten bills about $239, Modal $306, RunPod $149, against $1,110 for an always-on SageMaker endpoint. One cold start lands on the first user each morning, plus one after any long lunch gap.

Pattern B is a webhook that fires once every 20 minutes, around the clock, and it is the one that surprises people. Assume a 2-minute startup for the 7B container (your own number will differ; measure it). On Baseten's defaults, each request starts a replica, which then waits out the 900-second scale-down delay before stopping. You are billed for about 17 of every 20 minutes, roughly 620 hours or $749 a month, and every single request is still cold because 20 minutes is longer than 15. On Modal's 60-second default idle window the bill drops to about $169, and again every request is cold. Stretch the window to Modal's 20-minute maximum to stop the cold starts and you are back to a warm container at about $1,129.

The crossover rule that falls out of this: a gap between requests saves money only when it is longer than the idle window plus the startup time, and each such gap produces exactly one cold request. If your traffic has many short gaps, scale-to-zero gives you the cold starts without the savings. If it has a few long ones, it is the right call.

How Long Is a Cold Start for a Large Model?

For anything above about 10B parameters, plan for one to eight minutes without snapshotting, and treat any single-digit-second figure as a claim about small models or cached containers. The numbers below separate vendor claims from independent tests.

Source	Model and GPU	Cold start	Who measured
vllm-serverless on GitHub	Qwen 27B on A100 80GB, Modal	460 s baseline, 70 s with vLLM sleep mode + GPU snapshots	Independent user project
AWS Fast Model Loader post	Llama 3.1 70B on ml.p4d.24xlarge	Weights loaded in about 1 minute; adding a model copy on a new instance averaged 306-407 s end to end across AWS's runs	AWS, internal testing
Modal Ministral 3 post	Ministral 3 3B, GPU not named	118 s median, 12 s with GPU snapshots (alpha)	Modal
serverlessgpubench.com	Model not named on the page	Modal 1.8 s, RunPod 4.2 s, Replicate 6.5 s (medians, 500 runs)	Unnamed operator; page carries affiliate links, including RunPod's
RunPod Serverless page	Not stated	"Sub-200ms with FlashBoot"	RunPod

The independent 27B result is the one to plan around. It shows that GPU memory snapshots cut a large-model cold start by about 6.5x, and that the result after the cut is still over a minute. The README also warns that the snapshot only settles after three to five cold invocations. Baseten publishes no cold-start figure at all; its cold-start documentation describes image streaming, weight caching through its delivery network and compile caching, treats a replica as stalled only after 30 minutes, and tells you to set min_replica to 1 or higher to keep startup off the request path.

SageMaker has had its own scale down to zero since re:Invent 2024, and it behaves worse for interactive traffic. It works only with inference components, and while capacity scales out from zero, requests fail with NoCapacityInvocationFailures instead of waiting. Your client has to queue and retry. It suits an internal batch caller and does little for a user-facing endpoint.

---

Baseten: The Default Pick

Baseten is the closest like-for-like replacement for a SageMaker real-time endpoint. Its A10G at $1.21 an hour beats SageMaker's on-demand rate with no scaling tricks at all, so the move pays even if you never set min_replica to zero. The pricing page lists "SOC 2 Type II and HIPAA compliant" on the free Basic tier, and the Enterprise tier adds deployment into your own VPC, hybrid, or fully self-hosted so you can burn down your own cloud commitments.

Packaging is Truss, which is MIT-licensed and describes itself as running on Baseten "or your own infrastructure", so your exit cost is moderate.

What to watch: startup time is billed per minute even though the replica serves nothing, the default max_replica is 1 (a launch-day traffic spike queues instead of scaling until you raise it), and the 15-minute scale-down delay is the reason Pattern B costs what it does. Shorten it for spiky traffic.

Baseten is the wrong pick for a team that needs the model inside its own VPC but cannot sign an Enterprise contract. The VPC and self-host options are not on the self-serve plan.

Modal: Best for Batch, Weaker for Regulated Serving

Modal is the best developer experience here and the right home for asynchronous work: nightly embedding runs, evaluation sweeps, document backfills, anything where a 60-second idle window and per-second billing beat a reserved instance. Warm, it costs the same as SageMaker.

Three documented limits keep it out of the default slot for a regulated production endpoint:

1. GPU functions are preemptible and you cannot opt out. Modal's preemption guide says all functions are preemptible by default and the nonpreemptible flag "is not supported for GPU Functions." A serving replica can be interrupted, so your client needs retries.
2. The docs describe no private link to your VPC. The only mechanism is Proxies with up to five static egress IPs for allow-listing, available on Team and Enterprise. VPC peering and PrivateLink are not mentioned.
3. HIPAA is Enterprise-only, and the cold-start fix is out of scope for it. The security page limits HIPAA to Enterprise (the pricing page lists SOC 2 on all plans), and it says Memory Snapshots should not be used for PHI. A PHI workload loses the feature that makes Modal's cold starts tolerable.

Lock-in is higher than the others. Your images, functions and scaling settings are defined in Modal's Python SDK, so leaving means rewriting the deployment layer, and egress is $0.04 per GiB. Modal's own Substack case study describes a team that left SageMaker for developer speed and gives no dollar figure. It is vendor marketing, though the reason it gives is a believable one.

Modal is the wrong pick for anyone serving PHI on a Team plan, anyone whose network team requires private connectivity rather than IP allow-lists, and anyone whose client cannot retry a preempted request.

RunPod: Cheapest per Hour, Lightest on Enterprise Controls

RunPod is the cheapest way to put a 24 GB GPU behind an endpoint, at $0.75 an hour for the tier that includes the L4, and its serverless billing is the most transparent here. You pay from worker start until it fully stops, rounded up to the second; the 5-second default idle timeout is billed; pulling the image and caching the model happen while scaled down and are not billed. Discounted always-on "active" workers are available only through sales.

The trust center lists SOC 2 Type 2, SOC 3, HIPAA and ISO 27001, with some documents behind approved access. It does not say which of RunPod's cloud tiers those reports cover, so ask for the scope in writing before any regulated data goes near it. Private networking into your VPC is not documented. Packaging is a standard Docker image plus a handler function, so the exit cost is low.

RunPod is the wrong pick for a team whose vendor review needs documented private networking, or one that cannot get the report scope confirmed in writing.

Replicate: The Loser for This Job

Replicate loses for a SageMaker migration because it is the most expensive per hour and the weakest on enterprise documentation. The 7B workload lands on an L40S at $3.51 an hour, more than double SageMaker's on-demand rate, because the pricing page lists no 24 GB GPU. Private models on dedicated hardware bill for setup, idle and active time, so scale-to-zero on a private deployment still charges for the boot.

The deployments documentation mentions private endpoints and audit logging but says nothing about SOC 2, HIPAA or VPC connectivity. Ownership also changed: Replicate joined Cloudflare in a deal announced in November 2025, and a vendor being folded into a new parent is a weak bet for a three-year platform decision. Its packaging tool, Cog, produces a standard Docker image that runs "anywhere that Docker images run", which is the best thing about it.

Replicate is the wrong pick for anyone moving a private, steady production model. It suits public model APIs and prototypes.

Who Should Stay on SageMaker

Stay if any one of these is true, because each one erases the price gap.

• You have a SageMaker Savings Plan or an AWS commit to burn down. SageMaker Savings Plans cover Real-Time Inference at up to 64% off on one- or three-year terms. AWS does not say every instance reaches that ceiling, but if the g5.2xlarge got the full discount it would be about $0.55 an hour, below every warm price in the table. If your EDP drawdown depends on SageMaker spend, moving it off AWS costs you twice.
• The model must run inside your VPC. SageMaker attaches the container to your own subnets and security groups through VpcConfig, with S3 and other AWS services reached over VPC endpoints. Only Baseten's Enterprise tier matches that.
• Your BAA is with AWS. SageMaker AI is on the HIPAA-eligible services list, so your existing agreement already covers it.

SageMaker's lock-in sits in IAM, networking and the commit. The container itself moves easily: the hosting contract is a plain web server on port 8080 answering /invocations and /ping, which ports to Baseten, RunPod or Cog with modest work.

The Criteria That Predict Regret

The regret in this migration comes from the request that arrives after a long gap, which a price comparison never shows. These questions change the answer:

1. What is your p99 startup time on the target platform, for your container? If it is over 30 seconds, every gap longer than the idle window is a timeout risk for a synchronous caller.
2. What does your gap distribution look like? Pull a week of invocation timestamps from CloudWatch and count gaps longer than idle window plus startup. That count is your cold-start count, and the summed excess is your savings.
3. Can your client retry? Modal preemption and SageMaker's scale-from-zero failures both need it.
4. Who signs the BAA, and does it cover the features you need? Modal's snapshots are out of PHI scope; RunPod's report scope is unstated.
5. Is your SageMaker spend counting toward a commit? If yes, model the commit shortfall before you model the savings.

For background on the per-hour numbers, see our GPU hour pricing breakdown, and for how a cold start becomes a product decision, the Nebius and Inferize piece.

What to Do Next

This Week:

1. Export a week of Invocations for the endpoint from CloudWatch and compute the gap distribution. If fewer than 10% of hours have traffic, the move is worth doing.
2. Check whether the endpoint is under a SageMaker Savings Plan or counted toward an AWS commit. If it is, stop here and price the Savings Plan instead.

This Month:

1. Deploy the same container to Baseten with min_replica: 1 and measure cold-start p50 and p99 by scaling a second replica from zero ten times. Write the number down.
2. Set a scheduled floor: min_replica 1 during working hours, 0 overnight, raised ahead of known spikes by your p99 startup time, as Baseten's own docs suggest.

Before Q1 Close:

1. Move batch and evaluation jobs to Modal, where per-second billing fits and preemption is tolerable.
2. Decommission the SageMaker endpoint only after a month of side-by-side latency data from your real callers.

The Bottom Line

Baseten and RunPod are cheaper than SageMaker's on-demand rate by the hour, and Modal is level with it once CPU and memory are counted. The scale-to-zero savings are real for traffic that stops for hours at a time, and they work by moving latency onto the first user after each gap. For a 7B model that is a minute or two; for a 27B model an independent test measured 460 seconds without snapshots and 70 with them. Price the warm replica first, then decide how many of your users you are willing to make wait.

Continue Reading

• GPU Hour Pricing: A $3.44 Rate Cost $68.80 to Use
• Nebius Acquires Inferize to Cut GPU Cold Starts
• E2B vs Modal vs Daytona: Idle Time Sets Sandbox Bills
• Together AI and Bedrock beat self-hosted gpt-oss-120b on cost
• Reserved vs Spot GPUs: Commit Six Months, Not Twelve
• LLM Data Residency: Which Providers Actually Keep Data In Region
• vLLM vs TensorRT-LLM vs SGLang: Default to vLLM