AI COGS Problem: SaaS Gross Margins Compress, but "Blocking and Tackling" Optimizations Begin to Drive Recovery

AI COGS Problem: SaaS Gross Margins Compress, but "Blocking and Tackling" Optimizations Begin to Drive Recovery

As enterprise software companies aggressively integrate artificial intelligence, the structural shift in cost of goods sold (COGS) is rewriting traditional SaaS unit economics. Historically, B2B SaaS companies enjoyed gross margins of 80% or higher. However, the heavy compute requirements of generative AI initially compressed AI-product gross margins down to the 45-50% range.¹

By mid-2026, a clear counter-trend has emerged. According to the July 2026 edition of ICONIQ's "State of AI: The Builder's Economy" report, which surveyed over 300 software executives, average gross margins on AI products are projected to expand from 45% in 2025 to 53% in 2026, and reach 59% by 2027. This recovery is not driven by magic, but by aggressive "blocking and tackling" optimizations at the engineering level.

Key Drivers of AI Unit Economic Recovery

To combat the AI COGS squeeze, software builders are implementing highly structured cost-containment strategies:

1. Model Routing and Multimodal Architectures: Rather than routing all queries to the most expensive proprietary models (such as GPT-4 or Claude 3.5 Sonnet), companies are deploying routing layers that direct simpler tasks to cheaper, lightweight, or open-source models (like Llama 3 or Claude Haiku). According to the report, companies now use an average of 3.3 model providers to optimize performance and cost.
2. Aggressive Query Cost Optimization: Engineering teams are actively optimizing prompt lengths, caching frequent responses, and fine-tuning smaller models to achieve equivalent accuracy at a fraction of the inference cost. The data shows that 66% of respondents improved their cost per query by 10% or more, with 19% achieving improvements of 30% or more.
3. The Rise of Open-Source and Distillation: As open-source models close the capability gap with proprietary giants, enterprises are shifting workloads to self-hosted or distilled open-source options, removing the markup charged by commercial API vendors.

Despite these efficiency gains, the underlying economics remain fundamentally different from legacy software. Generating AI-driven outputs involves ongoing variable compute costs, meaning AI products may never reach the 80%+ gross margins of pure software. However, the rapid stabilization of margins in the 50-60% range suggests that software companies can maintain viable business models as AI becomes their primary revenue driver. Indeed, for non-AI-native startups, the revenue mix from AI products is scaling at a blistering pace: representing 32% of total revenue in 2025, 42% in 2026, and projected to reach 53% by 2027.

Verbatim Quotes

From SaaSletter's analysis of the ICONIQ report:

"Looking strictly at customer-facing AI product costs, the 14% projected increase in AI Gross Margins (2025: 45% → 2027: 59%) seems questionable. ... However, the change in the unit economics survey suggests gross margin improvement is supportable: 66% of respondents improved their cost per query by 10%+, with 19% at 30%+." — SaaSletter - ICONIQ "State Of AI" Takeaways

From OnlyCFO's analysis of the report's model strategy:

"We want the best but we are starting to see now that much cheaper models get the job done just as well. Also, some of the anxiety around open-source models is disappearing as companies find solutions to reduce concerns." — ICONIQ Releases 2026 State of AI Report - by OnlyCFO