Signal vs. Noise: Why Your Gross Margins Are Hemorrhaging in 2026 (The Hidden AI Tax)

TL;DR

• 84% of enterprises are watching gross margins erode by 6% or more due to AI infrastructure costs. For companies with heavy AI workloads, that hit reaches 16%, translating to $12M+ in lost EBITDA on a $200M PortCo.
• Most companies have zero visibility into their actual AI costs. 50% don’t track LLM API costs. 75% can’t forecast spend within ±10%. This invisible cost structure is eroding your exit multiple.
• The problem isn’t AI itself. It’s that your infrastructure wasn’t built for scale. Inference costs spike exponentially as usage grows, but most finance teams treat AI like a fixed SaaS license, not a variable cost of goods sold.
• The fix isn’t complex, but it requires immediate action: Reclassify AI infrastructure costs to COGS, set real-time cost alerts, and establish cost-per-inference targets before you deploy another model.

Your First Reality Check of 2026

You’ve spent the last 12 months telling your board that AI is the path to efficiency and margin expansion. Now the bill is coming due, and it’s hitting your P&L in ways your CFO didn’t model.

Here’s the fact that should keep you up at night:

84% of your peers are watching their gross margins disappear into the “black hole” of AI infrastructure costs.

source: (Mavvrik, 2025 State of AI Cost Governance Report)

For a $200M revenue PortCo with 50% gross margins, a 6% hit isn’t a rounding error. It’s $12M in lost EBITDA. For companies with heavier AI workloads, that margin erosion reaches 16%. That’s $32M gone…

This is the “Hidden AI Tax,” and most mid-market leaders are paying it without realizing they have a choice. It’s simple, too. Keep reading to find a 30-day audit and the only two paths forward in 2026 for PE-backed firms struggling with AI.

Why This Is Happening (And Why Your Finance Team Missed It)

The root cause is deceptively simple. You’re treating AI infrastructure like a fixed cost when it’s actually a variable cost that scales with usage.

Here’s how the math breaks down:

The Invoice Problem

When you deploy an LLM, it doesn’t matter if it’s OpenAI’s API, Claude, open-source models, etc., on your own infrastructure; every single inference (every API call, every token generated) costs money. Not upfront. Not monthly. Per. Use.

Who’s finding this out today? DM me, we have a LOT to discuss.

This is fundamentally different from a SaaS license, where you pay $10K/month regardless of whether your team uses it once or 1,000 times.

With AI inference, your cost structure looks like this:

• Base infrastructure cost: $50-$200K/month for your data pipeline and integration layer (depending on how siloed your current systems are)
• Per-inference cost: $0.001-$0.10+ per API call, depending on the model and volume
• Data transfer cost: Network fees to cloud providers
• Compute cost: Running inference at scale (especially if you’re using GPU-based models)

When you have 100 customers using your AI feature, the cost is reasonable. When you have 10,000 customers, and each customer generates 10 inferences per day, you’re suddenly burning through $50K-$100K+ per week in tokens alone.

And most CFOs don’t see this coming because it’s not in their original budget model.

The Visibility Catastrophe

Here’s what the data shows:

• 50% of companies with AI-core products don’t include LLM API costs in their P&L reporting at all. (Mavvrik, 2025 State of AI Cost Governance Report) They’re treating inference costs as a feature development expense, not a cost of serving the customer.
• 15% of companies can forecast their AI spend within a ±10% range. (USM Systems, AI Project Cost Estimation: 2026 Pricing Breakdown) The rest? They’re blindsided monthly.
• 24% of leaders miss their AI cost forecasts by over 50%. (Monetizely, The Economics of AI-First B2B SaaS in 2026) Some miss by 80-100%.

This isn’t incompetence. It is a structural problem. Your finance systems and your engineering systems don’t talk to each other in real-time.

Your engineering team knows how many inferences they ran last week. Your CFO doesn’t. By the time the AWS or OpenAI bill arrives, it’s too late to course-correct.

The Compounding Problem

Things can always get worse… And this situation is no outlier. These costs don’t stay static. They compound.

56% of unexpected AI costs come from data platform usage escalation. (Mavvrik, 2025 State of AI Cost Governance Report) As your data grows, as your models run more frequently, as you add customers, the cost per query, the cost per training run, the cost per inference soars.

And most companies don’t have real-time cost alerts. So they don’t see the spike until it shows up in the monthly invoice. By then, the damage is done. Your gross margin for that month (and often the next 3-4 months) is already hit.

The Math That Should Alarm You

Let’s ground this in a real scenario.

On $16.7M/month revenue (200M/12), with 50% gross margins, you have $8.3M in gross profit. At month 1, AI costs are 1.2% of gross profit. At month 12, AI costs are 3.1% of gross profit.

If you have multiple AI features running at scale, if your data pipelines are inefficient, if you’re not optimizing model selection, that 3.1% becomes 6%, 8%, 12%. And suddenly, a 50% gross margin product becomes a 42% gross margin product.

For a $200M PortCo, that’s $16M in lost EBITDA.

The Margin Impact (The Numbers You Need to Know)

Here’s what the data shows about margin erosion:

• 84% of enterprises report AI infrastructure costs are eroding gross margins by 6% or more. (Mavvrik, 2025 State of AI Cost Governance Report)
• For companies with heavy AI adoption (more than 50% of customers using AI features), that margin hit reaches 16%. (PwC, 2026 AI Business Predictions)
• For some companies in data-heavy verticals (Healthcare, FinTech, Manufacturing), it reaches +20%. (USM Systems, AI Project Cost Estimation: 2026 Pricing Breakdown)

Why such a wide range? Because it depends on:

1. Model choice: GPT-4 is expensive. GPT-4 Turbo is cheaper. Open-source models on your own infrastructure are cheapest but require engineering overhead.
2. Usage patterns: How many inferences per customer? How long are the prompts? How complex are the queries?
3. Data infrastructure: Are you running efficient pipelines? Or are you re-running data prep jobs inefficiently?
4. Scale: At 100 customers, the margin hit is small. At 10,000 customers, it starts to get devastating.

What You Need to Do Immediately (The 30-Day Audit)

If you’re a Managing Director or C-Suite executive reading this, here’s what you need to do in the next 30 days:

Step 1: Get Visibility (Week 1)

1. Pull your last 12 months of cloud bills (AWS, Azure, Google Cloud, OpenAI, Anthropic, etc.)
2. Ask your engineering team: “What percentage of this bill is AI-related?” (Most will say “I don’t know.” That’s the problem.)
3. Create a simple spreadsheet: Month, Total Cloud Cost, Estimated AI Cost, AI Cost as % of Gross Profit

Do this for the last 12 months. You’ll see the trend. Put together your sheet, analyze it. See nothing? DM me, I’ll clear it up for you.

Step 2: Classify AI Costs as COGS (Week 2)

1. Work with your CFO to move AI infrastructure costs from OpEx/R&D to COGS.
2. This changes your reported gross margin. It will be lower than you thought.

But now you’re seeing reality, and reality is the only thing that can fix this.

Step 3: Set Up Real-Time Cost Alerts (Week 2-3)

1. With your engineering team, set up cost monitoring in your cloud provider (AWS Cost Explorer, Azure Cost Management, etc.)
2. Create alerts: “If inference costs spike 10% month-over-month, alert the CFO and CTO immediately.”
3. Establish a cost-per-inference target for each product: “Our AI feature should cost us no more than $0.05 per inference.”

Step 4: Calculate Your Margin Hit (Week 3)

1. Take your new COGS classification.
2. Calculate: (AI Costs / Total Gross Profit) × 100 = Your AI Margin Hit

If that number is above 5%, you have a problem. If it’s above 10%, you have a crisis.

Step 5: Have the Board Conversation (Week 4)

1. Present the data: “Our AI infrastructure is costing us X% of gross margin.”
2. Frame the question: “At exit, how much margin are we willing to sacrifice for AI adoption?”

Because that’s what you’re doing right now. You’re trading margin for adoption.

Your Decision Point

You have two paths forward:

Path 1: Do Nothing

• Keep treating AI like a fixed cost
• Watch margins erode 1-2% per quarter as usage scales
• Hit your exit with a 42% gross margin instead of 50%
• Lose $96M in enterprise value

Path 2: Act Now

• Reclassify AI costs to COGS
• Set real-time cost alerts and optimization targets
• Optimize model selection, prompt engineering, and data efficiency to reduce cost-per-inference
• Adjust pricing to reflect the true cost of serving AI customers
• Hit your exit with a 48-50% gross margin, protected and defensible

The choice is yours. But the math doesn’t lie.

Your Burning Question

If your AI infrastructure costs spiked by 50% tomorrow, would your product still be profitable? Or are you just subsidizing your customers’ curiosity at the expense of your exit multiple?

If you can’t answer that with confidence, you need to have the conversation with your CFO this week. I work with Managing Directors and CFOs at mid-market PortCos to audit their AI cost structure and build defensible margins into their AI roadmap.

Ready to quantify your AI Margin impact? Book a 30-minute AI Margin Audit to see exactly what your AI infrastructure is costing you, and what you should be targeting instead.

Sources

• Mavvrik: 2025 State of AI Cost Governance Report
• USM Systems: AI Project Cost Estimation: 2026 Pricing Breakdown
• PwC: 2026 AI Business Predictions
• Prismetric: AI Development Cost: A Comprehensive Overview for 2026
• Monetizely: The Economics of AI-First B2B SaaS in 2026
• EverWorker: AI Strategy Best Practices for 2026: Executive Guide