Skip to content

What is the Cost of Maintaining the Correctness of a GenAI Service? (268)

November 25, 2025 · 46 min

Show notes

This week’s podcast is about estimating the costs of providing GenAI products and services. And this is changing.

You can listen to this podcast here, which has the slides and graphics mentioned. Also available at iTunes and Google Podcasts.

Here is the link to the TechMoat Consulting.

Here is the link to our Tech Tours.

My approach to assessing GenAI operating costs is the below checklist.

  1. What is the cost of compute, including energy and cooling? Initial cost vs. ongoing?
    1. The core compute is going to drive a lot of the costs. Especially if you are using an AI cloud service provider. If you have a downloaded open-source model, then it’s mostly the initial cost.
    2. These compute costs dependent on the requirements of the AI workloads:
      1. The compute requirements
      2. The timing requirements
      3. The memory requirements
  2. What is the cost of creating and maintaining the desired correctness over time?
    1. How accurate and correct you need the foundation model to be is a big deal. And this factor determines how much of the operations are done by software versus humans. It can make the costs of GenAI products look a lot more like services than traditional software.
    2. My main questions are:
      1. How much does correctness matter? What level of correctness does the product need to compete? What is the cost of inaccuracy?
      2. How does correctness change over time? Is it stable and flat lining, advancing, naturally changing or deteriorating?
      3. How much of a long tail is there in the domain? Is the long tail a liability or a benefit?
      4. How much of the process is iterative? How much are humans in the loop?
      5. What is the initial cost of training and getting to the desired level of correctness?
      6. What is the ongoing cost of inference at the desired level of correctness? How much are humans in the loop?
      7. What is the cost and frequency of fine tuning and/or retraining?
  3. What are the cost implications of increasing scale?
    1. What are the scale advantages vs. disadvantages?
    2. Does the Jevon paradox apply?

--------

I am a consultant and keynote speaker on how to accelerate growth with improving customer experiences (CX) and digital moats.

I am a partner at TechMoat Consulting, a consulting firm specialized in how to increase growth with improved customer experiences (CX), personalization and other types of customer value. Get in touch here.

I am also author of the Moats and Marathons book series, a framework for building and measuring competitive advantages in digital businesses.

This content (articles, podcasts, website info) is not investment, legal or tax advice. The information and opinions from me and any guests may be incorrect. The numbers and information may be wrong. The views expressed may no longer be relevant or accurate. This is not investment advice. Investing is risky. Do your own research.

Support the show

All episodes of The Tech Strategy Podcast

Settings

Display

Language

Decides which stations you see first on the home page.

History

Recently played stations and episodes and your latest searches are only stored in this browser.

Notifications

A short notice when a podcast you follow has a new episode. Turn off single podcasts with the bell on their page.

Account

Save favorites – in the app and on the website.

Keyboard shortcuts

/
Open search
Space
Play or pause
← →
In the player: back 15 s, forward 30 s (episodes) · in a row: next tile
↑ ↓
In a row: target within the tile
Home / End
In a row: first or last tile
Esc
Close window or menu
?
This overview
Now playing

Go to station page

Play next

    Add episodes with “Play next” – they play after the current episode.