How I evaluate AI startups for investors: 8 technical signals

TL;DR

Before a full technical diligence, I screen an AI startup on eight signals: cost per request at 10x volume, how much of the product is someone else’s API, where the data comes from and how its quality is measured, why the approach is hard to copy, whether the team can both research and ship, whether there is real production traffic, how the team measures quality, and, for agents, what an attacker could make the product do. A startup that fails several of these is not ready for the full review.

Why AI startups need their own screen

Ordinary software diligence asks about code quality, architecture and scaling. AI startups need all of that, plus a harder question: is the intelligence real, and will it stay theirs?

Many AI products fail between demo and production. Gartner predicted in 2024 that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, weak risk controls, rising costs and unclear business value. MIT NANDA’s 2025 report on enterprise adoption found that 95% of the organizations it studied saw no measurable return from their generative AI pilots. Those numbers describe buyers, not startups, but they point at the same causes: cost, data, evaluation and risk. Most of them are visible early to someone who knows where to look.

The screen below takes about an hour with the technical founder. It does not replace a full review. It decides which companies deserve one.

Signal 1: does the cost per request survive 10x volume?

This comes first because it disqualifies fastest. Ask: “What does it cost to serve one request today, and what does that become at ten times your current volume?”

Inference is a direct cost of every sale, so AI products usually run lower gross margins than classic SaaS. Model prices keep falling, but usage per customer tends to rise faster as products add longer contexts, more agent steps and more retries. For illustration: 10,000 requests a day at one cent each is about $3,000 a month; at 100,000 requests a day it is about $30,000. Does the price per customer still cover it? For voice products, ask the same question per minute of conversation, because speech recognition and synthesis add their own per-minute costs.

Green flag: A cost model the founders can walk through, with cost per request or per minute, projections at 10x and 100x, and a plan to reduce it: routing simple requests to smaller models, caching, shorter prompts.

Red flag: No answer, “we’ll optimize later”, or a model that assumes today’s API prices and today’s usage per customer will both hold.

Signal 2: how much of the product is someone else’s API?

Using a provider’s API is fine. Being nothing more than a provider’s API is not. If the main provider raises prices, changes its terms or ships the same feature, what is left?

Dependency	Risk	Example
API as one input to a proprietary system	Low	An LLM summarizes inside a pipeline built on the startup’s own data
API as the core intelligence, with real assets around it	Medium	A fine-tuned model on provider infrastructure, with a proprietary evaluation set
API as the whole product	High	A UI and prompt templates over a general-purpose model

Green flag: Proprietary data, fine-tuned models, domain evaluation or infrastructure that would take a competitor a long time to rebuild, and a clear answer to “what survives if your provider disappears?”

Red flag: Removing the API leaves a login page. The differentiation is prompt engineering.

Signal 3: where does the data come from, and how good is it?

Gartner’s list of reasons for abandoned projects puts data quality first. I ask three things.

Where does the data come from? Data collected through the product, from domain experts or careful annotation, is an asset. Scraped public data is not, because everyone else can scrape it too. The strongest companies have a loop where using the product produces data that improves the model.

How is quality measured? Annotation guidelines, agreement between annotators and validation checks show a team that knows what it has. “We have a lot of data” without quality numbers usually means a lot of noise.

Does production feedback flow back? A product that never learns from its own usage degrades as the world changes around it.

Signal 4: why is the approach hard to copy?

Not every AI startup needs a novel model. Every one needs a convincing answer to “why can’t a well-funded competitor, or the model provider itself, build this in six months?”

Green flag: Domain knowledge encoded somewhere hard to copy: a fine-tuned model, a proprietary evaluation set, a data flywheel, integrations with systems of record. Benchmarks on the startup’s own task that beat strong general-purpose baselines.

Red flag: The team cannot explain the architecture in plain language, uses the same approach as several competitors, or compares itself only against weak baselines on tasks unlike its product.

Signal 5: can the team both research and ship?

Teams made only of researchers build impressive demos that stall before production. Teams made only of product engineers ship fast and have nothing that is hard to copy. The companies that last usually have someone who can judge the research frontier, ideally with papers or patents in the area, alongside people who have run ML systems in production.

Green flag: In the same conversation, the team can explain the relevant research and walk through how they deploy, monitor and roll back models.

Red flag: Nobody has run a model in production, or nobody reads papers, or the expertise sits with an outside “AI advisor” instead of inside the team.

Signal 6: is there real production traffic?

The gap between demo and production is where most AI products die. This signal takes five minutes: ask to see production metrics.

Green flag: Real traffic, error rates, latency percentiles, a monitoring dashboard and customer feedback. For voice products, latency measured end to end on real calls, not a component benchmark.

Red flag: The demo runs on a laptop. There are no production metrics because there are no production users. A curated example set stands in for real-world results. Pilots that never convert to contracts usually mean something that worked in the demo fails in production.

Signal 7: how does the team measure quality?

Investors skip this signal most often, because it needs technical depth. It predicts product quality better than almost anything else.

Green flag: A held-out test set that is never used for training or prompt tuning. Metrics that match what customers care about: task success, latency, cost per query and known failure modes, not a single accuracy number. Regular evaluation runs with a history the team can show. For voice products, evaluation on real audio with real accents and phone-quality calls.

Red flag: One number on a public benchmark that does not match the product. Quality judged by someone reading a few outputs. No answer to “how would you know if the last change made things worse?”

Signal 8: for agent products, what could an attacker make it do?

This signal is new, and it matters more every quarter. An agent that can call tools turns text into actions: sending emails, editing records, issuing refunds, placing calls. If untrusted content can reach it (a customer email, a web page it reads, a document it summarizes, a caller’s speech), someone outside the company can try to steer it. That is prompt injection, and no current model is immune to it.

Ask three questions. Which tools can the agent call, and with what permissions? Can untrusted content reach the agent that holds those tools? What stops an irreversible action: an approval step, limits, or nothing?

Green flag: Tools scoped to what each task needs, untrusted input separated from privileged actions, human approval or hard limits on irreversible steps, and tool-call logs good enough to reconstruct an incident. Bonus: the team has tested its own product against injection and can show the results.

Red flag: One agent with broad credentials reads the open internet or customer messages and can act on them without approval. Nobody on the team can explain how they would detect or contain a misused tool call.

Putting it together

flowchart LR
    S1[1. Cost per request] --> S2[2. API dependency]
    S2 --> S3[3. Data]
    S3 --> S4[4. Hard to copy]
    S4 --> S5[5. Team]
    S5 --> S6[6. Production traffic]
    S6 --> S7[7. Evaluation]
    S7 --> S8[8. Agent security]
    S8 --> V{How many pass?}
    V -->|Most| Full[Worth a full review]
    V -->|Some| Maybe[Ask whether the gaps are fixable]
    V -->|Few| Pass[Pass for now]

What goes wrong later	Signal that shows it early
Margins collapse as usage grows	1. Cost per request
A provider change wipes out the product	2. API dependency
Model quality stops improving	3. Data
Competitors or the provider catch up	4. Hard to copy
The demo never becomes a product	5. Team, 6. Production traffic
Quality claims that do not hold	7. Evaluation
A security incident through the agent’s tools	8. Agent security

Key takeaways

• Start with cost per request at 10x volume. It disqualifies fastest.
• Ask what survives if the main model provider changes prices or ships the same feature.
• Proprietary data with measured quality beats a clever architecture on scraped data.
• Look for a team that can both judge research and run production systems.
• Ask for production metrics and the evaluation history, not a curated demo.
• For agent products, find out what an attacker could make the agent do, and what stops it.

FAQ

What is the most common technical red flag in an AI startup?

A product that is a thin layer over one model provider’s API with no proprietary data, evaluation or infrastructure underneath. If the provider changes prices or ships the same feature, there is nothing left to defend.

How long does technical due diligence on an AI startup take?

The screen in this post takes about an hour with the technical founder. A full review, with code access, infrastructure and data inspection and interviews, takes days. The screen decides whether the full review is worth doing.

How do you tell whether an AI startup is just an API wrapper?

Ask what happens if the main model provider raises prices fivefold or ships the same feature. Ask to see the data pipeline and the evaluation set. If the differentiation lives only in prompts, it is not a moat.

Why include security in a due diligence screen?

Agent products turn text into actions: sending emails, changing records, moving money. If untrusted content can reach the agent and its tools have broad permissions, one prompt injection can become a customer incident. That risk belongs in the valuation.

What team shape should an AI startup have?

At least one person who can judge research, ideally with papers or patents in the area, and people who have run ML systems in production. Pure research teams demo well and ship slowly. Pure product teams ship quickly with nothing hard to copy.

Further reading

• Fractional CTO playbook: week one: the same checks, done from the inside
• AI agents that actually make money: judging whether an agent business model works
• AI security assessments: evaluating an LLM system you did not build
• Agent evaluation frameworks: what a serious evaluation setup looks like
• Build vs buy for AI in 2026: where proprietary work is worth it

If you are looking at an AI, voice or agent startup and want a second opinion before committing, this is how my due diligence works.