Fractional CTO playbook: how I assess an AI startup’s technical health in week one

TL;DR

Week one of a fractional CTO engagement is diagnosis, not building. I run five checks in parallel: architecture and failure points at 10x load, code and test health, the deployment path, inference cost per request at today’s and tomorrow’s volume, and who on the team can do what. For agent products I add a sixth: what the agent can reach and who can steer it. The output is a short report with the top risks, the quick wins, and an honest call on whether a longer engagement makes sense.

Why the first week is only diagnosis

In week one I don’t write production code, redesign the architecture or hire anyone. Changes made before I understand the system tend to be confident and wrong. The first week answers one question for the founders: what will hurt this company in the next few months, and what is cheap to fix now?

The checks below run in parallel. Most of them take a day or two of reading and a few conversations.

Check 1: architecture and the first thing that breaks

I read the entry points, the data flow, the infrastructure configuration and the error handling. Not the whole codebase. Three questions drive the reading.

What breaks first at 10x traffic? Early AI products often have one bottleneck nobody has hit yet: a single GPU serving every request, one database behind every query, one queue for every job. I name it and estimate what fixing it costs.

Can you change the model without rewriting the product? If the provider’s SDK is called from inside every API handler, switching models means touching every endpoint. Model prices and quality change every few months, so a thin internal interface around model calls pays for itself quickly.

Where does user data go? How many systems touch it, and is personal data logged, cached or sent to third parties by accident? In week one I look for accidental exposure, not attacks. Attacks are check 6.

Check 2: code and test health

I run the linter, the type checker and the test suite. If there is no test suite, that is the first finding. I look at three things, with rough thresholds rather than rules:

• Test coverage of the paths that matter. The number itself is less useful than the question behind it: can the team change the core logic and know within minutes whether they broke something?
• Dead code. A codebase that pivoted twice usually carries large unused modules. They slow every change and hide real behavior.
• Dependency age. Major dependencies far behind their current release are both security exposure and future migration pain.

Check 3: the deployment path

“How do you deploy?” tells me more than any architecture diagram. Answers range from automated tests, a staging environment and gradual rollout, to one engineer connecting to the server and running a script.

I follow one real deploy. Is there a CI pipeline, and does it run tests? Is there a staging environment? Can the team roll back a bad release in minutes? Is the process written down, or does it live in one person’s head?

Check 4: inference cost per request

For AI startups this is the check most often skipped. I pull the cloud bill and the model API usage, work out the cost of one request, then multiply by the volume the company is planning for.

A typical shape: a demo that costs a few hundred dollars a month in API calls turns into tens of thousands a month at the volume in the pitch deck, and nobody has checked whether the price per customer covers it. The fix is rarely dramatic. Caching, routing simple requests to smaller models and trimming prompts often cut the bill a lot, but only after someone measures it per request. For voice products I also measure cost per minute of conversation, because speech recognition and synthesis add their own per-minute charges on top of the LLM.

Check 5: who can do what

I talk to every engineer individually. This is not a performance review. I want to know each person’s strongest skill and where the gaps are. Can anyone deploy and monitor a model in production? Can anyone read a paper and judge whether it applies? Is there someone senior enough to push back on architecture decisions?

The gap I look for first in AI teams is production experience: people who can train or prompt a model but have never containerized, monitored or scaled one. The second is the opposite: a team that ships quickly but has nobody tracking what has changed in the field.

Check 6, for agent products: what the agent can reach

If the product includes an LLM agent that calls tools, I add one more check, because this is where the most expensive surprises hide.

I list every tool the agent can call and the permissions behind each one. Then I trace where the agent’s input comes from. If untrusted content can reach it (a customer email, a web page it browses, a document it summarizes, a caller’s speech on a phone line), then someone outside the company can try to steer it. The questions are simple. Which actions are irreversible? Is there an approval step before them? Are tool calls logged well enough to reconstruct what happened? A day of reading here often finds an agent with far more access than its job needs. My longer treatment of this is in the AI security collection.

The week-one report

The report is three to five pages with three parts.

Top risks. What will hurt the company in the next 30 to 90 days if nothing changes, each with a rough cost to fix.

Quick wins. Changes that take less than a week and make a visible difference: basic monitoring and alerts, a CI pipeline that runs the tests, cost tracking per request, an approval step in front of the agent’s riskiest tool.

A recommendation. Either “continue, and here is the plan for the next three months,” or “fix these fundamentals first.” The second answer is sometimes the most useful thing I can give a founder.

When I recommend not continuing

The founders want validation, not a diagnosis. If “your inference cost grows faster than your revenue” gets defensiveness instead of questions, outside technical leadership won’t help much.

The core needs a rewrite. If the architecture cannot support the product, for example a single synchronous script trying to serve real-time voice calls, incremental advice won’t fix it. The better plan is to hire a strong senior engineer, rebuild the core, and bring in advice for the new design.

Nobody can carry out the recommendations yet. If no one can set up CI, containerize a model or reason about cloud networking, a fractional CTO turns into a full-time engineering manager. That is a different role with different costs, and the honest advice is to hire for it.

Key takeaways

• Spend the first week diagnosing. Changes made before you understand the system are usually wrong.
• Find the first thing that breaks at 10x load. Every early system has one.
• Model inference cost per request, and per minute for voice, at the volume the company is planning for.
• Map who can do what. Missing skills need hires, not tools.
• For agent products, map what the agent can reach and who can steer it.
• End the week with risks, quick wins and an honest recommendation, even if the recommendation is to stop.

FAQ

What does a fractional CTO do in the first week?

Diagnose. I read the code and infrastructure, run the tests, follow a deploy, model inference cost, map the team’s skills and, for agent products, map what the agent can reach. The result is a short report with the top risks, quick wins and a recommendation on whether to continue.

How is a fractional CTO engagement priced?

I start with a short fixed-price assessment like the one in this post. If we continue, it becomes a monthly retainer sized to the hours the company actually needs. We agree the details on a call before anything starts.

What technical problems show up most often in early AI startups?

Three come up again and again: nobody has modeled inference cost at the volume in the pitch deck, deploying means one person running a script by hand, and quality is judged by reading a few outputs instead of a fixed test set. Each is fixable in weeks, but only once someone looks.

Why check security in a technical health assessment?

An LLM agent that can call tools turns text into actions. If untrusted content such as an email, a web page or a caller’s speech can reach that agent, someone outside the company can steer it. Finding that in week one is far cheaper than finding it after an incident.

When do you recommend against a longer engagement?

When the founders want validation rather than a diagnosis, when the core system needs a rewrite before advice can help, or when nobody on the team can carry out the recommendations yet. In those cases the week-one report is the deliverable, along with what to hire or fix first.

Further reading

• How I evaluate AI startups for investors: the same lens, compressed into a first-pass screen
• Build vs buy for AI in 2026: the build-or-buy calls that come up in most engagements
• Human-in-the-loop agent design: where to put approval steps in an agent
• When self-hosted TTS beats the API: a worked cost model for one voice component

If your team would benefit from this kind of first week, here is how I work as an advisor or fractional CTO. It starts with a 30-minute call.