Choosing AI developers: how do you build AI that works inside a product?

Choose AI developers around a concrete user task and a way to test its outcome. Before judging a polished demo, define permitted data, quality criteria, permissions and operating costs—and ask what happens when the model is wrong.

What separates AI development from connecting a model?

Connecting a model can produce a plausible answer quickly. Building AI into a product also requires appropriate context, protection of information, evaluation and a workflow the user understands. When choosing AI developers, begin with a real piece of work rather than a model name. Ask the team to explain what the user is trying to achieve, which data is needed and which mistakes could cause harm, wasted effort or duplicate actions. A persuasive demonstration answers only part of that question.

Distinguish developers who use AI to write code from teams building AI behavior for customers. One describes the team’s working tools; the other describes the product. They can overlap, but experience with one does not establish expertise in the other. Ask to see model-behavior tests alongside ordinary software tests. Our guide to reviewing agent-written code explains why engineering decisions still deserve deliberate review even when code generation is fast.

How should you define success before the demo?

Choose a narrow task with an input and an outcome that somebody can assess. Turning a request into a task the user approves before saving is one example. Collect ordinary cases, incomplete cases and cases where asking a question is the right response. Decide who judges the output and what counts as a mistake. No single quality score suits every product: an awkward heading is different from an incorrect action that changes a business record or sends a customer message.

OpenAI’s evaluation documentation describes testing behavior against defined cases. Even when choosing another provider, the product question is the same: did a change to the model or instructions improve the task or break an important scenario? Make the evaluation set a project deliverable rather than a one-off demonstration. Protect the examples with appropriate access, and avoid real personal information when synthetic or anonymized examples can test the behavior adequately.

Which boundaries should an agent have?

Separate a suggestion from an action. An agent may propose an update without receiving permission to execute it across every connected system. Define which actions require approval, who can approve them and how permission is checked when execution occurs. Show clearly what has been saved. When a request is retried after a failure, the system should avoid duplicate actions or provide a state that an operator can investigate instead of silently repeating the operation.

The La Vision specification treats suggestions, consent and memory as separate product decisions. It requires user control over stored information and over proposals that become actions. These are documented requirements, not a claim that every component has already been validated in production. The project page gives context. The same questions apply to a business product: who owns the decision, what permission exists, and how can an incorrect outcome be corrected?

What belongs in the development proposal?

Request a breakdown of information sources, integrations, evaluation work and responsibility after launch. Operating cost is broader than model requests: it can include storage, retrieval, retries, support and ongoing testing. Without promising a price before usage is understood, the team should explain how these costs will be measured and how unexpected consumption will be detected. Agree on the response to a provider changing its availability or permitted use.

Bring Julius a recurring business task and examples of how a person handles it today. We can examine whether AI fits, what should be tested first and which steps need human approval. If the requirements are unclear, begin with a technology product specification. A capable team should also be able to recommend a straightforward business rule when it offers a reliable, sufficient result. Adding a model everywhere creates more behavior to evaluate without necessarily improving the customer’s experience.