How to choose an AI software development company: 12 questions

You have a defined use case, a budget, and a list of possible suppliers who claim to build production AI. Their websites and demos often look alike. The real differences between these firms become clear later. Those differences cover how they handle your data, their testing approaches, and their availability when the system needs updates.

An AI software development company designs, builds, and operates AI applications for your organisation, such as assistants, agents, classifiers, and forecasting models. To pick one, ask how they prepare your data, how they test quality before launch, how the system fits your workflows and access controls, who owns the code and data, and what support is provided after go-live.

This guide targets individuals who are responsible for selecting a supplier and justifying that choice. The role is usually held by a head of data, a director of technology, or a chief executive in a smaller organisation. The document lists twelve questions arranged into four groups. For each question it outlines the preferred answer and the answer that signals a possible issue. It also describes how to assess a proposal, how to choose between fixed-outcome and time-and-materials pricing, what to check in references, and how to run a brief selection process.

Shipshape Data builds AI applications for clients. We have written this as the checklist we would want a client to use on us.

What an AI software development company does

An AI software development company builds software in which a model does part of the work. That can take several forms:

• A chat assistant that answers from your documents.
• An agent that completes steps in a workflow.
• A classifier that routes tickets.
• A forecast that feeds a planning team.

The model usually accounts for a small share of the development effort. The majority of work falls into four other areas: the data the model consumes, the systems it connects to, the tests that confirm its performance, and the controls that manage its access and actions.

AI development services typically comprise four categories of work:

• Data preparation involves locating, cleaning, and structuring the data that the system uses.
• Model development involves choosing a model, writing prompts, and setting up retrieval. In some cases it also includes fine-tuning or training.
• Integration involves linking the system to your applications, your identity and access controls, and your existing workflows.
• Deployment and support involve moving the system into production, monitoring its performance, and updating it as your data and requirements evolve.

Suppliers differ most on the first and last of these. Many are strong on the model and the demo. Fewer take responsibility for the data underneath or for the system after launch. Those are the parts that decide whether the system still works a year later. Our guide to moving a model from training into production explains what that last stage involves.

The model is the visible part of an AI system. The data and the controls around it decide whether the system is reliable.

Questions about data and scope

Start with data. Most stalled AI builds stall because of the data, not the model. A supplier's answers on data also tell you how they will run the whole engagement.

1. What do you need from our data first?

A trustworthy supplier will ask to review your data before giving a price for the project.

• The answer you want: they will review the sources, check quality and ownership, and tell you what must be fixed before a model is useful.
• A concerning answer is that they will fix the issues during the build process, or that the model will handle any data it receives.

2. What would make you advise us not to build?

• You need a clear list of conditions. Examples are missing data, a lack of process ownership, or a problem that ordinary software could solve at lower cost.
• The answer that should worry you: nothing. A supplier who never says no will build whatever is asked, including things that should not be built.

3. How will you choose the first use case?

• The answer you want: a use case tied to a metric you already track, small enough to ship, and chosen so that the first win pays for the second.
• The answer that should worry you: the largest possible scope in a single programme, or a use case picked because it makes a good demo.

Questions about evaluation and quality

AI systems may generate varying outcomes for identical input. This is unlike the behaviour of standard software. You need to know the methods the supplier employs to assess quality both before and after the system is released.

4. How will we know it works before launch?

Ask to see the evaluation plan in writing. Our guide on how to evaluate large language models covers the methods a good plan should name.

• The answer you want: a test set built from your real cases, scores agreed with your team before the build, and a threshold the system must pass before it goes live.
• Be wary if the response consists of a demonstration, a generic benchmark score, or a simple assertion that the model is accurate.

5. What happens when the model is wrong?

All models err. The design issue is what occurs afterwards. Our guide to detecting and reducing AI hallucination outlines the typical controls.

• The answer you want: a design that expects errors. Answers are grounded in your sources, uncertain cases go to a person, and users have a way to flag mistakes.
• A concerning answer is a claim that the model never makes errors, or a promise that a future version will fix current problems.

6. How will you monitor it after launch?

• The answer you want: named measures for quality, cost and response time, with alerts and an owner for each. Our guide to model monitoring in production explains what to watch.
• Be concerned if monitoring is presented as optional or if your team is expected to configure it after handover.

Questions about integration and governance

A model that cannot securely connect to your systems will stay in the pilot phase. These questions assess whether the supplier can integrate the system into your workflows while preserving your data control.

7. How will it fit the workflows we run?

• They will map the existing process with the staff who carry out the work. They will then embed the system into the tools the staff already use.
• The answer that should worry you: a new standalone interface that staff must remember to open.

8. How will the AI reach our data?

This question separates suppliers who prioritise security from those who prioritise speed. The model needs data access, and that access must be subject to the same controls as any other user. One approach is the Model Context Protocol (MCP), an open standard for linking models to tools and data. MCP operates via servers that enforce access rules and record all requests. We describe how we construct these governed routes on our MCP connections page.

• Access is provided through a governed layer. This layer adheres to your existing permissions, applies your business rules, and logs each request.
• A concerning answer is that the supplier copies your data into their environment, or that the model uses a single account with access to all information.

9. How will you handle personal data and audit?

• The desired answer: a data protection impact assessment when personal data is involved, processing retained in regions you approve under UK GDPR, and a record of which sources informed each output.
• A concerning response is one that depends wholly on the model provider's terms of service.

Questions about ownership and maintenance

These questions clarify what you own once the build is complete and whether you can change suppliers later.

10. Who owns the code, prompts and data?

• The answer you want: you own the code, prompts, evaluation sets, configuration, and any trained model weights. All of these reside in your repositories and cloud accounts from the beginning.
• Be concerned if the system runs on the supplier's platform or if ownership only shifts after a lengthy licence period.

11. What happens after go-live?

• Expect a defined support period, a plan for updating the model and data, and the same team to maintain and enhance the system.
• The answer that should worry you: a handover document, with a new contract required for any change.

12. Who will actually do the work?

• Ask for the names of the individuals who will build the system and request a meeting with them before signing the contract.
• The troubling response is that a senior team runs the sales meeting and an unnamed team manages the delivery.

If a supplier never says no, ask what else they have agreed to build.

How to read a proposal

A proposal shows the supplier's approach. You ought to assess it according to the specific commitments it contains instead of its broad AI descriptions. A high-quality proposal contains the following elements:

• Data work constitutes a separate phase that yields a defined output prior to the start of the modelling phase.
• Success is defined by business metrics such as hours saved or tickets resolved, not by the mere delivery of a model.
• Evaluation has its own section, covering the test set, the scores and the go-live threshold.
• All assumptions are documented in writing. This covers what you must provide, the delivery deadline, and the repercussions of a late delivery.
• Ownership and support terms are stated in plain words.
• Exclusions are listed. A proposal with no exclusions has not been thought through.

Be careful with proposals that name a model or a platform before they describe your problem. The choice of model should follow from the use case and the data, not come before them.

Fixed outcome or time and materials

Most suppliers offer one of two commercial models, or a blend of both.

Fixed outcome

You pay a fixed price for a defined outcome. This model suits initial projects with a clear scope, for example an assessment or a single use case using verified data. It delivers predictable costs. Nevertheless, a supplier may raise the price to cover unknown variables or restrict the scope so that the final result lacks utility.

Time and materials

You pay for the hours expended. This model is suitable for situations where data or requirements are still undefined. It can also be applied to later project stages when priorities need to shift based on new information. This approach offers flexibility. The work lacks a fixed end date. Regular reviews using agreed metrics are required to manage risk.

A sensible combination

Many buyers combine the two. The first phase is fixed, and it checks the data and proves one use case. Further phases are priced once the unknowns are known. Whichever shape you choose, cost is driven mainly by four things: the state of your data, the number of systems to integrate, the level of evaluation and governance required, and the support you need after launch.

What to verify in references

Ask for references of work that has been in production for a substantial period instead of pilot projects. In each conversation, confirm the following points:

• Find out whether the system is still in use and whether the original target audience matches the current users.
• Which metric it moved, and how the client measured it.
• What went wrong during the build, and how the supplier responded.
• Whether the supplier raised data problems early or late.
• What support looked like in the months after go-live.
• Whether the client could change supplier without rebuilding the system.

Published case studies are useful when they provide a number and explain the mechanism behind it. For example, our work with 1NCE is described by a single outcome: 60% of tickets handled to a human standard. A useful reference enables you to ask how such a figure was measured and what it depended on.

A short selection process

A long tender process is not necessary to make a good choice. Five steps suffice for most organisations:

• Draft a one-page brief. Include the problem, the metric, the involved systems, and the data you believe the system requires.
• First, verify whether your organisation is prepared. The free AI readiness assessment assesses strategy, people, data, and change management through 16 questions and does not require an email address.
• Choose three suppliers. Send each of them the brief and the twelve questions.
• Hold a working session with the people who would do the build, using a sample of your real data.
• Request a proposal from each supplier for the first phase only, with a fixed price. Compare the proposals using the checklist above.

You may have to determine what to build before selecting a builder. Our buyer's checklist for choosing an AI consultancy describes how to make that first decision.

What this means for you

The right AI software development company is the one whose answers on data, evaluation, access and ownership hold up when you test them. The model matters less than the work around it. Use the twelve questions in your first meeting and again when you review the proposal. Give weight to the supplier who is clear about what they will not do.

Start small. Choose a first phase with a clear measure, and check that the result is real. Then let that result decide the next phase.

Our AI development service creates custom AI applications, agents, and chatbots with governed data and deploys them into production. We first assess the data foundation, set success metrics, and offer support after launch. If you wish to assess us with the twelve questions in this guide, Talk to us. We will respond to those questions for your particular project and let you know whether we are a suitable fit.

Frequently asked questions