The evaluation process: seven questions to ask every vendor
1. Can I see a working system in production — not a demo environment?
The gap between a demo and a production system is enormous. Demos run on clean, curated data. Production systems deal with OCR errors, missing fields, contradictory records, and edge cases the team never anticipated. Ask to access a live system, even briefly. If they cannot provide that, ask for a reference customer you can speak with directly — not a written testimonial.
2. What does your technical discovery process look like?
Every credible AI project starts with data assessment: what data exists, what quality it is in, whether it is sufficient to train or fine-tune a model, and what the realistic accuracy ceiling is given the data quality. If a vendor is willing to spec out and price a project without seeing your data, that is a warning sign. Discovery typically takes 1–3 weeks and should produce a written assessment with honest constraints.
3. How do you handle model drift and ongoing maintenance?
AI models degrade over time as the world changes and your data distribution shifts. A customer service model trained on 2024 conversations may perform measurably worse on 2026 conversations. Ask for the monitoring and retraining plan before you sign. Good vendors include alerting on key metrics (accuracy, confidence scores, user satisfaction) and a clear escalation path when metrics fall below threshold.
4. What is your approach when the AI gets it wrong?
Every AI system will occasionally produce incorrect outputs. The question is not whether errors will happen — it is how the system and the team handle them. Look for: human-in-the-loop workflows for high-stakes decisions, confidence thresholds that route uncertain cases to a human, and a clear process for logging errors and improving the model from them.
5. Who owns the models and data after the project ends?
This is a contract point many clients miss. If a vendor builds a fine-tuned model on your proprietary data, do you own those weights? What happens if you want to switch vendors in two years? Get explicit IP ownership clauses in writing before signing.
6. Can you show me your cost model for production — not just build cost?
API costs for AI systems can be significant at scale. A system that costs $20,000 to build but $2,000 per month to run because of inefficient prompt design is a bad deal. Ask for a cost model that includes expected API spend at your projected query volume, and ask how they optimize for inference cost.
7. What will you tell me NOT to use AI for?
The most trustworthy vendors are the ones who push back. If a vendor agrees that AI is the right solution for every problem you bring them, they are either not thinking critically or they need the revenue. Good teams will tell you when a simpler rule-based system, a better database query, or just a cleaner internal process would outperform an AI solution at lower cost and risk.
Technical capability checklist
Beyond the conversations, you can evaluate technical capability directly:
- Do they have a clear evaluation methodology? Ask how they measure model performance. If the answer is vague — "we test it" — that is a problem. Look for precision, recall, F1, or domain-specific metrics with defined thresholds.
- Do they understand your data privacy requirements? For Israeli companies, this includes Privacy Protection Law compliance. For fintech or healthcare, it includes additional sector-specific requirements. A vendor who cannot name the relevant regulation in your sector has not worked in it seriously.
- Can they explain their architecture choices in plain language? You do not need to understand transformer attention mechanisms. But a good vendor should be able to explain why they chose RAG over fine-tuning, why they picked a specific vector database, and what the trade-offs are — in terms a non-technical founder can follow. If they hide behind jargon, they may not understand it themselves.
- Do they have a staging environment and deployment process? Production AI systems need the same software engineering rigor as any other software: version control, staging environments, rollback plans, and monitoring. Ask what their deployment process looks like.
For context on the build-vs-buy decision — including when a custom AI system makes more sense than a SaaS product — see our build vs. buy framework. And if you are evaluating how AI agents specifically fit into your operations, our guide to AI agents for business covers the use cases and realistic timelines.
Pricing structures: what is normal in 2026
AI development pricing varies significantly by scope. As a rough reference for scoping conversations:
| Project type | Typical range | What drives the high end |
|---|
| RAG chatbot / knowledge base (single department) |
$8,000–$20,000 |
Document complexity, access controls, UI requirements |
| Custom AI agent (multi-step workflow automation) |
$20,000–$60,000 |
Number of integrations, error-handling complexity, approval workflows |
| Full product with AI core (new application) |
$60,000–$200,000+ |
Scope of the product, infrastructure requirements, custom model work |
| AI integration into existing system |
$15,000–$50,000 |
Quality of existing codebase, API availability, change management |
These are market estimates for 2026, not guarantees. Projects with unclear scope, poor data quality, or significant compliance requirements consistently run over initial estimates — which is why a technical discovery phase with a fixed price is worth paying for separately.
Red flags and green flags at a glance
- Red: Guarantees specific accuracy or ROI before seeing your data
- Red: No case studies with verifiable outcomes
- Red: Uses "AI" as a feature list without explaining the underlying approach
- Red: Cannot explain who maintains the system after launch
- Red: Pushes you to sign before completing a discovery phase
- Green: Asks hard questions about your data quality before scoping
- Green: Offers a paid discovery phase with written outputs
- Green: Can name specific accuracy metrics from past projects
- Green: Tells you when AI is the wrong tool for a given problem
- Green: Has a clear plan for post-launch monitoring and model maintenance
FAQ
What is the difference between an AI development company and a general software agency?
A general software agency builds applications using established patterns — databases, APIs, user interfaces. An AI development company additionally works with probabilistic systems: language models, computer vision, prediction models. The key difference is expertise in data preparation, model evaluation, and handling the inherent uncertainty in AI outputs. Many agencies now claim AI capability; the questions above will help you verify whether that claim is real.
Should I hire an agency or build an in-house AI team?
In-house makes sense when AI is core to your product and you need it to evolve continuously. Agency makes sense for defined projects, when you need to move faster than hiring allows, or when you want to validate a concept before committing to a team. Many companies start with an agency to build and prove a system, then hire internally to maintain and extend it. See our dedicated breakdown of dedicated team vs. outsourcing for the full decision framework.
How do I evaluate an AI company's Hebrew or local-market expertise?
Ask for examples of systems that handle Hebrew content — not just translated UI, but actual Hebrew NLP (entity extraction, classification, or generation in Hebrew). Ask about their experience with Israeli data privacy requirements. And ask whether their team includes people who have built for the Israeli market specifically — cultural context matters for product decisions even when the underlying technology is the same.
What should a technical discovery phase include?
A proper discovery phase should deliver: an assessment of your existing data (quality, quantity, format, gaps), a proposed technical architecture with rationale, a realistic accuracy estimate given the data, a risk register (what could go wrong and how it would be handled), and a phased project plan with clear milestones and success metrics. If the output is just a quote, you did not get a real discovery.
How long does a typical AI project take from start to production?
A focused, well-scoped AI project — a chatbot, a classification system, a recommendation engine — typically takes 8–16 weeks from discovery to production. Complex multi-component systems or projects with significant data preparation work can take 6–12 months. Any vendor promising a full production AI system in two weeks without a prior discovery phase should be viewed with skepticism.
How do I get started evaluating options?
Start by defining what success looks like in concrete, measurable terms: what task should the AI perform, how will you measure whether it works, and what is the cost of it failing? With that clarity, vendor conversations become much more productive. If you want a structured way to think through your AI opportunity before talking to vendors, Palmidos's free AI Blueprint maps your situation to a concrete technical approach — or book a 30-minute call to work through it together.
