How to evaluate an AI trading vendor: 10 questions buyers must ask

Buyers need to know if the data is proprietary, third-party, or a mix — and the latency of updates. Decision criteria: source reliability, licensing cost coverage, refresh frequency, and asset-class fit.

What to ask for: a data-source matrix (one row per provider, columns for asset class, coverage, refresh interval). Compare it against your own pipeline so you know which sources are new vs. duplicated.

What a good answer looks like: equity tick data refreshed every 5 seconds; crypto books streamed continuously; macro indicators pulled hourly; news wires ingested in real-time with a clearly documented retention window.

Look for out-of-sample testing (the model must be evaluated on data it never saw during training), walk-forward analysis (re-fit on rolling windows to simulate live conditions), and cross-validation with explicit confidence intervals.

Decision criteria: statistical rigor, reproducibility, and whether the methodology is documented enough for an independent quant to replicate.

Ask for: a recent performance report with confidence intervals, the exact backtest window, and the specific assumptions (slippage model, commission schedule, fill assumptions). A vendor that hands over a one-page "we got 87% win rate" without methodology is hiding the fragile assumptions underneath.

This includes position limits, stop-loss rules, real-time exposure monitoring, and circuit breakers that pause the bot when risk thresholds breach. Decision criteria: alignment with your firm's risk policy + configurability of every threshold.

Ask for: a demo of the risk dashboard, a list of configurable parameters, and a sandbox where you can set custom limits and watch them enforced live. The vendor should also document what happens when a bot fails — does it close positions, does it freeze, does it alert?

Regulatory compliance demands traceability of model changes. Decision criteria: version control on model code, documentation standards, access logs, and the ability to reconstruct any historical decision.

Ask for: a sample audit log showing who changed what and when, retention policy, and the export format (CSV, JSON, S3 sync). A vendor that can't produce an audit trail in under 5 minutes is not ready for institutional use.

For high-frequency strategies, milliseconds matter. Decision criteria: measured in microseconds for HFT, milliseconds for systematic equity, sub-second for swing strategies.

Ask for: a latency benchmark with median, 95th percentile, and worst-case numbers, plus the network topology (co-located, cloud region, retail broker round-trip). A vendor that quotes a single average is hiding tail latency.

What good looks like: signal-to-broker latency under 500 ms for non-HFT strategies, with a clear breakdown of every hop.

Look for encryption at rest and in transit, role-based access control, and third-party security audits. Decision criteria: compliance with GDPR, CCPA, and any industry-specific frameworks your firm operates under.

Ask for: a copy of the most recent SOC 2 report (or equivalent), redacted excerpts highlighting access controls, and the data-retention policy. A vendor without these documents should not be allowed near your credentials.

Buyers often need to tweak factor exposures, weighting schemes, or risk parameters. Decision criteria: API flexibility, UI configurability, and how deep you can go without writing custom code.

Ask for: a walkthrough of the parameter-tuning interface, the API surface for programmatic control, and whether configuration changes require redeployment.

Transparency here prevents budget overruns. Decision criteria: subscription tiers, usage-based fees, support cost, and overage triggers. A vendor that obscures pricing behind a "talk to sales" wall is signalling that the price is whatever they think you'll pay.

Ask for: a detailed price sheet broken down by component, and a calculator that models cost at different trade volumes.

Seamless connectivity reduces operational risk. Decision criteria: supported protocols (FIX, REST, MetaAPI), SDK availability, onboarding timeline, and whether the integration survives broker outages gracefully.

Ask for: the full list of supported brokers, the integration guide, a sample code snippet for order routing, and the failover behavior when the connection drops.

Future-proofing is essential. Decision criteria: release cadence, beta-testing program, client feedback loops, and whether breaking changes get advance notice.

Ask for: a product roadmap with quarterly milestones, the process for early-access participation, and the deprecation policy for any feature you depend on.