How to vet an AI provider: Or what three rounds of jury questions taught us

In early October, the Bundesverband für KI-Transformation e.V. honored visibus with the Venture AI Excellence Award 2026. I am pleased about it, and I will say right away what it is and what it is not: not a seal of approval, not a certification, but the verdict of a jury that looked very closely for three weeks.

visibus Chat was honored in the startup category as a documented platform for the controlled use of commercial AI models in mid-sized companies: technical safeguards, controlled access and the rollout within the company, working together. That is how the association put it. This article is nonetheless less about the award than about the three weeks before it. Because what the jury demanded of us does not only exist for awards. It is the examination any company can demand of an AI solution before introducing it, and one that rarely happens in practice, because nobody knows what to ask. After three rounds, we do.

What the jury did not want

After the application came neither a yes nor a no, but an email with questions. Then another. Then another. The jury had read our documents, the user manual, the technical and organizational measures, the list of sub-processors, and in every round it wanted the same thing, only more precisely: evidence that what we describe actually runs that way on the productive instances.

What it did not want was slides.

The first round asked about all data paths, including the ones that tend to get lost in an application: the memory function, images, tool connections, local models, backups, administrative access. The second round wanted the filter status of every single instance with date and build, a binding profile to be assessed, and confirmation from an external test user. The third round wanted a clearly named product state: which release runs on which instance since when, what happens to voice recordings after seven days, proven with timestamps, and what happens when the privacy filter fails, proven with a log.

Built is not rolled out

The hardest question was the seemingly simplest one: which state is actually being assessed? Between the second and the third round we rolled out several times, not because that is our usual pace, but because every round surfaced points we did not want to leave standing as "planned". To a jury that still looks like a moving target, and it offered us two ways: assess the older state everyone knew, or the current one, but then with evidence per instance.

Our answer was a cut-off date. What is assessed is what runs on all productive instances on the day of our reply. Whatever comes after does not count. And whatever is built but not rolled out everywhere on that day does not count either. That sounds obvious, but it is not, if you are used to thinking in roadmaps. A target state is not a product state.

So the day before our reply we cut a release and rolled it out to all instances, so that what the jury had asked about was not "planned" on the cut-off date, but running. And where a function was still in transition, we declared it as a transition and left it to the jury whether it belongs in the assessment.

The rule that came out of it has applied since then to everything we say publicly: built is not rolled out. What does not run on all instances is not reported as done.

Pulling the plug

One question in the third round was, in essence: how did you test the fail-closed case? Fail-closed means: if the detection service for personal data cannot be reached, the gateway rejects the request instead of sending it unfiltered to the model. We had automated tests for that and the setting on every instance. The jury wanted to have seen the case.

So we stopped the service during live operation and sent a request with a test name. The interface showed an error message, the gateway log the rejection, the cost log no call to the model provider. Service restarted, the same request answered normally, the name as a placeholder. The whole test took a few minutes. A mechanism you only know from tests is a promise. One you have seen under load is evidence.

The same pattern with the deletion period for voice recordings. The rule was in the TOMs, the deletion job ran daily. The jury wanted an extract with creation time, deletion time and result. We tracked one recording across its deadline: the run before the seven full days had passed left it in place, the first run after that deleted it, both readable in the system journal. A timestamp is worth more than a sentence in a document.

Not every statement in our documents survived the three rounds unchanged. Where we made things more precise, the updated versions are in the contract documents today, with date and release line. That is the real return on such an examination: documents that describe the state that is running, not the one you have in your head.

Six questions every AI solution should answer

Everything the jury wanted from us comes down to six questions. Three concern the product, three the documents. None of them requires you to understand a line of code. What matters is that the answers exist in writing and are consistent with each other.

1. Can we switch the model when a better one comes along, without a new subscription and without a new contract?
2. Do our data and our chat histories stay with us, on an instance that belongs to us?
3. What leaves the building, how long does the provider keep it, and may it be used for training?
4. Is there a data processing agreement with technical and organizational measures, and which product state do they apply to?
5. Which sub-processors are actually active for our instance?
6. What happens to our data, the backups and the snapshots when the contract ends?

And above all six stands the question that gave us the most work: Which state is running for us, today, with a date? Whoever gets an answer to that with a log knows more about their AI solution than any award can tell, ours included.

Eight questions from the user side

A second perspective, developed independently of us: swissAI, the Swiss user association for artificial intelligence, of which we are a member, asks eight questions of every offering in its comparison of Swiss AI chats. Four of them overlap with ours; four extend the list and belong on it: Where does the model come from? Does Swiss data protection law apply, or only the GDPR? What do the terms of the free tier say? And what liability and availability does the provider commit to? The answers belong in the offer, the contract and the terms of the trial, not in a blog post. That is where we give them, too.

In his article on data protection with ChatGPT, Copilot and Co., association president Chris Beyeler also breaks data location down into four questions: Where is it stored, where does the model compute, who has access, who receives the data. It is the most precise version of what the jury asked us as "all data paths". We applied the four to our own product page and closed a gap in the process: where the large cloud models compute is now stated there explicitly. Audit questions work on yourself, too.

The point

AI belongs to company access, like email, Office and file storage. Company access comes with contracts and evidence, not promises. For email and file storage that goes without saying; for AI it is not there yet, and that is exactly why the examination the jury demanded of us is worth it.

What it honored in the end was not only a product but also an attitude: logs instead of slides, a cut-off date instead of a roadmap, naming limits instead of glossing over them. That attitude can be expected of any AI solution. The six questions are a start.

If you are currently facing the question of how AI should become official in your company: write to us, we will look at it together. Why your own access is the right question in the first place is in the article Why your company needs its own AI access.