An AI chatbot based on a generic off-the-shelf solution provides quick answers but offers little insight into their quality. On the other hand, those who develop their own chatbot and systematically evaluate its responses gain something else: a tool that continuously improves the underlying website content. It is precisely this content that ChatGPT, Perplexity, and Gemini also draw on when generating information about the company.
Why Companies Develop Their Own Chatbots
When deciding between in-house development and a standard solution, cost and speed are usually the deciding factors. In the case of enterprise AI, the main advantages of in-house development are generally independence from vendors and customizability. With chatbots, there is an additional factor that is often overlooked: control over the quality and content of the responses.
We saw firsthand how this advantage plays out in practice during one of our projects. The goal was to create a service chatbot that provides customers with immediate answers and reduces the load on other support channels—using the website’s content as its knowledge base. We deliberately chose not to use a standard solution. Three reasons tipped the scales: building AI expertise within our own organization, avoiding dependence on external systems, and maintaining full control over the responses.
Evaluation as the true driving force behind development
The chatbot was implemented using an agent-based approach, in which multiple language models, each assigned different tasks, are designed to process user inquiries and respond based on the website’s content. Once the technical foundation was in place, the real work began: the evaluation phase, which took up the majority of the development time.
In this context, “evaluation” means systematically reviewing and assessing answers and using that information to identify areas for improvement. To this end, three testing procedures ran in parallel, each with its own focus and metrics. Internal experts verified the factual accuracy of individual answers. External users assessed whether the answers were understandable and helpful even without specialized knowledge. Automated tests ran continuously to verify that queries were assigned to the correct subject area and that the answers were generated technically correctly.
The results of all three processes led to specific optimization steps that affected the technical structure or the knowledge base. It quickly became apparent that the majority of problematic answers could not be improved through technical adjustments. The problem lay primarily in the content: contradictory statements between individual pages, outdated information that no one had updated, and content lacking context. Such issues led to statements in the answers being misrepresented or missing entirely.
This precision in error detection distinguishes evaluation from a traditional content review. Those who manage the website are generally familiar with their own content, but they do not automatically know every error. Evaluation makes every single one visible. The most effective lever for improving answers therefore usually lay in the knowledge base: the website content that the chatbot accesses. And a better knowledge base doesn’t just benefit the chatbot: AI search systems also analyze the same content (see Fig. 1).
What specific effects this has on GEO
Every gap, every unclear term, and every piece of outdated information uncovered by an evaluation also affects other AI systems that access the same content. Conversely, every adjustment that improves a company’s own chatbot makes that company a more reliable source for generative search systems.
This is exactly where Generative Engine Optimization (GEO) comes in: It prepares content for AI-powered search and response systems, not for traditional rankings. SEO aims for clicks and rankings, while GEO aims for inclusion in a generated response—often without the user ever visiting a website.
Three effects are particularly evident here:
When the search shifts to the response section of AI systems, users receive an answer there instead of a list of links. Whether they go on to visit a website afterward depends on whether it is cited as a source in the answer. It is crucial that the content be clear, up-to-date, and consistent.
Against this backdrop, FAQs, glossaries, and step-by-step explanations are more than just good editorial practice. When they result from a thorough chatbot evaluation, they help companies stand out in AI search results.
Conclusion: Plan for evaluation early on
A chatbot developed in-house offers more than just independence from the feature set of an off-the-shelf solution. It allows you to systematically test, learn from the results, and continuously improve your own content—with benefits that extend beyond the chatbot itself. Evaluation is often underestimated as a mere quality check right before launch. Anyone considering developing their own chatbot today should therefore plan for evaluation from the very beginning.
Would you like to know how an evaluation-driven chatbot can also improve your company's visibility in AI search?