How to: AI chatbot in seven weeks

In this article you can read

• how the AI chatbot of T Cloud Public came about,
• what added value it creates for cloud users,
• and what role Retrieval Augmented Generation (RAG) played in this.

What is the potential of artificial intelligence? T Cloud Public – as a user – has dared to try. Our website has been offering the first LLM-based, freely available AI chatbot from a cloud provider as of July 2024. We combine the strengths of artificial intelligence and Retrieval Augmented Generation (RAG). The chatbot was developed in just seven weeks together with our partner amberSearch.

Everyone is talking about artificial intelligence and many companies want to answer two questions in particular:

• Where are AI use cases that will bring us tangible added value?
• How can we implement AI use cases easily, quickly and cost-effectively?

“As T Cloud Public, we have various points of contact with artificial intelligence,” says Norbert Brüll, the project manager for the ChatBot frontend, ”As an AI provider, we are pursuing various AI initiatives with our partners.” These include the provision of development and training environments (e.g., via ModelArts), data pipelines, the GDPR-compliant operation of AI services and the offer of an LLM Hub.

“With the current project, we have changed our perspective and put ourselves in the shoes of an AI user. We wanted to explore the possibilities of AI for ourselves and our customers,” says Brüll. The use case was quickly found - T Cloud Public also has a typical enterprise problem: a lot of data is “scattered” in many different places. Users (both internal and external) often have to search until they find the right information. The solution – in times of AI – lies in an intelligent search assistant that taps into all sources and summarizes the information in a handy answer. 

“An enterprise search or intelligent knowledge management reduces search efforts in such cases by an average of 40 percent,” explains Bastian Maiworm, founder and CEO of amberSearch, ”The productivity gains are obvious.”

In May 2024, amberSearch and T Cloud Public launched the project to get the new AI-based chatbot up and running. The basis for this were services that the start-up had already developed, which are now available quickly and easily as a standard product. These include the semantic search “amberSearch” and the question-answer system “amberAI”, which are based on a self-developed and trained Large Language Model (LLM). In the basic version, the amberSearch services function as a kind of ChatGPT that can be used to chat about any topic.

“But that wasn't the intention of our T Cloud Public Chatbot,” says the project manager, ‘it was designed to help our users find relevant information and use it for simple cloud applications, in line with the motto ’Find it fast'.” The aim was to quickly answer questions such as “How do I deploy a Kubernetes cluster?” or “What functionalities does the backup service offer?”.

To do this, the existing amberSearch system needed access to the relevant sources. Specifically, these are T Cloud Public website, the document library (Doc Library), community content and a collection of PDFs such as the service description. “To give the chatbot access to this information, we used a technology known as Retrieval Augmented Generation (RAG),” explains Maiworm. When using RAG, an AI model is not retrained or fine-tuned – which also eliminates the corresponding effort and costs. Instead, the chatbot cooperates with an enterprise search. When a question is asked, a search is first triggered in the connected sources. The information is made available to the AI model depending on the situation and then condensed into an answer. It thus remains within the company or in the repositories where it is stored anyway. amberSearch offers various methods to make the data accessible to the bot. Specifically, an API is used for the multimedia content of the community (videos etc.), while the other sources are searched several times a day using a crawler. The information is collected and standardized in a vector database. This database then acts as a consolidated basis for the concentrated T Cloud Public knowledge from which the bot draws. The automated updates of the vector database ensure that the bot's knowledge stays up to date on a daily basis.

The chatbot was created in just seven weeks and has been running since mid-July. It provides users of T Cloud Public with easy access to the latest knowledge they need; tedious searches are a thing of the past. The bot also documents the sources from which it has obtained the relevant information – this reduces the likelihood of hallucinations. “Our chatbot is easily accessible via the support tab on the right-hand side of the screen (AI). No registration is necessary,” says Brüll. “By operating the vector DB and the model within T Cloud Public, the team has full control over the AI. This also includes GDPR compliance,” adds Maiworm, ”so that a second stage can also be implemented if necessary, in which personal data is processed via a login.”

So – approach the bot and ask it what you always wanted to know about T Cloud Public. Incidentally, the bot is not just an information assistant, but can also help you in a very practical way. Norbert Brüll: “If you want to deploy a service, you don't have to wade through API documentation or community input, you can simply ask the bot for a deployment setup on a container basis – including code”.