How Reliable Are GPT-based Document Search Tools?

Before We Get Started: What Is ChatGPT?

GPT stands for “Generative Pre-trained Transformer” and refers to a specific architecture of artificial neural networks developed by OpenAI. OpenAI has been doing this for some time, but with the release of the AI chatbot “ChatGPT-3” at the end of 2022, it entered the mass market. The special thing about GPT-based chatbots is their ability to understand and generate natural language. It feels like a real conversation. This is, among other things, due to OpenAI training GPT with language models (so-called LLMs – Large Language Models)

‍

How Can Medium-Sized Businesses Benefit From LLMs?

Working with AI has already become prevalent in medium-sized businesses. However, this is frequently limited to optimizing core business operations (smart products, IIoT in production, etc.). With GPT, medium-sized businesses can use AI for internal processes and operations by making internal knowledge more accessible and efficiently utilized. This can speed up or simplify numerous processes, such as:

…Onboarding

…Sales

…Maintenance

…Legal Advice

…Customer Support

The best use case depends on several factors. Fill out this questionnaire to get an individual assessment where the greatest potential lies for your business.

Until now, the digitalization of these areas has not been profitable for medium-sized businesses. Developing the necessary quality of a chatbot in these areas simply required a very high level of development effort. In relation to the number of salespeople, customer service representatives, and so on, the costs would not have been proportionate. With GPT, this has changed.

‍

The Benefits

A GPT-based tool that processes and indexes internal technical documents can save a lot of time and resources. Why? Because it takes a long time to find the right document to answer the request. According to this McKinsey study, highly skilled knowledge workers spent 19% of their worktime searching and gathering information. The documents are difficult to grasp – it takes a professional to understand what actions need to be derived… and relevant information is usually scattered in different places. These are things that GPT is good at:

• Finding relevant information quickly.
• Consolidating all relevant information, even if it is located in different places or in different documents.
• When needed, information can be better formulated for understanding, as GPT can also translate tech lingo. This is especially helpful for newcomers or non-experts.

In addition, GPT-based services are accessible 24/7 and take care of “tedious” to-dos. Searching through documents is generally considered to be a less inspiring task. Instead, employees can focus on exciting questions (problem-solving, product innovation, etc.).

‍

Nice, But How Does It Work?

The advantages of GPT-based document search are obvious. Therefore, there are already a couple of providers that offer PoCs (proof of concepts) for this. They all basically follow the same approach and use a pre-trained language model (LLM) to generate answers based on a specific document or paragraphs. This is how they operate:

Preprocessing: The document or section is preprocessed to remove unnecessary formatting and divide it into smaller, more manageable segments such as paragraphs or sentences.

Indexing: These segments are then encoded with an embedding model and stored in a database (e.g. Pinecone).

Question encoding: The asked question is encoded with the same embedding model.

Contextual search: Usually, the embedding of the question is compared with the embeddings of the segments to identify relevant contexts. More advanced approaches use multiple search methods here.

Input formatting: The encoded question and context segments are concatenated and formatted into a single prompt. Essentially, the LLM is asked to answer the following question based on the following context.

LLM query: Finally, the model is prompted to formulate the answer based on the provided context.

Moderation (optional): Most solutions do not yet include this part. However, you should consider adding moderation to check the answer for policy violations.

At Motius, we also developed an application that follows this principle.

‍

‍

Arising Challenges

So far, so good. However, as always, there are challenges, and currently few existing solutions address them. As we have been working with LLMs for years, we have thoroughly analyzed the following challenges:

→ Reliability of the output: If you follow the schema explained above, it is essential to find the “right” paragraph for the correctness of your answer. Without the paragraph that is relevant for answering the question, there is no meaningful answer. We recommend using not only the commonly used “Embedded Search” but also a conventional “Keyword Search.” In our experience, this significantly increases the reliability of the output.

→ Clean embedding: Embedding Document Q&A tools into the existing infrastructure is not trivial. In addition to a fragmented tool landscape, good access management is essential. Individual tools, as they are currently popping up everywhere, are not tailored to your (operational) environment. A customized tooling significantly increases security. At Motius, we have already supported several integrations and developed a Document Q&A solution that can be quickly and precisely tailored to your environment.

→ Data security: If you are processing sentsitive data with your Document Q&A tool, you want to be sure that it is done safely. There are many reasons (DSGVO, ethical concerns, reputation, …) to ensure your data safety. In our experience, the best way to do this os by assessing your data security requirements upfront.

Document Q&A for Critical Applications

The accuracy of the output, secure access management, and data security are particularly important for some areas, for example for legal or medical applications. Here it’s is crucial that only certain individuals have data access, and information or decisions must be traceable. It must be clear what information decisions are based on and, of course, ensured that they are correct.

As mentioned above, we see opportunities to ensure these points. However, if certifiability is important, so-called knowledge graphs can be a better approach for your Document Q&A tool, either in addition or alone.

A knowledge graph is a semantic data structure that represents knowledge in the form of entities, attributes, and relationships between these entities. This enables organizing information in a way that is readable for humans and machines. Hence, humans can verify the information before it is unlocked for the bot. Which means, connections extracted by AI from the documents can be reviewed before they are provided to the user.

With LLMs like GPT, context-aware knowledge graphs can be developed, eliminating false statements from a Document Q&A tool. Moreover, it is clear what entities a response is based on, for example which legal precedents the tool considered to make its decision.

Which leaves us to the last challenge mentioned above: data security. Depending on your requirements, there are different ways to ensure the security of your data. If you use data that shouldn’t leave your company, you need to deploy on-premise. If your data can leave your company and also Europe, you can go for commercial cloud providers. However, oftentimes this isn’t the case (e.g. due du DSGVO regulations). In those cases you should check if your data can be shared with a commercial could provider from Europe. Not an option? Then we’d suggest a VPC deploment.

‍

Leverage GPT for Your Business

GPT will change a lot – especially for small and medium-sized enterprises (SMEs), as GPT is an opportunity to advance their digitization relatively easily. Resource-intensive processes can be accelerated and used for more value-added processes. Given the growing shortage of skilled workers, companies can relieve their employees with more efficient GPT-supported processes and create a more attractive working environment. This is a not to be underestimated competitive advantage in the notorious “War for Talents”.

However, there are some challenges that you should keep in mind. We can help you to ensure reliable output and a watertight access management. Let’s define and implement a value-adding use case for GPT in your company!