Retriever performance is a function that allows you to fine-tune the RAG according to the types of documents and questions uploaded to the project. By improving the Retriever’s performance, you can receive more accurately generated answers.
To use this feature, you must first go to Knowledge Base > Retriever Optimization tab and click the ‘Request’ button.
Requesting an optimization is very simple. Upload directly and/or utilize the feedback function. The optimization request is sent to your account manager and is executed after review, and the degree of performance improvement as a result of the execution is then given in a final report.
- Enable retriever training by utilizing the documents that are uploaded within the project.
- Utilize the template to upload questions and documents that you would like used for retreiver optimization. The template is shown below. The data set uploaded should closely match the queries and style of queries that will be asked for your specific use case.
- Utilize user feedback within the optimization
- Utilize an LLM to create additional data for optimizing the retreiver
- Cancel or click “Next” to review the training and optimization
The basic structure of the file consists of ① a question, ② a file name containing the answer to the question, and ③ a page number.
If there are three documents for the same question, they can be divided into three lines.
The recommended number of data sets is 50 or more. If the number of data sets is small, optionally, you can use the LLM model to automatically generate a question dataset with similar patterns to the uploaded data + aggregated feedback data. However, additional credits will be used if you use this option.
*Optimization requests are possible even if the number of data sets is less than 50, but the more data there is, the better the effect on Retriever optimization.
Feedback data can be more effective when used with a directly uploaded data set from the conversation history that aggregates the history of feedback given by the agent for the page on which the answer is based.
Once we have clicked “Next”, we are given a review of the request we are about to create. Once we click “Confirm”, your optimization request is complete. The account manager then works on improving the retriever. This process will take 2-3 business days. After requesting optimization, cancellation is not possible on the dashboard. If you wish to cancel, you must contact your account manager separately.
Once improvements are complete, you can view performance before and after the optimization in the dashboard. you can also download a final report to see where your retriever has improved.
