How Algomo reduced AI hallucinations with LangWatch
Manouk
Jun 11, 2024
Last week we met with one of our first customers - Algomo, a platform specializing in AI agents that automate and personalize customer interactions. They have integrated LangWatch into their AI operations to improve performance and maintain high-quality interactions for their customers.
Who did we meet with?
Kenneth: Data Scientist at Algomo
Nikhil: AI Researcher focused on securing generative AI at Algomo
LangWatch was integrated into Algomo's production environment over a month ago, logging over 10’s of thousands of conversations since its implementation. The team uses LangWatch daily to monitor and improve the performance of their AI agents.
Alerting on problematic messages to debug immediately
One of the most impactful features for Algomo is LangWatch's alert system. Kenneth highlights, “Alerts are one of the best things about LangWatch. When there’s a problematic message, we get an alert and can investigate it quick and easily.” The typical workflow involves receiving an alert in their Slack channel, reviewing the chatbot’s response and context, and determining if there’s a hallucination or error. This process has streamlined Algomo's debugging and resolution workflow significantly. It's like having a monitoring system such as Sentry on your software development, but now better and optimized for LLM-powered products.
An instance of LangWatch’s utility was when it detected outdated data sources being used by the chatbot. Nikhil explains, “One of our clients updated their FAQs, but our system was still using the old data. LangWatch alerted us to this issue, leading us to discover corrupted data in our vector database, which we then refreshed to resolve the problem.” Obviously when hearing this, LangWatch was delighted we could help them solving this inaccurate data.
Real deep insights into the product performance
Nikhil spends two to three hours daily retrieving real deep insights using LangWatch, finding the labels and custom analytics particularly beneficial. “Labels come in handy for manipulating the visibility of how the platform works,” he says. Algomo uses labels to log request inputs (such as client and data source details) and to monitor internal processes. Previously he had no idea where to look at or the AI was a complete "blackbox". Now they have the ability to spend their time wisely using LangWatch.
Custom analytics, is a feature in LangWatch to build your own graphs based on all metrics available, allows Algomo to develop tailored graphs for monitoring various metrics, such as the number of messages, response quality, and cost per customer. Nikhil elaborates, “We have developed four to five custom graphs on the production instance to track message counts, threads, and cost per customer.”
On the other hand, the flexible filter options and fast search functionality are essential for day-to-day operations. “If you want to track back a week, the filters are super fast, allowing us to easily filter out and investigate specific cases,” Nikhil notes. This feature aids in quickly identifying and addressing issues, enhancing the efficiency of their monitoring process.
Algomo plans to enhance their reporting capabilities with LangWatch. They envision using custom analytics and filters to generate detailed reports for clients. Nikhil mentions, “We’re preparing to create reports using custom analytics, which will give us more visibility into the performance of our chatbots and help in reporting to internal team members and customers.” LangWatch is excited to hear Algomo is using the custom graphs to build extensive reporting for their customers to show how their AI is performing and plan Customer QBR sessions with them showing these results.
Preventing Data leakgage
LangWatch has significantly improved Algomo’s ability to detect and resolve errors quickly. Kenneth highlights a critical instance where a chatbot used data from a different client, causing a data leakage. “Without LangWatch, it would be hard to find such issues. Manually going through data would be very time-consuming,” he says.
The team has noted significant improvements in their ability to monitor and maintain the quality of their AI interactions. The alerts going into slack provided by LangWatch have allowed them to identify and resolve issues that have gone unnoticed otherwise.
Conclusion
Algomo looks forward to further refining its use of LangWatch, especially in reporting and analytics.
LangWatch has proven to be an invaluable tool for Algomo, offering enhanced monitoring, rapid error detection, and detailed analytics. As Kenneth succinctly puts it, “LangWatch gives us an easy way to get more visibility into the problems of our application.” With ongoing use and planned enhancements, Algomo expects LangWatch to continue playing a crucial role in maintaining and improving their AI-driven customer interactions.
Are you curious to learn how LangWatch can have this impact on your AI products?
Get a demo now
Last week we met with one of our first customers - Algomo, a platform specializing in AI agents that automate and personalize customer interactions. They have integrated LangWatch into their AI operations to improve performance and maintain high-quality interactions for their customers.
Who did we meet with?
Kenneth: Data Scientist at Algomo
Nikhil: AI Researcher focused on securing generative AI at Algomo
LangWatch was integrated into Algomo's production environment over a month ago, logging over 10’s of thousands of conversations since its implementation. The team uses LangWatch daily to monitor and improve the performance of their AI agents.
Alerting on problematic messages to debug immediately
One of the most impactful features for Algomo is LangWatch's alert system. Kenneth highlights, “Alerts are one of the best things about LangWatch. When there’s a problematic message, we get an alert and can investigate it quick and easily.” The typical workflow involves receiving an alert in their Slack channel, reviewing the chatbot’s response and context, and determining if there’s a hallucination or error. This process has streamlined Algomo's debugging and resolution workflow significantly. It's like having a monitoring system such as Sentry on your software development, but now better and optimized for LLM-powered products.
An instance of LangWatch’s utility was when it detected outdated data sources being used by the chatbot. Nikhil explains, “One of our clients updated their FAQs, but our system was still using the old data. LangWatch alerted us to this issue, leading us to discover corrupted data in our vector database, which we then refreshed to resolve the problem.” Obviously when hearing this, LangWatch was delighted we could help them solving this inaccurate data.
Real deep insights into the product performance
Nikhil spends two to three hours daily retrieving real deep insights using LangWatch, finding the labels and custom analytics particularly beneficial. “Labels come in handy for manipulating the visibility of how the platform works,” he says. Algomo uses labels to log request inputs (such as client and data source details) and to monitor internal processes. Previously he had no idea where to look at or the AI was a complete "blackbox". Now they have the ability to spend their time wisely using LangWatch.
Custom analytics, is a feature in LangWatch to build your own graphs based on all metrics available, allows Algomo to develop tailored graphs for monitoring various metrics, such as the number of messages, response quality, and cost per customer. Nikhil elaborates, “We have developed four to five custom graphs on the production instance to track message counts, threads, and cost per customer.”
On the other hand, the flexible filter options and fast search functionality are essential for day-to-day operations. “If you want to track back a week, the filters are super fast, allowing us to easily filter out and investigate specific cases,” Nikhil notes. This feature aids in quickly identifying and addressing issues, enhancing the efficiency of their monitoring process.
Algomo plans to enhance their reporting capabilities with LangWatch. They envision using custom analytics and filters to generate detailed reports for clients. Nikhil mentions, “We’re preparing to create reports using custom analytics, which will give us more visibility into the performance of our chatbots and help in reporting to internal team members and customers.” LangWatch is excited to hear Algomo is using the custom graphs to build extensive reporting for their customers to show how their AI is performing and plan Customer QBR sessions with them showing these results.
Preventing Data leakgage
LangWatch has significantly improved Algomo’s ability to detect and resolve errors quickly. Kenneth highlights a critical instance where a chatbot used data from a different client, causing a data leakage. “Without LangWatch, it would be hard to find such issues. Manually going through data would be very time-consuming,” he says.
The team has noted significant improvements in their ability to monitor and maintain the quality of their AI interactions. The alerts going into slack provided by LangWatch have allowed them to identify and resolve issues that have gone unnoticed otherwise.
Conclusion
Algomo looks forward to further refining its use of LangWatch, especially in reporting and analytics.
LangWatch has proven to be an invaluable tool for Algomo, offering enhanced monitoring, rapid error detection, and detailed analytics. As Kenneth succinctly puts it, “LangWatch gives us an easy way to get more visibility into the problems of our application.” With ongoing use and planned enhancements, Algomo expects LangWatch to continue playing a crucial role in maintaining and improving their AI-driven customer interactions.
Are you curious to learn how LangWatch can have this impact on your AI products?
Get a demo now
Last week we met with one of our first customers - Algomo, a platform specializing in AI agents that automate and personalize customer interactions. They have integrated LangWatch into their AI operations to improve performance and maintain high-quality interactions for their customers.
Who did we meet with?
Kenneth: Data Scientist at Algomo
Nikhil: AI Researcher focused on securing generative AI at Algomo
LangWatch was integrated into Algomo's production environment over a month ago, logging over 10’s of thousands of conversations since its implementation. The team uses LangWatch daily to monitor and improve the performance of their AI agents.
Alerting on problematic messages to debug immediately
One of the most impactful features for Algomo is LangWatch's alert system. Kenneth highlights, “Alerts are one of the best things about LangWatch. When there’s a problematic message, we get an alert and can investigate it quick and easily.” The typical workflow involves receiving an alert in their Slack channel, reviewing the chatbot’s response and context, and determining if there’s a hallucination or error. This process has streamlined Algomo's debugging and resolution workflow significantly. It's like having a monitoring system such as Sentry on your software development, but now better and optimized for LLM-powered products.
An instance of LangWatch’s utility was when it detected outdated data sources being used by the chatbot. Nikhil explains, “One of our clients updated their FAQs, but our system was still using the old data. LangWatch alerted us to this issue, leading us to discover corrupted data in our vector database, which we then refreshed to resolve the problem.” Obviously when hearing this, LangWatch was delighted we could help them solving this inaccurate data.
Real deep insights into the product performance
Nikhil spends two to three hours daily retrieving real deep insights using LangWatch, finding the labels and custom analytics particularly beneficial. “Labels come in handy for manipulating the visibility of how the platform works,” he says. Algomo uses labels to log request inputs (such as client and data source details) and to monitor internal processes. Previously he had no idea where to look at or the AI was a complete "blackbox". Now they have the ability to spend their time wisely using LangWatch.
Custom analytics, is a feature in LangWatch to build your own graphs based on all metrics available, allows Algomo to develop tailored graphs for monitoring various metrics, such as the number of messages, response quality, and cost per customer. Nikhil elaborates, “We have developed four to five custom graphs on the production instance to track message counts, threads, and cost per customer.”
On the other hand, the flexible filter options and fast search functionality are essential for day-to-day operations. “If you want to track back a week, the filters are super fast, allowing us to easily filter out and investigate specific cases,” Nikhil notes. This feature aids in quickly identifying and addressing issues, enhancing the efficiency of their monitoring process.
Algomo plans to enhance their reporting capabilities with LangWatch. They envision using custom analytics and filters to generate detailed reports for clients. Nikhil mentions, “We’re preparing to create reports using custom analytics, which will give us more visibility into the performance of our chatbots and help in reporting to internal team members and customers.” LangWatch is excited to hear Algomo is using the custom graphs to build extensive reporting for their customers to show how their AI is performing and plan Customer QBR sessions with them showing these results.
Preventing Data leakgage
LangWatch has significantly improved Algomo’s ability to detect and resolve errors quickly. Kenneth highlights a critical instance where a chatbot used data from a different client, causing a data leakage. “Without LangWatch, it would be hard to find such issues. Manually going through data would be very time-consuming,” he says.
The team has noted significant improvements in their ability to monitor and maintain the quality of their AI interactions. The alerts going into slack provided by LangWatch have allowed them to identify and resolve issues that have gone unnoticed otherwise.
Conclusion
Algomo looks forward to further refining its use of LangWatch, especially in reporting and analytics.
LangWatch has proven to be an invaluable tool for Algomo, offering enhanced monitoring, rapid error detection, and detailed analytics. As Kenneth succinctly puts it, “LangWatch gives us an easy way to get more visibility into the problems of our application.” With ongoing use and planned enhancements, Algomo expects LangWatch to continue playing a crucial role in maintaining and improving their AI-driven customer interactions.