In the grand tradition of the tech industry, I currently see a number of folks jumping onto the latest bandwagon without a seatbelt. And a number of riders are about to fall into the trap of believing that the latest and greatest tools are the perfect fit for their enterprise needs.
Some are now claiming that a simple vector database, a sprinkle of LangChain, and a dash of OpenAI are the magic recipe for a perfect Retrieval Augmented Generation (RAG) Stack.
Sure, let’s throw in a unicorn and a pot of gold at the end of the rainbow while we’re at it. I’ve seen this movie. Back in the late nineties many self-respecting CIOs and CTOs of Forbes 1’000 companies started to build their own Content Management Systems.
I remember a very large bank to join the bandwagon, too, only to realize that multi-language content rendering is not a bank USP. A decade later we were back at the same game: The eCommerce wave produced a number of large scale fails when retailer CIOs discovered that for a successful web shop you need more than a digital shopping basket.
So here we go again: Just because it’s new and glitzy doesn’t mean it’s golden. A vector database? Sure, it’s a robust system for certain tasks. LangChain? Stellar at simplyfing the cobbling together of such solutions. OpenAI? Undeniably, a giant in text generation. But slap them together and expect enterprise magic? That’s like expecting a random mix of haute couture pieces to make you the next fashion icon. Good luck with that.
The Misconception of Synergy
Just because individual components excel in their domains doesn’t necessarily mean their combined utility will take you ever beyond 80% of the road. And it’s the last 20% of the road that are the hardest. Outlining just a few of those issues:
- While vector databases excel at retrieving relevant data based on similarities, much of the retrieval has to do with keywords. Not something an out-of-the-box vector database is good at. The way forward will be hybrid search.
- Additionally, the accuracy depends largely on the quality of embeddings and the algorithm’s ability to discern subtle differences. Retrieving a slightly incorrect piece of information can drastically affect the output’s quality. To deal with that you need a lot of experience in classic information retrieval (IR).
- Users might pose queries with multiple valid interpretations. Catering to such ambiguity and ensuring the retrieved knowledge aligns with the user’s intended context can be a significant challenge. You need an entire query parsing and user profiling / scoring setup.
- On top of that most foundational models are trained on vast swaths of internet text, making them susceptible to inherent biases present in the data. Merging retrieved information with such models can sometimes inadvertently perpetuate or even amplify biases.
- In addition, no foundational model will ever understand the finer points of an enterprise’s specific language (e.g. non-public product catalogue). You need a refined IR and eventually graph approach to get to the required precision levels for enterprise use.
- The capability to connect disparate databases can be both a boon and a bane. While it allows for broad knowledge access, ensuring the cohesiveness and consistency of data from different sources can be challenging.
- Which leads to a few other challenges such as data lifecycle, enterprise security, data access control, scaling of such solutions, user interfaces and integration into the existing enterprise landscape, total cost of operation aspects, etc. E.g., as the system evolves and the underlying databases grow, ensuring efficient retrieval without compromising on speed and accuracy becomes challenging.
Conclusion
A generic approach may work and get you 80% of the way. The last 20% to perfection encapsulates the complexities outlined above. It involves iterative refinement, extensive validation, and often, domain-specific adaptations. It’s the nuanced challenges that make the journey to 100% a demanding endeavor.
PS:
We’ve been in this business for very long. Gartner thinks of us as the Visionary in the space. So we got a few insights about build versus buy, shared here. And our SquirroGPT solution brings the points made above to life. Try it for yourself.
