This week in AI, NVIDIA released Chat with RTX, which lets you to build a custom chatbot that you can run locally. In this guide, we'll look at what it is and how to get started it.
NVIDIA released Chat with RTX. Why is it significant? It's NVIDIA's first step towards the vision of "LLM as Operating System" - a locally running, heavily optimized AI assistant that deeply integrates with your file systems, features retrieval as the first-class citizen,… pic.twitter.com/0aJTHwqymV
— Jim Fan (@DrJimFan) February 14, 2024
What is Chat with RTX?
Chat with RTX allows you to build a custom GPT-based large language model (LLM) that is trained on your documents, notes, videos, and more.
Chat With RTX is a demo app that lets you personalize a GPT large language model (LLM) connected to your own content—docs, notes, videos, or other data.
Chat with RTX uses retrieval augmented generation (RAG), TensorRT-LLM, and RTX acceleration in order to provide fast, contextually relevant answers based on your content. Given that it's also runs locally, this also. means your private data is secure.
Users can quickly, easily connect local files on a PC as a dataset to an open-source large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers.
How to use Chat with RTX
In order to run Chat with RTX, you'll need a Windows PC with the following system requirements:
- GPU: NVIDIA GeForce RTX 30 or 40 Series, or NVIDIA RTX Ampere or Ada Generation, with a minimum of 8GB VRAM
- RAM: 16GB or more
- OS: Windows 11
- Driver: Version 535.11 or newer
If your system meets those requirements, here's how you can get started with Chat with RTX:
- Installation: Download and install Chat with RTX from NVIDIA's website
- Loading Content: Chat with RTX's personalized chatbot becomes more useful with your content. The app supports text, PDF, DOC/DOCX, and XML files. Direct the app to your content's folder for it to be indexed. You can also add YouTube playlists; the app will load video transcriptions, making them searchable through the chatbot.
- Chatting with your AI: With your content loaded, you can start chatting with your personalized AI. Whether you're looking for specific information from your documents or want insights based on your video content, Chat with RTX can provide quick, accurate responses.
Developer Opportunities
Chat with RTX is not just for end-users; it's also a platform for developers. Based on the TensorRT-LLM RAG developer reference project, it offers a foundation for developing and deploying RAG-based applications optimized for RTX hardware.
Summary: Chat with RTX
NVIDIA's Chat with RTX offers a unique opportunity to enhance your productivity with a personalized chatbot trained on your own data. Given that it the LLM runs locally, it's ideal for security-conscious developers and AI practitioners. Now if they would just release a Mac version that would be even more ideal...
