NVIDIA Releases Chat with RTX

This week in AI, NVIDIA released Chat with RTX, which lets you to build a custom chatbot that you can run locally.

8 months ago   •   2 min read

By Peter Foy
Table of contents

This week in AI, NVIDIA released Chat with RTX, which lets you to build a custom chatbot that you can run locally. In this guide, we'll look at what it is and how to get started it.

What is Chat with RTX?

Chat with RTX allows you to build a custom GPT-based large language model (LLM) that is trained on your documents, notes, videos, and more.

Chat With RTX is a demo app that lets you personalize a GPT large language model (LLM) connected to your own content—docs, notes, videos, or other data.

Chat with RTX uses retrieval augmented generation (RAG), TensorRT-LLM, and RTX acceleration in order to provide fast, contextually relevant answers based on your content. Given that it's also runs locally, this also. means your private data is secure.

Users can quickly, easily connect local files on a PC as a dataset to an open-source large language model like Mistral or Llama 2, enabling queries for quick, contextually relevant answers.

How to use Chat with RTX

In order to run Chat with RTX, you'll need a Windows PC with the following system requirements:

  • GPU: NVIDIA GeForce RTX 30 or 40 Series, or NVIDIA RTX Ampere or Ada Generation, with a minimum of 8GB VRAM
  • RAM: 16GB or more
  • OS: Windows 11
  • Driver: Version 535.11 or newer

If your system meets those requirements, here's how you can get started with Chat with RTX:

  • Installation: Download and install Chat with RTX from NVIDIA's website
  • Loading Content: Chat with RTX's personalized chatbot becomes more useful with your content. The app supports text, PDF, DOC/DOCX, and XML files. Direct the app to your content's folder for it to be indexed. You can also add YouTube playlists; the app will load video transcriptions, making them searchable through the chatbot.
  • Chatting with your AI: With your content loaded, you can start chatting with your personalized AI. Whether you're looking for specific information from your documents or want insights based on your video content, Chat with RTX can provide quick, accurate responses.

Developer Opportunities

Chat with RTX is not just for end-users; it's also a platform for developers. Based on the TensorRT-LLM RAG developer reference project, it offers a foundation for developing and deploying RAG-based applications optimized for RTX hardware.

Summary: Chat with RTX

NVIDIA's Chat with RTX offers a unique opportunity to enhance your productivity with a personalized chatbot trained on your own data. Given that it the LLM runs locally, it's ideal for security-conscious developers and AI practitioners. Now if they would just release a Mac version that would be even more ideal...

Spread the word

Keep reading