ChatGPT Plugins: Key Concepts & Use Cases

Another week, another huge announcement from OpenAI: ChatGPT plugins have arrived.

In this guide, we'll discuss exactly what ChatGPT plugins are, what's possible with plugins, and why many are calling this the "App Store moment of AI".

Specifically, we'll discuss:

• What are ChatGPT plugins?
• Examples of ChatGPT plugins
• ChatGPT web browsing
• ChatGPT code interpreter
• ChatGPT retrieval plugins
• ChatGPT third-party plugins
• How to create a ChatGPT plugin

Let's get started.

What are ChatGPT plugins?

As OpenAI highlights in their blog post:

Plugins are tools designed specifically for language models with safety as a core principle, and help ChatGPT access up-to-date information, run computations, or use third-party services.

Plugins are essentially chat extensions that tell ChatGPT what they can do and when to invoke them, enabling the LLM to access up-to-date information, perform computations, interpret code, browse the web.

Though not a perfect analogy, plugins can be “eyes and ears” for language models, giving them access to information that is too recent, too personal, or too specific to be included in the training data.

In other words, plugins open up the power of ChatGPT to the rest of the internet...and if that isn't enough to get you excited (and a bit terrified), I'm not sure what will.

A plugin consists of an API, an API schema (i.e. OpenAPI JSON or YAML file), and a manifest that describes what the plugin can do for both humans and models, as well as some other plugin metadata.

Examples of ChatGPT plugins

So far, the following plugins have already been created, although it's quite obvious there will be a Cambrian explosion of ChatGPT plugins in the coming weeks and months...

Aside from third-party ChatGPT plugins, OpenAI mentions they're hosting two plugins internally to start: web browsing and a code interpreter. Lastly, they've open sourced the code for a retrieval plugin.

Let's check out each of these in a bit more detail.

Become a Prompt Engineer: Prompt Engineering & LLMs Track

In this Prompt Engineering track, you’ll learn essential skills, tools, and techniques for working with LLM-enabled applications.

MLQ.aiPeter Foy

ChatGPT web browsing

An experimental model that knows when and how to browse the internet

OpenAI has already done a significant amount of research into adding web browsing to LLMs, and since it's such an essential utility of opening up ChatGPT to the rest of the internet, it makes sense they're handling this one internally.

Similar to the LangChain library, when using this plugin you can see exactly what browsing steps the LLM is taking, including what it searched, clicked on, and read.

As we can see, the plugin also provides links and cites the sources used in the answer.

They also mention that due to safety considerations, the plugin is only to make GET requests to reduce certain risks, which means it can only retrieve information and not perform "transactional" requests such as filling out a form.

ChatGPT code interpreter

An experimental ChatGPT model that can use Python, handle uploads and downloads

Whereas before ChatGPT could understand and write code, now it can actually interpret and run code in the browser.

For now, the models are just provided a Python interpreter, although I expect they'll add support for other coding languages at some point. It also supports uploading files to the conversation workspace and downloading the results. As OpenAI writes:

Having access to a very eager junior programmer working at the speed of your fingertips can make completely new workflows effortless and efficient, as well as open the benefits of programming to new audiences.

They also mention a few cases where the code interpreter is particularly useful:

• Solving mathematical problem
• Data analysis and visualization
• Converting files between multiple formats

ChatGPT retrieval plugins

OpenAI also open-sourced a ChatGPT retrieval plugin, which:

...provides a flexible solution for semantic search and retrieval of personal or organizational documents using natural language queries.

In other words, the retrieval plugin enables you to find relevant document snippets from files, notes, emails, and so on. Companies can also make their internal documents available to employees via ChatGPT with this plugin.

As expected, the retrieval plugin uses text-embedding-ada-002 embeddings model to compute the embeddings of each document chunk, and then stores these at a vector database, a process that I've gone through several times in previous articles.

Given that it's open-sourced and self-hosted, developers can deploy their own retrieval plugins and register them with ChatGPT. They also support multiple vector databases including Pinecone, Weaviate, and several others.

ChatGPT third-party plugins

An experimental model that knows when and how to use plugins.

Third-party plugins can also be used by reading the plugin descriptions and deciding when and how to use them. Third-party plugins are described by a manifest file, for example:

{
  "schema_version": "v1",
  "name_for_human": "TODO Manager",
  "name_for_model": "todo_manager",
  "description_for_human": "Manages your TODOs!",
  "description_for_model": "An app for managing a user's TODOs",
  "api": { "url": "/openapi.json" },
  "auth": { "type": "none" },
  "logo_url": "https://example.com/logo.png",
  "legal_info_url": "http://example.com",
  "contact_email": "hello@example.com"
}

How to create a ChatGPT plugin

As mentioned I'll do a full technical tutorial on this topic shortly, although at a high level, the steps that OpenAI highlights to create a plugin include:

1. Build an API with endpoints for the LLM to call, these can be new APIs, existing APIs, or a wrapper around existing APIs that are specifically designed for ChatGPT and language models
2. Submit an OpenAPI specification documentation as well as a manifest file that describes the API for both models and humans, along some other metadata

After creating a ChatGPT plugin, users can choose which plugin to enable which can be invoked by the above-mentioned API documentation.

Summary: ChatGPT plugins

In summary, ChatGPT plugins open up the power of large language models to an incredibly wide range of new use cases and possibilities.

Just providing web browsing alone is a big step forward, so there's no surprise that people are so excited about this one, calling it an even bigger announcement than last week's GPT-4 release.

Stay tuned for a developer-focused tutorial on how to build ChatGPT plugins, until then, here are 10 plugin app ideas from GPT-4: