Prompt Engineering with Claude

Peter Foy -

Along with the recent release of Claude 3, Anthropic released several useful resources for prompt engineering. Specifically, they've now got a prompt library as well as a Colab notebook for their helper Metaprompt, which:

...can guide Claude to generate a high-quality prompts tailored to your specific tasks.

In this guide, we'll walk through prompt engineering with Claude, and how to use this Metaprompt to quickly develop optimal prompts for your use case.

What is prompt engineering?

We've covered the art and practice of prompt engineering before, but as a quick review as the Claude docs highlight:

Prompt engineering is an empirical science that involves iterating and testing prompts to optimize performance.

Not to be confused with prompting, which is simply the input text you give to a model, prompt engineering involves the careful design, refinement, and evaluation of the LLMs performance.

The prompt engineering workflow

Here's how the Anthropic recommends building an empirical, test-driven workflow for prompt engineering:

Source
  1. Define the task & criteria: First, we need to clearly define the task at hand and establish measurable criteria for success, such as the cost, accuracy, and latency.
  2. Develop test cases: Next, we want to develop a diverse set of test cases of both typical and edge cases for the task at hand. These will be used to objectively assess the performance of your prompts.
  3. Write initial prompts: With the task and test cases defined, next we want to write our initial prompts. These should outline the task clearly, characteristics of an ideal response, and any other relevant context. Here we can also provide examples out ideal input/output responses, which is known as few-shot learning.
  4. Test prompts: Next we want to test these initial prompts with the test cases and evaluate them against consistent grading metrics. This step can involve human evaluation, automatic evaluation with an answer key, and so on.
  5. Refine prompts: Based on the test results, we can then reifine our input prompts to improve the performance, and continue this iterative improvement until we reach a desired level of performance.

Prompt engineering techniques

We won't go into detail on techniques here, but at a high-level, here are a few useful tips for improving prompts:

  • Be as clear & direct in your instructions as possible
  • Include examples of desired outputs in your prompts
  • Give the model a role, for example as an expert prompt engineer
  • Tell the model to think step-by-step to improve quality
  • Specify the output format you want the model to return
  • Use XML tags, for example "answer the question based on the document in the <doc><doc> tags below..."

Those are just a few examples of useful prompting techniques, and you can find more in the docs here.

Using the helper Metaprompt

Alright now that we've reviewed prompt engineering at a high-level, let's jump into their helper metaprompt Colab notebook. As the authors describe:

This is a prompt engineering tool designed to solve the "blank page problem" and give you a starting point for iteration

The Metaprompt

The Metaprompt Anthropic provides is a very long multi-shot prompt that gives the model examples of ideal prompts for various tasks. This is meant to guide Claude on what an optimal prompt looks like, so that it can be tailored to your specific task.

I recommend reading through it to get a better understanding of just how specific and comprehensive it is. For now, let's move on to using this Metaprompt to craft prompts for our own tasks.

Prompt Template 1: Educational Study Plans

The next step is to define our task and, optionally, and relevant input variables to choose. This can also be an empty list.

Here are the example tasks that Anthropic provides:

  • Drafting an email responding to a customer complaint
  • Explaining complex topics in simple terms
  • Developing a marketing strategy for a new product launch

For this example, let's set our task for an educational AI assistant, which will: Create a personalized study plan for learning a new skill.

We'll also add a few variables for the prompt:

  • "TOPIC": The subject or skill you want to learn (i.e. prompt engineering)
  • "CURRENT_SKILL_LEVEL": Your current level of proficiency in the topic (i.e. beginner, intermediate, advanced)
  • "AVAILABLE_TIME": How much time you want to spend on a daily or weekly basis to this study.
  • "LEARNING_GOALS": What you want to achieve by the end of the study plan (i.e. be able to explain the basics in simple terms, master advanced techniques, etc.).
  • "PREFERRED_LEARNING_RESOURCES": Your preference for the type of learning materials & resources (i.e. textbooks, online courses, YouTube, etc.).
TASK = "Create a personalized study plan for learning a new skill or topic"

VARIABLES = ["TOPIC", "CURRENT_SKILL_LEVEL", "AVAILABLE_TIME", "LEARNING_GOALS", "PREFERRED_LEARNING_RESOURCES"]

Next, the notebook inserts your task and variables into the Metaprompt as follows:

prompt = metaprompt.replace("{{TASK}}", TASK)
assistant_partial = "<Inputs>"
if variable_string:
    assistant_partial += variable_string + "\n</Inputs><Instructions Structure>"

message = CLIENT.messages.create(
    model=MODEL_NAME,
    max_tokens=4096,
    messages=[
        {
            "role": "user",
            "content":  prompt
        },
        {
            "role": "assistant",
            "content": assistant_partial
        }
    ],
    temperature=0
).content[0].text

After prettifying the output, here's the initial prompt template from Claude:

Variables:

{'LEARNING_GOALS', 'AVAILABLE_TIME', 'TOPIC', 'PREFERRED_LEARNING_RESOURCES', 'CURRENT_SKILL_LEVEL'}

************************

Prompt:
You will be creating a personalized study plan to help someone learn a new skill or topic. The plan should be tailored to the individual's current skill level, available time, learning goals, and preferred learning resources. Here is the information you will be working with:

Topic: <topic>{TOPIC}</topic>
Current skill level: <current_skill_level>{CURRENT_SKILL_LEVEL}</current_skill_level>
Available time: <available_time>{AVAILABLE_TIME}</available_time>
Learning goals: <learning_goals>{LEARNING_GOALS}</learning_goals>
Preferred learning resources:
<preferred_resources>{PREFERRED_LEARNING_RESOURCES}</preferred_resources>

First, acknowledge the topic the individual wants to learn. Then, consider their current skill level in this area and ensure that the plan you create is appropriate for their starting point.

Next, take into account the amount of time they have available to dedicate to learning. Use this information to determine the scope and pace of the plan.

Review their learning goals carefully and make sure that the plan you develop aligns with what they hope to achieve.

Incorporate their preferred learning resources into the plan wherever possible. If they have not provided any preferred resources, recommend high-quality resources that you think would be helpful for learning this topic.

Break the plan down into clear, manageable steps or phases. Each step should build upon the previous one and move the individual closer to their learning goals.

Include a variety of learning activities and resources in the plan, such as reading materials, videos, practice exercises, projects, and so on. Aim to create an engaging and well-rounded learning
experience.

Provide an estimated timeline for completing the plan based on the individual's available time and learning goals. Be realistic in your estimates and allow for some flexibility.

Throughout the plan, emphasize the importance of consistent practice, seeking feedback, and applying new knowledge and skills in real-world situations.

Remember to tailor the plan to the individual's unique needs and preferences. Avoid a one-size-fits-all approach and strive to create a plan that is both challenging, practical, and achievable.

After you have created the personalized study plan, present it inside <study_plan></study_plan> tags.

Quite impressive, Claude!

Testing the prompt template

Alright let's test it out with the following variables for learning generative AI:

  • Topic: Generative AI
  • Current Skill level: Intermediate
  • Available Time: 10 hours per week
  • Learning goals: Learn how to apply generative AI in practical business settings & workflows.
  • Preferred learning resources: Blog posts and YouTube videos

Here's the output:

Generative AI Study Plan

Thank you for your interest in learning about Generative AI! Given your intermediate skill level, we'll focus on expanding your knowledge and practical application of these technologies in business
settings.

Week 1-2 (2 hours per week):
- Read introductory blog posts on Generative AI applications in business, such as "10 Real-World Applications of Generative AI" and "How Generative AI is Transforming Business Processes."
- Watch YouTube videos that provide an overview of Generative AI concepts and techniques, like "Generative AI Explained" and "Introduction to Generative Adversarial Networks (GANs)."

Week 3-4 (3 hours per week):
- Dive deeper into specific Generative AI models and their use cases by reading articles such as "Understanding Variational Autoencoders (VAEs)" and "Generative AI for Content Creation."
- Follow along with YouTube tutorials that demonstrate how to implement basic Generative AI models using popular libraries like TensorFlow or PyTorch.

Week 5-7 (4 hours per week):
- Apply your knowledge by working on a small project that involves using a Generative AI model for a specific business task, such as generating product descriptions, creating realistic images, or enhancing data privacy.
- Document your project process and results in a blog post or video to share with others and gather
feedback.

Week 8-10 (1 hour per week):
- Explore real-world examples of businesses successfully implementing Generative AI by reading case studies and watching interviews with industry experts.
- Reflect on your learning journey and identify areas where you'd like to deepen your understanding or expand your skillset further.

Throughout this 10-week plan, aim to consistently engage with the learning materials and allocate time for hands-on practice. Don't hesitate to seek guidance or clarification when needed, and consider joining online communities focused on Generative AI to learn from others' experiences and
insights.

By the end of this plan, you should have a solid understanding of how Generative AI can be applied in practical business settings and workflows, as well as hands-on experience implementing these
technologies yourself. Keep exploring and experimenting beyond this plan to continue growing your skills in this exciting field!

Not bad. It didn't exactly get the time per week right...but as a starting point this is still quite useful, and definitely way better than a single shot prompt...

In order to refine this, however, the next step would be to make the task much more detailed and clearly defined, as Create a personalized study plan for learning a new skill is too broad and generic.

Also, it'd be useful to add several more examples to the Metaprompt itself with ideal input/output responses for your specific task at hand.

Summary: Prompt Engineering with Claude

In summary, this metaprompt helper is definitely a useful resource in the prompt engineers toolkit. If you're looking for inspiration for useful tasks + example prompts, definitely check out Anthropics prompt library

As a next step, I think I'll put a simple frontend on this metaprompt and give the same prompt engineering task to both Claude 3 and GPT-4 to compare the results...but we'll save that for another article.