Overview:

These sources offer a comprehensive technical guide for integrating and optimising Claude models within the Amazon Bedrock ecosystem. They detail the programmatic implementation of AI services using the Boto3 library, covering essential functionalities such as inference configuration, real-time streaming, and structured JSON output control. Beyond simple deployment, the text emphasises a rigorous five-step evaluation workflow to objectively measure performance through automated datasets and hybrid grading systems. Furthermore, it outlines advanced prompt engineering strategies, including the use of XML delimiters and multi-shot prompting, to refine model accuracy and reliability. By combining practical coding examples with systematic testing methodologies, the documentation provides a blueprint for building production-ready, high-performance AI applications.

Bedrock API

This course about integrating and deploying Claude through Amazon Bedrock1. Accroding to the Claude2, there are three models from the Claude:

Cluade Models Image from Claude Academy

I am using Amazon Bedroc in ap-southeast-2 which is the closest to the Sydney. It is important to find the available models and the Model IDs in the region. Currently my system has the following versions:

%%bash
aws bedrock list-foundation-models --by-provider anthropic --query "modelSummaries[*].modelId" --output table

-----------------------------------------------
|            ListFoundationModels             |
+---------------------------------------------+
|  anthropic.claude-haiku-4-5-20251001-v1:0   |
|  anthropic.claude-fable-5                   |
|  anthropic.claude-sonnet-4-6                |
|  anthropic.claude-opus-4-6-v1               |
|  anthropic.claude-opus-5                    |
|  anthropic.claude-opus-4-8                  |
|  anthropic.claude-opus-4-7                  |
|  anthropic.claude-sonnet-4-5-20250929-v1:0  |
|  anthropic.claude-fable-5-1                 |
|  anthropic.claude-sonnet-5                  |
|  anthropic.claude-opus-4-5-20251101-v1:0    |
|  anthropic.claude-sonnet-4-20250514-v1:0    |
+---------------------------------------------+

The command aws bedrock list-foundation-models --by-provider anthropic --query "modelSummaries[*].modelId" --output table lists all available Anthropic foundation models in Amazon Bedrock.

Breakdown:

Part Description
aws bedrock AWS CLI service for Amazon Bedrock
list-foundation-models Operation to retrieve available foundation models
--by-provider anthropic Filters results to only show models from Anthropic
--query "modelSummaries[*].modelId" JMESPath query to extract only the modelId field from each model summary
--output table Formats output as an ASCII table for readability

Choose Sonnet when you need balance. Most applications benefit from Sonnet’s combination of intelligence, speed, and reasonable cost. {.ok}

Essential component to connect to the Bedrock Model:

  1. Bedrock runtime client
  2. Model ID
  3. Prompt message

You can create client conntecting the Bedrock runtime:

import boto3

client = boto3.client('bedrock-runtime', region_name='ap-southeast-2')

User Inference profile Image from Claude Academy

Inference profile automatically route the request to the region where your choosen model is available.

As per above you can use the default Opus, Sonnet, or

user_message = {
    "role": "user",
    "content": [
        {"text": "What is the capital of Sri Lanka?"}
    ]
}

response = client.converse(
    modelId='au.anthropic.claude-opus-4-8',
    messages=[user_message],
)

print(response["output"]["message"]["content"][0]["text"])

Sri Lanka has two capitals:

1. **Sri Jayawardenepura Kotte** – This is the official (administrative) capital, where the parliament and legislative functions are located. It's often considered the "official" capital.

2. **Colombo** – This is the commercial capital and largest city. It serves as the executive and judicial center and is frequently referred to as the capital in casual contexts.

Sri Jayawardenepura Kotte is actually a suburb of the larger Colombo metropolitan area, which is why there's often some confusion. The capital was officially moved from Colombo to Sri Jayawardenepura Kotte in 1982.

Multi-turn conversation

Both Bedrock runtime and Claude model don’t store any messages. Therefore, for a conversation where you need to store the history. You can mannualy or programmatically maintain the history of all the messages in the follow up prompt: this is called context.

Conversation should follow the user → assistant → user → assistant pattern.

System Prompts

The Problem with User Instructions: Putting rules in user messages is unwieldy, cluttering, requires anticipating every edge case, and forces repetitive instructions. The System Prompt is a Solution, Instructing Claude to adopt a specific persona/role naturally aligns its knowledge, tone, and constraints without long lists of rule exceptions.

model_id = "au.anthropic.claude-haiku-4-5-20251001-v1:0"
user_message = {
    "role": "user",
    "content": [
        {"text": "What are the best tourist locations in Port Villa?"}
    ]
}

response = client.converse(
    modelId=model_id,
    messages=[user_message],
    system=[{"text": """You are a helpful tourist guide who provides information about tourist locations in Port Villa."""}]
)

The system prompt cannot be empty string. At least one character need. System prompts are processed before any user messages in the conversation.

print(response["output"]["message"]["content"][0]["text"])

# Best Tourist Locations in Port Vila

Here are the must-visit attractions in Port Vila, Vanuatu:

## **Beaches & Water Activities**
- **Erakor Beach** - Beautiful sandy beach with calm waters, perfect for swimming and water sports
- **Irikiki Island** - Nearby island with pristine beaches, snorkeling, and day trips available
- **Port Vila Waterfront** - Scenic promenade for walks, dining, and ocean views

## **Cultural & Historical Sites**
- **Vanuatu National Museum** - Showcases local history, artifacts, and cultural exhibits
- **Port Vila Market** - Vibrant local market with traditional crafts, produce, and souvenirs
- **Chief Roi Mata's Domain** - UNESCO World Heritage Site with historical significance (day trip)

## **Nature & Adventure**
- **Mele Cascades** - Stunning waterfall with natural pools for swimming (about 15 minutes from town)
- **Hideaway Island** - Snorkeling, diving, and underwater post office
- **Local Gardens** - Various botanical gardens showcasing tropical flora

## **Dining & Entertainment**
- **Waterfront restaurants** - Fresh seafood with ocean views
- **Local craft shops** - Handmade souvenirs and traditional items

## **Tips**
- Best visited during the dry season (May-October)
- Hire a local guide for cultural insights
- Many attractions are within 15-30 minutes of the city center

Would you like more specific information about any of these locations?

Temperature

The temperature is a 0 to 1 dial for the creativity. Lower value make the heghest probability tokens much more and higher temperature is more about token distributed probability.

Low temperature (More deterministic output)

Selection Probability
   ▲   
   │ █   
   │ █   
   │ █   
   │ █   
   │ █   
───┴─┴─┴─┴─┴─┴─┴──►   
     a w o i w m w  Tokens

Hight temperature (More random output)

Selection Probability 
   ▲   
   │ █ █    
   │ █ █ █    
   │ █ █ █ █ █   
   │ █ █ █ █ █ █   
   │ █ █ █ █ █ █ █  
───┴─┴─┴─┴─┴─┴─┴──►
     a w o i w m w  Tokens

Claude recommendations are:

Low Temperature (0.0 - 0.3)

  • Factual responses
  • Coding assistance
  • Data extraction
  • Content moderation

Medium Temperature (0.4 - 0.7)

  • Summarization
  • Educational content
  • Problem-solving
  • Creative writing with constraints

High Temperature (0.8 - 1.0)

  • Brainstorming
  • Creative writing
  • Marketing content
  • Joke generation

Claude’s temperature is set to 1.0.

model_id = "au.anthropic.claude-haiku-4-5-20251001-v1:0"
user_message = {
    "role": "user",
    "content": [
        {"text": "What are the best travel plan to follow tourist attractions in Port Villa within a day (10 am - 4 pm)?"}
    ]
}

response = client.converse(
    modelId=model_id,
    messages=[user_message],
    system=[{"text": """You are a helpful tourist guide who provides travel advice about Port Villa tourist attractions."""}],
    inferenceConfig={"temperature": 1.0}
)

print(response["output"]["message"]["content"][0]["text"])

# One-Day Port Vila Tourist Guide (10 AM - 4 PM)

Here's an efficient itinerary to maximize your time:

## **10:00 AM - Efate Water Park**
- Start with water activities or relax by the pools
- *Location:* Central Port Vila
- *Duration:* 1-1.5 hours

## **11:30 AM - Port Vila Market**
- Browse local crafts, fresh produce, and souvenirs
- Experience authentic local culture
- *Duration:* 45 minutes

## **12:15 PM - Lunch**
- Eat at a local restaurant near the market or waterfront
- Try Vanuatu specialties like fresh seafood

## **1:15 PM - Vanuatu Cultural Centre**
- Learn about indigenous culture and history
- Browse handicrafts and art
- *Duration:* 1 hour

## **2:15 PM - Shol's Beach or Seaside Promenade**
- Relax and enjoy ocean views
- Take photos of the sunset area
- *Duration:* 45 minutes

## **3:00 PM - Local Shops & Handicrafts**
- Browse boutique shops near the waterfront
- Pick up last-minute souvenirs

## **4:00 PM - Wrap-up**

### **Pro Tips:**
- Book water activities in advance
- Wear sunscreen and stay hydrated
- Use taxis between locations
- Wear comfortable walking shoes

Would you like specific recommendations for restaurants or accommodation?

Streaming

Standard requests force users to wait 10–30 seconds for a complete AI response. Streaming provides immediate visual feedback by transmitting response fragments as they are generated, shifting the user experience from “submit and wait” to “submit and watch response appear”.

Calling client.converse_stream() returns an initial response containing a generator stream object. Iterating over this stream yields real-time event objects as chunks arrive.

sequenceDiagram
    autonumber
    actor User / App
    participant Bedrock as Amazon Bedrock API
    
    User / App->>Bedrock: converse_stream(messages, modelId)
    activate Bedrock
    Bedrock-->>User / App: Returns Stream Object (Generator)
    deactivate Bedrock
    
    loop Stream Event Processing
        Bedrock-->>User / App: messageStart
        loop For each generated chunk
            Bedrock-->>User / App: contentBlockDelta (text chunk)
            Note over User / App: Display/Process text chunk in real-time
        end
        Bedrock-->>User / App: contentBlockStop
        Bedrock-->>User / App: messageStop
        Bedrock-->>User / App: metadata (usage statistics, stop reason)
    end

When you call converse_stream, you immediately get back an initial response that contains a stream object.

model_id = "au.anthropic.claude-haiku-4-5-20251001-v1:0"
user_message = {
    "role": "user",
    "content": [
        {"text": "What are the best travel plan to follow tourist attractions in Mistery Island, Vanuatu within a day (10 am - 4 pm)?"}
    ]
}

response = client.converse_stream(messages=[user_message], modelId=model_id)

text = ""
for event in response["stream"]:
    if "contentBlockDelta" in event:
        chunk = event["contentBlockDelta"]["delta"]["text"]
        print(chunk, end="", flush=True)
        text += chunk

# print("\n\nTotal Message:\n" + text)

# One-Day Itinerary for Mystery Island, Vanuatu (10 AM - 4 PM)

## Quick Overview
Mystery Island is a small, uninhabited island accessible by daily catamaran. Here's an optimized plan:

## Suggested Schedule

**10:00 AM - Arrival & Settlement**
- Disembark and settle into the beach area
- Store belongings, apply sunscreen
- Get oriented with facilities

**10:30 AM - 12:00 PM - Beach & Snorkeling**
- Explore the pristine white-sand beach
- Snorkel in crystal-clear waters (gear usually provided)
- Spot tropical fish and coral
- Visit the wreck of the SS President Coolidge (if snorkeling)

**12:00 PM - 1:30 PM - Lunch**
- Enjoy lunch at island facilities or packed meal
- Rest in the shade
- Optional: explore the island's interior trails

**1:30 PM - 3:00 PM - Activities**
- Glass-bottom boat tour (if available)
- Further snorkeling
- Beach volleyball or relaxation
- Photography at scenic spots

**3:00 PM - 4:00 PM - Final Hours**
- Last swim or snorkel
- Collect belongings
- Prepare for departure

## Tips
✓ Book tours through Port Vila operators (Island Cruises, Captain Cook Cruises)
✓ Bring reef shoes, high SPF sunscreen, and underwater camera
✓ Water is warm year-round
✓ Catamaran ride takes ~45 minutes each way

Would you like specific operator recommendations?

Output control with biasness

Two core techniques for steering and constraining model generations beyond basic prompt engineering: Prefilled Assistant Messages and Stop Sequences.

graph TD
    A[Control Techniques for Claude] --> B[Prefilled Assistant Messages]
    A --> C[Stop Sequences]
    
    B --> B1[Steer direction & tone]
    B --> B2[Force specific output format]
    B --> B3[Claude continues directly after prefill]
    
    C --> C1[Truncate output at specific string]
    C --> C2[Exclude stop string from response]
    C --> C3[Enforce natural breakpoints / length limits]
  1. Prefilled Assistant Messages (Output Steering):
    • Mechanism: You insert an assistant role message at the end of the messages array containing the exact starting text you want Claude to begin with.
    • Behavior: Claude assumes it already wrote that opening fragment and continues directly from where you left off. It does not repeat the prefilled text in its response.
    • Use Case: Biasing sentiment, setting specific starting formats (e.g., forcing JSON opening {), or guiding response structure. 
      sequenceDiagram
    autonumber
    actor User as User Application
    participant Bedrock as Claude (Amazon Bedrock)

    User->>Bedrock: Send messages array:<br/>1. user: "Is coffee or tea better?"<br/>2. assistant: "Tea is better because"
    Note over Bedrock: Claude sees prefill and continues generation from where it left off.
    Bedrock-->>User: Returns continuation: "it has more caffeine."
    Note over User: Full Response = Prefill + Output:<br/>"Tea is better because it has less caffeine."
  2. Stop Sequences (Output Truncation):
    • Mechanism: Passed under inferenceConfig -> stopSequences as an array of strings (e.g., ["5"], ["\n\n"]).
    • Behavior: As soon as Claude generates any string in the list, generation halts immediately. The stop sequence string itself is omitted from the returned output.
    • Use Case: Preventing responses from running past boundaries, stopping at specific delimiters, or capping output length cleanly. 
      sequenceDiagram
    autonumber
    actor Client as Client Application
    participant Bedrock as Claude API (Bedrock)

    Client->>Bedrock: Send Request:<br/>messages = [<br/>  {role: "user", content: "Is coffee or tea better?"},<br/>  {role: "assistant", content: "Tea is better because"}<br/>]<br/>stopSequences = ["**Consider:**"]
    Note over Bedrock: Generates tokens for Coffee vs Tea bullet points...<br/>Detects target stop string "**Consider:**"
    Note over Bedrock: Halts generation immediately.<br/>Strips "**Consider:**" from final output.
    Bedrock-->>Client: Returns Continuation:<br/>"coffee can cause jitters... [Tea specs]"<br/>(Truncated right before **Consider:**)

Here the example:

model_id = "au.anthropic.claude-haiku-4-5-20251001-v1:0" 
# 1. Setup messages with a prefilled assistant start
messages = [
    {"role": "user", "content": [{"text": "Is coffee or tea better for breakfast?"}]},
    {"role": "assistant", "content": [{"text": "Tea is better because"}]}
]

# 2. Invoke Bedrock Converse API with stop sequences
response = client.converse(
    modelId=model_id,
    messages=messages,
    inferenceConfig={
        "temperature": 1.0,
        "stopSequences": ["**Consider**"]
    }
)

# Output continuation from prefilled text
continuation = response["output"]["message"]["content"][0]["text"]
full_response = "Tea is better because" + continuation

print(full_response)

Tea is better because the caffeine kicks in more gradually, giving you stable energy without the jitters. It's also easier on the stomach.

Actually, I should be more balanced: **it depends on what works for you.**

**Coffee** offers:
- Faster energy boost
- More caffeine per serving
- Bold flavor some prefer

**Tea** offers:
- Gentler caffeine release
- L-theanine (promotes calm focus)
- Often easier on digestion
- Less likely to cause crashes

**Better approach:** Consider your own digestion, caffeine sensitivity, and what taste you enjoy. Some people do great with coffee; others feel jittery. Neither is objectively "better"—it's personal.

What matters more is eating actual food with your drink rather than caffeine alone.

Above has stopped at **consider** in the following text something similar to the following text:

Tea is better because coffee can cause jitters and crashes, while tea provides a gentler caffeine boost.

Actually, ...:

**Coffee** tends to offer:
- ...

**Tea** tends to offer:
- ...


**Consider:**
- Your caffeine sensitivity
- What flavor appeals to you
- How your body responds
- Whether you eat food with it (helps either go down easier)

...

When you passed {"role": "assistant", "content": "Tea is better because"} in the messages array:

  • What Claude saw: Claude treats the prefilled text as tokens it has already written. It does not re-generate "Tea is better because".

  • What Claude generated: It picked up immediately after the word "because" with:

    `coffee can cause jitters and crashes, while tea provides a gentler caffeine boost...`
  • The Impact: Even though your user prompt asked an open-ended question (“Is coffee or tea better?”), the prefill forced Claude to immediately argue in favor of tea in its opening sentence. Combining the prefill string with Claude’s API payload output yields the complete first line.

Structured Output

A common challenge when integrating Claude into automated software pipelines is to ensuring the model returns clean, pure structured data (such as JSON, CSV, or code) without conversational filler, headers, or markdown wrappers.

Instead of relying strictly on prompt instructions, the standard technique uses two API mechanisms together:

  1. Assistant Message Prefilling: Pass ```json (or the opening syntax for your desired format) as the starting text in the assistant role message within the messages array.
    • Effect: Claude assumes it has already begun outputting the response inside a markdown block and immediately starts writing the raw data payload, skipping intros and headers.
  2. Stop Sequences (stop_sequences=[”```”]): Configure ``` as a stop sequence in the API request call.
    • Effect: When Claude finishes generating the JSON payload and attempts to output the closing markdown tag ( ```), the API immediately halts token generation.
    sequenceDiagram
        autonumber
        actor App as Client Application
        participant API as Amazon Bedrock API
        participant Model as Claude Model Context

        App->>API: Send Request<br/>• User: 'Generate EventBridge rule as JSON'<br/>• Assistant: '```json'<br/>• stop_sequences: ['```']
        API->>Model: Load message history & prompt context
        Note over Model: Sees '```json' as already written.<br/>Skips conversational intro & headers.<br/>Generates raw JSON content directly.
        Model->>API: Stream tokens: '{\n  "source": ["aws.ec2"], ...'
        Note over Model: Completes JSON structure and attempts<br/>to generate closing markdown delimiter: '```'
        API-->>Model: Halt generation (Stop Sequence matched)
        API->>App: Return response string (Pure JSON content)
        App->>App: clean_data = json.loads(text.strip())

Example code

import json
# Opening delimiter prefill
prefill_text = "```json"

messages = [
    {
        "role": "user",
        "content": [
            {
                "text": (
                    "Generate a JSON list of 3 sample AWS Bedrock providers for an enterprise environment. "
                    "Each entry must include: provider and model summary. "
                )
            }
        ],
    },
    {
        "role": "assistant",
        "content": [{"text": prefill_text}],
    },
]

# Invoke Bedrock Converse API with closing code block delimiter as a stop sequence
response = client.converse(
    modelId=model_id,
    messages=messages,
    inferenceConfig={
        "temperature": 0.1,  # Low temperature for deterministic output
        "stopSequences": ["```"],  # Stops execution right when Claude tries to close the code block
    },
)

# Extract output continuation and clean the payload
continuation = response["output"]["message"]["content"][0]["text"]

# Combine prefill (optional, depending on if you parse continuation directly)
raw_json = continuation.strip()

# Parse directly into Python data structures without regex
rj_output = json.loads(raw_json)

# Pretty-print the validated JSON output
print(json.dumps(rj_output, indent=2))

{
  "bedrock_providers": [
    {
      "provider": "Anthropic",
      "model": "Claude 3 Opus",
      "summary": "Advanced large language model optimized for complex reasoning, analysis, and enterprise applications. Supports 200K token context window, ideal for document processing and multi-turn conversations in regulated industries."
    },
    {
      "provider": "Meta",
      "model": "Llama 2 70B",
      "summary": "Open-source large language model designed for enterprise deployment with strong performance on coding, reasoning, and instruction-following tasks. Cost-effective option with good throughput for high-volume workloads."
    },
    {
      "provider": "Cohere",
      "model": "Command R Plus",
      "summary": "Enterprise-grade model specialized in retrieval-augmented generation (RAG), semantic search, and knowledge-intensive tasks. Optimized for business applications with strong multilingual support and low latency requirements."
    }
  ]
}

Evals

Writing a prompt is only the start of building AI applications. While Prompt Engineering focuses on crafting instructions to help Claude understand requirements, Prompt Evaluation provides automated, objective testing to measure how well those prompts perform across diverse scenarios before reaching production.

Concept Primary Focus Objective Key Techniques / Activities
Prompt Engineering Craft & Construction Crafting effective instructions so Claude understands intent. Multishot prompting, XML tag structuring, role setting, formatting constraints.
Prompt Evaluation Measurement & Testing Generating objective metrics to measure real-world performance. Automated test runs against datasets, output scoring, error analysis, version comparison.

When developing an AI application, engineers generally follow one of three paths3 after writing an initial prompt:

    graph TD
        A[Draft Initial Prompt] --> B{Evaluation Path}
        
        B -->|Path 1| C[Test Once]
        C --> C_Risk["⚠️ High Production Risk<br/>Breaks when users input unexpected text"]
        
        B -->|Path 2| D[Ad-hoc Tweaks]
        D --> D_Risk["⚠️ Vulnerable<br/>Handles obvious corner cases but fails on unconsidered inputs"]
        
        B -->|Path 3| E[Automated Eval Pipeline]
        E --> F[Score against test dataset & benchmark metrics]
        F --> G[Iterate prompt based on objective data]
        G --> H["✅ High Reliability<br/>Catches edge cases before deployment"]

        style C_Risk fill:#fee,stroke:#f66,stroke-width:1px
        style D_Risk fill:#ffe,stroke:#fc0,stroke-width:1px
        style H fill:#efe,stroke:#3b3,stroke-width:1px

Claude Academy (Claude with Amazon Bedrock) outlines a systematic, 5-step evaluation workflow4 designed to objectively measure, score, and iterate on LLM prompt performance rather than relying on subjective intuition.

  1. Draft a prompt: initial prompt for baseline
  2. Create an Eval dataset: Prepare manually or generate via Claude
  3. Feed through Claude: collect the Claude’s repsonse
  4. Feed through a Grader: Q&A pair need to grade from 1 to 10. Calculate the avarage
  5. Change prompt and repeat: Base on the avarage of the above repeate to get better result.
flowchart TD
    S1["Step 1: Draft Initial Prompt Template"] --> S2["Step 2: Create Evaluation Dataset"]
    S2 --> S3["Step 3: Feed Inputs & Prompt through Claude"]
    S3 --> S4["Step 4: Score Responses via Grader"]
    S4 --> S5{"Analyze Aggregate Score"}
    S5 -->|"Refine Prompt (v2, v3...)"| S3

Evals

Decoupling dataset generation from prompt evaluation is standard practice in LLM benchmarking. Saving the dataset to disk ensures your prompt variations (v1​,v2​,…) are evaluated against the exact same static inputs while saving unnecessary API calls. No need of headers, footer or explanation.

DATASET_FILE = "eval_dataset.json"
# --- File Persistence Helpers ---
def save_dataset(dataset, filepath=DATASET_FILE):
    """Saves the generated dataset list to a local JSON file."""
    with open(filepath, "w", encoding="utf-8") as f:
        json.dump(dataset, f, indent=2)
    print(f"✓ Dataset saved to '{filepath}'.")


def load_dataset(filepath=DATASET_FILE):
    """Loads the dataset list from a local JSON file."""
    if not os.path.exists(filepath):
        raise FileNotFoundError(
            f"Dataset file '{filepath}' not found. Generate it first."
        )
    with open(filepath, "r", encoding="utf-8") as f:
        dataset = json.load(f)
    print(f"✓ Dataset loaded from '{filepath}' ({len(dataset)} tasks found).")
    return dataset

Here the pipeline functionality to generate dataset:

# --- Core Pipeline Functions ---
def generate_dataset():
    """Generates synthetic tasks using Bedrock Claude."""
    dataset_prompt = """
    Generate 3 AWS-related tasks that require Python, JSON, or Regex solutions.
    Focus on tasks solvable by a single Python function or JSON object.
    
    Example output format:
    [
        {"task": "Write a Regex to match an AWS S3 bucket name."}
    ]
    """

    messages = [
        {"role": "user", "content": [{"text": dataset_prompt.strip()}]},
        {"role": "assistant", "content": [{"text": "```json"}]},
    ]

    response = client.converse(
        modelId=model_id,
        messages=messages,
        inferenceConfig={"temperature": 0.1, "stopSequences": ["```"]},
    )
    return json.loads(response["output"]["message"]["content"][0]["text"].strip())


def solve_task(task_description):
    """Runs a task through the candidate prompt."""
    formatted_prompt = EVAL_PROMPT_TEMPLATE.format(task=task_description)
    messages = [{"role": "user", "content": [{"text": formatted_prompt.strip()}]}]

    response = client.converse(
        modelId=model_id,
        messages=messages,
        inferenceConfig={"temperature": 0.1},
    )
    return response["output"]["message"]["content"][0]["text"].strip()

GRADER_PROMPT_TEMPLATE = """
You are an expert software engineer evaluating an AI's response to a task.

Task: {task}
Solution: {solution}

Evaluate the solution against the task description. Return a valid JSON object matching EXACTLY this structure:
score
"""


def grade_solution(task_description, solution_text):
    """Grades a solution using Claude as LLM judge with prefilled JSON structure."""
    formatted_prompt = GRADER_PROMPT_TEMPLATE.format(
        task=task_description, solution=solution_text
    )

    messages = [
        {"role": "user", "content": [{"text": formatted_prompt.strip()}]},
        # Prefill forces Claude to start with the JSON opening brace and 'score' key
        {"role": "assistant", "content": [{"text": '```json\n{\n  "score":'}]},
    ]

    response = client.converse(
        modelId=model_id,
        messages=messages,
        inferenceConfig={"temperature": 0.0, "stopSequences": ["```"]},
    )

    # Reconstruct the raw JSON string by prepending the prefilled prefix
    completion_text = response["output"]["message"]["content"][0]["text"].strip()
    full_json_str = '{\n  "score":' + completion_text

    # Parse JSON cleanly
    data = json.loads(full_json_str)

    # Defensive key lookup (handles case-sensitivity or key variance)
    score = data.get("score") or data.get("Score") or 0
    reasoning = data.get("reasoning") or data.get("Reasoning") or "No reasoning provided."

    return {
        "score": int(score),
        "reasoning": reasoning,
        "strengths": data.get("strengths", []),
        "weaknesses": data.get("weaknesses", []),
    }

STEP 1 is to generate the Dataset and save to a file:

import os

if os.path.exists(DATASET_FILE):
    print(f"Found existing dataset file. Loading from '{DATASET_FILE}'...")
    dataset = load_dataset(DATASET_FILE)
else:
    print("No dataset file found. Generating dataset from Bedrock...")
    dataset = generate_dataset()
    save_dataset(dataset, DATASET_FILE)

Found existing dataset file. Loading from 'eval_dataset.json'...
✓ Dataset loaded from 'eval_dataset.json' (3 tasks found).

Here the file contents:

%%bash
cat eval_dataset.json

[
  {
    "task": "Write a Python function that parses an AWS CloudFormation template (JSON) and extracts all resource logical IDs that have type 'AWS::Lambda::Function'."
  },
  {
    "task": "Write a Regex pattern to validate an AWS IAM role ARN format (arn:aws:iam::123456789012:role/RoleName)."
  },
  {
    "task": "Write a Python function that takes an AWS CloudWatch Logs query result (JSON array of log events) and filters events where the 'level' field equals 'ERROR', returning only the 'message' and '@timestamp' fields."
  }
]

Then run Evaluation Loop using the retrieved dataset file

EVAL_PROMPT_TEMPLATE = """
Please provide a solution to the following task in JSON format:

{task}
"""

results = []
print(f"\nEvaluating prompt across {len(dataset)} tasks...\n" + "=" * 50)

for idx, item in enumerate(dataset, 1):
    task_text = item["task"]
    print(f"\n[Task {idx}]: {task_text}")

    # Solve task loaded from file
    solution = solve_task(task_text)
    print(f"Solution generated.")

    # Grade solution
    evaluation = grade_solution(task_text, solution)
    print(f"Grade: {evaluation['score']}/10")
    print(f"Reason: {evaluation['reasoning']}")

    results.append(
        {
            "task": task_text,
            "solution": solution,
            "score": evaluation["score"],
            "evaluation": evaluation,
        }
    )





Evaluating prompt across 3 tasks...
==================================================

[Task 1]: Write a Python function that parses an AWS CloudFormation template (JSON) and extracts all resource logical IDs that have type 'AWS::Lambda::Function'.
Solution generated.
Grade: 9/10
Reason: The solution effectively addresses the task requirements with a well-implemented, production-ready function. It correctly parses CloudFormation templates and extracts Lambda function logical IDs. The code is clean, properly documented, and includes comprehensive test cases. Minor areas for improvement exist around edge case handling and validation.

[Task 2]: Write a Regex pattern to validate an AWS IAM role ARN format (arn:aws:iam::123456789012:role/RoleName).
Solution generated.
Grade: 9/10
Reason: The solution provides a well-crafted regex pattern that accurately validates AWS IAM role ARNs with comprehensive documentation, multiple language implementations, and thoughtful edge cases. The pattern correctly enforces the 12-digit account ID requirement and includes valid special characters for role names. Minor weaknesses exist around AWS documentation alignment and path handling nuances.

[Task 3]: Write a Python function that takes an AWS CloudWatch Logs query result (JSON array of log events) and filters events where the 'level' field equals 'ERROR', returning only the 'message' and '@timestamp' fields.
Solution generated.
Grade: 9/10
Reason: The solution comprehensively addresses the task with multiple well-implemented approaches, proper documentation, and thorough testing. The basic function correctly filters CloudWatch logs for ERROR level events and returns only the specified fields. The solution goes beyond requirements by providing alternative implementations and advanced features, though the advanced version introduces optional complexity not requested in the task.

Output aggregate metrics:

avg_score = sum(r["score"] for r in results) / len(results)
print("\n" + "=" * 50)
print(f"EVALUATION RESULT: {avg_score:.2f} / 10.0")
print("=" * 50)

==================================================
EVALUATION RESULT: 9.00 / 10.0
==================================================

Model-Based Grading

Model-based grading uses an AI model as an objective judge to evaluate response quality when programmatic rules are too rigid. It provides a measurable score (typically from 1 to 10) to assess subjective or complex criteria.

Grader Type Mechanism Best Used For
Code Graders Programmatic checks Length, exact keywords, syntax validation (JSON, Python, Regex)
Model Graders another LLM judge Task-following, response quality, completeness, helpfulness, safety
Human Graders Manual review High-level nuance, depth, relevance (time-intensive), Conciseness

Before implementing any grader, you need clear evaluation criteria.

Here the example code;

def grade_by_model(test_case, output):
    eval_prompt = f"""
    You are an expert code reviewer. Evaluate this AI-generated solution.

    Task: {test_case['task']}
    Solution: {output}

    Provide your evaluation as a structured JSON object with:
    - "strengths": An array of 1-3 key strengths
    - "weaknesses": An array of 1-3 key areas for improvement
    - "reasoning": A concise explanation of your assessment
    - "score": A number between 1-10
    """

    messages = [
        {"role": "user", "content": [{"text": eval_prompt.strip()}]},
        {"role": "assistant", "content": [{"text": '```json\n{\n  "score":'}]},
    ]

    response = client.converse(
        modelId=model_id,
        messages=messages,
        inferenceConfig={"temperature": 0.0, "stopSequences": ["```"]},
    )

    completion_text = response["output"]["message"]["content"][0]["text"].strip()
    full_json_str = '{\n  "score":' + completion_text
    return json.loads(full_json_str)

Code-Based Grading

Code-based grading provides deterministic syntax and format validation without requiring additional LLM calls. It checks two primary criteria:

  1. Format Compliance: Verifies whether the output contains strictly the target format (Python, JSON, or Regex) without conversational text or markdown headers.
  2. Valid Syntax: Confirms that the output parses or compiles successfully.

Programmatic Validation Functions

Validation helper functions return a binary score (10 for successful parsing, 0 for failure) using standard Python libraries:

import ast
import json
import re

def validate_json(text):
    try:
        json.loads(text.strip())
        return 10
    except json.JSONDecodeError:
        return 0

def validate_python(text):
    try:
        ast.parse(text.strip())
        return 10
    except SyntaxError:
        return 0

def validate_regex(text):
    try:
        re.compile(text.strip())
        return 10
    except re.error:
        return 0

def grade_syntax(output, test_case):
    fmt = test_case.get("format", "python")
    if fmt == "json":
        return validate_json(output)
    elif fmt == "regex":
        return validate_regex(output)
    else:
        return validate_python(output)

Hybrid Evaluation Pipeline

To balance semantic quality with technical correctness, combine the model grader score with the code grader score into a composite score:

def run_hybrid_eval(dataset):
    results = []
    for test_case in dataset:
        solution = solve_task(test_case["task"])

        # 1. Model-based grading (Semantic quality & task adherence)
        model_eval = grade_by_model(test_case, solution)
        model_score = model_eval["score"]

        # 2. Code-based grading (Syntax correctness)
        syntax_score = grade_syntax(solution, test_case)

        # 3. Hybrid score calculation
        final_score = (model_score + syntax_score) / 2

        results.append(
            {
                "task": test_case["task"],
                "model_score": model_score,
                "syntax_score": syntax_score,
                "final_score": final_score,
                "reasoning": model_eval["reasoning"],
            }
        )

    avg_score = sum(r["final_score"] for r in results) / len(results)
    print(f"Overall Benchmark Score: {avg_score:.2f} / 10.0")
    return results

Prompt Engineering

Prompt Engineering is how to systematically build, evaluate, and refine prompts through an iterative process.

flowchart TD
    A[Set Goal] --> B[Write Baseline Prompt]
    B --> C[Generate Test Dataset]
    C --> D[Run Evaluation Pipeline]
    D --> E[Analyze Scores & HTML Report]
    E --> F[Apply Engineering Techniques]
    F -->|Re-evaluate & Iterate| D
  • Iterative Cycle: Prompt engineering relies on setting clear goals, establishing baseline performance, applying systematic techniques, and re-evaluating to verify improvements.
  • Evaluation Pipeline: Uses a PromptEvaluator class to manage dataset creation and model grading. It supports concurrent task execution (starting at 3–5 concurrent tasks) to speed up testing while managing API rate limits.
  • Generating Test Data: Uses .generate_dataset() to create synthetic test cases by defining a task description (e.g., meal planning) and specifying input variables (height, weight, goal, restrictions).
  • Initial Prompt Baseline: Starts with a simple, naive prompt template to establish a performance benchmark. In the example provided, the initial baseline scored 2.3 out of 10.
  • Grading Criteria & Analysis: Evaluation runs output against custom parameters (e.g., requiring caloric totals, macro breakdowns, and meal timing) and outputs an output.html visual report with scores, reasoning, and response text.
  • Future Techniques: Subsequent lessons focus on improving low baseline scores by applying explicit instructions, structured output formatting, and multi-shot examples.

create boto3 client:

import boto3

client = boto3.client('bedrock-runtime', region_name='ap-southeast-2')
model_id = "au.anthropic.claude-haiku-4-5-20251001-v1:0" 

def add_user_message(messages, text):
    user_message = {
        "role": "user",
        "content": [
            {"text": text}
        ]
    }
    messages.append(user_message)

def add_assistant_message(messages, text):
    assistant_message = {
        "role": "assistant", 
        "content": [
            {"text": text}
        ]
    }
    messages.append(assistant_message)

def chat(messages):
    response = client.converse(
        modelId=model_id,
        messages=messages
    )
    return response["output"]["message"]["content"][0]["text"]

The PromptEvaluator wrap the dataset generation and the model grading discussed above. Create a evaluator using PromptEvaluator defined in the utils.py file.

import utils 
utils.set_model_id(model_id)
utils.set_client(client)

Setup the evalation pipeline:

evaluator = utils.PromptEvaluator(max_concurrent_tasks=5)

The generate_dataset method creates test cases for your prompt. You need to specify:

  • A task description explaining what your prompt should do
  • A specification of the inputs your prompt requires
  • The number of test cases to generate

Following prompt5 is created for the one-day meal plans for athletes based on their height, weight, physical goals, and dietary restrictions. Following code create test cases your prompt:

  • Task description
  • specification of the inputs
  • Number of test cases to generate

dataset = evaluator.generate_dataset(
    task_description="Write a compact, concise 1 day meal plan for a single athlete",
    prompt_inputs_spec={
        "height": "Athlete's height in cm",
        "weight": "Athlete's weight in kg", 
        "goal": "Goal of the athlete",
        "restrictions": "Dietary restrictions of the athlete"
    },
    num_cases=3
)

Generated 1/3 test cases
Generated 2/3 test cases
Generated 3/3 test cases


[
  {
    "prompt_inputs": {
      "height": "180",
      "weight": "78",
      "goal": "Match day preparation with 2800 calorie target and sustained energy for 90-minute soccer performance",
      "restrictions": "Vegetarian"
    },
    "solution_criteria": [
      "Meal plan contains exactly 3 meals that total approximately 2800 calories",
      "All meals are vegetarian with no meat, poultry, or fish",
      "Plan includes carbohydrate-rich options for energy and protein sources suitable for match day performance"
    ],
    "task_description": "Write a compact, concise 1 day meal plan for a single athlete",
    "scenario": "Testing with a team sport athlete (soccer player) with dietary restrictions (vegetarian) and specific caloric targets for match day preparation"
  },
  {
    "prompt_inputs": {
      "height": "178 cm",
      "weight": "72 kg",
      "goal": "Marathon training - maximize carbohydrate intake with proper timing around a 90-minute morning run",
      "restrictions": "Vegetarian, no nuts"
    },
    "solution_criteria": [
      "Meal plan is compact and covers all meals for one day",
      "Includes high carbohydrate content appropriate for endurance athlete (~8-10g per kg body weight)",
      "Pre-workout meal provided before the 90-minute run and post-workout meal after",
      "All dietary restrictions (vegetarian, no nuts) are respected"
    ],
    "task_description": "Write a compact, concise 1 day meal plan for a single athlete",
    "scenario": "Testing with an endurance athlete (marathon runner) who requires high carbohydrate intake and specific timing around training sessions"
  },
  {
    "prompt_inputs": {
      "height": "178 cm",
      "weight": "85 kg",
      "goal": "Maximize strength gains and recovery post-workout with optimized protein distribution",
      "restrictions": "None"
    },
    "solution_criteria": [
      "Meal plan spans exactly 1 day with 4-5 meals",
      "Protein distributed across all meals with at least 25-30g per meal and elevated post-workout nutrition",
      "Includes a substantial post-workout meal within 1-2 hours of training",
      "Compact format (concise descriptions, no excessive detail)"
    ],
    "task_description": "Write a compact, concise 1 day meal plan for a single athlete",
    "scenario": "Testing with a strength/power athlete (weightlifter) who prioritizes protein distribution and recovery nutrition post-workout"
  }
]

Write initial prompt to establish a baseline:

def run_prompt(prompt_inputs):
    prompt = f"""
    What should this person eat?
    
    - Height: {prompt_inputs["height"]}
    - Weight: {prompt_inputs["weight"]}
    - Goal: {prompt_inputs["goal"]}
    - Dietary restrictions: {prompt_inputs["restrictions"]}
    """
    
    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

Run the evalation:

results = evaluator.run_evaluation(
    run_prompt_function=run_prompt,
    dataset_file="dataset.json",
    extra_criteria="""
    The output should include:
    - Daily caloric total
    - Macronutrient breakdown  
    - Meals with exact foods, portions, and timing
    """
)

Graded 1/3 test cases
Graded 2/3 test cases
Graded 3/3 test cases
Average score: 2.6666666666666665

Here the output.html file with analysis results:

According to the above low avarage result, the intial prompt need to be improved. Time to systematically apply prompt engineering techniques

  • being more specific,
  • adding output formatting
  • structured prompt
  • Implementing multishot examples

Clarity and Direction

Use the simple language that leaves no room for ambiguity about what you want Claude6 to do.

  • Clear
    • simple language
    • state what you want explicitly
    • simple statement of the model’s task
  • Direct
    • Use instructions, not questions
    • Use direct action verbs

Being specific

Provide clear guidelines or steps that direct Claude toward the kind of output you’re looking for. Guidelines are the way to specific. There are 2 types of guidelines:

  1. List qualities that the output should have
  2. Provide process steps the model should follow

Use step guidelines when troubleshooting hard problem, decision making, critical thinking so on where Claude to consider wider view.

def run_prompt(prompt_inputs):
    prompt = f"""
   Generate a one-day meal plan for an athlete that meets their dietary restrictions.

    - Height: {prompt_inputs["height"]}
    - Weight: {prompt_inputs["weight"]}
    - Goal: {prompt_inputs["goal"]}
    - Dietary restrictions: {prompt_inputs["restrictions"]}

    Guidelines:
    1. Include accurate daily calorie amount
    2. Show protein, fat, and carb amounts
    3. Specify when to eat each meal
    4. Use only foods that fit restrictions
    5. List all portion sizes in grams
    6. Keep budget-friendly if mentioned
    """
    
    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

results = evaluator.run_evaluation(
    run_prompt_function=run_prompt,
    dataset_file="dataset.json",
    extra_criteria="""
    The output should include:
    - Daily caloric total
    - Macronutrient breakdown  
    - Meals with exact foods, portions, and timing
    """
)

Graded 1/3 test cases
Graded 2/3 test cases
Graded 3/3 test cases
Average score: 6

XML tags

When prompts contain large datasets, system instructions, or multiple distinct types of content, Claude can struggle to distinguish where instructions end and data begins.

XML tags act as explicit delimiters that wrap specific blocks of text. Using them yields several benefits:

  • Separation of Concerns: Clearly distinguishes model instructions from external, interpolated data.
  • Reduces Ambiguity: Prevents Claude from mistaking context or input text for new instructions (mitigating prompt injection/confusion).
  • Improved Accuracy: Helps Claude parse multi-part prompts systematically, leading to more reliable outputs.

The tag names don’t need to follow any official XML specification.

For example just add the <athlete_information>...</athlete_information>:

def run_prompt(prompt_inputs):
    prompt = f"""
   Generate a one-day meal plan for an athlete that meets their dietary restrictions.

   <athlete_information>
    - Height: {prompt_inputs["height"]}
    - Weight: {prompt_inputs["weight"]}
    - Goal: {prompt_inputs["goal"]}
    - Dietary restrictions: {prompt_inputs["restrictions"]}
   </athlete_information>
   
    Guidelines:
    1. Include accurate daily calorie amount
    2. Show protein, fat, and carb amounts
    3. Specify when to eat each meal
    4. Use only foods that fit restrictions
    5. List all portion sizes in grams
    6. Keep budget-friendly if mentioned
    """
    
    messages = []
    add_user_message(messages, prompt)
    return chat(messages)

results = evaluator.run_evaluation(
    run_prompt_function=run_prompt,
    dataset_file="dataset.json",
    extra_criteria="""
    The output should include:
    - Daily caloric total
    - Macronutrient breakdown  
    - Meals with exact foods, portions, and timing
    """
)

Graded 1/3 test cases
Graded 2/3 test cases
Graded 3/3 test cases
Average score: 7


/home/ojitha/Github/learn-bedrock/.venv/lib/python3.13/site-packages/IPython/core/display.py:448: UserWarning: Consider using IPython.display.IFrame instead
  warnings.warn("Consider using IPython.display.IFrame instead")

Average score is increased 🚀.

Best Practices for XML Tags:

  • Custom Tags Are Allowed: Tag names do not need to follow strict XML schemas (e.g., <sales_records>, <my_code>, <docs>).
  • Be Specific: Prefer descriptive tag names over generic ones (e.g., <customer_feedback> is better than <data>).
  • Match Opening & Closing Tags: Ensure every block has matching opening (<tag_name>) and closing (</tag_name>) tags.

Use of Examples

Providing sample input/output pairs—commonly known as one-shot (one example) or multi-shot (multiple examples) prompting—is one of the most effective techniques for engineering reliable prompts. It helps Claude handle tricky edge cases (such as sarcasm), adhere strictly to formatting rules, and understand what constitutes a high-quality response.

flowchart TD
    A[Prompt / Task Request] --> B[XML Tagged Example]
    B --> C["Sample Input: ≺sample_input≻"]
    B --> D["Ideal Output: ≺ideal_output≻"]
    D --> E[Optional Context: Why output is good]
    E --> F[Claude Generates Desired Output]

Key Elements of Example Prompting

  1. Structured Format with XML Tags: Wrap sample pairs clearly using tags like <sample_input> and <ideal_output>.
  2. Clear Introductions: Explicitly introduce example blocks (e.g., “Here is an example input with an ideal response”).
  3. Contextual Explanations: Include a brief rationale after an example output explaining why it is high quality so Claude understands the underlying criteria.

Key Use Cases

  • Handling Corner Cases & Nuance: Particularly useful for tasks like sentiment analysis, where edge cases like sarcasm (e.g., “Oh yeah, I really needed a flight delay tonight!”) look positive literally but are negative in context.
  • Enforcing Complex Output Formats: Demonstrating custom schemas, nested JSON structures, or structured report layouts.
  • Defining “Good” Standards: Providing a benchmark of what constitutes a complete, high-scoring output.
sequenceDiagram
    autonumber
    actor User
    participant Eval as Prompt Evaluation
    participant Prompt as Prompt Template

    User->>Eval: Run evaluation suite
    Eval-->>User: Generate HTML Report
    User->>Eval: Identify top-scoring outputs (e.g., Score 10/10)
    User->>Prompt: Copy high-scoring Input/Output pair into XML tags
    Prompt-->>User: Deployment-ready Multi-Shot Prompt

When running prompt evaluations, look for your highest-scoring outputs in the HTML report. These make excellent examples to include in your prompt7.<>

  1. Claude with Amazon Bedrock ↩

  2. Overview of Claude Models - Claude with Amazon Bedrock ↩

  3. Prompt evaluation - Claude with Amazon Bedrock ↩

  4. A typical eval workflow · Claude with Amazon Bedrock · Claude Academy ↩

  5. Prompt engineering · Claude with Amazon Bedrock · Claude Academy ↩

  6. Being clear and direct · Claude with Amazon Bedrock · Claude Academy ↩

  7. Providing examples · Claude with Amazon Bedrock · Claude Academy ↩