RaindropA practical guide to choosing decision modelsGitHub ↗

Unrelated examples · No worked example (absence-control recipe) variant

Original question · Other versions of this question · All variants

This version keeps the original reference answer. Its context, question instructions or answer choices were changed as described below.

Changed fields: context.

How the edits were constructed

Context

Read the full context

{"user_request": "I am interested in seeing an Imax show, preferably a cartoon movie. Would you search for this?", "available_tools": [{"name": "Movies_1_BuyMovieTickets", "description": "Purchase tickets for a specific movie showing, including the number of tickets, show date and time, and location.", "parameters": {"type": "dict", "required": ["movie_name", "number_of_tickets", "location"], "properties": {"movie_name": {"type": "string", "description": "The title of the movie for which tickets are being purchased."}, "number_of_tickets": {"type": "integer", "description": "The total number of tickets to be bought."}, "show_date": {"type": "string", "description": "The date on which the movie is showing, in the format 'YYYY-MM-DD'.", "default": null}, "location": {"type": "string", "description": "The city in which the movie theater is located, in the format of 'City, State', such as 'Los Angeles, CA'."}, "show_time": {"type": "string", "description": "The start time of the movie showing, in 24-hour format 'HH:MM'.", "default": "20:00"}, "show_type": {"type": "string", "description": "The format of the movie showing.", "enum": ["regular", "3d", "imax"], "default": "regular"}}}}, {"name": "Movies_1_FindMovies", "description": "Search for movies based on location, genre, and show type at specific theaters.", "parameters": {"type": "dict", "required": ["location"], "properties": {"location": {"type": "string", "description": "The city where the theatre is located, in the format of 'City, State', such as 'Berkeley, CA' or 'New York, NY'."}, "theater_name": {"type": "string", "description": "The name of the theatre. If unspecified, all theatres are considered.", "default": "dontcare"}, "genre": {"type": "string", "description": "The genre of the movie. If unspecified, all genres are considered.", "enum": ["World", "Offbeat", "Mystery", "Supernatural", "Horror", "Animation", "Sci-fi", "Documentary", "Drama", "War", "Family", "Action"], "default": "dontcare"}, "show_type": {"type": "string", "description": "The type of movie show. If unspecified, all show types are considered.", "enum": ["regular", "3d", "imax"], "default": "dontcare"}}}}, {"name": "Movies_1_GetTimesForMovie", "description": "Retrieves the show times for a specific movie at a particular theater location on a specified date.", "parameters": {"type": "dict", "required": ["movie_name", "location", "show_date"], "properties": {"movie_name": {"type": "string", "description": "The title of the movie for which to find show times."}, "location": {"type": "string", "description": "The city and state where the theater is located, in the format of 'City, State', such as 'Berkeley, CA' and 'New York, NY'."}, "show_date": {"type": "string", "description": "The date of the show in the format 'YYYY-MM-DD', for example, '2023-04-15'."}, "theater_name": {"type": "string", "description": "The name of the theater where the movie is showing. If not specified, any theater will be considered.", "default": "Any Theater"}, "show_type": {"type": "string", "description": "The format of the movie showing.", "enum": ["regular", "3D", "IMAX"], "default": "regular"}}}}]}

The notebook has a linen cover and stitched binding.

Question

Which candidate response is most appropriate, given the user's request and the available tools?

Answer choices

  • direct. Absolutely, I found an IMAX showing of "The Secret Life of Pets 2" at the Roxy Stadium 14 in Santa Rosa. It's a fun, animated movie.
  • request_for_info. To help you with that, I'll need to know the location of the theater. Could you please provide the city and state?
  • cannot_answer. I'm sorry for the inconvenience, but I'm unable to provide real-time information or perform web searches. I recommend checking local listings for IMAX showtimes in Santa Rosa.
  • tool_call. {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}}

Reference answer: request_for_info.

Exact evaluated input as JSON
{
  "state": "{\"user_request\": \"I am interested in seeing an Imax show, preferably a cartoon movie. Would you search for this?\", \"available_tools\": [{\"name\": \"Movies_1_BuyMovieTickets\", \"description\": \"Purchase tickets for a specific movie showing, including the number of tickets, show date and time, and location.\", \"parameters\": {\"type\": \"dict\", \"required\": [\"movie_name\", \"number_of_tickets\", \"location\"], \"properties\": {\"movie_name\": {\"type\": \"string\", \"description\": \"The title of the movie for which tickets are being purchased.\"}, \"number_of_tickets\": {\"type\": \"integer\", \"description\": \"The total number of tickets to be bought.\"}, \"show_date\": {\"type\": \"string\", \"description\": \"The date on which the movie is showing, in the format 'YYYY-MM-DD'.\", \"default\": null}, \"location\": {\"type\": \"string\", \"description\": \"The city in which the movie theater is located, in the format of 'City, State', such as 'Los Angeles, CA'.\"}, \"show_time\": {\"type\": \"string\", \"description\": \"The start time of the movie showing, in 24-hour format 'HH:MM'.\", \"default\": \"20:00\"}, \"show_type\": {\"type\": \"string\", \"description\": \"The format of the movie showing.\", \"enum\": [\"regular\", \"3d\", \"imax\"], \"default\": \"regular\"}}}}, {\"name\": \"Movies_1_FindMovies\", \"description\": \"Search for movies based on location, genre, and show type at specific theaters.\", \"parameters\": {\"type\": \"dict\", \"required\": [\"location\"], \"properties\": {\"location\": {\"type\": \"string\", \"description\": \"The city where the theatre is located, in the format of 'City, State', such as 'Berkeley, CA' or 'New York, NY'.\"}, \"theater_name\": {\"type\": \"string\", \"description\": \"The name of the theatre. If unspecified, all theatres are considered.\", \"default\": \"dontcare\"}, \"genre\": {\"type\": \"string\", \"description\": \"The genre of the movie. If unspecified, all genres are considered.\", \"enum\": [\"World\", \"Offbeat\", \"Mystery\", \"Supernatural\", \"Horror\", \"Animation\", \"Sci-fi\", \"Documentary\", \"Drama\", \"War\", \"Family\", \"Action\"], \"default\": \"dontcare\"}, \"show_type\": {\"type\": \"string\", \"description\": \"The type of movie show. If unspecified, all show types are considered.\", \"enum\": [\"regular\", \"3d\", \"imax\"], \"default\": \"dontcare\"}}}}, {\"name\": \"Movies_1_GetTimesForMovie\", \"description\": \"Retrieves the show times for a specific movie at a particular theater location on a specified date.\", \"parameters\": {\"type\": \"dict\", \"required\": [\"movie_name\", \"location\", \"show_date\"], \"properties\": {\"movie_name\": {\"type\": \"string\", \"description\": \"The title of the movie for which to find show times.\"}, \"location\": {\"type\": \"string\", \"description\": \"The city and state where the theater is located, in the format of 'City, State', such as 'Berkeley, CA' and 'New York, NY'.\"}, \"show_date\": {\"type\": \"string\", \"description\": \"The date of the show in the format 'YYYY-MM-DD', for example, '2023-04-15'.\"}, \"theater_name\": {\"type\": \"string\", \"description\": \"The name of the theater where the movie is showing. If not specified, any theater will be considered.\", \"default\": \"Any Theater\"}, \"show_type\": {\"type\": \"string\", \"description\": \"The format of the movie showing.\", \"enum\": [\"regular\", \"3D\", \"IMAX\"], \"default\": \"regular\"}}}}]}\n\nThe notebook has a linen cover and stitched binding.",
  "question": "Which candidate response is most appropriate, given the user's request and the available tools?",
  "choices": {
    "direct": "Absolutely, I found an IMAX showing of \"The Secret Life of Pets 2\" at the Roxy Stadium 14 in Santa Rosa. It's a fun, animated movie.",
    "request_for_info": "To help you with that, I'll need to know the location of the theater. Could you please provide the city and state?",
    "cannot_answer": "I'm sorry for the inconvenience, but I'm unable to provide real-time information or perform web searches. I recommend checking local listings for IMAX showtimes in Santa Rosa.",
    "tool_call": "{\"name\": \"Movies_1_FindMovies\", \"arguments\": {\"location\": \"Santa Rosa, CA\", \"genre\": \"Animation\", \"show_type\": \"imax\"}}"
  }
}

Recorded model answers

The original column uses the control response paired with this variant. Missing responses and inputs outside a model’s limit remain visible.

Model Original answer Variant answer Change
GLiDE v1.2 request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
Decider v1.1 tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} Same answer
Jev request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
Microsoft Decision-1 request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
Clef request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
Luna Decisions request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
Clef Flash tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} Same answer
Strands Decider request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
GLiNER2.5 Decide cannot_answer — I'm sorry for the inconvenience, but I'm unable to provide real-time information or perform web searches. I recommend checking local listings for IMAX showtimes in Santa Rosa. cannot_answer — I'm sorry for the inconvenience, but I'm unable to provide real-time information or perform web searches. I recommend checking local listings for IMAX showtimes in Santa Rosa. Same answer
Laya request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? request_for_info — To help you with that, I'll need to know the location of the theater. Could you please provide the city and state? Same answer
CalDec v1 tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} tool_call — {"name": "Movies_1_FindMovies", "arguments": {"location": "Santa Rosa, CA", "genre": "Animation", "show_type": "imax"}} Same answer
Identifiers and recorded conditions

Variant ID: when2call/f3c687e305eecb7e3d56/examples/absent/clean-bare_when2call_example_controls_full_0064

Input fingerprint: a9c5146ba20452e5f9fe0d7c921d8048184b8fcb5735f9199b1be62ddd9bf2ba

Change Recorded version
Unrelated examples No worked example (absence-control recipe)

Source attribution and dataset terms are linked from the original question.

Type to search questions, variants and documentation.