RaindropA practical guide to choosing decision modelsGitHub ↗

Information seeking · question 52

Back to Information seeking · All capabilities

Question ID: when2call/c452709a8e024fee8370
Source: When2Call
Input format: native choices

Explore this question’s altered versions

Context

{"user_request": "I'd like to modify my order, please.", "available_tools": [{"name": "change_food", "description": "Modifies the food item based on the customer's request, such as changing ingredients or preparation methods.", "parameters": {"type": "dict", "required": ["food_item", "modification_request"], "properties": {"food_item": {"type": "string", "description": "The name of the food item to be modified."}, "modification_request": {"type": "string", "description": "Specific request detailing the changes to be made to the food item, such as 'no peanuts' or 'extra spicy'. If there are multiple requests, separate using comma with no space between comma"}}}}]}

Question

Which candidate response is most appropriate, given the user's request and the available tools?

Answer choices

  • request_for_info: Of course, I can help with that. Could you please specify the modifications you'd like for your food item?
  • tool_call: {"name": "change_food", "arguments": {"food_item": "chicken dish", "modification_request": "extra spicy"}}
  • cannot_answer: Apologies, I'm unable to modify your order as I'm an assistant designed to provide information and answer questions. Please contact the restaurant directly for order changes.
  • direct: Absolutely, I can help with that. I'll make sure your chicken dish is prepared extra spicy. Your updated order will reflect this change.

Reference answer: Of course, I can help with that. Could you please specify the modifications you'd like for your food item? (request_for_info)

Exact evaluated input as JSON
{
  "state": "{\"user_request\": \"I'd like to modify my order, please.\", \"available_tools\": [{\"name\": \"change_food\", \"description\": \"Modifies the food item based on the customer's request, such as changing ingredients or preparation methods.\", \"parameters\": {\"type\": \"dict\", \"required\": [\"food_item\", \"modification_request\"], \"properties\": {\"food_item\": {\"type\": \"string\", \"description\": \"The name of the food item to be modified.\"}, \"modification_request\": {\"type\": \"string\", \"description\": \"Specific request detailing the changes to be made to the food item, such as 'no peanuts' or 'extra spicy'. If there are multiple requests, separate using comma with no space between comma\"}}}}]}",
  "question": "Which candidate response is most appropriate, given the user's request and the available tools?",
  "choices": {
    "request_for_info": "Of course, I can help with that. Could you please specify the modifications you'd like for your food item?",
    "tool_call": "{\"name\": \"change_food\", \"arguments\": {\"food_item\": \"chicken dish\", \"modification_request\": \"extra spicy\"}}",
    "cannot_answer": "Apologies, I'm unable to modify your order as I'm an assistant designed to provide information and answer questions. Please contact the restaurant directly for order changes.",
    "direct": "Absolutely, I can help with that. I'll make sure your chicken dish is prepared extra spicy. Your updated order will reflect this change."
  }
}

Recorded model answers

Model Answer Outcome
Astra Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Opus 5.5 Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
GLiDE v1.2 Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Decider v1.1 Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Jev Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Microsoft Decision-1 Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Clef Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Luna Decisions Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Clef Flash Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
Strands Decider Of course, I can help with that. Could you please specify the modifications you'd like for your food item? Correct
GLiNER2.5 Decide Apologies, I'm unable to modify your order as I'm an assistant designed to provide information and answer questions. Please contact the restaurant directly for order changes. Incorrect
Laya Absolutely, I can help with that. I'll make sure your chicken dish is prepared extra spicy. Your updated order will reflect this change. Incorrect
CalDec v1 {"name": "change_food", "arguments": {"food_item": "chicken dish", "modification_request": "extra spicy"}} Incorrect

Source and sharing

Attribution: NVIDIA; When2Call authors.
Terms: CC-BY-4.0. Source sharing decision.
The capability assignment and choice-format normalization are part of this experiment. Source data retain their own terms.

Exact source locator and input fingerprint
{
  "source_locator": {
    "path": "data/prepared/when2call.jsonl",
    "record": "2143",
    "kind": "prepared_normalizer_record",
    "native_id": "when2call/338ff826-f73c-48a8-88e1-67339ddf9fff/baseline"
  },
  "source_revision": "0582f7749df63a96fdc3070932e83e72396ace53",
  "input_hash": "18b9447988bbe9fed27132a5a8e984bc054f2e7ee4fc2aafb088c54a23358cbb"
}

Type to search questions, variants and documentation.