zjunlp/alfworld-device-operator

Instructions

1. Skill Trigger

Activate this skill when the task goal explicitly requires using a device or appliance to interact with a target object. Common indicators include phrases like:

• "look at [object] under the [device]"
• "heat [object] with [device]"
• "cool [object] with [device]"
• "clean [object] with [device]"

2. Core Execution Flow

Follow this sequence when the skill is triggered:

Phase 1: Locate the Device

1. Identify the device from the task description (e.g., desklamp, microwave, fridge).
2. Search common receptacles where such a device is typically found (e.g., desks, sidetables, countertops).
3. Use the go to {recep} action to navigate to and inspect these locations until the device is found.
4. Note the device's exact name (e.g., desklamp 1).

Phase 2: Locate the Target Object

1. Identify the target object from the task description (e.g., laptop, mug, plate).
2. Search the environment for this object. It may not be near the device initially.
3. Use the go to {recep} and visual inspection to find the object.
4. Once found, use take {obj} from {recep} to pick it up.

Phase 3: Co-locate Object and Device

1. Navigate to the receptacle where the target device is located using go to {recep}.
2. Ensure you are in the same location as the device before proceeding.

Phase 4: Operate the Device

1. Execute the final use action. The specific action is determined by the device-object pair:
   • For a desklamp and a viewable object (like a laptop, book), use: toggle {device} {recep}.
   • For a microwave and a heatable object, use: heat {obj} with {device}.
   • For a fridge and a coolable object, use: cool {obj} with {device}.
   • For a faucet/spraybottle and a cleanable object, use: clean {obj} with {device}.
2. Action Format: The action must use the exact object and device names discovered during Phases 1 & 2.

3. Key Assumptions & Rules

• Co-location Required: The skill assumes the final action requires the target object and the operating device to be in the same location (on the same receptacle).
• Device First: Prioritize finding the device before extensively searching for the target object, as the device's location is often a fixed landmark.
• Invalid Actions: If the environment responds with "Nothing happened," re-evaluate your object/device names and your location. Ensure you are using the correct action verb for the device type.

Example

Scenario: Task is "look at laptop under the desklamp."

Thought: I need to find the desklamp first. It's likely on a sidetable or desk.
Action: go to sidetable 1
Observation: On the sidetable 1, you see a desklamp 1.
Thought: Found the desklamp. Now I need to find the laptop.
Action: go to desk 1
Observation: On the desk 1, you see a laptop 1, a pen 2.
Action: take laptop 1 from desk 1
Observation: You pick up the laptop 1 from the desk 1.
Action: go to sidetable 1
Observation: On the sidetable 1, you see a desklamp 1.
Action: use desklamp 1
Observation: You turn on the desklamp 1.

Result: The laptop is now being examined under the desklamp, completing the task.