Desktop Control

Install resolver

{
  "artifact": {
    "downloadUrl": "/api/v1/download?slug=desktop-control&version=1.0.0&ownerHandle=matagul",
    "format": "zip",
    "generated": true,
    "kind": "skillArchive",
    "sha256": "85277edef5f72a4e92f4a405c22775780927828f041f385b04a1177d87e3f1e4",
    "size": 15198
  },
  "owner": {
    "displayName": "matagul",
    "handle": "matagul",
    "id": "aded540a-62ef-474c-a729-673cdd712890",
    "verified": false
  },
  "skill": {
    "displayName": "Desktop Control",
    "latestVersion": "1.0.0",
    "license": "unknown",
    "name": "desktop-control",
    "ownerHandle": "matagul",
    "slug": "desktop-control",
    "stats": {
      "downloads": 59964,
      "installs": 1860,
      "stars": 373
    },
    "summary": "Advanced desktop automation with mouse, keyboard, and screen control",
    "tags": [
      "automation"
    ],
    "type": "skill",
    "updatedAt": "2026-07-06T10:48:55.230Z"
  },
  "sourceHandoff": null,
  "version": {
    "checksum": null,
    "checksumAlgorithm": null,
    "id": "c6764c0b-c1d2-41b3-bffc-3792ea87b5c0",
    "publishedAt": "2026-02-05T01:33:20.863Z",
    "size": null,
    "version": "1.0.0",
    "artifactStorageKey": null,
    "changelog": "Version 1.0.0\n\n- Initial release of the Desktop Control skill for OpenClaw.\n- Provides advanced automation: mouse movement/clicks, keyboard input, hotkeys, and typing speed control.\n- Supports screen capture, region-based screenshots, image/template matching, and pixel color detection.\n- Includes window management (list, activate, move, resize, minimize/maximize).\n- Safety features: failsafe abort, logging, approval mode, bounds checks, and emergency pause.\n- Detailed documentation with examples and complete API reference.",
    "manifest": {
      "clawhub": {
        "tags": {
          "latest": "1.0.0"
        },
        "owner": "matagul",
        "stats": {
          "stars": 373,
          "comments": 4,
          "installs": 1860,
          "versions": 1,
          "downloads": 59964
        },
        "topics": [],
        "categories": [
          "automation"
        ]
      },
      "install": "openclaw skills install @matagul/desktop-control"
    },
    "readme": "---\r\ndescription: Advanced desktop automation with mouse, keyboard, and screen control\r\n---\r\n\r\n# Desktop Control Skill\r\n\r\n**The most advanced desktop automation skill for OpenClaw.** Provides pixel-perfect mouse control, lightning-fast keyboard input, screen capture, window management, and clipboard operations.\r\n\r\n## 🎯 Features\r\n\r\n### Mouse Control\r\n- ✅ **Absolute positioning** - Move to exact coordinates\r\n- ✅ **Relative movement** - Move from current position\r\n- ✅ **Smooth movement** - Natural, human-like mouse paths\r\n- ✅ **Click types** - Left, right, middle, double, triple clicks\r\n- ✅ **Drag & drop** - Drag from point A to point B\r\n- ✅ **Scroll** - Vertical and horizontal scrolling\r\n- ✅ **Position tracking** - Get current mouse coordinates\r\n\r\n### Keyboard Control\r\n- ✅ **Text typing** - Fast, accurate text input\r\n- ✅ **Hotkeys** - Execute keyboard shortcuts (Ctrl+C, Win+R, etc.)\r\n- ✅ **Special keys** - Enter, Tab, Escape, Arrow keys, F-keys\r\n- ✅ **Key combinations** - Multi-key press combinations\r\n- ✅ **Hold & release** - Manual key state control\r\n- ✅ **Typing speed** - Configurable WPM (instant to human-like)\r\n\r\n### Screen Operations\r\n- ✅ **Screenshot** - Capture entire screen or regions\r\n- ✅ **Image recognition** - Find elements on screen (via OpenCV)\r\n- ✅ **Color detection** - Get pixel colors at coordinates\r\n- ✅ **Multi-monitor** - Support for multiple displays\r\n\r\n### Window Management\r\n- ✅ **Window list** - Get all open windows\r\n- ✅ **Activate window** - Bring window to front\r\n- ✅ **Window info** - Get position, size, title\r\n- ✅ **Minimize/Maximize** - Control window states\r\n\r\n### Safety Features\r\n- ✅ **Failsafe** - Move mouse to corner to abort\r\n- ✅ **Pause control** - Emergency stop mechanism\r\n- ✅ **Approval mode** - Require confirmation for actions\r\n- ✅ **Bounds checking** - Prevent out-of-screen operations\r\n- ✅ **Logging** - Track all automation actions\r\n\r\n---\r\n\r\n## 🚀 Quick Start\r\n\r\n### Installation\r\n\r\nFirst, install required dependencies:\r\n\r\n```bash\r\npip install pyautogui pillow opencv-python pygetwindow\r\n```\r\n\r\n### Basic Usage\r\n\r\n```python\r\nfrom skills.desktop_control import DesktopController\r\n\r\n# Initialize controller\r\ndc = DesktopController(failsafe=True)\r\n\r\n# Mouse operations\r\ndc.move_mouse(500, 300)  # Move to coordinates\r\ndc.click()  # Left click at current position\r\ndc.click(100, 200, button=\"right\")  # Right click at position\r\n\r\n# Keyboard operations\r\ndc.type_text(\"Hello from OpenClaw!\")\r\ndc.hotkey(\"ctrl\", \"c\")  # Copy\r\ndc.press(\"enter\")\r\n\r\n# Screen operations\r\nscreenshot = dc.screenshot()\r\nposition = dc.get_mouse_position()\r\n```\r\n\r\n---\r\n\r\n## 📋 Complete API Reference\r\n\r\n### Mouse Functions\r\n\r\n#### `move_mouse(x, y, duration=0, smooth=True)`\r\nMove mouse to absolute screen coordinates.\r\n\r\n**Parameters:**\r\n- `x` (int): X coordinate (pixels from left)\r\n- `y` (int): Y coordinate (pixels from top)\r\n- `duration` (float): Movement time in seconds (0 = instant, 0.5 = smooth)\r\n- `smooth` (bool): Use bezier curve for natural movement\r\n\r\n**Example:**\r\n```python\r\n# Instant movement\r\ndc.move_mouse(1000, 500)\r\n\r\n# Smooth 1-second movement\r\ndc.move_mouse(1000, 500, duration=1.0)\r\n```\r\n\r\n#### `move_relative(x_offset, y_offset, duration=0)`\r\nMove mouse relative to current position.\r\n\r\n**Parameters:**\r\n- `x_offset` (int): Pixels to move horizontally (positive = right)\r\n- `y_offset` (int): Pixels to move vertically (positive = down)\r\n- `duration` (float): Movement time in seconds\r\n\r\n**Example:**\r\n```python\r\n# Move 100px right, 50px down\r\ndc.move_relative(100, 50, duration=0.3)\r\n```\r\n\r\n#### `click(x=None, y=None, button='left', clicks=1, interval=0.1)`\r\nPerform mouse click.\r\n\r\n**Parameters:**\r\n- `x, y` (int, optional): Coordinates to click (None = current position)\r\n- `button` (str): 'left', 'right', 'middle'\r\n- `clicks` (int): Number of clicks (1 = single, 2 = double)\r\n- `interval` (float): Delay between multiple clicks\r\n\r\n**Example:**\r\n```python\r\n# Simple left click\r\ndc.click()\r\n\r\n# Double-click at specific position\r\ndc.click(500, 300, clicks=2)\r\n\r\n# Right-click\r\ndc.click(button='right')\r\n```\r\n\r\n#### `drag(start_x, start_y, end_x, end_y, duration=0.5, button='left')`\r\nDrag and drop operation.\r\n\r\n**Parameters:**\r\n- `start_x, start_y` (int): Starting coordinates\r\n- `end_x, end_y` (int): Ending coordinates\r\n- `duration` (float): Drag duration\r\n- `button` (str): Mouse button to use\r\n\r\n**Example:**\r\n```python\r\n# Drag file from desktop to folder\r\ndc.drag(100, 100, 500, 500, duration=1.0)\r\n```\r\n\r\n#### `scroll(clicks, direction='vertical', x=None, y=None)`\r\nScroll mouse wheel.\r\n\r\n**Parameters:**\r\n- `clicks` (int): Scroll amount (positive = up/left, negative = down/right)\r\n- `direction` (str): 'vertical' or 'horizontal'\r\n- `x, y` (int, optional): Position to scroll at\r\n\r\n**Example:**\r\n```python\r\n# Scroll down 5 clicks\r\ndc.scroll(-5)\r\n\r\n# Scroll up 10 clicks\r\ndc.scroll(10)\r\n\r\n# Horizontal scroll\r\ndc.scroll(5, direction='horizontal')\r\n```\r\n\r\n#### `get_mouse_position()`\r\nGet current mouse coordinates.\r\n\r\n**Returns:** `(x, y)` tuple\r\n\r\n**Example:**\r\n```python\r\nx, y = dc.get_mouse_position()\r\nprint(f\"Mouse is at: {x}, {y}\")\r\n```\r\n\r\n---\r\n\r\n### Keyboard Functions\r\n\r\n#### `type_text(text, interval=0, wpm=None)`\r\nType text with configurable speed.\r\n\r\n**Parameters:**\r\n- `text` (str): Text to type\r\n- `interval` (float): Delay between keystrokes (0 = instant)\r\n- `wpm` (int, optional): Words per minute (overrides interval)\r\n\r\n**Example:**\r\n```python\r\n# Instant typing\r\ndc.type_text(\"Hello World\")\r\n\r\n# Human-like typing at 60 WPM\r\ndc.type_text(\"Hello World\", wpm=60)\r\n\r\n# Slow typing with 0.1s between keys\r\ndc.type_text(\"Hello World\", interval=0.1)\r\n```\r\n\r\n#### `press(key, presses=1, interval=0.1)`\r\nPress and release a key.\r\n\r\n**Parameters:**\r\n- `key` (str): Key name (see Key Names section)\r\n- `presses` (int): Number of times to press\r\n- `interval` (float): Delay between presses\r\n\r\n**Example:**\r\n```python\r\n# Press Enter\r\ndc.press('enter')\r\n\r\n# Press Space 3 times\r\ndc.press('space', presses=3)\r\n\r\n# Press Down arrow\r\ndc.press('down')\r\n```\r\n\r\n#### `hotkey(*keys, interval=0.05)`\r\nExecute keyboard shortcut.\r\n\r\n**Parameters:**\r\n- `*keys` (str): Keys to press together\r\n- `interval` (float): Delay between key presses\r\n\r\n**Example:**\r\n```python\r\n# Copy (Ctrl+C)\r\ndc.hotkey('ctrl', 'c')\r\n\r\n# Paste (Ctrl+V)\r\ndc.hotkey('ctrl', 'v')\r\n\r\n# Open Run dialog (Win+R)\r\ndc.hotkey('win', 'r')\r\n\r\n# Save (Ctrl+S)\r\ndc.hotkey('ctrl', 's')\r\n\r\n# Select All (Ctrl+A)\r\ndc.hotkey('ctrl', 'a')\r\n```\r\n\r\n#### `key_down(key)` / `key_up(key)`\r\nManually control key state.\r\n\r\n**Example:**\r\n```python\r\n# Hold Shift\r\ndc.key_down('shift')\r\ndc.type_text(\"hello\")  # Types \"HELLO\"\r\ndc.key_up('shift')\r\n\r\n# Hold Ctrl and click (for multi-select)\r\ndc.key_down('ctrl')\r\ndc.click(100, 100)\r\ndc.click(200, 100)\r\ndc.key_up('ctrl')\r\n```\r\n\r\n---\r\n\r\n### Screen Functions\r\n\r\n#### `screenshot(region=None, filename=None)`\r\nCapture screen or region.\r\n\r\n**Parameters:**\r\n- `region` (tuple, optional): (left, top, width, height) for partial capture\r\n- `filename` (str, optional): Path to save image\r\n\r\n**Returns:** PIL Image object\r\n\r\n**Example:**\r\n```python\r\n# Full screen\r\nimg = dc.screenshot()\r\n\r\n# Save to file\r\ndc.screenshot(filename=\"screenshot.png\")\r\n\r\n# Capture specific region\r\nimg = dc.screenshot(region=(100, 100, 500, 300))\r\n```\r\n\r\n#### `get_pixel_color(x, y)`\r\nGet color of pixel at coordinates.\r\n\r\n**Returns:** RGB tuple `(r, g, b)`\r\n\r\n**Example:**\r\n```python\r\nr, g, b = dc.get_pixel_color(500, 300)\r\nprint(f\"Color at (500, 300): RGB({r}, {g}, {b})\")\r\n```\r\n\r\n#### `find_on_screen(image_path, confidence=0.8)`\r\nFind image on screen (requires OpenCV).\r\n\r\n**Parameters:**\r\n- `image_path` (str): Path to template image\r\n- `confidence` (float): Match threshold (0-1)\r\n\r\n**Returns:** `(x, y, width, height)` or None\r\n\r\n**Example:**\r\n```python\r\n# Find button on screen\r\nlocation = dc.find_on_screen(\"button.png\")\r\nif location:\r\n    x, y, w, h = location\r\n    # Click center of found image\r\n    dc.click(x + w//2, y + h//2)\r\n```\r\n\r\n#### `get_screen_size()`\r\nGet screen resolution.\r\n\r\n**Returns:** `(width, height)` tuple\r\n\r\n**Example:**\r\n```python\r\nwidth, height = dc.get_screen_size()\r\nprint(f\"Screen: {width}x{height}\")\r\n```\r\n\r\n---\r\n\r\n### Window Functions\r\n\r\n#### `get_all_windows()`\r\nList all open windows.\r\n\r\n**Returns:** List of window titles\r\n\r\n**Example:**\r\n```python\r\nwindows = dc.get_all_windows()\r\nfor title in windows:\r\n    print(f\"Window: {title}\")\r\n```\r\n\r\n#### `activate_window(title_substring)`\r\nBring window to front by title.\r\n\r\n**Parameters:**\r\n- `title_substring` (str): Part of window title to match\r\n\r\n**Example:**\r\n```python\r\n# Activate Chrome\r\ndc.activate_window(\"Chrome\")\r\n\r\n# Activate VS Code\r\ndc.activate_window(\"Visual Studio Code\")\r\n```\r\n\r\n#### `get_active_window()`\r\nGet currently focused window.\r\n\r\n**Returns:** Window title (str)\r\n\r\n**Example:**\r\n```python\r\nactive = dc.get_active_window()\r\nprint(f\"Active window: {active}\")\r\n```\r\n\r\n---\r\n\r\n### Clipboard Functions\r\n\r\n#### `copy_to_clipboard(text)`\r\nCopy text to clipboard.\r\n\r\n**Example:**\r\n```python\r\ndc.copy_to_clipboard(\"Hello from OpenClaw!\")\r\n```\r\n\r\n#### `get_from_clipboard()`\r\nGet text from clipboard.\r\n\r\n**Returns:** str\r\n\r\n**Example:**\r\n```python\r\ntext = dc.get_from_clipboard()\r\nprint(f\"Clipboard: {text}\")\r\n```\r\n\r\n---\r\n\r\n## ⌨️ Key Names Reference\r\n\r\n### Alphabet Keys\r\n`'a'` through `'z'`\r\n\r\n### Number Keys\r\n`'0'` through `'9'`\r\n\r\n### Function Keys\r\n`'f1'` through `'f24'`\r\n\r\n### Special Keys\r\n- `'enter'` / `'return'`\r\n- `'esc'` / `'escape'`\r\n- `'space'` / `'spacebar'`\r\n- `'tab'`\r\n- `'backspace'`\r\n- `'delete'` / `'del'`\r\n- `'insert'`\r\n- `'home'`\r\n- `'end'`\r\n- `'pageup'` / `'pgup'`\r\n- `'pagedown'` / `'pgdn'`\r\n\r\n### Arrow Keys\r\n- `'up'` / `'down'` / `'left'` / `'right'`\r\n\r\n### Modifier Keys\r\n- `'ctrl'` / `'control'`\r\n- `'shift'`\r\n- `'alt'`\r\n- `'win'` / `'winleft'` / `'winright'`\r\n- `'cmd'` / `'command'` (Mac)\r\n\r\n### Lock Keys\r\n- `'capslock'`\r\n- `'numlock'`\r\n- `'scrolllock'`\r\n\r\n### Punctuation\r\n- `'.'` / `','` / `'?'` / `'!'` / `';'` / `':'`\r\n- `'['` / `']'` / `'{'` / `'}'`\r\n- `'('` / `')'`\r\n- `'+'` / `'-'` / `'*'` / `'/'` / `'='`\r\n\r\n---\r\n\r\n## 🛡️ Safety Features\r\n\r\n### Failsafe Mode\r\n\r\nMove mouse to **any corner** of the screen to abort all automation.\r\n\r\n```python\r\n# Enable failsafe (enabled by default)\r\ndc = DesktopController(failsafe=True)\r\n```\r\n\r\n### Pause Control\r\n\r\n```python\r\n# Pause all automation for 2 seconds\r\ndc.pause(2.0)\r\n\r\n# Check if automation is safe to proceed\r\nif dc.is_safe():\r\n    dc.click(500, 500)\r\n```\r\n\r\n### Approval Mode\r\n\r\nRequire user confirmation before actions:\r\n\r\n```python\r\ndc = DesktopController(require_approval=True)\r\n\r\n# This will ask for confirmation\r\ndc.click(500, 500)  # Prompt: \"Allow click at (500, 500)? [y/n]\"\r\n```\r\n\r\n---\r\n\r\n## 🎨 Advanced Examples\r\n\r\n### Example 1: Automated Form Filling\r\n\r\n```python\r\ndc = DesktopController()\r\n\r\n# Click name field\r\ndc.click(300, 200)\r\ndc.type_text(\"John Doe\", wpm=80)\r\n\r\n# Tab to next field\r\ndc.press('tab')\r\ndc.type_text(\"john@example.com\", wpm=80)\r\n\r\n# Tab to password\r\ndc.press('tab')\r\ndc.type_text(\"SecurePassword123\", wpm=60)\r\n\r\n# Submit form\r\ndc.press('enter')\r\n```\r\n\r\n### Example 2: Screenshot Region and Save\r\n\r\n```python\r\n# Capture specific area\r\nregion = (100, 100, 800, 600)  # left, top, width, height\r\nimg = dc.screenshot(region=region)\r\n\r\n# Save with timestamp\r\nimport datetime\r\ntimestamp = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\r\nimg.save(f\"capture_{timestamp}.png\")\r\n```\r\n\r\n### Example 3: Multi-File Selection\r\n\r\n```python\r\n# Hold Ctrl and click multiple files\r\ndc.key_down('ctrl')\r\ndc.click(100, 200)  # First file\r\ndc.click(100, 250)  # Second file\r\ndc.click(100, 300)  # Third file\r\ndc.key_up('ctrl')\r\n\r\n# Copy selected files\r\ndc.hotkey('ctrl', 'c')\r\n```\r\n\r\n### Example 4: Window Automation\r\n\r\n```python\r\n# Activate Calculator\r\ndc.activate_window(\"Calculator\")\r\ntime.sleep(0.5)\r\n\r\n# Type calculation\r\ndc.type_text(\"5+3=\", interval=0.2)\r\ntime.sleep(0.5)\r\n\r\n# Take screenshot of result\r\ndc.screenshot(filename=\"calculation_result.png\")\r\n```\r\n\r\n### Example 5: Drag & Drop File\r\n\r\n```python\r\n# Drag file from source to destination\r\ndc.drag(\r\n    start_x=200, start_y=300,  # File location\r\n    end_x=800, end_y=500,       # Folder location\r\n    duration=1.0                 # Smooth 1-second drag\r\n)\r\n```\r\n\r\n---\r\n\r\n## ⚡ Performance Tips\r\n\r\n1. **Use instant movements** for speed: `duration=0`\r\n2. **Batch operations** instead of individual calls\r\n3. **Cache screen positions** instead of recalculating\r\n4. **Disable failsafe** for maximum performance (use with caution)\r\n5. **Use hotkeys** instead of menu navigation\r\n\r\n---\r\n\r\n## ⚠️ Important Notes\r\n\r\n- **Screen coordinates** start at (0, 0) in top-left corner\r\n- **Multi-monitor setups** may have negative coordinates for secondary displays\r\n- **Windows DPI scaling** may affect coordinate accuracy\r\n- **Failsafe corners** are: (0,0), (width-1, 0), (0, height-1), (width-1, height-1)\r\n- **Some applications** may block simulated input (games, secure apps)\r\n\r\n---\r\n\r\n## 🔧 Troubleshooting\r\n\r\n### Mouse not moving to correct position\r\n- Check DPI scaling settings\r\n- Verify screen resolution matches expectations\r\n- Use `get_screen_size()` to confirm dimensions\r\n\r\n### Keyboard input not working\r\n- Ensure target application has focus\r\n- Some apps require admin privileges\r\n- Try increasing `interval` for reliability\r\n\r\n### Failsafe triggering accidentally\r\n- Increase screen border tolerance\r\n- Move mouse away from corners during normal use\r\n- Disable if needed: `DesktopController(failsafe=False)`\r\n\r\n### Permission errors\r\n- Run Python with administrator privileges for some operations\r\n- Some secure applications block automation\r\n\r\n---\r\n\r\n## 📦 Dependencies\r\n\r\n- **PyAutoGUI** - Core automation engine\r\n- **Pillow** - Image processing\r\n- **OpenCV** (optional) - Image recognition\r\n- **PyGetWindow** - Window management\r\n\r\nInstall all:\r\n```bash\r\npip install pyautogui pillow opencv-python pygetwindow\r\n```\r\n\r\n---\r\n\r\n**Built for OpenClaw** - The ultimate desktop automation companion 🦞\r\n"
  }
}

Versions