Project ideas from Hacker News discussions.

Show HN: Let your AI agents paint big arrows, boxes and text on your screen

📝 Discussion Summary (Click to expand)

Theme 1 – Guidance and teaching
Many commenters see the big‑arrow tool as a way for agents (or humans) to point out UI elements, making tutorials or remote help easier.
- inanutshellus: ““Teach me to do this” kinda stuff. ‘Guiding agent’ rather than ‘doing agent’.”
- ghm2180: “Pointing my aging aunt to the right place on the screen to click without having to take control of her laptop, of course.”
- mistersquid: “Claude walked me through how to use Apple's Compressor.app… It walked me through dragging the file into compressor and clicking on the appropriate UI elements.”

Theme 2 – Dismissed as low‑effort joke or “slop”
A frequent reaction is that the project is frivolous, requiring little thought and adding little value.
- socializer: “It's just the epitome of living in an era where code is so cheap to generate that we don't invest any effort into even fully fleshing out the idea. It's just ‘Claude, make me a program that draws arrows on screen’.”
- mococa: “Another slop project on front page.”
- isoprophlex (joking): “rainbow dripping arrows… angrily pointing arrows… flame‑surrounded text boxes… airhorn.wav at max volume.”

Theme 3 – Security, permissions, and focus concerns
Several users worry about letting an agent draw over sensitive dialogs (e.g., permission prompts) and about the tool’s interaction with system safeguards.
- hn8726: “If it can draw on top of permission prompts, what's stopping it from drawing box that hides the ‘decline’ button and changing the ‘approve’ button copy?”
- jaapz: “isn’t --dangerously-skip-permissions enough?”
- franze: “Claude refuses to do certain actions (enter passwords, change security settings, create new accounts on external services) even in Yolo mode running as sudo.”
- sitkack shows the flag --norms-taboos-off, indicating users can bypass built‑in safety checks.


🚀 Project Ideas

Agent-Guided UI Assistant

Summary

  • A cross-platform daemon that lets LLM agents draw precise, accessible UI annotations (arrows, labels, voice prompts) on any application, even over security dialogs, using system accessibility APIs.
  • Enables agents to guide non‑technical users through complex workflows without taking control of the mouse or keyboard.

Details

Key Value
Target Audience Developers building LLM agents, remote support teams, and caregivers assisting aging parents
Core Feature Real‑time, permission‑aware overlay drawing with semantic labels and optional speech output
Tech Stack Electron/Tauri for cross‑platform UI, macOS Accessibility API / Windows UI Automation / Linux AT-SPI, WebSocket API for agent communication
Difficulty Medium
Monetization Revenue-ready: Freemium (free tier for personal use, $5/mo per agent for commercial SDK)
#### Notes
- HN users highlighted the need for agents to point at buttons in permission prompts (franze: “Claude refuses to do certain actions… even in Yolo mode”) and to help aging parents (ghm2180: “Pointing my aging aunt to the right place on the screen to click”). This tool directly addresses those pain points.
- Provides a programmable skill that agents can invoke without burning tens of thousands of tokens, as suggested by ballofrubber1’s comment about avoiding token waste.
- Enables discussion on safe agent‑UI interaction standards and could become a de‑facto “pointing” protocol for LLM‑driven automation.

Remote Help Overlay Service

Summary

  • A WebRTC‑based screen‑sharing service where a helper can draw live arrows, highlights, and voice annotations on a remote user’s screen to walk them through tasks.
  • No installation required on the user's side beyond a simple viewer link; works across browsers and native apps.

Details

Key Value
Target Audience Family tech support, IT helpdesks, and tutors assisting non‑technical or elderly users
Core Feature Live collaborative overlay with persistent annotations and optional audio guidance
Tech Stack WebRTC (mediasoup), React.js frontend, Node.js signaling server, optional native wrapper for macOS/Windows using WebView
Difficulty Medium
Monetization Revenue-ready: Subscription ($10/mo per helper seat, free for viewers)
#### Notes
- Commenters repeatedly mentioned the pain of guiding parents over the phone (ghm2180: “I’ve lost count of how many times I’ve had to repeat this over the phone to my parents”; lbreakjai: “I would pay good money for something like this on iPad… a godsend for aging parents”). This service gives a visual, hands‑free alternative.
- Solves the focus‑stealing issue noted by inanutshellus (“if the said 'big arrow' automatically gains focus…”) by keeping the overlay in a separate, non‑focus‑stealing layer.
- Could spark HN discussion about privacy‑preserving remote assistance and accessibility standards for elder care.

Interactive Tutorial Creator for Agents

Summary

  • A recorder that captures user interactions (clicks, keystrokes, UI changes) and automatically generates annotated step‑by‑step tutorials with arrows, labels, and explanations.
  • Outputs machine‑readable JSON (or Markdown) that LLM agents can consume to guide users, or humans can use as documentation.

Details

Key Value
Target Audience Content creators, software trainers, and agent developers building guided workflows
Core Feature Interaction recorder → auto‑generated visual tutorial with exportable agent‑friendly format
Tech Stack Desktop recorder (Electron/Framer), OpenCV for UI element detection, annotation engine (SVG/Canvas), export to JSON/Markdown
Difficulty Low
Monetization Hobby
#### Notes
- Users like ravila4 expressed frustration when learning Blender: “I spent a lot of time trying to figure out which button the model was talking about.” This tool turns those moments into reusable tutorials.
- alansaber and melvinroest discussed giving LLMs visual pointing capabilities; this provides the curated content (tutorials) that agents can leverage without needing to reason from raw screenshots.
- Encourages discussion on the balance between agent‑generated guidance and human‑authored tutorials, a topic raised by several commenters.

Read Later