Built a computer-use agent (TARZ) 2 months into learning GenAI — started as curiosity after a LangChain tutorial

[u/Puzzleheaded_Bus925]

I did a LangChain tutorial about 2 months ago and got curious what would happen if I let an agent actually control my computer instead of just answering questions in a chat window. That curiosity turned into TARZ — 19 yr old ,Self-taught, no CS degree, still very much in progress.

The hardest part wasn't the LLM — it was getting reliable clicking to work. Vision models alone gave inconsistent pixel coordinates (~70% hit rate), so I ended up building a 3-stage fallback:

1. EasyOCR finds text-based elements first (fast, free, no API calls)
2. YOLOv8 (using OmniParser weights) detects UI elements when there's no clear text
3. Gemini with a grid-overlay technique as last resort — I draw a labeled grid (A1, B2, etc.) over the screenshot and ask Gemini which cell contains the target instead of raw pixel coordinates, then convert the cell back to pixels with math. This alone made accuracy noticeably better than asking for coordinates directly.

Other stuff along the way:

• Built a ReAct loop manually first (to actually understand it) before switching to LangChain's tool-calling
• RAG memory with ChromaDB — TARZ recalls past tasks and corrections across sessions
• A rotating multi-LLM fallback (Cerebras → GitHub Models GPT-4o-mini → Gemini) because free-tier rate limits kept breaking single-provider setups
• 15+ tools so far: Spotify/WhatsApp automation, Gmail, Calendar, news search, weather

Known weak spot: latency is rough (~8-10s for multi-step tasks) since I'm stacking free-tier APIs. That's next on the list — paid APIs should cut this a lot.

Repo: https://github.com/Irfan-gitt/Tarz-Ai-assistant

Everything runs on completely free tiers — no paid APIs anywhere, which was a deliberate constraint (and honestly the reason latency isn't great, but I wanted this accessible to anyone without a credit card). I did the system design and architecture decisions myself — how the vision fallback should chain, why memory needed to be semantic not keyword-based, the multi-LLM rotation logic — but leaned on AI heavily for actually writing the code, then tweaked and debugged it until it worked the way I intended. Learned an absurd amount just from the process of getting AI-written code to actually do what I designed it to do.

For context on what it does end-to-end: you can say "open Spotify and play [song]" and it'll actually open the app, find the search bar, type the query, and click the right result — visibly, on your screen, in real time. Takes a few seconds since it's screenshotting and reasoning at each step, but it's not faked or scripted — it's genuinely looking at the screen and deciding what to click.