Project Description:

Development and support of a bot that will interact with a web service using only screen images and simulate user actions.

The task is to teach the bot to navigate through predefined paths (sequences of screens) by performing specified actions:

• Screen/window capture
• Mouse and keyboard control (clicks, scrolling, input, key combinations)
• Clipboard operations (reading copied text)
• Screen/interface state recognition
• Finding elements by image (buttons, icons) and by text. Use OCR if necessary
• Storing scenarios in YAML/JSON (I define the path in words, the bot executes)
• Ability to describe: expected screen type, what to search for (text/scheme number/synonyms), what to do (click, scroll, input data, copy)
• Handling timeouts, errors, pop-up windows
• Implement a "learning from experience" mechanism: log successful/unsuccessful steps, expand lists of acceptable options, optimize routes (cache successful actions, reduce the number of searches).

Experience Expectations:

• Practical experience in UI automation by image, not by DOM • Experience working via RDP/remote desktop • Experience with OpenCV, template matching, element search by template • Experience with OCR (Tesseract or similar) and text processing • Ability to build configurable scenarios • Describe steps in YAML/JSON and write an engine that executes them

Will be a plus:

• Experience developing RPA solutions with real practical custom code (not just drag-and-drop) • Experience in computer vision and/or text embeddings (fuzzy search by names) • Any projects like “a bot that plays/works based on screenshots”