Setting up Sikuli to work with TagUI on Windows


TagUI has built-in integration with Sikuli (based on OpenCV) to allow identification of web elements and desktop user interface elements for interaction.

However, Sikuli is excluded from TagUI packaged installation due to complex dependencies that are handled by its installer. You need to do some setup first before this feature is available.

Commands that support visual automation include: click, hover, type, select, read, show, save, snap

Setting up Sikuli

  1. Make sure Java JDK v8 is installed. Open a command prompt and type the following:
  2. java -version
    You will see something like the following:
    java version "1.8.0_131"
    Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
    The above means you have Java v8.0.131, and you are good to go.

  3. If you need to install Java, please follow the instructions here: How To Install JDK on Windows. After Java is installed, you will have to restart your command prompt or terminal to use it.
  4. Download this sikuli installer file: Unzip contents of this file to tagui/src/tagui.sikuli folder (so that sikulixsetup-1.1.3.jar is there)
  5. Right-click on sikulixsetup-1.1.3.jar to run as administrator. If there is no option to run this java file when you right-click, then open a command prompt as administrator, cd to the tagui/src/tagui.sikuli folder, and run the sikulixsetup-1.1.3.jar using the following command:
  6. java -jar sikulixsetup-1.1.3.jar
  7. Choose "Option 1 - Pack1", and click "yes" to the pop-up dialog boxes during installation.
  8. If everything's ok, you should see a pop-up message like this:
  9. sikuli-setup-ok

  10. Double check that there is a new file runsikulix.cmd in the tagui/src/tagui.sikuli folder.

Testing Sikuli

  1. Use the Window's Snipping Tool to capture the Windows logo in the bottom-left corner:
  2. toolbar-start-menu

  3. Save this file as start-menu.png
  4. For Sikuli image recognition, make sure to save all images in png format
  5. Use the notepad to create a script as follows.
  6. click start-menu.png
    type page.png as notepad[enter]
  7. Save the above file as test1
  8. Make sure both the script "test1" and the image "start-menu.png" are in the same folder
  9. Open a command prompt, cd to the folder where the script and image is, and run using the following:
  10. tagui test1
  11. If your Sikuli is set up correctly, you should see the robot automatically move the cursor to the Start Menu, click on the Start Menu, type the word "notepad, and press the "Enter" key. You should see notepad being launched.

If this is the first time you learn about TagUI, would strongly encourage you to read the article: hello world, TagUI!

Where to go from here?

TagUI Tutorials

Comments (0)

There are no comments posted here yet

Leave your comments

Posting comment as a guest. Sign up or login to your account.
Attachments (0 / 3)
Share Your Location