03 17
Selenium with Sikuli
Darshan Chudasama

Selenium with Sikuli


“Sikuli can Automate anything that you see on the screen, It’s like an GOD’s eye”.

  • Sikuli is a GUI automation tool, it has the image recognition method which help us to track the object with the help of screenshots. It matches the visual image and gets the location of the object to perform actions through Selenium script.
  • Sometimes it is very difficult to automate objects which do not have any id, name or sometimes it happens that object keep changing its locators. In that case using Sikuli can be very much helpful, it will trigger GUI interactions based on the visual match method and allow you to track the location of the object.
  • With Sikuli you can even automate the Windows screen and its applications


  • It is an Open source Tool and free of cost.
  • And one of the biggest advantage of Sikuli is that, it can easily automate Flash objects or any image.
  • It makes easy to automate windows application.
  • When you’re testing an application under development and you don’t know the ID/name of the elements, then you can go with Sikuli. It will check the appearance of the image and if match found, it will interact with the image accordingly and perform all the actions.
  • We can integrate with Selenium and with other tools as well


To start with sikuli, we need the following tools, with the help of them we can create an sikuli script:

  • Any screenshot capturing tool
  • JDK
  • Eclipse
  • Download Sikuli jars
  • Add Sikuli jar into your project.
  • Once Setup is complete then we can start writing selenium/sikuli scripts.
  • We will use some Sikuli classes that we will get once we will add that jars into the project.
  • Screen – This is a class, which will focus on the screen.
  • Pattern- This is another class, which will focus on images.

How it’s done:

  1. Crop the image of the object which could not be identified by Selenium and put it into data pool folder.
  2. Pass the path of that image into selenium script.
  3. At the run time, Sikuli searches the image on the screen and matches it to the image in the data pool. If found then it performs the desirable action on that object.


  • First Download this setup of the SikuliX (https://launchpad.net/sikuli)
  • Create a folder in the C:// with folder name SikuliX and copy the setup file to this folder.
  • After that run this setup and select no. 2 option, to develop automation script using Java language.
  • Add the jar file which is extracted from the setup file to the Project.

Methods to use in the development of the automation script

  • Screen s=new Screen();
  • click(“test.png”); or You can use the path C://windows//abcd.png
  • rightClick(“test.png”);
  • find(“test.png”);
  • doubleClick(“test.png”);
  • exists(“test.png”); (This Method is used to check whether the specified element is present on the screen.)
  • wheel(“test.png”,25,0); (This method is used to perform wheeling action on the element image.)
  • dragDrop(“test.png”,”test1.png”);
  • hover(“test.png”);    (This method is used to perform roll hover event on the specified image.)
  • paste(“test.png”,”test”); (This method used to paste text on the specified textbox)

Code Example

package sikuli;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.sikuli.script.FindFailed;

import org.sikuli.script.Screen;

public class sikulialert {

public static void main(String[] args) throws FindFailed, InterruptedException {


WebDriver driver=new ChromeDriver();



Screen s=new Screen();










    • In the above code, We opened the Url with webdriver and followed by creating object for Screen class which will allow to perform action on screen. In the class, we have passed the reference of any image on which we have to perform any operation (like click/double click/type, etc.) While executing this program, Selenium searches the image on the screen which we have passed as reference and performs the desired action on that if it exists.


  • Nowadays Selenium is widely used tool for automating web-application. In real life, products are enterprise application, so we can expect application would be an integration of Silverlight, Java Applets, Raphael Java Script, Canvas Objects, Java Scripts, JQuery, etc. However, we cannot expect everything gets identified with Selenium and can face challenges in automating the few of the above mentioned UI with Selenium. So this time you get stuck, add a flavor of Sikuli in your script.


6 thoughts on “Selenium with Sikuli

  1. I just want to mention I am just newbie to weblog and certainly loved your web site. Most likely I’m want to bookmark your site . You definitely have excellent well written articles. Thanks a bunch for revealing your website page.

  2. You undoubtedly help it become search really easy with all your speech nevertheless i discover this specific topic to be seriously a very important factor i believe I’d personally certainly not recognize. It appears as well complex and very large in my situation.. experienced SMM I am writing about ahead of time in your up coming submit, Let me try to get the hang than it!

Leave a Reply

Your email address will not be published. Required fields are marked *