Sikuli : Screenshot-driven automation
Submitted by reckoner on Sat, 01/30/2010 - 02:09.
What's SIKULI?
Sikuli is a visual technology to search and automate graphical user interfaces (GUI) using images (screenshots). The first release of Sikuli contains Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API's support. You can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X, or even an iphone application running in an emulator.
http://groups.csail.mit.edu/uid/sikuli
I wonder if there would be
I wonder if there would be any way to integrate this utility with Vocola. That would make it very powerful and polyvalent when a specific command with no shortcut is required.
John
pantoute wrote: I wonder if
I wonder if there would be any way to integrate this utility with Vocola. That would make it very powerful and polyvalent when a specific command with no shortcut is required.
John
I'm thinking the same (except for dragonfly/natlink). This actually runs Jython so maybe it's not out of the question.
I think it may only be a
I think it may only be a matter of being able to call their editor from within a natlink script and then save and link the resulting Sikuli script so it can be called by a natlink function.
pantoute wrote: I think it
I think it may only be a matter of being able to call their editor from within a natlink script and then save and link the resulting Sikuli script so it can be called by a natlink function.
The scripts that it produces are actually directories that contain Jython files. There is an additional step for you create an "executable" which can be run outside of the editor. The batch files that drive this have long java invocations that may be runnable from within dragonfly/natlink.
That's as far as I've gotten.
The downside is that the computer vision algorithms that it uses is not so great as the demos suggest. I think it really needs a way to cordon off part of the screen to search for the relevant icons. There is a method for doing this already, but it is once again driven by another graphic (more reliance on the computer vision algorithms) instead of being purely text based.
Do you mean that there is no
Do you mean that there is no way to restrain the script to a specific part of the screen by, for instance, specifying the x,y coordinates of the top left and bottom right corners ?
pantoute wrote: Do you mean
Do you mean that there is no way to restrain the script to a specific part of the screen by, for instance, specifying the x,y coordinates of the top left and bottom right corners ?
The only way to do this, as far as I can tell, is to use another screenshot. I looked through the forums and this topic has come up before. Hopefully, it is something they will add.