1

Guys, i m planningg to do some sorts of automation with Python, my main ground would be pyautokey moddule, i will ask a sighted person to use mouse and i will save coordinates of each mouse movement to similate mouse clicks and moves.

This way i i intend to over come some accessibility problems as well.

So my question is; how can i record all these mouse  drag and drops, clicks, scrawls etc? if i can right down all coordinates as campaigns then it means, i can use pyautokey aas welll.

Thumbs up

2

you can try using pygame.mouse.get_pos() and pygame.mouse.set_pos().
for the complete documentation of the mouse look here

If you want to get in touch with me you can follow me on Twitter
have a nice day.
Paul

Thumbs up

3

PS, you can try it if you don't get any luck with the library you're currently using

If you want to get in touch with me you can follow me on Twitter
have a nice day.
Paul

Thumbs up

4 (edited by Turkce_Rap 2018-06-26 19:27:06)

pauliyobo wrote:

PS, you can try it if you don't get any luck with the library you're currently using


i always thought this lib meant for gaming pourpices, thank you  for the doc.

Nevertheless how can i save the positions with this function while a sighted person deal with the mouse in the mean time?

Thumbs up

5

I suppose with a name like Pygame its easy to think its only for games, but its actually a wrapper around the Simple DirectMedia Layer API, or SDL library for short. There are other libraries that can also do the same, like Pyglet or WxPython, among others. In Pyglet you can draw the mouse as an icon using its captured coordinates and move it around, for example:

import pyglet
from pyglet.window import mouse


class Example(pyglet.window.Window):
    def __init__(self):
        super(Example, self).__init__(640, 480, resizable=False, fullscreen=False, caption="Example")
##        self.set_exclusive_mouse(True)
        self.clear()

   #mouse cursor position
        self.mouse = [0,0]

        pyglet.clock.schedule_interval(self.update, .01)

    def update(self,dt):
    #purge mouse clicks
        if len(self.mouse) > 2:
            self.mouse = self.mouse[:2]
    #draw screen
        self.draw()

    def draw(self):
        self.clear()
        #If you set exclusive mouse you can draw the mouse with an image
        #using the mouses current position for example:

        #self.icon.blit(self.mouse[0],self.mouse[1])

#Mouse Motion Input
    def on_mouse_motion(self,x,y,dx,dy):
        self.mouse[0] += dx
        self.mouse[1] += dy

#Mouse Drag Input
    def on_mouse_drag(self,x,y,dx,dy,buttons,modifiers):
        self.mouse[0] += dx
        self.mouse[1] += dy

#Mouse Press Input
    def on_mouse_press(self,x,y,button,modifiers):
        self.mouse.append(mouse.buttons_string(button) + " press")

#Mouse Release Input
    def on_mouse_release(self,x,y,button,modifiers):
        self.mouse.append(mouse.buttons_string(button) + " release")



if __name__ == '__main__':
    window = Example()
    pyglet.app.run()

You can also do the same in WxPython by using Get Mouse Position and Warp Pointer:

import wx

class Frame(wx.Frame):
    def __init__(self):
        wx.Frame.__init__(self, None, title='Example', pos=(150,150), size=(350,200))

    #get the mouses current coordinates relative to the windows upper left corner
        print self.ScreenToClient(wx.GetMousePosition())

    #move mouse to the upper left of the window
        self.WarpPointer(0,0)


app = wx.App()
top = Frame()
top.Show()
app.MainLoop()
-BrushTone v1.3.1: Accessible Paint Tool
-AudiMesh3D v1.0.0: Accessible 3D Model Viewer

Thumbs up

6

magurp244 wrote:

I suppose with a name like Pygame its easy to think its only for games, but its actually a wrapper around the Simple DirectMedia Layer API, or SDL library for short. There are other libraries that can also do the same, like Pyglet or WxPython, among others. In Pyglet you can draw the mouse as an icon using its captured coordinates and move it around, for example:

import pyglet
from pyglet.window import mouse


class Example(pyglet.window.Window):
    def __init__(self):
        super(Example, self).__init__(640, 480, resizable=False, fullscreen=False, caption="Example")
##        self.set_exclusive_mouse(True)
        self.clear()

   #mouse cursor position
        self.mouse = [0,0]

        pyglet.clock.schedule_interval(self.update, .01)

    def update(self,dt):
    #purge mouse clicks
        if len(self.mouse) > 2:
            self.mouse = self.mouse[:2]
    #draw screen
        self.draw()

    def draw(self):
        self.clear()
        #If you set exclusive mouse you can draw the mouse with an image
        #using the mouses current position for example:

        #self.icon.blit(self.mouse[0],self.mouse[1])

#Mouse Motion Input
    def on_mouse_motion(self,x,y,dx,dy):
        self.mouse[0] += dx
        self.mouse[1] += dy

#Mouse Drag Input
    def on_mouse_drag(self,x,y,dx,dy,buttons,modifiers):
        self.mouse[0] += dx
        self.mouse[1] += dy

#Mouse Press Input
    def on_mouse_press(self,x,y,button,modifiers):
        self.mouse.append(mouse.buttons_string(button) + " press")

#Mouse Release Input
    def on_mouse_release(self,x,y,button,modifiers):
        self.mouse.append(mouse.buttons_string(button) + " release")



if __name__ == '__main__':
    window = Example()
    pyglet.app.run()

You can also do the same in WxPython by using Get Mouse Position and Warp Pointer:

import wx

class Frame(wx.Frame):
    def __init__(self):
        wx.Frame.__init__(self, None, title='Example', pos=(150,150), size=(350,200))

    #get the mouses current coordinates relative to the windows upper left corner
        print self.ScreenToClient(wx.GetMousePosition())

    #move mouse to the upper left of the window
        self.WarpPointer(0,0)


app = wx.App()
top = Frame()
top.Show()
app.MainLoop()

Sorry for my ignorance but, let's say i've asked a sighted person to click a spesiphic software and he/she done afew tricks by mouse on the software, will i be able to record everything to similate?

Thumbs up

7

If I understand you correctly, you'd like to log mouse input globally across other applications? That would be a bit trickier to do, as the functions in the libraries mentioned are designed largely to only track the mouse when its within your programs own focus, not others. This is more of a security feature, as things that typically capture mouse or keyboard events outside the OS could be considered a form of malware such as with keyloggers. I did turn up a library [here] though that may work for what you have in mind.

-BrushTone v1.3.1: Accessible Paint Tool
-AudiMesh3D v1.0.0: Accessible 3D Model Viewer

Thumbs up

8

magurp244 wrote:

If I understand you correctly, you'd like to log mouse input globally across other applications? That would be a bit trickier to do, as the functions in the libraries mentioned are designed largely to only track the mouse when its within your programs own focus, not others. This is more of a security feature, as things that typically capture mouse or keyboard events outside the OS could be considered a form of malware such as with keyloggers. I did turn up a library [here] though that may work for what you have in mind.


it seems that lib  doesn't listen coordinates.

Thumbs up