How do I call bot.loadURL('...')?

Ric Werme's Avatar

Ric Werme

08 Apr, 2021 07:41 PM

When I start the MoneyBot console, the left panel/pane/component/whatever says:

bot.loadURL('https://infinitekind.com/developer')
15:06:23.504 webbot:dispatching action on [page=loading action=finished] background thread, after waiting for load webtask:LOAD_URLval=https://infinitekind.com/developer;

So far, I haven't been able to find the magic incantation that makes this work in the snippet window.

bot.loadURL('file:///tmp/networth.html')

gives me:

  File "<string>", line 1, in <module>
NameError: name 'bot' is not defined

In the apidoc pages, I see:

Package com.moneydance.apps.md.controller.bot
Interface RobotWebSession

public interface RobotWebSession

Interface for a browser session, allowing for a script to navigate the web programatically with a full web browser including javascript and UI capabilities.

    Method Summary

Modifier and Type Method Description
...
void loadURL​(java.lang.String url)

About the closest I can get includes:

from com.moneydance.apps.md.controller import bot
print dir(bot)
print dir(bot.RobotWebSession)
# bot.loadURL('file:///tmp/networth.html')
rws = bot.RobotWebSession()
rws.loadURL('file:///tmp/networth.html')

The dir()s confirm that there's a loadURL in RobotWebSession, but creating one says:

TypeError: No visible constructors for class (com.moneydance.apps.md.controller.bot.RobotWebSession)

Are mere mortals supposed to be able to use that?

  1. 1 Posted by Stuart Beesley ... on 08 Apr, 2021 08:10 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    something like this:

    x=moneybot.getWebBot()
    x.loadURL(file:///tmp/networth.html)
    

    in console

  2. 2 Posted by Stuart Beesley ... on 08 Apr, 2021 08:24 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    as bot is an interface, I suspect it's something like this:

    from com.moneydance.apps.md.view.gui.bot import RobotBrowser
    x = RobotBrowser()
    x.loadURL("file:///Users/stu/Desktop/file.html")
    

    But I've never played with it... Perhaps this will get you started.. Perhaps Sean will jump in here too..?

  3. 3 Posted by Stuart Beesley ... on 08 Apr, 2021 08:40 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    I also suspect it does the handling for you, but you will need a method to display on screen...

  4. 4 Posted by Ric Werme on 09 Apr, 2021 12:41 AM

    Ric Werme's Avatar

    Great, this works as a snippet, and displays in the browser on the right half of the Moneybot window:

    x=moneybot.getWebBot()
    x.loadURL('file:///tmp/networth.html')

    I'll go try it in my networth program, hang on a sec....

  5. 5 Posted by Ric Werme on 09 Apr, 2021 01:45 AM

    Ric Werme's Avatar

    Hmm, did I mess up my reply? Let me try again.

    This worked in my program:

        # Suppress printing the account records that have zero value for all dates of interest
        checkPrint(accountRecs)
        if parameters.want_text:
            maketext(parameters.text_file, datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)
        if parameters.want_html:
            makehtml(parameters.html_file, datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)
            mwb = moneybot.getWebBot()
            mwb.loadURL('file://' + parameters.html_file)
        if parameters.want_onscreen:
            maketable(datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)

    You'll have to imagine the output from these parameters. (I may still have an ancient test .md file from 2005 I could dust off.)

  6. 6 Posted by Stuart Beesley ... on 09 Apr, 2021 12:05 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    I just had a play with RobotBrowser, but gave up... It seemed too complicated for a simple task.. So I tried this and it works OK.. Take a look. It's not refined, but perhaps a start.....

  7. Support Staff 7 Posted by Sean Reilly on 09 Apr, 2021 12:07 PM

    Sean Reilly's Avatar

    Calling moneybot.getWebBot() will give you an object that you can use in your script to control a web session in a virtual browser. If you run it from within the moneybot console window then you will be controlling the actual visible browser which can be useful for debugging.

    So basically you can script a (usually) headless browser to login to web sites, navigate the site, extract information from web pages, and register callbacks to intercept downloaded files or http responses with specific mime types.

    For example, here is a script that opens hsbc.co.uk logs in using their challenge-response forms. It doesn't yet automate the downloading of transactions, but that's not far off.

    IMO this is incredibly powerful and will soon allow us to download or sync transactions with any bank that has a web site. ...once a few small remaining quirks are worked out.

        web = moneybot.getWebBot()
        web.setImplicitWait(120) # wait at most 20 seconds after load/click operations
        username = moneybot.getField(None, "hsbc.co.uk/user", "HSBC UK IB#", None, None, False)
        if username==None:
            return
        web.loadURL('https://www.hsbc.co.uk/1/2/')
    
        if not web.clickItemWithXPath('//*[text()=\"\n                                            Log On\n                                        \"]'):
            web.reportError('Error clicking page element')
            return
        web.waitForLoadCompletion(5)
        if not web.setValueForXPath('//*[@id=\'Username1\']', username):
            web.reportError('Error locating username input')
            return
    
        # click the login link after entering the username
        web.clickItemWithXPath('//*[@id=\'ibLogonForm\']/div[2]/div[2]/span/input')
        web.waitForLoadCompletion(5)
    
        # click on the "no secure key" option
        print("clicking no-secure-key option...")
        if web.clickItemWithXPath("//span[contains(.,'Without Secure Key')]"):
            print("clicked")
        else:
            print("failed to engage no-secure-key option")
            return
        web.waitForLoadCompletion(5)
    
        memQNode = web.nodeForXPath('//*[@id=\'dijit_form_Form_0\']/div[1]/div/div/label')
        memQ = None
        if memQNode:
            memQ = memQNode.getText()
        if not memQ:
            web.reportError("Unable to retrieve memorable question")
            return
        
        memAnswer = moneybot.getField(None, "hsbc.co.uk/memans:%s"%(memQ), "HSBC UK Memorable Question: "+memQ, None, None, True)
        if not memAnswer:
            return
        if web.setValueForXPath('//*[@id=\'memorableAnswer\']', memAnswer):
            print("set memorable answer")
        else:
            web.reportError("Couldn't get or set memorable answer")
            return
    
        passCharNodes = web.nodesForXPath("//*[@id='hsbcwidget_CharacterChallenge_0']/div/h4/span")
        if not passCharNodes or len(passCharNodes) < 3:
            web.reportError("Couldn't locate password challenge")
            return
        
        passwordFields = web.nodesForXPath("//fieldset/input[@type='password']")
        numEnabledFields = 0
        if not passwordFields or len(passwordFields) < 3:
            web.reportError("Couldn't locate password challenge fields")
            return
        for passwordField in passwordFields:
            if passwordField.isEnabled():
                numEnabledFields += 1
                print("enabled field: %s"%(passwordField))
        if numEnabledFields != len(passCharNodes):
            print("found %s password field nodes and %s requested password characters"%(numEnabledFields, len(passCharNodes)))
            web.reportError("Mismatch between number of enabled password character fields and number of requested characters")
            return
        
        userPass = moneybot.getField(None, "hsbc.co.uk/pass", "HSBC UK Password:", None, None, True)
        if not userPass:
            return
        passChars = []
        for passCharNode in passCharNodes:
            passwdChar = tikmoneynet.get_named_character(userPass, passCharNode.getText())
            if not passwdChar:
                web.reportError("Couldn't find character ", passCharNode.getText()," in password ")
                return
            else:
                passChars.append(passwdChar)
        passwordCharIdx = 0
        
        for passwordField in passwordFields:
            if passwordField.isEnabled():
                if passwordCharIdx >= len(passChars):
                    web.reportError("More password fields were enabled than characters requested");
                    return
                web.setValueForNode(passwordField, passChars[passwordCharIdx])
                passwordCharIdx += 1
        print("set %s password fields in challenge-response"%(passwordCharIdx))
    

    Thanks,
    Sean

    --
    Sean Reilly
    Developer, The Infinite Kind
    https://infinitekind.com

  8. Support Staff 8 Posted by Sean Reilly on 09 Apr, 2021 12:09 PM

    Sean Reilly's Avatar

    Stuart,
    For what you want to do - display html in a visible web view, using a JEditorPane is probably the right approach. Since I've got the embedded browser I could pretty easily make that accessible as a viewer for html generated by extensions, so that will go onto my to-do list.

    Thanks,
    Sean

    --
    Sean Reilly
    Developer, The Infinite Kind
    https://infinitekind.com

  9. 9 Posted by Stuart Beesley ... on 09 Apr, 2021 12:23 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    Yes, I figured JEditorPane() would be simpler. It would be great to get access to the embedded browser via extensions sometime.

    Thanks for the Robot code too.

    Thanks.

  10. 10 Posted by Stuart Beesley ... on 09 Apr, 2021 02:35 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    Ric, given javafx is also available, I thought I would try this too... Works very well and renders web pages better than JEditorPane()... See latest version attached.

  11. 11 Posted by Ric Werme on 12 Apr, 2021 01:12 AM

    Ric Werme's Avatar

    Stuart, I tried your javafx link. I like it. It may work well with --invoke_and_quit, at least when I run html_viewer.py with my networth.html page, that page stays up after my program exits.

    Thanks.

  12. 12 Posted by Stuart Beesley ... on 12 Apr, 2021 05:04 AM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    Hi Ric, I presume you have built an mxt and installed as an extension if you are using invoke and quit? Out of interest, which options in script_info are you using?

  13. 13 Posted by Ric Werme on 12 Apr, 2021 12:09 PM

    Ric Werme's Avatar

    So far, all my work has been in the MoneyBot console - Load Script, Run, gaze upon diagnostics messages and stack traces that remind me that I don't know beans about Java Swing (don't know Java Beans, either). Recent decades have been all Python and C.

    I haven't looked at the mxt stuff yet. I haven't encountered script_info yet.

    I'll clean up what I have now some, add a call to invokeLater() (and contemplate/look for multiple java threads, I assume that's why it's important). And contemplate the evils of invoking a second copy of Moneydance from a script while the interactive version is live but idle. I don't change anything in the database, and I never saw problems in the pre-2017 releases. It should be a safe thing to run.

    I'll post cleaned up code on my web site or here on my networth discussion, though I might make a stab at running from a script first. Oh, I also should install Moneydance on my Windows-10 system and try it there. I don't have a Mac.

    In real life, I only run it once or twice a month after updating month-end data, so I'm not too motivated to make a mxt version.

  14. 14 Posted by Stuart Beesley ... on 12 Apr, 2021 12:17 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    Ric, I was curious how you had or were going to get --invoke_and_quit to work with your Python script..?

  15. 15 Posted by Ric Werme on 13 Apr, 2021 12:43 PM

    Ric Werme's Avatar

    The intermediate answer is "-invoke_and_quit isn't working for me on the command line."

    I'll play around more and maybe start a new discussion.

    I will comment on your Moneydance start script right now.

  16. 16 Posted by Stuart Beesley ... on 13 Apr, 2021 04:39 PM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    -invoke_and_quit and -invoke do work. You need to call your extension

    -invoke_and_quit=moneydance:fmodule:myextensionname:parameter
    

    But as the 'wizard' says, with ?s you might need to split as:

    -invoke_and_quit "moneydance:fmodule:myextensionname:parameter"
    
  17. 17 Posted by Dan P on 27 Jun, 2021 05:15 PM

    Dan P's Avatar

    Sean,
    I am very interested in using the embedded browser, but I can't get your example to run. I'm getting the "return outside of function'" at line 5. I am not a python programmer and reading the error/solution is not sinking in. I agree you that "this is incredibly powerful and will soon allow us to download or sync transactions with any bank that has a web site"
    The web site I'm trying in the web bot browser works until I input the user/password. The bot has a problem with that response. Until I can get a script running, I can't really debug it.

    Thoughts?

    PS: I have a scraper extension using a whole bunch of the Apache libraries which is overkill. But I need them to save and reload a CookieStore for my banks credentials. Please add a cookiehandler to your todo list.

  18. 18 Posted by Stuart Beesley ... on 01 Jul, 2021 08:46 AM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    .

  19. 19 Posted by Stuart Beesley ... on 01 Jul, 2021 08:47 AM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    That’s easy to fix. Put the code inside a function. E.g.

    def runbot():
       Code here

    runbot()

  20. 20 Posted by Stuart Beesley ... on 01 Jul, 2021 08:50 AM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    the reason is that you cannot call return from a top level code. The alternative is to call raise to force an exit. E.g. replace return with

    raise Exception("username cannot be empty")

  21. 21 Posted by Stuart Beesley ... on 01 Jul, 2021 08:52 AM

    Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’'s Avatar

    PPS - MD ofx downloads do handle and store cookies. Maybe Sean can comment on whether these functions can be used…

  22. 22 Posted by Dan P on 01 Jul, 2021 01:18 PM

    Dan P's Avatar

    Stuart,
    Thanks for the runbot() tip. I assumed it had to be enclosed in something.

    On the cookie issue. Yes, MD does handle cookies but only in the current session. Once the program is closed all cookies are lost. At least that's my observation. I have used the Apache libraries to process cookies in my extension to log in and download QFX files. I save the cookie store to a file and reload it next time. When I next log in, the cookies are there and my bank does not challenge me for verification.
    My comment for Sean was to allow some access that we could use to save and load cookies. Perhaps there is another way?

    Using a snippet and changing the python return to exit did work and I managed to connect to the FI and enter my username. I successfully found the button to click, however, the site didn't respond. Using the moneybot console to manually log in, it seems two of my FI's are using code that the internal browser doesn't handle.

    I have given up for now on developing this with moneybot.

    Thanks again for your help.

  23. Ric Werme closed this discussion on 22 Sep, 2021 04:37 AM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac