Home → Extension Development →

How do I call bot.loadURL('...')?

• Edit

Ric Werme

08 Apr, 2021 07:41 PM

When I start the MoneyBot console, the left panel/pane/component/whatever says:

bot.loadURL('https://infinitekind.com/developer')
15:06:23.504 webbot:dispatching action on [page=loading action=finished] background thread, after waiting for load webtask:LOAD_URLval=https://infinitekind.com/developer;

So far, I haven't been able to find the magic incantation that makes this work in the snippet window.

bot.loadURL('file:///tmp/networth.html')

gives me:

  File "<string>", line 1, in <module>
NameError: name 'bot' is not defined

In the apidoc pages, I see:

Package com.moneydance.apps.md.controller.bot
Interface RobotWebSession

public interface RobotWebSession

Interface for a browser session, allowing for a script to navigate the web programatically with a full web browser including javascript and UI capabilities.

    Method Summary

Modifier and Type Method Description
...
void loadURL​(java.lang.String url)

About the closest I can get includes:

from com.moneydance.apps.md.controller import bot
print dir(bot)
print dir(bot.RobotWebSession)
# bot.loadURL('file:///tmp/networth.html')
rws = bot.RobotWebSession()
rws.loadURL('file:///tmp/networth.html')

The dir()s confirm that there's a loadURL in RobotWebSession, but creating one says:

TypeError: No visible constructors for class (com.moneydance.apps.md.controller.bot.RobotWebSession)

Are mere mortals supposed to be able to use that?

1. 1 Posted by Stuart Beesley ... on 08 Apr, 2021 08:10 PM

   

   something like this:

   x=moneybot.getWebBot()
   x.loadURL(file:///tmp/networth.html)
   

   in console

   • Edit

2. 2 Posted by Stuart Beesley ... on 08 Apr, 2021 08:24 PM

   

   as bot is an interface, I suspect it's something like this:

   from com.moneydance.apps.md.view.gui.bot import RobotBrowser
   x = RobotBrowser()
   x.loadURL("file:///Users/stu/Desktop/file.html")
   

   But I've never played with it... Perhaps this will get you started.. Perhaps Sean will jump in here too..?

   • Edit

3. 3 Posted by Stuart Beesley ... on 08 Apr, 2021 08:40 PM

   

   I also suspect it does the handling for you, but you will need a method to display on screen...

   • Edit

4. 4 Posted by Ric Werme on 09 Apr, 2021 12:41 AM

   

   Great, this works as a snippet, and displays in the browser on the right half of the Moneybot window:

   x=moneybot.getWebBot()
   x.loadURL('file:///tmp/networth.html')

   I'll go try it in my networth program, hang on a sec....

   • Edit

5. 5 Posted by Ric Werme on 09 Apr, 2021 01:45 AM

   

   Hmm, did I mess up my reply? Let me try again.

   This worked in my program:

       # Suppress printing the account records that have zero value for all dates of interest
       checkPrint(accountRecs)
       if parameters.want_text:
           maketext(parameters.text_file, datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)
       if parameters.want_html:
           makehtml(parameters.html_file, datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)
           mwb = moneybot.getWebBot()
           mwb.loadURL('file://' + parameters.html_file)
       if parameters.want_onscreen:
           maketable(datestr, accountRecs, typeWorth, netWorth, toCurrencyType.formatSemiFancy)

   You'll have to imagine the output from these parameters. (I may still have an ancient test .md file from 2005 I could dust off.)

   • foo.jpg 84.1 KB
   • Edit

6. 6 Posted by Stuart Beesley ... on 09 Apr, 2021 12:05 PM

   

   I just had a play with RobotBrowser, but gave up... It seemed too complicated for a simple task.. So I tried this and it works OK.. Take a look. It's not refined, but perhaps a start.....

   • html_viewer.py 1.25 KB
   • Edit

7. Support Staff 7 Posted by Sean Reilly on 09 Apr, 2021 12:07 PM

   

   Calling moneybot.getWebBot() will give you an object that you can use in your script to control a web session in a virtual browser. If you run it from within the moneybot console window then you will be controlling the actual visible browser which can be useful for debugging.

   So basically you can script a (usually) headless browser to login to web sites, navigate the site, extract information from web pages, and register callbacks to intercept downloaded files or http responses with specific mime types.

   For example, here is a script that opens hsbc.co.uk logs in using their challenge-response forms. It doesn't yet automate the downloading of transactions, but that's not far off.

   IMO this is incredibly powerful and will soon allow us to download or sync transactions with any bank that has a web site. ...once a few small remaining quirks are worked out.

       web = moneybot.getWebBot()
       web.setImplicitWait(120) # wait at most 20 seconds after load/click operations
       username = moneybot.getField(None, "hsbc.co.uk/user", "HSBC UK IB#", None, None, False)
       if username==None:
           return
       web.loadURL('https://www.hsbc.co.uk/1/2/')
   
       if not web.clickItemWithXPath('//*[text()=\"\n                                            Log On\n                                        \"]'):
           web.reportError('Error clicking page element')
           return
       web.waitForLoadCompletion(5)
       if not web.setValueForXPath('//*[@id=\'Username1\']', username):
           web.reportError('Error locating username input')
           return
   
       # click the login link after entering the username
       web.clickItemWithXPath('//*[@id=\'ibLogonForm\']/div[2]/div[2]/span/input')
       web.waitForLoadCompletion(5)
   
       # click on the "no secure key" option
       print("clicking no-secure-key option...")
       if web.clickItemWithXPath("//span[contains(.,'Without Secure Key')]"):
           print("clicked")
       else:
           print("failed to engage no-secure-key option")
           return
       web.waitForLoadCompletion(5)
   
       memQNode = web.nodeForXPath('//*[@id=\'dijit_form_Form_0\']/div[1]/div/div/label')
       memQ = None
       if memQNode:
           memQ = memQNode.getText()
       if not memQ:
           web.reportError("Unable to retrieve memorable question")
           return
       
       memAnswer = moneybot.getField(None, "hsbc.co.uk/memans:%s"%(memQ), "HSBC UK Memorable Question: "+memQ, None, None, True)
       if not memAnswer:
           return
       if web.setValueForXPath('//*[@id=\'memorableAnswer\']', memAnswer):
           print("set memorable answer")
       else:
           web.reportError("Couldn't get or set memorable answer")
           return
   
       passCharNodes = web.nodesForXPath("//*[@id='hsbcwidget_CharacterChallenge_0']/div/h4/span")
       if not passCharNodes or len(passCharNodes) < 3:
           web.reportError("Couldn't locate password challenge")
           return
       
       passwordFields = web.nodesForXPath("//fieldset/input[@type='password']")
       numEnabledFields = 0
       if not passwordFields or len(passwordFields) < 3:
           web.reportError("Couldn't locate password challenge fields")
           return
       for passwordField in passwordFields:
           if passwordField.isEnabled():
               numEnabledFields += 1
               print("enabled field: %s"%(passwordField))
       if numEnabledFields != len(passCharNodes):
           print("found %s password field nodes and %s requested password characters"%(numEnabledFields, len(passCharNodes)))
           web.reportError("Mismatch between number of enabled password character fields and number of requested characters")
           return
       
       userPass = moneybot.getField(None, "hsbc.co.uk/pass", "HSBC UK Password:", None, None, True)
       if not userPass:
           return
       passChars = []
       for passCharNode in passCharNodes:
           passwdChar = tikmoneynet.get_named_character(userPass, passCharNode.getText())
           if not passwdChar:
               web.reportError("Couldn't find character ", passCharNode.getText()," in password ")
               return
           else:
               passChars.append(passwdChar)
       passwordCharIdx = 0
       
       for passwordField in passwordFields:
           if passwordField.isEnabled():
               if passwordCharIdx >= len(passChars):
                   web.reportError("More password fields were enabled than characters requested");
                   return
               web.setValueForNode(passwordField, passChars[passwordCharIdx])
               passwordCharIdx += 1
       print("set %s password fields in challenge-response"%(passwordCharIdx))
   

   Thanks,
   Sean

   --
   Sean Reilly
   Developer, The Infinite Kind
   https://infinitekind.com

   • Edit

8. Support Staff 8 Posted by Sean Reilly on 09 Apr, 2021 12:09 PM

   

   Stuart,
   For what you want to do - display html in a visible web view, using a JEditorPane is probably the right approach. Since I've got the embedded browser I could pretty easily make that accessible as a viewer for html generated by extensions, so that will go onto my to-do list.

   Thanks,
   Sean

   --
   Sean Reilly
   Developer, The Infinite Kind
   https://infinitekind.com

   • Edit

9. 9 Posted by Stuart Beesley ... on 09 Apr, 2021 12:23 PM

   

   Yes, I figured JEditorPane() would be simpler. It would be great to get access to the embedded browser via extensions sometime.

   Thanks for the Robot code too.

   Thanks.

   • Edit

10. 10 Posted by Stuart Beesley ... on 09 Apr, 2021 02:35 PM

    

    Ric, given javafx is also available, I thought I would try this too... Works very well and renders web pages better than JEditorPane()... See latest version attached.

    • html_viewer.py 2.06 KB
    • Edit

11. 11 Posted by Ric Werme on 12 Apr, 2021 01:12 AM

    

    Stuart, I tried your javafx link. I like it. It may work well with --invoke_and_quit, at least when I run html_viewer.py with my networth.html page, that page stays up after my program exits.

    Thanks.

    • Edit

12. 12 Posted by Stuart Beesley ... on 12 Apr, 2021 05:04 AM

    

    Hi Ric, I presume you have built an mxt and installed as an extension if you are using invoke and quit? Out of interest, which options in script_info are you using?

    • Edit

13. 13 Posted by Ric Werme on 12 Apr, 2021 12:09 PM

    

    So far, all my work has been in the MoneyBot console - Load Script, Run, gaze upon diagnostics messages and stack traces that remind me that I don't know beans about Java Swing (don't know Java Beans, either). Recent decades have been all Python and C.

    I haven't looked at the mxt stuff yet. I haven't encountered script_info yet.

    I'll clean up what I have now some, add a call to invokeLater() (and contemplate/look for multiple java threads, I assume that's why it's important). And contemplate the evils of invoking a second copy of Moneydance from a script while the interactive version is live but idle. I don't change anything in the database, and I never saw problems in the pre-2017 releases. It should be a safe thing to run.

    I'll post cleaned up code on my web site or here on my networth discussion, though I might make a stab at running from a script first. Oh, I also should install Moneydance on my Windows-10 system and try it there. I don't have a Mac.

    In real life, I only run it once or twice a month after updating month-end data, so I'm not too motivated to make a mxt version.

    • Edit

14. 14 Posted by Stuart Beesley ... on 12 Apr, 2021 12:17 PM

    

    Ric, I was curious how you had or were going to get --invoke_and_quit to work with your Python script..?

    • Edit

15. 15 Posted by Ric Werme on 13 Apr, 2021 12:43 PM

    

    The intermediate answer is "-invoke_and_quit isn't working for me on the command line."

    I'll play around more and maybe start a new discussion.

    I will comment on your Moneydance start script right now.

    • Edit

16. 16 Posted by Stuart Beesley ... on 13 Apr, 2021 04:39 PM

    

    -invoke_and_quit and -invoke do work. You need to call your extension

    -invoke_and_quit=moneydance:fmodule:myextensionname:parameter
    

    But as the 'wizard' says, with ?s you might need to split as:

    -invoke_and_quit "moneydance:fmodule:myextensionname:parameter"
    

    • Edit

17. 17 Posted by Dan P on 27 Jun, 2021 05:15 PM

    

    Sean,
    I am very interested in using the embedded browser, but I can't get your example to run. I'm getting the "return outside of function'" at line 5. I am not a python programmer and reading the error/solution is not sinking in. I agree you that "this is incredibly powerful and will soon allow us to download or sync transactions with any bank that has a web site"
    The web site I'm trying in the web bot browser works until I input the user/password. The bot has a problem with that response. Until I can get a script running, I can't really debug it.

    Thoughts?

    PS: I have a scraper extension using a whole bunch of the Apache libraries which is overkill. But I need them to save and reload a CookieStore for my banks credentials. Please add a cookiehandler to your todo list.

    • Edit

18. 18 Posted by Stuart Beesley ... on 01 Jul, 2021 08:46 AM

    

    .

    • Edit

19. 19 Posted by Stuart Beesley ... on 01 Jul, 2021 08:47 AM

    

    That’s easy to fix. Put the code inside a function. E.g.

    def runbot():
       Code here

    runbot()

    • Edit

20. 20 Posted by Stuart Beesley ... on 01 Jul, 2021 08:50 AM

    

    the reason is that you cannot call return from a top level code. The alternative is to call raise to force an exit. E.g. replace return with

    raise Exception("username cannot be empty")

    • Edit

21. 21 Posted by Stuart Beesley ... on 01 Jul, 2021 08:52 AM

    

    PPS - MD ofx downloads do handle and store cookies. Maybe Sean can comment on whether these functions can be used…

    • Edit

22. 22 Posted by Dan P on 01 Jul, 2021 01:18 PM

    

    Stuart,
    Thanks for the runbot() tip. I assumed it had to be enclosed in something.

    On the cookie issue. Yes, MD does handle cookies but only in the current session. Once the program is closed all cookies are lost. At least that's my observation. I have used the Apache libraries to process cookies in my extension to log in and download QFX files. I save the cookie store to a file and reload it next time. When I next log in, the cookies are there and my bank does not challenge me for verification.
    My comment for Sean was to allow some access that we could use to save and load cookies. Perhaps there is another way?

    Using a snippet and changing the python return to exit did work and I managed to connect to the FI and enter my username. I successfully found the button to click, however, the site didn't respond. Using the moneybot console to manually log in, it seems two of my FI's are using code that the internal browser doesn't handle.

    I have given up for now on developing this with moneybot.

    Thanks again for your help.

    • Edit

23. Ric Werme closed this discussion on 22 Sep, 2021 04:37 AM.