Home → Extension Development →

Headless Access for Moneydance Data in Python: Demonstration and Code

• Edit

dalefurrow (Fellow User)

07 Feb, 2021 07:18 PM

Hello;

So I if you're like me, you spend a good portion of your day in python, particularly for work on investments. So I wanted to see if one could access moneydance files from python, and even run applications like investment reports in an interactive python environment. Turns out, with JPype you can.  So I created a demonstration repo out on github md_python_headless_demo, and this video to show how it works.  If anyone else is interested, I'd happily take any comments or pull requests you have on that demo project.

Thanks,

Dale (Not Staff, Just Another User)

1. 1 Posted by Stuart Beesley ... on 07 Feb, 2021 07:33 PM

   

   Very nice! I’m going to have a play with these!!

   • Edit

2. 2 Posted by dalefurrow (Fel... on 07 Feb, 2021 07:44 PM

   

   Absolutely! You've done so much good work with python scripts in md, felt it was only fair that I should contribute something.

   • Edit

3. 3 Posted by Stuart Beesley ... on 07 Feb, 2021 08:11 PM

   

   I have been trying to work out how to call / run memorised reports and I think your scripts may do this, so I will take a peek.

   • Edit

4. 4 Posted by george.dobbs on 09 Feb, 2021 12:12 PM

   

   Brilliant! Exactly what I have been lokking for!

   • Edit

5. 5 Posted by george.dobbs on 12 Feb, 2021 09:06 PM

   

   Dale, I was able to get this to work (OK, mostly). Good stuff. I notice that it works well when no master password is set on the data file. I tried to work out a way to do that, but there is some hashing and crypto going on that I don't think I'll be able to figure out.

   Looks like the place to pass in the password should be here:
   wrapper.loadDataModel(None) where None is replaced by something that implements
   public interface com.moneydance.security.SecretKeyCallback

   Might need to know more about how that interface works. By any chance did you have a go at this?

   • Edit

6. 6 Posted by Stuart Beesley ... on 12 Feb, 2021 09:16 PM

   

   I've had a nose around, it can be done, but it's a bit fiddly, certainly not standard, and requires some 'deep' java calls. You also have to reconstruct some code to make it work (so liable to brake in the future).. As Sean says on the next post, I have no idea what 'problems' this may create for your dataset, so probably not advisable... It would be great if Sean can make some standard API calls available in a future release..

   • Edit

7. Support Staff 7 Posted by Sean Reilly on 12 Feb, 2021 09:58 PM

   

   Please don't call methods like wrapper.loadDataModel()... bad things could happen. Let's work out a nice way to do this in a way that I can add to the official interface.

   Would the simplest way to do this be to pass a URI on the command-line that is invoked once the data file is loaded? Presumably the URL would invoke an extension, but it could also reference a report or graph URL or some other common view.

   Thanks,
   Sean

   • Edit

8. 8 Posted by dalefurrow (Fel... on 12 Feb, 2021 10:21 PM

   

   Hey George, I'll defer to Sean, possibly there's some sort of lock issue that might be a problem here...none of my data has been under password protection, and my primary use case for this has been either running regression tests for investment reports, or updating prices or security transactions, where it's helpful to be able to run java/python separate from the application with all the debug, logging, and inspection capability that you get from a regular ide. If there's a better way to do this, I'm all for it.

   • Edit

9. 9 Posted by george.dobbs on 12 Feb, 2021 10:29 PM

   

   I'm happy to take your advice on this.

   I guess I was intrigued with the idea of being able to get access to my data and reports from "outside". That would improve some of my use-cases. For instance I could schedule the investment data loads and more easily export report data out to Excel.

   I'm not sure I'm understanding what your URI scheme would entail, but it sort of sounds like Moneydance exposing an API to be called from the outside. Or maybe you are thinking of passing in a reference along with the command to start Moneydance in the first place.

   I for one am definitely in favor of supported interfaces. In the meanwhile I can continue with Python 2.7 (Jython) internal and some cut and paste. Happy to take a look any ideas and provide on man's feedback. // George

   • Edit

10. 10 Posted by Stuart Beesley ... on 12 Feb, 2021 10:35 PM

    

    FYI - just a quick point - your data is always encrypted. If you haven’t set a master passphrase then an internal / default ‘secret’ encryption passphrase is used. As the moneydance code knows this ‘secret’ then it’s able to open the dataset of one is not set by the user.

    • Edit

11. 11 Posted by george.dobbs on 13 Feb, 2021 12:29 PM

    

    Thanks for that note, Stuart. Good to know.

    George Dobbs

    • Edit

12. 12 Posted by Stuart Beesley ... on 13 Feb, 2021 05:48 PM

    

    Hi Sean, I guess the several points/questions are:
    1/ Some users would like to be able to load the dataset from Python outside of MD... I.e. dale is trying to do this with JPype (to link with the JVM). So can an API be exposed to do this (load the dataset and allow API calls to be made)?
    2/ I would like to be able to execute MD from the command line and pass a Python script parameter, extension parameter, report, to execute and then exit.. I've tried the method that's supposed to exist, but I could not get it working....?
    3/ For both 1 & 2 above (which are different requests), can a user specified passphrase be passed as an argument to decrypt the dataset if a user specified passphrase is set?

    Thanks

    • Edit

13. 13 Posted by Stuart Beesley ... on 12 Apr, 2021 03:09 PM

    

    @dalefurrow, @george.dobbs. I though I would have a play with this and also resolve the question posed about how to open and access the MD dataset headless when there is a user specific passphrase encryption set.. It works - see my revised code attached..

    But do NOTE Sean's warning above. He advises against doing this. I take no responsibility for its usage.....!

    As already discussed, we really need an API method and also a command line method to do this...

    Let me know if you use this, and that it works ok?

    • test-mdheadless-with-passphrase.py 5.99 KB
    • Edit

14. 14 Posted by dalefurrow (Fel... on 12 Apr, 2021 07:49 PM

    

    @Stuart, Thanks! I didn't have any examples of that. I haven't done much with this except to add a script based on conversation with another user, trying to automatically trigger an import (which I definitely think would be better handled as a callable to some kind of url). As far as data safety, yeah, I use this only when the app is not open, so I'm only touching the data with either the app or the application but never both simultaneously, and of course I back up regularly. I've used this method for about 5 years with a self-built java app to download investment transactions, haven't had a problem yet. Ironically, it was Sean who gave me the original code, used for regression testing for investment reports extension here. There's absolutely no way I would have continued to work on that project without headless access to do regression testing.

    • Edit

15. 15 Posted by Stuart Beesley ... on 12 Apr, 2021 07:57 PM

    

    To run a an import for example. Fairly easy. You could create a Python extension that loads at runtime and sits there waiting. Use the -invokeandquit parameter which will call .invoke() on your extension and the extension would receive the .invoke(), check the parameter and run the import etc.

    • Edit

16. 16 Posted by dalefurrow (Fel... on 12 Apr, 2021 09:58 PM

    

    @ Stuart, yeah, I got that, but the user wanted to run the import as a scheduled job, outside of MD...any way you know to do that?

    • Edit

17. 17 Posted by Stuart Beesley ... on 13 Apr, 2021 07:05 AM

    

    Most Os have a scheduling system. Eg Cron, task scheduler. You would have to play with these. I believe you can pass MD a ofx or qif import file name as a parameter and it will import that.

    • Edit

18. 18 Posted by Stuart Beesley ... on 14 Apr, 2021 01:11 PM

    

    An update... I've just installed Jython and I think this is perhaps a better approach to Python + JPype as it gives you more options (as it's all java based).

    I have created a modified version of my launch script which sets up the JVM environment properly as Moneydance requires/expects when Jython launches.

    You can run the launch shell script, which launches Jython (and thus the JVM with all the right settings etc), and runs your/my headless script(s) with a couple of tweaks. If anyone wants examples, shout....

    • Edit

19. System closed this discussion on 14 Jul, 2021 01:20 PM.

20. Stuart Beesley - JUST A FELLOW USER and Toolbox ‘guy’ re-opened this discussion on 05 Aug, 2021 06:38 PM

21. 19 Posted by Stuart Beesley ... on 05 Aug, 2021 06:38 PM

    

    FYI in PREVIEW 2021.2(3089) - you can now set the environment variables

    md_passphrase=yourpassphrase
    or
    md_passphrase_=yourpassphrase

    and MD will read this to bypass the encryption passphrase popup entry box..

    It seems to work running the normal app. I haven't yet tested using API.....

    • Edit

22. 20 Posted by Stuart Beesley ... on 07 Aug, 2021 02:12 PM

    

    I'm very pleased to report that the new MD2021.2(3088) feature that reads an environment viable to bypass the 'enter your password' prompt works well when loading Moneydance via a shell script, and then accessing the data using Jython from the shell.

    One thing I've found is that you have to 'export' the variable from the shell script, and then it will work... Trying to set it within program just doesn't....

    This means you can avoid the 'deep' methods that Sean advised against using above (post 7), and just use normal calls...

    • Edit

23. 21 Posted by george.dobbs on 07 Aug, 2021 05:38 PM

    

    Fantastic news, Stuart. I’ll definitely give it a try!

    George Dobbs

    • Edit

24. 22 Posted by george.dobbs on 22 Aug, 2021 07:45 PM

    

    Stuart,

    I had good luck with the environment variable. It started up fine from a shell command

    export md_passphrase
    /Applications/Moneydance\ 2.app/Contents/MacOS/Moneydance

    But I'm not seeing how to do this from python.

    I'm wondering if you would be so kind as to share how you invoked moneydance?

    //George Dobbs

    • Edit

25. 23 Posted by Stuart Beesley ... on 22 Aug, 2021 07:53 PM

    

    Look at the files/folders here:

    https://yogi1967.github.io/MoneydancePythonScripts/

    The links are at the bottom of the web site under the heading ‘Experimental / advanced’.

    Let me know?

    • Edit

26. 24 Posted by Stuart Beesley ... on 22 Aug, 2021 08:44 PM

    

    To clarify further. You probably want these files:

    https://github.com/yogi1967/MoneydancePythonScripts/blob/master/ext...

    and

    https://github.com/yogi1967/MoneydancePythonScripts/blob/master/ext...

    • Edit

27. 25 Posted by george.dobbs on 24 Aug, 2021 10:49 AM

    

    Thanks for these links. Good stuff!

    George Dobbs

    • Edit

28. System closed this discussion on 23 Nov, 2021 10:50 AM.