tag:infinitekind.tenderapp.com,2009-01-14:/discussions/general-questions/114244-how-to-export-data-from-net-worth-chartInfinite Kind: Discussion 2022-01-10T04:50:20Ztag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-26T21:55:09Z2021-09-10T03:54:49ZHow to export data from Net Worth chart?<div><p>Hi,</p>
<p>I want to export the daily net worth data from the net worth chart, but looks like the "Copy to Clipboard" button only saves the chart itself, not the backend daily data. Anyone knows how can I get it? Thanks!</p>
<p>(I have attached the screenshot of the chart, but don't read too much on the chart, as the Y-axis is not indexed to 0.)</p>
<p>Jerry</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-26T23:47:54Z2021-08-26T23:47:54ZHow to export data from Net Worth chart?<div><p>I'm a fellow user.<br>
Have you tried the Net Worth <strong>Report</strong>?</p>
<p>If yes, then perhaps you could explain why that's not suitable.</p></div>tgilbert666tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-26T23:54:46Z2021-09-10T03:54:49ZHow to export data from Net Worth chart?<div><p>Thanks tgilbert666!</p>
<p>I think Net Worth Report only gets the snapshot of one day. I need get the daily snapshot for the whole history. One net worth number per day.</p>
<p>Jerry</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T01:08:30Z2021-08-27T01:08:30ZHow to export data from Net Worth chart?<div><p>Aaargh - my bad. I forgot to put my brain into gear.<br>
Perhaps take a look at user Kevin N. 's suggestion in post 23 of this thread:</p>
<p><a href="https://infinitekind.tenderapp.com/discussions/suggestions/427-net-worth-reports-for-past-dates/page/1">https://infinitekind.tenderapp.com/discussions/suggestions/427-net-...</a></p>
<p>Not sure if it will help. Perhaps someone more knowledgeable can jump in.</p></div>tgilbert666tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T16:24:27Z2021-09-10T03:54:49ZHow to export data from Net Worth chart?<div><p>Thank you again!</p>
<p>Tried out Kevin's approach. It could be useful for comparison by year or so. But not exactly what I wanted to get daily numbers and be able to download to do additional analysis.</p>
<p>Thanks anyway!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T16:53:47Z2021-08-27T16:53:47ZHow to export data from Net Worth chart?<div><p>Ric Werme Has built a networth data extract tool. He might share it?</p></div>Stuart Beesley (Mr Toolbox)tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T16:55:32Z2021-09-10T03:54:49ZHow to export data from Net Worth chart?<div><p>Yeah, would love to see an extensions or something to get more flexibility! Thanks!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T18:05:35Z2021-08-27T18:05:35ZHow to export data from Net Worth chart?<div><p>I'll send him an email...</p></div>Stuart Beesley (Mr Toolbox)tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T19:46:38Z2021-08-27T19:46:38ZHow to export data from Net Worth chart?<div><p>My extension that Stuart refers to is not quite what Jerry is looking for. It could be a good start for a Python programmer who isn't well skilled int the art but wants to learn.</p>
<p>What I wanted was an extension I could run once a month to track the value of my accounts listed on the Moneydance Summary page (i.e. not Expense and Income) and compare it to four earlier months. I wrote one ages ago in the pre 2007? Jython, and finally had time to get to the current system. Then decided to add a Java Swing interface since Stuart had all the code I could learn from.</p>
<p>I've uploaded it to <a href="http://wermenh.com/moneydance/networth_matrix.mxt">http://wermenh.com/moneydance/networth_matrix.mxt</a> and <a href="http://wermenh.com/moneydance/networth_matrix.zip">http://wermenh.com/moneydance/networth_matrix.zip</a> . It's unsigned, so if you install the .mxt file, MD will complain twice but accept it and add it to the Extensions menu. Oh - I can attach files, see attached. I can't attached a .mxt file, so fish it out of the .zip file.</p>
<p>I was going to make a new post about it soon, but the main code is done.</p>
<p>It should be safe to run as it doesn't change the database. I've had no problems during development. Developed on Linux, seems happy on Windows, Stuart has run an earlier version on MacOS. Ultimately I'd like for MD/IK to sign it and distribute it as contributed code.</p>
<p>I'll post a screen shot in a second.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-27T19:51:48Z2021-08-27T19:51:48ZHow to export data from Net Worth chart?<div><p>Here's the screenshot - When you start networth-matrix, it displays this, then does all the work when you click Proceed.</p>
<p>It can also be run from a Linux command line if Moneydance isn't running already.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-30T09:17:40Z2021-08-30T09:17:40ZHow to export data from Net Worth chart?<div><p>This looks a great feature - it would be amazing if IK could get it signed and the then integrated to a future version of MD as an approved extension.</p></div>Rich Wtag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-30T17:50:51Z2021-09-10T03:54:50ZHow to export data from Net Worth chart?<div><p>This extension is almost as what I expect. I hope the setting UI change can be more flexible, e.g. allows to specify a date range, and granularity (Say Jan 2015 - Dec 2020, by month). I am ok to copy from the HTML and paste to Excel by myself.</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-31T04:23:23Z2021-08-31T04:23:23ZHow to export data from Net Worth chart?<div><p>Jerry,</p>
<p>It sounds like you expect something designed for initializing a local database whenever you run it. I wanted something that I could run every month and smooth out big transactions like monthly mortgage payments, but also look at a reasonable set of previous months. The screen shot shows the default selections when run this month - i.e. the current year, the current month, and a set of previous months. In January its the previous years, after a quarter's end, its the previous quarters, in November nothing really makes sense, but I like triangle numbers, hence the 4321 pattern. That makes the first month reported to be January, so it's a look at the year to date.</p>
<p>The onscreen report destination lets me check for things that aren't quite ready, e.g. missed downloads of the latest charges or retirement accounts. The HTML prints well and Emails well.</p>
<p>Displaying more than five months doesn't print so well, a granularity of less than a month shows spikes due to monthly transactions. My last job paid me twice a month, so no spikes from that either.</p>
<p>The intent is to run the extension and click proceed. Or run it from the command line and skip the option panel.</p>
<p>To accommodate your wishes, I'd be inclined to create a separate "Data dump" options panel with date widgets for start & stop, granularity, and an all account flag. Then maybe make the sole output be a .csv file.</p>
<p>On thing I do want to do is replace using the AccountUtil.getBalanceAsOfDate() API. That may scan the entire database on each call. Many of my accounts are expired old loans or 401ks from past employers, but I have to check all of them in case there's a non-zero balance in one of the months, and I have to call it for each month.</p>
<p>My ancient version predated that API and I had code for scanning the database once, putting things in some 500 bins (100 accounts, 5 months) and was about 500X faster. Someone noted it had problems with currency, but I never understood the details.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-08-31T21:20:05Z2021-09-10T03:54:50ZHow to export data from Net Worth chart?<div><p>Hi Ric,</p>
<p>Really appreciate your help and the extension! I think a data dump into CSV would be enough, if it allows me to choose date range and granularity (ideally daily and monthly), and by accounts.</p>
<p>Since I want to get the NET WORTH at different time, so I don't really need to filter any income or expense categories.</p>
<p>I usually like to do further analysis in Excel, so a csv file would be enough for me.</p>
<p>Appreciate again!<br>
Jerry</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-09-01T16:36:11Z2021-09-01T16:36:11ZHow to export data from Net Worth chart?<div><p>Well, my goal in March was to adapt a 15 year-old non-GUI program of some 350 lines to get me off an ancient MD with big restrictions with Python.</p>
<p>After that, I saw all of Stuart's Swing code, so I decided I should take the dive, knowing full well it would take more time to write than it would save me editing the source code every month. It's now 1,500 lines.</p>
<p>Meanwhile, I had hoped to be deep into analyzing wind turbine infrasound, do <em>something</em> about my poor, neglected web site, get serious about scanning hundreds of 35 mm slides from the 1970s, and get in a fair amount of hiking and bicycling. A current distraction is updating some climate reference pages before the Arctic melt season ends, but that's held up by some image caching issues. I probably shouldn't attack the infrasound project (steep learning curves) until after New Hampshire's foliage season.</p>
<p>Adding a data dump options page should be pretty easy at this point (no learning curve) so I'll see what I can do. I won't speed up the data gathering code. It will report the same accounts the current code does. Have you tried using the current extension to get data five months at a time? It wouldn't take you much long to get a few year's worth of data.</p>
<p>More pesto sauce making and minimal work on my website's home page to reference my Covid-19 exponential growth/decay pages take priority, though.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-09-01T16:53:01Z2021-09-10T03:54:50ZHow to export data from Net Worth chart?<div><p>Yes, I was able to install the extension and dump 5 months data into HTML, and then copy to Excel. Though I was trying to see if I can change the python code to extend the 5 months limit, but my limited experience in programming (only knows Python to deal with data analysis) lands on nothing...</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-09-22T04:33:20Z2021-09-22T04:33:20ZHow to export data from Net Worth chart?<div><p>I've scrounged some time to much around with a data dump function. The interface window will look like the attached. There's a few things to tweak, and I want to add a batch/invoke functions so it will work from the command line.</p>
<p>I'll likely impose on Stu to try it out first, but it'll likely be ready to share in a few days.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-09-22T04:40:52Z2021-09-22T04:40:53ZHow to export data from Net Worth chart?<div><p>Thanks Ric! The UI looks awesome!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-02T18:02:44Z2021-10-02T18:02:44ZHow to export data from Net Worth chart?<div><p>Jerry,</p>
<p>I have something that should work well. (Note to self: I should try this on Windows....) At least it seems to work well on Linux, as an (unsigned) extension started from Moneydance or on a Linux command line.</p>
<p>The data dump code is a poor cousin to the original code. To get to it from the extension, start it, then click on the button in the lower right corner labelled "Data dump instead". Then you'll see the options screen above.</p>
<p>Try it out when you get a chance. I've also updated the files on my web site. The attached .zip file includes the .mxt file Moneydance's extension installer needs. (I can't upload the .mxt file - the security system doesn't recognize it.)</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-02T18:32:51Z2021-10-02T18:32:51ZHow to export data from Net Worth chart?<div><p>Well, networth_matrix runs on Windows, but the .csv file has lines terminated with CR, CR, LF instead of CR,LF. Notepad and excel (from the glimpse Microsoft let me see), shows everything is double spaced. There's probably a config thing for Python's .csv library.</p>
<p>I'll go look, it's okay to try it out anyway.</p>
<p>A hex dump shows 0D 0D 0A line termination.</p>
<p>00070 31 5F 41 75 67 5F 31 35 2C 32 30 32 31 5F 53 65 <em>1_Aug_15,2021_Se</em><br>
00080 70 5F 31 35 0D 0D 0A 49 6E 76 65 73 74 6D 65 6E <em>p_15...Investmen</em></p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-02T22:19:11Z2021-10-02T22:19:11ZHow to export data from Net Worth chart?<div><p>Well, between looking into tweaking the options screen some, and discovering that .csv files are happier in Python 3, I told the csv code to write '\n' and Python expands that to '\r\n' on Windows.</p>
<p>I've updated my web site, but won't upload a new file here for a bit.</p>
<p>It looks like Excel sees fields like <code>+subaccount</code> and doesn't think its a string. I may need to change that leading plus-sign or simply not use it. I use it because it makes things more readable (see the text file from the normal output).</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-03T06:07:43Z2021-10-03T06:07:47ZHow to export data from Net Worth chart?<div><p>Oh my! You are super hero, Ric!</p>
<p>This updated extension works on my Windows (Didn't get to test Mac yet).</p>
<p>I quickly exported few days, I think that looks perfect, except one small bug:<br>
Your data seems 1 day behind MoneyDance. E.g. Your 1/10/2021 data is indeed 1/9/2021 data in MoneyDance. Don't know if that's related to timezone? I use PST/PDT time.</p>
<p>I will continue to test and keep you updated!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-03T06:50:29Z2021-10-03T06:50:29ZHow to export data from Net Worth chart?<div><p>Excel won’t like leading minus or plus.</p></div>Stuart Beesley (Mr Toolbox)tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-03T13:07:17Z2021-10-03T13:07:17ZHow to export data from Net Worth chart?<div><p>"Your data seems 1 day behind MoneyDance. E.g. Your 1/10/2021 data is indeed 1/9/2021 data in MoneyDance."</p>
<p>That must be a holdover from a kludge I did for the normal output. While convention says to report data as of the end of the month, I report data at the <em>start</em> of the month and can do that expeditiously with:</p>
<pre>
<code> for date in dates:
# "date - 1" is a cheap way to get values at the opening of the date
balance = AccountUtil.getBalanceAsOfDate(book, acct, date - 1)
values.append(CurrencyUtil.convertValue(balance, fromCurrencyType, toCurrencyType, date))</code>
</pre>
<p>Moneydance uses an internal date form that is a number. Today is 20211003. If "date" is 20211001, then one less, 20211000, is treated as Sep 30.</p>
<p>It may make the most sense do that hack whenever monthly data is requested and not adjust it if daily data is requested. Financial purists will be annoyed.</p>
<p>OTOH, the csv file always has the day of month, and there's a better way to handle the end of month.</p>
<p>There's one weird difference between getting csv data when run interactively vs. batch mode from the command line. Interactively, I use Java's SimpleDateFormat and Moneydance's DateField. In batch mode, I do a lame job of parsing the date myself, and treat the day part of mm/dd/yyyy as something to be reduced to fit. So a date 4/31/2021 will be limited to 4/30. If you dump out 13 months for a year, you can use a from date like 12/31/2019 and to date of 12/31/2020 and it will report the end of each month letting you compare the start of the year to the end.</p>
<p>The interactive code accepts 4/31/2021 but makes it be 5/1/2021 or makes 2/31/2021 be 3/3/2021.</p>
<p>I think I'll improve the syntax checking on the batch mode dates and copy that into the interactive mode, then only do my date-1 hack in normal mode. Inconsistent, but financial purists might accept it.</p>
<p>That suggests I better handle date formats like mm/dd/yyyy and dd/mm/yyyy and yyyy/mm/dd. Probably worth the effort.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-03T13:19:49Z2021-10-03T13:19:49ZHow to export data from Net Worth chart?<div><p>"Excel won’t like leading minus or plus."</p>
<p>Perhaps I'll tell Microsoft that LibreOffice Calc accepts it Just Fine. Or not.</p>
<p>Thanks for that csv_ric.py example. I'll try employing "QUOTE_NONNUMERIC" instead, and look into adding codecs.BOM_UTF8.</p>
<p>I get the feeling that Python's V2 csv support was written as something minimalist, then "improved" by someone who uses Excel, but not well.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-03T13:36:27Z2021-10-03T13:36:27ZHow to export data from Net Worth chart?<div><p>The +/- problem should go away with the quoting solution I supplied</p>
<p>👍</p></div>Stuart Beesley (Mr Toolbox)tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-04T19:00:55Z2021-10-04T19:00:55ZHow to export data from Net Worth chart?<div><p>Stuart,</p>
<p>The +/- issue is weirder than I thought. I used QUOTE_NONNUMERIC which quotes the subaccount names as I wanted. However, Excel appears to be treating strings like that as formulas and displays |#NAME| because it doesn't have matching functions. Sigh.</p>
<p>So I just changed the prefix from '+' to '.' . I probably should have just stopped using it.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-04T19:19:24Z2021-10-04T19:19:24ZHow to export data from Net Worth chart?<div><p>jerryxe,</p>
<p>I've uploaded new code to <a href="http://wermenh.com/moneydance/networth_matrix.mxt">http://wermenh.com/moneydance/networth_matrix.mxt</a> and <a href="http://wermenh.com/moneydance/networth_matrix.zip">http://wermenh.com/moneydance/networth_matrix.zip</a> . Can you access those okay?</p>
<p>That off-by-one date issue should be fixed, and the interactive UI for the CSV data is now using my own data parsing code. I'm also doing better date syntax checking than in the past, and it will gripe about errors before giving you a new (blank) form. Cute enough.</p>
<p>There are a couple things I should clean up in the command line interface, but that won't impact you.</p>
<p>Try it out when you get a chance, it may be "done."</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-04T20:27:15Z2021-10-04T20:27:19ZHow to export data from Net Worth chart?<div><p>Yes, just tested. It's perfect. date offset issued resolved.</p>
<p>Thank you so much, Ric!!!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-05T23:35:06Z2021-10-05T23:35:08ZHow to export data from Net Worth chart?<div><p>Hi Ric,</p>
<p>I found a small bug, and I don't understand why it happened.</p>
<p>So I first exported 7 years daily history, and then I exported only few weeks daily history (7/28/2021 - 9/10/2021 to be exact). I compared daily total net worth of these 2 dumps. I found discrepancy.</p>
<p>E.g. on 8/1/2021, the first dump gives a number that is significanly lower than second dump.</p>
<p>Comparing line by line, I found that I have a security (which does not have tickers, so I manually add price history every day in the past) that caused the issue.<br>
1. In the first dump (with 7 years history), it only shows I have value since 8/17/2021.<br>
2. But I actually first bought this security on 6/15/2021, and it should have price history since 6/15/2021 (may not daily, but at least weekly).<br>
3. In the second dump, it does show this security have value in every single day between 7/28/2021 and 9/10/2021.</p>
<p>So I don't understand why this security shows 0 value before 8/17/2021. I only bought this security once on 6/15/2021, and never sold it.</p>
<p>Let me know if you need additional info to de-bug. Thanks!</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-06T04:26:37Z2021-10-06T04:26:37ZHow to export data from Net Worth chart?<div><p>That's really weird. With luck, we might have found a Moneydance bug. (Well, lucky for me, perhaps.)</p>
<p>The heart of my data collection relies on a couple MD functions:</p>
<p>for each account:<br>
for each date of interest: balance = AccountUtil.getBalanceAsOfDate(book, acct, date) values.append(CurrencyUtil.convertValue(balance, fromCurrencyType, toCurrencyType, date))</p>
<p>BTW, this why the script is so slow - I think AccountUtil.getBalanceAsOfDate() scans all the transactions and accumulates the value from all those before and including the date. My original code tried to do all this in one pass, an optimization worth trying in version 2.</p>
<p>So, I can't explain what you're seeing.</p>
<p>Can you poke around with Moneydance's net worth tools and see if you can find a similar anomaly?</p>
<p>Also, from the summary page, select the security, and that should open a screen that has a "History" button. Clicking that will open your history of per share prices. I doubt that will provide any clues, but it's worth a check.</p>
<p>I have an old account for my IRA from 1980 or so that has the full history (of transactions!), and also some more recent accounts. I'll poke around with them when I can - the next few days may be glorious NH fall weather and there are grapes to pick. Some things take precedence. :-)</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-06T04:37:00Z2021-10-06T04:37:01ZHow to export data from Net Worth chart?<div><p>Thanks Ric. Since this issue happens when I dump different date range. I will also test different combinations to see what date range caused it.</p>
<p>Kinda busy this week, but will try to hunt it down.</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T05:44:11Z2021-10-07T05:44:14ZHow to export data from Net Worth chart?<div><p>Hi Ric,</p>
<p>While I am still investigating into it, I want to leave a note. I think the issue is broader than I thought.</p>
<p>I tried to dump multiple daily csv files with different date range, and then compare the total balance on each day, I found the discrepancy is quite often, and happens on different stocks (including the ones with tickers). E.g. in one dump, a stock that I have ~10K balance actually shows negative balance for some days, then suddenly shows correct balance.</p>
<p>There is nothing special in this stock's price history (by looking at the price history chart).</p>
<p>I will report back if I have more findings</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T06:36:28Z2021-10-07T06:36:35ZHow to export data from Net Worth chart?<div><p>Hope you enjoy the beautiful fall in NH.</p>
<p>Here is one more update. I noticed that the data dump constantly changes. I tried to download 3 months data (7/1/2021 - 9/30/2021) 5 times consecutively, without any changes to MD.</p>
<p>Then I imported into Google sheet, and compare their Total Net Worth day by day, below is the result (Dump 2-5 comparing to Dump 1). As you can see every time, the dump is different.<br>
- The numbers here are delta comparing to dump 1 - I hided dates that all 5 dump have the same numbers. - For 7/16, I guess dump 1 got wrong number, but dump 2-5 got the correct number. - For 7/20-7/24, dump 2 got wrong number - For 8/19, both dump 1 and 5 got wrong number.</p>
<p>I guess this is enough for you to test on your side?</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T09:51:07Z2021-10-07T09:51:07ZHow to export data from Net Worth chart?<div><p>Ric, a quick tip in case you haven't found it:</p>
<p><a href="https://infinitekind.com/dev/apidoc/com/infinitekind/util/DateUtil.html">https://infinitekind.com/dev/apidoc/com/infinitekind/util/DateUtil....</a></p>
<pre>
<code>from com.infinitekind.util import DateUtil
dateInt=DateUtil.getStrippedDateInt()
newDate = DateUtil.incrementDate(dateInt, 0, 0, -1)</code>
</pre>
<p>and many other methods to play with the date....</p></div>Stuart Beesley (Mr Toolbox)tag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T13:32:33Z2021-10-07T13:32:33ZHow to export data from Net Worth chart?<div><p>Jerry,</p>
<p>Thanks for that effort and clear description.</p>
<p>I tried restarting Moneydance, then dumped first 1/1/2021 to 4/1/2021 and then 1/1 to 7/1. I did each three times, and all three matched, e.g:</p>
<p>sl:matrix$ mv ~/Desktop/networth.csv dump.3<br>
sl:matrix$ cksum dump.*<br>
1430670731 48993 dump.1<br>
1430670731 48993 dump.2<br>
1430670731 48993 dump.3<br>
sl:matrix$ diff dump.1 dump.3<br>
sl:matrix$</p>
<p>1430670731 is a checksum, 48993 is the size of the file in bytes.</p>
<p>I'll think about it more after breakfast.</p>
<p>I have stocks and mutual funds with and without ticker symbols, but I update them once a month manually. Do you see problems with bank accounts, or credit cards?</p>
<p>The next step will probably be to include the account's balance in shares and the conversion rate used that day. After that, maybe try calling each routine twice and comparing the result then.</p>
<p>Are you downloading the .mxt file, or do you have to download the .zip file and extract the .mxt file?</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T13:54:58Z2021-10-07T13:54:58ZHow to export data from Net Worth chart?<div><p>Stuart,</p>
<p>I used DateUtil until my last update and dropped it because I preferred my approach to using "2/31" to denote the last day of February.</p>
<p>Documentation for DateUtil is poor, e.g. there's nothing for incrementMonth​(java.util.Date date) and I'd have to experiment to see what happens if I call that once or twice for 1/30.</p>
<p>Besides, DateUtil uses <a href="https://docs.oracle.com/javase/8/docs/api/java/util/Date.html">https://docs.oracle.com/javase/8/docs/api/java/util/Date.html</a> and most of that is deprecated.</p>
<p>Moneydance dates are simple enough that it's easy to roll my own. Look at my get_csv_datestr(), that controls the daily/monthly incrementing for the .csv dumps.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-07T17:13:24Z2021-10-07T17:13:26ZHow to export data from Net Worth chart?<div><p>Hi Ric,</p>
<p>I re-tested and followed a cleaner path:<br>
1. Installed MD on both Windows and Mac, and update to 3069 the latest non-beta version.<br>
2. Used Dropbox Sync and wait until the sync is done.<br>
3. Remove old networth extension.<br>
4. Restart MD<br>
5. Reinstall the networth extension on #27 of this thread (The .mxt one)<br>
6. Dump 3 months daily data 3 times (7/1/2021 - 9/30/2021) on both Mac and Win<br>
7. Use cksum on Mac and Hash on Win to check the 3 dumps.<br>
8. Looks like all the 3 are different in each platform.</p>
<p>Below is from windows.<br>
PS C:\Users\Jerry\Downloads> Get-FileHash networth*</p>
<p>Algorithm Hash Path</p>
<hr>
<p>SHA256 8CA518FDDF726DBB84261FADED579A00185A476682DBE5D621094E345C95BF02 C:\Users\Jerry\Downloads\networth1.csv<br>
SHA256 0316E6A2D9236AB28CB65FE2FD54B1D804F212AA886F5755C7ACE49E92FD2489 C:\Users\Jerry\Downloads\networth2.csv<br>
SHA256 D43F54DDFA52CBC790832A57D1771B6855FB738D128C5A95854624D44DB04A9A C:\Users\Jerry\Downloads\networth3.csv</p>
<p>Below is from Mac:<br>
​jerry@Jerrys-MBP NetWorthTest % cksum networth* 1552848269 141777 networth1.csv<br>
185114022 141762 networth2.csv<br>
1758807976 141783 networth3.csv</p>
<p>I checked the date range in all the files, they are correct</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-08T17:12:18Z2021-10-08T17:12:18ZHow to export data from Net Worth chart?<div><p>Jerry,</p>
<p>I'm going to make a test version that will include the currency/share balance and value with each amount to see which of those is at fault (this presumes my code is right) and also the time to generate data for each account.</p>
<p>Some people think computer software always does the same thing, but software engineers rarely use absolute terms. When that doesn't happen even though the code looks right, eventually I start worrying about scary things with the operating system's virtual memory support, Java's garbage collection, or interlocking issues within Moneydance. OTOH, I'd expect symptoms more like Moneydance crashing.</p>
<p>That you see problems on both Windows and MacOS means it's probably not a VM problem, that's the least likely situation any, that code gets too well exercised. Ditto for Java's garbage collection.</p>
<p>I was going to start writing the new code, but it's too nice a day, I'm going for a bicycle ride instead. I might have something tonight, or at least have some questions about memory consumption.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-10T13:04:06Z2021-10-10T13:04:06ZHow to export data from Net Worth chart?<div><p>Is it tonight yet?</p>
<p>Okay, I've uploaded both a new .mxt file <a href="http://wermenh.com/moneydance/networth_matrix.mxt">http://wermenh.com/moneydance/networth_matrix.mxt</a> and the program source code <a href="http://wermenh.com/moneydance/networth_matrix.py">http://wermenh.com/moneydance/networth_matrix.py</a> .</p>
<p>The .csv results are pretty ugly, but now each date has three columns:</p>
<pre>
<code>Balance * Rate = 2021_Jan_30 Balance * Rate = 2021_Jan_31
6600000 0.6469 42695.40 6600000 0.6043 39883.80</code>
</pre>
<p>The Balance (shares/currency scaled to be an integer) times a conversion rate (in this case a share price like $64.69 $/sh) equals the real value in $. I don't know if the repeating column names will annoy Excel, they don't bother LibreCalc.</p>
<p>Also, at the end of each account line is the number of milliseconds of "wall clock" time it took to extract all the data. I'm really interested in seeing if there are spikes on the lines that have errors. However, that means we can't look for changes in checksums of the file.</p>
<p>Looking for glitches in the subtotal line might work for eyeballing, perhaps your comparison above will still work.</p>
<p>You can suppress the msec column by using the .py file and deleting the 'm' in:</p>
<pre>
<code>debug = 'fm'</code>
</pre>
<p>Then run that file via the MoneyBot console Window, yell if you need a walk-through.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-10T13:21:35Z2021-10-10T13:21:35ZHow to export data from Net Worth chart?<div><p>A couple other things:</p>
<p>Virtual memory stuff seems uninteresting, at least on my Linux system.</p>
<p>From a typical run:</p>
<pre>
<code>$ ps xl | head -1
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
Start MD:
$ ps xl | grep '[j]ava' | cut -c 1-80
0 1001 18672 2669 20 0 6321512 532276 futex_ Sl ? 0:22 /opt/Moneyd
Start networth_matrix:
0 1001 18275 2669 20 0 6396284 598156 futex_ Sl ? 0:25 /opt/Moneyd
After dumping 7/1/2021 - today:
0 1001 18672 2669 20 0 6331768 601388 futex_ Sl ? 0:56 /opt/Moneyd
VSZ is the virtual memory used, the amount Java may want to use. RSZ is the real memory that's allocated to Java, all in 1 KB pages. 6 GB VSZ, 600 MB RSZ, huge my some standards, but my system (16 GB) has plenty of space.
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
1 0 244500 2279268 359584 5432708 0 0 1 22 1 1 2 0 98 0 0
$ vmstat -a
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free inact active si so bi bo in cs us sy id wa st
1 0 244500 2278956 2664944 9939972 0 0 1 22 1 1 2 0 98 0 0</code>
</pre>
<p>MacOS has similar but different ps and vmstat commands. On Windows, I watch the Resource Monitor app.</p>
<p>The other thing is I'll be at a conference for part of next week, I'd like to do a fair amount of reading beforehand, so I won't have much time for this puzzle until afterwards.</p></div>Ric Wermetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-11T04:37:47Z2021-10-11T04:38:40ZHow to export data from Net Worth chart?<div><p>Thank you Ric! I was able to update to your latest extension, and did a comparison line by line, and I actually find some clue.</p>
<p>On 2 dumps on 7/9 data, there is one line does not match, which is related to a stock ICLN. +/- 7 days of 7/9 actually match.</p>
<p>Dump 1 shows (Balance * Rate = 2021_Jul_09)<br>
7037950 0.2322 16342.12</p>
<p>Dump 2 shows (Balance * Rate = 2021_Jul_09)<br>
-2500000 0.2322 -5805</p>
<p>I checked the full transactions history of ICLN:<br>
02/18/2021, buy ICLN, 400 shares, total $11162.00, per share price 27.905<br>
02/22/2021, buy ICLN, 300 shares, total $8109.90, per share price 27.033<br>
06/16/2021, DivReinvest ICLN, 3/795 shares, total $86.95, per share price 22.91172596<br>
09/15/2021, buy ICLN, 250 shares, total $5628.75, per share price 22.515</p>
<p>Looking at dump 2's number on 7/9, the balance should be 7037950, but it shows a negative number of -2500000, this number conincides with the 9/15/2021 transaction. (Though 9/15 is a future date and it's buy, not sell)</p></div>jerryxetag:infinitekind.tenderapp.com,2009-01-14:Comment/493600992021-10-11T04:47:57Z2021-10-11T04:48:28ZHow to export data from Net Worth chart?<div><p>Another example of discrepancy. On 7/20, another stock HD.</p>
<p>Dump 1 shows (Balance * Rate = 2021_Jul_20)<br>
-5440 3.253492 -176.99</p>
<p>Dump 2 shows (Balance * Rate = 2021_Jul_20)<br>
1106020 3.253499 35984.36</p>
<p>Here is my full HD history on the same account:<br>
03/14/2021, buy HD, 20 shares, total 4960.97<br>
03/25/2021, Div (Cash), total 33<br>
05/11/2021, buy HD, 15 shares, total 4942.80<br>
05/11/2021, buy HD, 15 shares, total 4953.11<br>
05/11/2021, buy HD, 15 shares, total 9892.68<br>
05/12/2021, buy HD, 15 shares, total 4758.83<br>
05/12/2021, buy HD, 15 shares, total 4777.28<br>
06/17/2021, DivReinvest, 0.602 shares, total 181.5<br>
09/16/2021, DivReinvest, 0.544 shares, total 182.49</p>
<p>Dump 1's balance -5440 suspiciously conincides with the 9/16 DivReinvest, though 9/16 should be future date.<br>
Dump 2's 1106020 is correct</p></div>jerryxe