Home → General Questions →

How to export data from Net Worth chart?

• Edit

jerryxe

26 Aug, 2021 09:55 PM

Hi,

I want to export the daily net worth data from the net worth chart, but looks like the "Copy to Clipboard" button only saves the chart itself, not the backend daily data. Anyone knows how can I get it? Thanks!

(I have attached the screenshot of the chart, but don't read too much on the chart, as the Y-axis is not indexed to 0.)

Jerry

• Screen_Shot_2021-08-26_at_2.54.15_PM.png 201 KB

Showing page 2 out of 2. View the first page

1. 31 Posted by jerryxe on 06 Oct, 2021 04:37 AM

   

   Thanks Ric. Since this issue happens when I dump different date range. I will also test different combinations to see what date range caused it.

   Kinda busy this week, but will try to hunt it down.

   • Edit

2. 32 Posted by jerryxe on 07 Oct, 2021 05:44 AM

   

   Hi Ric,

   While I am still investigating into it, I want to leave a note. I think the issue is broader than I thought.

   I tried to dump multiple daily csv files with different date range, and then compare the total balance on each day, I found the discrepancy is quite often, and happens on different stocks (including the ones with tickers). E.g. in one dump, a stock that I have ~10K balance actually shows negative balance for some days, then suddenly shows correct balance.

   There is nothing special in this stock's price history (by looking at the price history chart).

   I will report back if I have more findings

   • Edit

3. 33 Posted by jerryxe on 07 Oct, 2021 06:36 AM

   

   Hope you enjoy the beautiful fall in NH.

   Here is one more update. I noticed that the data dump constantly changes. I tried to download 3 months data (7/1/2021 - 9/30/2021) 5 times consecutively, without any changes to MD.

   Then I imported into Google sheet, and compare their Total Net Worth day by day, below is the result (Dump 2-5 comparing to Dump 1). As you can see every time, the dump is different.
   - The numbers here are delta comparing to dump 1
   - I hided dates that all 5 dump have the same numbers.
   - For 7/16, I guess dump 1 got wrong number, but dump 2-5 got the correct number.
   - For 7/20-7/24, dump 2 got wrong number
   - For 8/19, both dump 1 and 5 got wrong number.

   I guess this is enough for you to test on your side?

   • Screen_Shot_2021-10-06_at_11.31.05_PM.png 190 KB
   • Edit

4. 34 Posted by Stuart Beesley ... on 07 Oct, 2021 09:51 AM

   

   Ric, a quick tip in case you haven't found it:

   https://infinitekind.com/dev/apidoc/com/infinitekind/util/DateUtil....

   from com.infinitekind.util import DateUtil
   dateInt=DateUtil.getStrippedDateInt()
   newDate = DateUtil.incrementDate(dateInt, 0, 0, -1)
   

   and many other methods to play with the date....

   • Edit

5. 35 Posted by Ric Werme on 07 Oct, 2021 01:32 PM

   

   Jerry,

   Thanks for that effort and clear description.

   I tried restarting Moneydance, then dumped first 1/1/2021 to 4/1/2021 and then 1/1 to 7/1. I did each three times, and all three matched, e.g:

   sl:matrix$ mv ~/Desktop/networth.csv dump.3
   sl:matrix$ cksum dump.*
   1430670731 48993 dump.1
   1430670731 48993 dump.2
   1430670731 48993 dump.3
   sl:matrix$ diff dump.1 dump.3
   sl:matrix$

   1430670731 is a checksum, 48993 is the size of the file in bytes.

   I'll think about it more after breakfast.

   I have stocks and mutual funds with and without ticker symbols, but I update them once a month manually. Do you see problems with bank accounts, or credit cards?

   The next step will probably be to include the account's balance in shares and the conversion rate used that day. After that, maybe try calling each routine twice and comparing the result then.

   Are you downloading the .mxt file, or do you have to download the .zip file and extract the .mxt file?

   • Edit

6. 36 Posted by Ric Werme on 07 Oct, 2021 01:54 PM

   

   Stuart,

   I used DateUtil until my last update and dropped it because I preferred my approach to using "2/31" to denote the last day of February.

   Documentation for DateUtil is poor, e.g. there's nothing for incrementMonth​(java.util.Date date) and I'd have to experiment to see what happens if I call that once or twice for 1/30.

   Besides, DateUtil uses https://docs.oracle.com/javase/8/docs/api/java/util/Date.html and most of that is deprecated.

   Moneydance dates are simple enough that it's easy to roll my own. Look at my get_csv_datestr(), that controls the daily/monthly incrementing for the .csv dumps.

   • Edit

7. 37 Posted by jerryxe on 07 Oct, 2021 05:13 PM

   

   Hi Ric,

   I re-tested and followed a cleaner path:
   1. Installed MD on both Windows and Mac, and update to 3069 the latest non-beta version.
   2. Used Dropbox Sync and wait until the sync is done.
   3. Remove old networth extension.
   4. Restart MD
   5. Reinstall the networth extension on #27 of this thread (The .mxt one)
   6. Dump 3 months daily data 3 times (7/1/2021 - 9/30/2021) on both Mac and Win
   7. Use cksum on Mac and Hash on Win to check the 3 dumps.
   8. Looks like all the 3 are different in each platform.

   Below is from windows.
   PS C:\Users\Jerry\Downloads> Get-FileHash networth*

   Algorithm Hash Path
   --------- ---- ----
   SHA256 8CA518FDDF726DBB84261FADED579A00185A476682DBE5D621094E345C95BF02 C:\Users\Jerry\Downloads\networth1.csv
   SHA256 0316E6A2D9236AB28CB65FE2FD54B1D804F212AA886F5755C7ACE49E92FD2489 C:\Users\Jerry\Downloads\networth2.csv
   SHA256 D43F54DDFA52CBC790832A57D1771B6855FB738D128C5A95854624D44DB04A9A C:\Users\Jerry\Downloads\networth3.csv

   Below is from Mac:
   ​jerry@Jerrys-MBP NetWorthTest % cksum networth*
   1552848269 141777 networth1.csv
   185114022 141762 networth2.csv
   1758807976 141783 networth3.csv

   I checked the date range in all the files, they are correct

   • Edit

8. 38 Posted by Ric Werme on 08 Oct, 2021 05:12 PM

   

   Jerry,

   I'm going to make a test version that will include the currency/share balance and value with each amount to see which of those is at fault (this presumes my code is right) and also the time to generate data for each account.

   Some people think computer software always does the same thing, but software engineers rarely use absolute terms. When that doesn't happen even though the code looks right, eventually I start worrying about scary things with the operating system's virtual memory support, Java's garbage collection, or interlocking issues within Moneydance. OTOH, I'd expect symptoms more like Moneydance crashing.

   That you see problems on both Windows and MacOS means it's probably not a VM problem, that's the least likely situation any, that code gets too well exercised. Ditto for Java's garbage collection.

   I was going to start writing the new code, but it's too nice a day, I'm going for a bicycle ride instead. I might have something tonight, or at least have some questions about memory consumption.

   • Edit

9. 39 Posted by Ric Werme on 10 Oct, 2021 01:04 PM

   

   Is it tonight yet?

   Okay, I've uploaded both a new .mxt file http://wermenh.com/moneydance/networth_matrix.mxt and the program source code http://wermenh.com/moneydance/networth_matrix.py .

   The .csv results are pretty ugly, but now each date has three columns:

   Balance *  Rate =  2021_Jan_30  Balance *  Rate =  2021_Jan_31
     6600000  0.6469     42695.40    6600000  0.6043     39883.80
   

   The Balance (shares/currency scaled to be an integer) times a conversion rate (in this case a share price like $64.69 $/sh) equals the real value in $. I don't know if the repeating column names will annoy Excel, they don't bother LibreCalc.

   Also, at the end of each account line is the number of milliseconds of "wall clock" time it took to extract all the data. I'm really interested in seeing if there are spikes on the lines that have errors. However, that means we can't look for changes in checksums of the file.

   Looking for glitches in the subtotal line might work for eyeballing, perhaps your comparison above will still work.

   You can suppress the msec column by using the .py file and deleting the 'm' in:

   debug = 'fm'
   

   Then run that file via the MoneyBot console Window, yell if you need a walk-through.

   • Edit

10. 40 Posted by Ric Werme on 10 Oct, 2021 01:21 PM

    

    A couple other things:

    Virtual memory stuff seems uninteresting, at least on my Linux system.

    From a typical run:

    $ ps xl | head -1
    F   UID   PID  PPID PRI  NI    VSZ   RSS WCHAN  STAT TTY        TIME COMMAND
    Start MD:
    $ ps xl | grep '[j]ava' | cut -c 1-80
    0  1001 18672  2669  20   0 6321512 532276 futex_ Sl ?          0:22 /opt/Moneyd
    
    Start networth_matrix:
    0  1001 18275  2669  20   0 6396284 598156 futex_ Sl ?          0:25 /opt/Moneyd
    
    After dumping 7/1/2021 - today:
    0  1001 18672  2669  20   0 6331768 601388 futex_ Sl ?          0:56 /opt/Moneyd
    
    VSZ is the virtual memory used, the amount Java may want to use.  RSZ is the real memory that's allocated to Java, all in 1 KB pages.  6 GB VSZ, 600 MB RSZ, huge my some standards, but my system (16 GB) has plenty of space.
    
    $ vmstat
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
     1  0 244500 2279268 359584 5432708    0    0     1    22    1    1  2  0 98  0  0
    
    $ vmstat -a
    procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
     r  b   swpd   free  inact active   si   so    bi    bo   in   cs us sy id wa st
     1  0 244500 2278956 2664944 9939972    0    0     1    22    1    1  2  0 98  0  0
    

    MacOS has similar but different ps and vmstat commands. On Windows, I watch the Resource Monitor app.

    The other thing is I'll be at a conference for part of next week, I'd like to do a fair amount of reading beforehand, so I won't have much time for this puzzle until afterwards.

    • Edit

11. 41 Posted by jerryxe on 11 Oct, 2021 04:37 AM

    

    Thank you Ric! I was able to update to your latest extension, and did a comparison line by line, and I actually find some clue.

    On 2 dumps on 7/9 data, there is one line does not match, which is related to a stock ICLN. +/- 7 days of 7/9 actually match.

    Dump 1 shows (Balance * Rate = 2021_Jul_09)
    7037950 0.2322 16342.12

    Dump 2 shows (Balance * Rate = 2021_Jul_09)
    -2500000 0.2322 -5805

    I checked the full transactions history of ICLN:
    02/18/2021, buy ICLN, 400 shares, total $11162.00, per share price 27.905
    02/22/2021, buy ICLN, 300 shares, total $8109.90, per share price 27.033
    06/16/2021, DivReinvest ICLN, 3/795 shares, total $86.95, per share price 22.91172596
    09/15/2021, buy ICLN, 250 shares, total $5628.75, per share price 22.515

    Looking at dump 2's number on 7/9, the balance should be 7037950, but it shows a negative number of -2500000, this number conincides with the 9/15/2021 transaction. (Though 9/15 is a future date and it's buy, not sell)

    • Edit

12. 42 Posted by jerryxe on 11 Oct, 2021 04:47 AM

    

    Another example of discrepancy. On 7/20, another stock HD.

    Dump 1 shows (Balance * Rate = 2021_Jul_20)
    -5440 3.253492 -176.99

    Dump 2 shows (Balance * Rate = 2021_Jul_20)
    1106020 3.253499 35984.36

    Here is my full HD history on the same account:
    03/14/2021, buy HD, 20 shares, total 4960.97
    03/25/2021, Div (Cash), total 33
    05/11/2021, buy HD, 15 shares, total 4942.80
    05/11/2021, buy HD, 15 shares, total 4953.11
    05/11/2021, buy HD, 15 shares, total 9892.68
    05/12/2021, buy HD, 15 shares, total 4758.83
    05/12/2021, buy HD, 15 shares, total 4777.28
    06/17/2021, DivReinvest, 0.602 shares, total 181.5
    09/16/2021, DivReinvest, 0.544 shares, total 182.49

    Dump 1's balance -5440 suspiciously conincides with the 9/16 DivReinvest, though 9/16 should be future date.
    Dump 2's 1106020 is correct

    • Edit

13. System closed this discussion on 10 Jan, 2022 04:50 AM.