Schwab downloads work, but merging doesn't

v+ik's Avatar

v+ik

11 Jun, 2021 06:18 AM

After trying to do manual entry for Schwab investment accounts, I was glad to upgrade to the new version that does it automatically. But then I discovered that confirming investment transactions doesn't allow merging. Found a thread that talks about ticket #4694 but don't know how to find out what is in there, or what the current status is.

The first download grabbed years of old stuff, unconfirmed, which, since I couldn't merge, I had to delete. Is that stuff going to keep showing up in future downloads? Or will stuff older than my first download be ignored in the future?

  1. 1 Posted by dtd on 11 Jun, 2021 08:41 PM

    dtd's Avatar

    just a user

    Unfortunately, investment transactions have never allowed merging in MD. There have been requests for that to be added, but that request has been around for some time.

    Schwab normally downloads 18 months of data, but I do think it jumped to 4 years recently. Here's a post of my own:

    https://infinitekind.tenderapp.com/discussions/investments/5971-info-schwab-now-downloads-up-to-48-months-4-years-vs-18-months-15-years

    To go with your second paragraph, you need to keep the NEW downloaded transactions (yes it is 4 years maybe - SIGH) as there is a transaction id to tell Schwab not to download again in the new items, but not in the manually entered ones. (If you could merge, it would be fixed).

    As an alternative, there is a "last transaction date downloaded" field, which within MD can only be reset to the beginning of time. If you are using 2021.1(3069), there is an extension called "Toolbox" (by user Stuart Beesley) which allows you to set it to any date, and then Schwab will only download from that date you set forward. That sounds like what you need.

  2. 2 Posted by v+ik on 11 Jun, 2021 09:50 PM

    v+ik's Avatar

    Interesting. Well, I mostly deleted the downloaded transactions, because they didn't correspond to transfers between accounts, whereas the ones imported from Quicken did have that correct, and were confirmed in other accounts.

    From what you say, it seems like I need to use Toolbox to set the "last transaction date downloaded" field for each of my Schwab accounts, to avoid ređownloading those old transactions again.... but why wouldn't MD update that date (hopefully date/time) on each download, to avoid re-downloading?

    Probably for 3 of the accounts, I can just accept the downloaded transactions as-is, in the future. Except that IIRC the downloaded transactions didn't have the right Categories, so I guess that would have to be fixed manually (would that be done before, during, or after the confirmation process?)

    But for the other one, which includes transfers to and from various other accounts, I'm not sure what to do... I had been entering some of that transaction data as part of updating other accounts, some by reminder transactions, but without merge capability, I'm not sure how to handle it. On the days of such transactions, would I downnload the transaction from Schwab first, and then "fix" the transaction Category to be the transfer account?

  3. 3 Posted by dtd on 11 Jun, 2021 10:13 PM

    dtd's Avatar

    just a user

    I have no idea about why MD doesn't necessarily update the last transaction date downloaded field - because they do - at least when investigating what I do day to day (using Toolbox to check). But I've observed that if I download and then delete old transactions on an account 'set to zero' - they download again and again if I delete the new transactions until I accept them (pre-toolbox).

    You'd have to get Stuart (or maybe dwg) to answer the above more definitively. I'm just an observer.

    Going to your third paragraph, USUALLY I can just accept downloaded transactions as is (including category), but not all action groups are accepted by Moneydance (now we are into dwg's expertise). As a simple example, a dividend followed by a rebuy of shares - I have to change the category of the div action to "dividend income" category, but the rebuy doesn't need a category since it goes into the security value.

    As to your fourth paragraph, I do this, so I let the download control the situation, vs. trying to do manual entries (which, with automatic downloads working, seems fine)
    "On the days of such transactions, would I download the transaction from Schwab first, and then "fix" the transaction Category to be the transfer account?"

  4. 4 Posted by dtd on 11 Jun, 2021 10:23 PM

    dtd's Avatar

    missed one question sorry...

    I guess that would have to be fixed manually (would that be done before, during, or after the confirmation process?)

    Happily, it can be fixed at any point.

  5. 5 Posted by dtd on 11 Jun, 2021 10:28 PM

    dtd's Avatar

    To continue, since you obviously get it, and even could write an extension (I used to be able to do that, in an earlier life).... a couple more bits for you.

    1) There is a request in to change "Reset Sync" to not be only "reset to zero", but to allow a date set. (similar to Stuart's toolbox)

    2) dwg has requested that more Action types be allowed for investment accounts, especially when they mirror supported Action types.

    3) The "allow merge" in investment accounts is a long term request, but the answer tends to be "investment accounts are quite different from other account types", so we will see.

    4) Once one accepts the limitations, the process can and does work, but could be better.

  6. 6 Posted by dwg on 11 Jun, 2021 10:57 PM

    dwg's Avatar

    I'm a fellow user.

    As far as categories are concerned the OFX specification has no provision for category information, so when you see a category suggested it is Moneydance suggesting it based on past transactions.

    While QIF does have support for categories, it is uncommon for FIs to use it as what they think and what you might have are usually two totally different things.

    For transfers OFX does have provision to handle these.

    In Moneydance Categories and transfers are handled in the one field, this is pretty normal in the Personal finance field an you also see it reflected in the QIF format, which may have influenced some vendors.

    In Moneydance it makes sense in that since the software follows accounting rules accounts and categories are really just different instances of the same type of structure. The differences are in the way you normally interact with each and in reporting.

  7. 7 Posted by v+ik on 12 Jun, 2021 03:01 AM

    v+ik's Avatar

    Interesting. So (Stuart's download page)[https://yogi1967.github.io/MoneydancePythonScripts/] claims his extensions are now signed and available from Moneydance, but Toolbox isn't listed when I look, although some of his other extensions are. Then it claims a link to version 1040 of Toolbox, hosted at Moneydance, but when downloaded and installed, it seems to be version 1038, and tells me version 1040 is available from his github site.

    Looks like Stuart's version control isn't quite as robust as it could be...

    But, version 1038 successfully tells me that my Schwab accounts have download dates of 2021/06/11 12:00:00 PDT, which is a little surprising, as I did the downloads yesterday (6/10 well before midnight PDT) But then, I don't know exactly what the date means... the time of the download? The time the data at the bank was last updated? The time the data at the bank will next be updated? So I downloaded for one account, there were no new transafctions downloaded, and the reported date didn't change.

    Several other (BANK and ASSET) accounts say "Never downloaded" and others "IS SET TO ZERO", both have the comment "Download all available dates", but I'm not sure what the distinction is.

    And a couple other CREDIT CARD accounts for which downloades are set up show dates which probably reflect the last time I attempted a download, but I'll pay closer attention now that I can.

    A couple other BANK accounts and one CREDIT CARD show downloads, but the reality is that no downloads were performed, so I wonder if it is reporting the last time I imported from downloaded QFX Peeking in the saved the QFX files, it seems likely that the date was taken from there, possibly from the tag.

  8. 8 Posted by dtd on 12 Jun, 2021 05:21 AM

    dtd's Avatar

    just a user - thanks to Stuart, we have the information you refer to - before toolbox, it was all a mystery. As for version control - Stuart is working on it with MD, takes time, I'd guess.

    Your questions seem legitimate, I think Stuart will welcome the questions.

  9. 9 Posted by Stuart Beesley ... on 12 Jun, 2021 06:25 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    @v+ik - some info regarding version control etc:

    I was alerted on 31st May that my Extensions were all going into the internal manage extensions list… Well they all made it except Toolbox.. Sean asked for a couple of tweaks and then he would add that too. I made the tweaks and then adjusted my web site accordingly.. Unfortunately, it’s been silent since then… So yes I made the error of updating my site assuming the code would be pushed that day, but it wasn’t…. Given your comments, I have reverted the webpage.. To be clear:

    - All my extensions except Toolbox are now within Manage Extensions. My page doesn’t claim all, it states 'As of 1st June 2021 - Most extensions listed below are listed within the Moneydance Manage Extensions menu - Check there first for updates'
    - However, all my extensions are signed by IK and all signed versions can be downloaded here: https://infinitekind.com/app/md/extensions/ <https://infinitekind.com/app/md/extensions/>
    - The latest versions of all my extensions will always be here: https://yogi1967.github.io/MoneydancePythonScripts/ <https://yogi1967.github.io/MoneydancePythonScripts/> This is where I push them to. The latest may be newer than hosted by IK and therefore unsigned.
    - There is always a delay between requesting signing and getting signing. Nothing I can do about this. If you always and only want signed, then always use Manage Extensions or grab from the IK server (link above).

    Regarding Toolbox:
    - My own page is hosting the latest 1040
    - You are correct that IK is hosting 1038 signed… The delta being the small tweaks I made to get it released into Manage Extensions, which hasn’t actually happened yet. I have reverted the web page text on this whilst I am awaiting IK - as you are right….

    NOTE: if you peak at: https://infinitekind.com/app/md/extensions.dct <https://infinitekind.com/app/md/extensions.dct> and scroll to the end, you will see Toolbox is actually there (in the Manage Extensions list) but hidden awaiting my change (done) and for IK to release...

    Regards, SCB

  10. 10 Posted by Stuart Beesley ... on 12 Jun, 2021 06:33 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    Regarding the last download date, some banks don't observe the last download date option.. You can sometimes see this in the connection profile (using Toolbox view dynamic service profiles - or view all OFX data etc). I also suspect if you get an error downloading, then the date doesn't update. When banks don't use this they either send all, or they maintain their own internal 'date'.

    When it's not set, it means the account is not yet using downloads
    When it's zero (but with no downloads), it simply means you have viewed the account in MD and when you do this MD primes it with a zero - even if no downloads...

  11. 11 Posted by v+ik on 13 Jun, 2021 03:44 AM

    v+ik's Avatar

    Thanks for all the clarifications, Stuart. The version stuff was a bit confusing as a new user of Toolbox, but I understand how coordinating with someone else can lead to discrepancies.

    There's a lot more inside Toolbox than I have looked at yet, very interesting extension. Thanks for that.

    I was surprised there were accounts thatI hadn't been "viewed the account in MD", but then, looking at the ones so marked, they are all accounts with simple, few, and/or old transactions that imported directly from Quicken with the proper balance (and therefore, assumed to have all proper transactions as well), so I probably didn't look at them... at least, not after the exporting and re-importing using MD only, to shrink the file size.

  12. 12 Posted by v+ik on 13 Jun, 2021 04:40 AM

    v+ik's Avatar

    And thanks, dtd, but helping me understand how to deal with the lack of merging. A couple transactions have happened since, and letting them download, and then adjusting them and confirming them works OK, and probably, in my cases, isn't particularly more cumbersome than doing the reminder entries, and then merging later after downloading (my Quicken workflow). Just different, although if the transactions were more complex than a simple one-destination transfer, entering a split each time would be very annoying.

  13. 13 Posted by dtd on 13 Jun, 2021 04:58 AM

    dtd's Avatar

    That's basically what I do for investment accounts, I do not do manual entry for investments, I let the online download work, and then since MD doesn't necessarily support all actions, I adjust them, then confirm them.

    Sounds like we are on the same page here.

  14. 14 Posted by Stuart Beesley ... on 15 Jun, 2021 05:11 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    FYI. Toolbox (1040) is now signed and available from Manage Extensions….

  15. 15 Posted by v+ik on 19 Jun, 2021 03:17 AM

    v+ik's Avatar

    Well, more time has passed, and downloaded from Schwab again.

    1 account downloaded just the new transaction since a week ago.
    3 accounts downloaded 4 years worth of transactions.

    The "last downloaded date" for all accounts was one week ago.

    I know dtd warned me about this situation. I was really hoping not to have to fix 4 years worth of proper and confirmed transactions. I deleted the unconfirmed ones again (except the new ones from this week). So I put off the fixing another week.

    But repeatedly deleting 4 years worth of transactions will probably make my database balloon in size, as well as being annoying to do for the next 4 years?

  16. 16 Posted by v+ik on 19 Jun, 2021 03:27 AM

    v+ik's Avatar

    Transactions older than "last download date" should be ignored, not re-inserted.

    Probably there is no hook in the download process to allow looking at the transactions before they get applied?

  17. 17 Posted by dtd on 19 Jun, 2021 03:45 AM

    dtd's Avatar

    @v+ik - just me again

    What this suggests to me is that Schwab is ignoring the "last downloaded date" field (unfortunately, some fi's (financial institutions) do that), in favor of the FITID field.

    I absolutely have no idea if I'm right, but you'll notice in that thread I mentioned about 4 years, I "blithely" just took the new entries (since for me it was about 1 year, i.e. year 0-1 of 4, as the other 3 years had the correct FITID), so I never explored "last downloaded date", as MD had reset it to zero.

    In doing so, I took the new transactions, and moved on to a different problem (I think it was sell allocations).

    So, IF I am right (and people like Stuart could help you analyze the downloads to see if I am wrong), then you need to accept the new transactions with the acceptable FITID. Sigh - I don't like it either, as is obvious in my thread posts.

    But, feel free to get more advice, mine is based on observation versus analysis.

    As to this statement:
    But repeatedly deleting 4 years worth of transactions will probably make my database balloon in size, as well as being annoying to do for the next 4 years?

    *Again, assuming I am correct in my observations...
    You would need to bite the bullet and accept the 4 years worth of transactions and since there is no merge function, make the new downloads match the originals via edit (i.e. action items and memos/notes) and delete the originals.

    You'll note I did that myself (although i only had 1 year to deal with out of the 4), and it took a bit less than an hour to do - because since it was history, I didn't worry too much about notes or memos except in things like IRA notes, and I'd already dealt with most action item issues (another story about me moving to MD).

    Yes I know with 4 years, and three accounts, that could be close to 12 hours - I don't have a clue how active those accounts of yours are. Hope not terribly busy. If 12 hours - yes, see if there are alternatives to my brute force suggestion.

    Hope that informs (it certainly can't "help")

  18. 18 Posted by dtd on 19 Jun, 2021 03:49 AM

    dtd's Avatar

    re -
    Transactions older than "last download date" should be ignored, not re-inserted.

    Probably there is no hook in the download process to allow looking at the transactions before they get applied?

    The whole point is the FITID field - if it is unique, and sent despite the last downloaded date field - for MD to ignore the downloaded transactions would suggest data manipulation or ignoring of the OFX data - the fi says you NEED this information, and I have had a couple of cases where an fi creates an "older" transaction to fix something - and indeed sends it since it is "new" despite the last downloaded date (and this is for an fi that usually "listens" to that field)

    Again, this is observation based, not analysis based.

  19. 19 Posted by dtd on 19 Jun, 2021 04:09 AM

    dtd's Avatar

    just a user -
    one last thought, and one I'd never do except twice when I HAD to (the fi downloaded the same transaction with two different FITIDs) - I kept both, (otherwise Id get both over and over and over) and zeroed out one of them, and just lived with it. Well, at least for six months (which is how long the fi sent data vs. 4 years)

    After the download from the FI expired, I deleted the zeroed out transaction as it bothered my sensibilities.

    Not a suggestion, just a reality I dealt with.

  20. 20 Posted by v+ik on 19 Jun, 2021 04:56 AM

    v+ik's Avatar

    I was surprised that only 3 of the 4 accounts downloaded 4 years of transactions; the 4th only downloaded the one new transaction.

    My opinion is that if a fi adds a back-dated transaction, I would far rather reconcile that manually based on a statement, than to download old transactions over and over.

    I respect your experience, and descriptions of how you dealt with it, but I do think that MD should at least provide the option of ignoring transactions older than the last download date, as a user option at least, if not by default. It just makes more sense than FITIDs for really ancient transactions.

    A compromise might be to depend on FITIDs for some time span, such as 30 or 60 days, but simply ignore older ones.

    Searching on FiTID I found this comment, which is describes an interesting technique, which could also be useful if implemented:

    QuickBooks keeps track of the transaction ids that have been imported and what you've done with them. For instance if you delete a transaction from the download transaction window QB will track that FITID number. Next time you pull transactions from the bank that same transaction may get pulled down again. If it is, QB will see that you already marked it "deleted" and it won't show up in the list of downloaded transactions.

  21. 21 Posted by dtd on 19 Jun, 2021 05:02 AM

    dtd's Avatar

    There are a LOT of things I wish MD would do differently, (don't get me started!)

    My comments are based on what I do, not what I want.

    I do think you should explore why one account seems to "work" while the other three give you the laundry list of transactions.

    There are others who will probably weigh in with better suggestions.

  22. 22 Posted by Stuart Beesley ... on 19 Jun, 2021 05:12 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    Load toolbox extension. Ofx tools menu. Then look at the two options to look at all your internal hidden MD ofx data. Compare the data across the two accounts with the same bank that are giving different results.

    (Not support, just a fellow user)

  23. 23 Posted by Stuart Beesley ... on 19 Jun, 2021 05:40 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    Also. On the acct where 4 years of data downloads, is this actually sent from the bank? Open console, download, check the console log and see if there are just a few txns actually being sent? If so, you might need to purge the internal MD download cache.

    Toolbox. Advanced mode. Ofx tools menu then :
    - Delete single cached OnlineTxnList record/Txns

    (It’s a very safe fix and you can run anytime, but always backup first)

  24. 24 Posted by v+ik on 19 Jun, 2021 05:55 AM

    v+ik's Avatar

    "two options"... "Search for stored OFX related data" must be one of them... "View all yoeur OFX last download txn dates" might be the other, but there are 4 other options in the menu, so I'm not sure which you are referring to.

    Under "stored OFX", the account that did not download excess transactions is very brief.

    The second one has about 4000 lines of data, under "txns.0.org_data"
    The third one has about 7000 lines of data.
    The fourth one has about 4200 lines of data.

    Is the above the "MD downlnoad cache"?

    While downloading (again), I don't see any data appear in the MoneyBot Console, but the transactions all reappeared.

  25. 25 Posted by dtd on 19 Jun, 2021 05:58 AM

    dtd's Avatar

    I think Stuart is referring to the normal console, not the moneybot console.

  26. 26 Posted by dtd on 19 Jun, 2021 06:00 AM

    dtd's Avatar

    Help/Console Window
    not
    Window/Show Moneybot Console

  27. 27 Posted by Stuart Beesley ... on 19 Jun, 2021 06:01 AM

    Stuart Beesley (Mr Toolbox)'s Avatar

    Use Help, Show console (Not moneybot).
    I’m not near a computer at the moment, but it sounds like your cache may not be empty. Run the delete cache option and try again.

  28. 28 Posted by v+ik on 19 Jun, 2021 06:04 AM

    v+ik's Avatar

    Hadn't found that before. Thanks.

    But without having cleared the cache (if there is one), the second try on my second account (with the right console open) didn't regenerated the 4 years of transactions.

    I had exited and restarted MD to generate a backup, not realizing that setting Toolbox Advanced mode offered a backup option. Hmm.

  29. 29 Posted by v+ik on 19 Jun, 2021 06:05 AM

    v+ik's Avatar

    I got 400 lines in the console. Seems to be values of holdings rather than transactions.

  30. 30 Posted by v+ik on 19 Jun, 2021 06:08 AM

    v+ik's Avatar

    Doing the 3rd account download got me 986 lines of console data, and all old transactions again.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac