Home → Problems →

CSV Importer tells me that duplicates were skipped

• Edit

Gerd

05 Sep, 2025 04:50 PM

Hi,
I use the CSV importer to import bank transactions.
I did an import with 3 records, but the imported records don't show up in the bank account. First I thought it was might not have clicked on the Process button, tried the import again, but now get a message, that 3 duplicates have been skipped.
in the bank account I have selected to show all transactions. The 3 transactions have today's date, thus, should be on top of the screen.
I did a search by date, but nothing was found.
I even used the search and replace extension to search for the transaction by date, but again, none were found.
I changed the sorting from date to description then back to date again. But I still don't see the 3 transactions.
What else can I do in MD to find the existing transactions that are not being displayed?

1. 1 Posted by Gerd on 05 Sep, 2025 05:02 PM

   

   And just after I posted the problem, I found the problem. I must have been blind, the transactions were loaded into the wrong account.

   • Edit

2. Gerd closed this discussion on 05 Sep, 2025 05:02 PM.

3. Gerd re-opened this discussion on 06 Sep, 2025 10:19 AM

4. 2 Posted by Gerd on 06 Sep, 2025 10:32 AM

   

   Yesterday, after getting the duplicate error, I thought I MUST have used the wrong account to import the transactions. So, today I repeated all the imports again. All imports until July imported into my TEST IMPORT account without any errors.
   But when I tried to import the last 131 transactions from the August and beginning of September, all 131 got rejected with the duplicate error message. I verified that I used the correct test import account, none of the transaction shows up in the bank register.

   The only explanation I would have, is, that they exist in my regular bank account. But this should not really matter, as I used the test import account for the import. I searched the account by date, and even some descriptions, but none were displayed. Also, no other account got "inflated", which would indicate the transactions were imported into a different account.

   For another test, I deleted the test account again, then recreated it. Did the import, but this time, 30 transactions got imported, 101 rejected as duplicates. The CSV I use has dated up to September 5th. The last transaction date shown in the test import account is August 15th.
   So, I can reproduce the error. Even though this test shows a bit different results.

   So, my question to the developer of the extension, is, how is a duplicate being determined?

   • Edit

5. Support Staff 3 Posted by Maddy on 06 Sep, 2025 01:22 PM

   

   Hi Gerd,
   Thank you for contacting Moneydance support.

   Usually we recommend using OFX and QFX files as these contain unique transaction identifiers and if you were to accidentally download the same transaction to Moneydance more than once, the ID would be recognised. Moneydance would then ignore the duplicate transactions and not download them again.

   Unfortunately QIF and CSV files don't contain these same ID's.

   If you haven't already, I'd recommend taking a look at this article from the knowledge base that details recognising and confirming your recently downloaded data.

   When transactions have been recently downloaded, they'll be 'unconfirmed' which is signified by a blue dot in the register.
   To begin the confirmation process, select the first transaction in the register with a 'blue dot'. A panel will appear on the right of your screen which shows the 'Current Transaction' details. You can amend the category within this panel, then select 'Confirm'. You'll see the solid blue dot change to a blue circle. Moneydance will then move to the next unconfirmed transaction in the register.

   As you go through this process it will become quicker and easier. Moneydance will begin to recognise the information you have previously confirmed, and start to suggest the correct categories within 'Similar Payees' in the panel on the right.
   The next time you download from the bank many of the correct categories will automatically be assigned, and this will improve and you confirm more transactions.

   ---

   Lasty, you can navigate to Online->Match Online Accounts.
   This feature is for the remembered account number links for manually imported files.

   I hope this information is helpful. Please let us know if you have further questions or need more assistance.

   --
   Maddy, Infinite Kind Support

   • Edit

6. 4 Posted by Gerd on 06 Sep, 2025 02:11 PM

   

   Thanks for your reply, Maddy.
   However, you don't comment on my question how some of the transactions are flagged as duplicates, even though they have not been imported, or are not visible, in the bank account.

   I have tried the OFX import, which works well, but the bank transactions have text in the description field that I either don't want, or some of the additional text I would like to have in the memo field.

   This was my reason to download the transactions as a CSV, run some VBA to delete test I don't want, and move text I still to the memo field. I also will add some additional logic to the VBA that determines the correct category for each transaction and adds it to the CSV.

   I have imported just about 3000 (no joke) into my test import account (the same 3000 also exist in my checking account) without the duplicate error messages, just the last month import shows the duplicates. And after I saved the balances of each bank account in the database I know they did not get added into anther but the selected test bank account.

   II had already mentioned that the same transactions I previously imported into my main checking account, but all of those were already categorized correctly, where the category in the test account were all the same set to ATM Withdrawal, which I use to identify the once I still have to categorize with the search and replace extension.

   I have checked the link about confirming the transactions, what I already did in the checking account. But none of the transactions in the test account were confirmed yet. The next step in the test account would have been to first categorize all transactions, then to confirm all of them.

   So, the question still remains, how does the importer know about duplicates? And where are they?

   I had tried to add some of the duplicate transactions manually, which was accepted without another duplicate message. I did not want to do this for the 131 duplicates, and my worry now is, that my import end of the month could trigger the same duplicate error.

   I still feel, that the developers of the extension should check this, and hopefully provide an update. FYI, my monthly bank transactions are between 100 and about 130 each month.

   And one last comment about confirming each transaction. It would be nice if there would be an option where I can select multiple unconfirmed transaction and confirm them, instead of doing it one by one.

   Gerd

   • Edit

7. 5 Posted by Gerd on 06 Sep, 2025 02:15 PM

   

   ... I should have tried before asking, I found the confirm all function

   • Edit

8. 6 Posted by dwg on 07 Sep, 2025 01:09 AM

   

   I'm not aware of the CSV importer trying to detect duplicates, it certainly supports the confirmation process.

   OFX/QFX files are the only files that contain transaction IDs so are the only ones that can be used to absolutely identify duplicates previously loaded via the same mechanism, as the IDs are used.

   • Edit

9. 7 Posted by Gerd on 07 Sep, 2025 09:45 AM

   

   I got the duplicates error several times for the CSV import.

   What do you mean by "it certainly supports the confirmation process"?

   • Edit

10. 8 Posted by dwg on 07 Sep, 2025 10:34 AM

    

    The confirmation process is discussed in the Knowledge Base article:

    https://infinitekind.tenderapp.com/kb/online-banking-and-bill-pay/confirm-downloaded-data

    • Edit

11. 9 Posted by Gerd on 07 Sep, 2025 12:29 PM

    

    Ok, I saw that KB already and all my transactions are confirmed already.
    But I don't see any relevance to the import issue.

    Here is a screen shot of a CSV import that reported duplicates did not get imported. This is from my test database.

    • Screenshot_2025-09-07_at_13.28.16.png 101 KB
    • Edit

12. 10 Posted by -Kevin N. on 07 Sep, 2025 12:47 PM

    

    Hi Gerd,

    Q. 1) For the account that the CSV Importer reports that there are duplicate transactions; does the 'blue dot' remain next to the account in the left sidebar after confirming all of the visible transactions in it?

    Q. 2) Is there a message in the lower margin of the account in question stating that you have 'x number' of unconfirmed transactions?

    If so, drag up that header bar with the message on it to hopefully reveal the hidden transactions.

    -Kevin N. (not a member of Moneydance support)

    • Edit

13. 11 Posted by Gerd on 07 Sep, 2025 01:39 PM

    

    Re 1) yes there is a blue dot
    Re 2) all the new imported transactions have a blue dot in the register. At the bottom of the screen is NO message, just the balance in the bottom right corner, and if I click the balance it tells me the amount of unconfirmed transactions
    Did you expect the message in that bottom line past the last transaction on the screen?

    • Edit

14. 12 Posted by -Kevin N. on 07 Sep, 2025 01:58 PM

    

    Hi Gerd,

    Did you expect the message in that bottom line past the last transaction on the screen?

    The message would indicate that you are using the split-register which culls out unconfirmed transactions (txns).

    The only other thing that I can think of is that there may be legitimate duplicate txns in the CSV file that the importer is flagging as duplicate txns that should be skipped.
    e.g. 2 or more transactions entered on the same date from the same vendor for the same amount. Is that a conceivable possibility?

    In that case, I don't think that there is much that can be done with regards to importing CSV files. This is where the OFX file format outperforms CSV wherein the aforementioned unique identifiers provide a measure of prevention of unwanted duplicates as well as proper identification of legitimate duplicate transactions.

    -Kevin N. (not a member of Moneydance support)

    • Edit

15. 13 Posted by Gerd on 07 Sep, 2025 02:55 PM

    

    Hi Kevin,
    yes, my CSV (but also the OFX) can have the same transaction multiple times on the same day. Like I go to my favorite coffee shop in the morning, in the afternoon and even in the evening, purchase always the same coffee for the same price. Well, I don't even drink coffee, but yes, we have legitimate duplicates on the same day from the same vendor.

    I am going to delete some months from my test database and do some more OFX imports. My problem with OFX is that I can't change the description field like I would want to before the import.

    Give me some time and I'll report back.

    And thanks for your continued help!!!

    Gerd

    • Edit

16. 14 Posted by -Kevin N. on 07 Sep, 2025 05:41 PM

    

    Hi Gerd,

    You're welcome. I'm glad I could help.

    I think you'll find that once you've imported the OFX file and edited the Description field to your liking & Confirmed the txn(s) that going forward, future imports will offer the edited Description in the Similar Payees section of the Confirmation sidebar.

    Moneydance may even automatically enter the edited Description based solely on your prior entries.

    Take note though that you must Confirm txns in order for either of these potential options to be offered.

    -Kevin N. (not a member of Moneydance support)

    • Edit

17. 15 Posted by Gerd on 07 Sep, 2025 06:06 PM

    

    Now, that would be very helpful, if the description would be changed automatically after I corrected some with the first import. I'll give this a try.

    I just finished to import 2 1/2 years of OFX without a problem. I am going to delete them again, then only import the first month, correct the description and confirm the changes. Then I'll import the next month to see if the description changes will be suggested.

    I learned a lot this past week and this weekend, and I sure like MD a lot, despite the issues I found so far.

    I'll report back when I have the next batch of imports done.

    • Edit

18. 16 Posted by Gerd on 07 Sep, 2025 06:26 PM

    

    It sure does work, but I have 2 questions.
    1) On the similar Payees I sometimes see duplicates (they are exactly the same), where I had expected to only see singles for the very same description, but this is not a problem
    2) why are those Merge options showing up? And is there a config setting to avoid the merge from being displayed? If you don't pay attention you might merge something you won't want to, and then might have a difference.
    I am just curious why I ever want to merge any transactions, after there are no duplicates. All transactions are single and individual transactions, thus, there would not be a need to ever merge some.

    • Edit

19. 17 Posted by Gerd on 07 Sep, 2025 07:19 PM

    

    It does not work so well, as for our coffee shop, where we have quite often the same amount, it now already shows about 2 pages of merge, and for a super market various duplicates of the same name under other payees.
    And with each confirmation of a new transactions, the merge and other payees list gets longer.
    I am not sure I want to do this for the 2 1/2 years of transactions I need to process.

    It probably is ok, for the new transactions I import about once a week. That will be manageable.

    • Edit

20. 18 Posted by -Kevin N. on 07 Sep, 2025 07:55 PM

    

    Hi Gerd,

    Try this to hopefully limit the Similar Payees and such...

    Go to Preferences on the Network tab and change the following option:
    'Only match downloaded transactions when they are at most 'X' days apart.

    Change the 'X' number of days to something low, e.g. 7.

    (If by chance you don't see the above option, drag the window's lower margin down to reveal it.)

    -Kevin N. (not a member of Moneydance support)

    • Edit

21. 19 Posted by Gerd on 08 Sep, 2025 05:11 PM

    

    I imported another month with OFX. I also changed the setting as you suggested
    But have a look at my attachment. Even though WAITROSE 192 is always the same, why do I have so many similar payees, and as you can see they are not similar, they are the same. I have the same with other Payees as well.

    I'll now update the current import and report back.

    • Edit

22. 20 Posted by Gerd on 08 Sep, 2025 05:13 PM

    

    Forgot the screen shot

    • Screenshot_2025-09-08_at_18.08.53.png 52 KB
    • Edit

23. 21 Posted by Gerd on 08 Sep, 2025 05:20 PM

    

    And this is another mysterious example for Sainsburrys
    One time it shows more in Similar Payees, other times less. Why is that? ( don't expect an answer.

    • Screenshot_2025-09-08_at_18.18.16.png 59.8 KB
    • Screenshot_2025-09-08_at_18.18.29.png 37.8 KB
    • Edit

24. 22 Posted by -Kevin N. on 08 Sep, 2025 08:58 PM

    

    Hi Gerd,

    No answers but some questions...

    These stores appear to be grocery stores.

    Are you buying groceries there or are you actually just using the ATM at these establishments?

    Are you Confirming all of these txns with the Category ATM Withdrawal or something like Food:Groceries?

    Confirming transactions is vital to Moneydance 'learning' the correlation between the Description and Category fields.

    One last thing, when you changed the 'X' number of days, did you restart Moneydance? Some settings require a restart for the changes to take effect.

    -Kevin N. (not a member of Moneydance support)

    • Edit

25. 23 Posted by Gerd on 09 Sep, 2025 07:46 AM

    

    Hi Kevin,
    those are groceries stores and for all the imports I use temporarily the existing category ATM Withdrawal. This category is used only for the import.

    And yes, I confirmed each and every one transaction as you had suggested earlier, paying attention that I don't use merge by accident. Further, we also buy groceries from the different Tesco stores, here I even keep the store number in the description, confirm the transaction, but still have all of them showing us as similar payees, where some with the same store number are duplicates, and the list is not sorted by the store number either.

    One thing that also stroke me odd, was, that sometimes you get a similar payee displayed with a totally different name. I just was wondering about this logic. It's not a problem, as I pay attention to what I select.

    I had copied my production database, then deleted all the transactions from the checking account, thus, I have all the correct categories, then use search and replace to assign the correct category. I had saved all the replacements for the payees with lots of transactions, like for Sainsburys or Tesco. I just love that extension, as it allows so many selections and replacement options. Thank you very much for whoever provided it.

    I still learn about MD functionality, by comments and suggestions from you, dwg, or Maddy. I also like the forum, as you get answers and help to all the question I might have.

    • Edit

26. 24 Posted by Gerd on 09 Sep, 2025 07:55 AM

    

    here is an example for Tesco

    • Screenshot_2025-09-09_at_08.54.10.png 59.4 KB
    • Edit

27. 25 Posted by Gerd on 09 Sep, 2025 07:59 AM

    

    and this is an example where the suggested Payee does not make any sense. The main transaction is a pet store, so why ASDA is suggested beats me.

    • Screenshot_2025-09-09_at_08.58.36.png 18.8 KB
    • Edit

28. 26 Posted by Stuart Beesley ... on 09 Sep, 2025 08:24 AM

    

    I just looked at the CSV Importer extension’s code. It uses this logic to assign an online id to each txn and then if a duplicate is detected, it skips it.

    String tmp = onlinetxn.getDateInitiatedInt() + ":" + currency.format( onlinetxn.getAmount(), '.' )
                               + ":" + (onlinetxn.getName() == null ? "" : onlinetxn.getName() )    // used payeeName once
                               + ":" + (onlinetxn.getCheckNum() == null ? "" : onlinetxn.getCheckNum().replaceAll( "^0*(.*)", "$1" ) )    // strip leading 0's
                               + ":" + (onlinetxn.getMemo() == null ? "" : onlinetxn.getMemo() )
                                      ;
    

    So, are your ‘duplicates’ triggering with this logic?

    • Edit

29. 27 Posted by Gerd on 09 Sep, 2025 10:48 AM

    

    Hi Stuart,
    the Tesco and Sainsbury stores don't use the memo field, Tesco uses the store number, and Sainsbury has 2 flavors of the name, Sainsburys S/MKT, Sainsbury's S/MTKS, and Sainsburys MKT.

    And to answer your question, I could not tell you if the duplicates are triggered by this code.

    • Edit

30. 28 Posted by Gerd on 09 Sep, 2025 11:08 AM

    

    Stuart,
    here is one more example where the similar payees is not working properly.

    I have a new transaction from Tesco 6446. I had this before in the previous month, confirmed it, but this time it did not show up under similar payees

    • Screenshot_2025-09-09_at_12.06.26.png 68.6 KB
    • Edit

31. 29 Posted by Stuart Beesley ... on 09 Sep, 2025 11:09 AM

    

    Upload the header row and actual two rows that trigger as duplicate to look at

    • Edit

32. 30 Posted by Gerd on 09 Sep, 2025 11:30 AM

    

    Here are 2 examples.

    Let me know if you any more.

    • Screenshot_2025-09-09_at_12.27.35.png 91.8 KB
    • Screenshot_2025-09-09_at_12.29.39.png 198 KB
    • Edit