While DIP can handle a file that size, I have had to split up work before. It all depends on your project constraints.
Many years ago I was working on a FileNet Backfile conversion, migrating more than 8 million records. This project was further complicated because we also had an Epic EMR integration that needed updated with these new records via HL7 Importer. Finally, I was not allowed to import during business hours (6am-6pm).
We started in a test environment, building out our process. Then we iteratively tested growing batches. EX: 100, 1000, 10000. Along the way we found small issues that had to be fixed (which is way easier with 100 docs).
Based on the sample processing speeds discovered, I ended up setting up 3 dedicated DIP Servers to import my batches in parallel. I found 100-500k document sized batches worked well based on my test environment testing.
Initially, I split the text file using an advanced Text Editor called Notepad++.
Eventually, I batch scripted most of the process to automate as much as possible.
At first, during the available night hours I was getting about 1.3Million imported into Production. At the start of the next day the customer's Database Admin team requested we reduce the amount as they could not keep up with the increased work related to Database Backups and Maintenance tasks. So we tried only 500k/night (it was either 2x 250k batches or 5x100k Batches) on the following night shift. The next day the DBAs reported the workload was now manageable. It took about 12 days with these new limits in place.
