Problem…
I was recently tasked with uploading a large number of documents from our previous document management system (Stored as an SQL FileStream) into our new Salesforce org.
The Apex Dataloader should be able to do the trick. So I wrote a simple windows form app to export and save the files to a local file system with an accompanying metadata CSV. All good so far. After letting the program run overnight to export all our documents I began trying to use the Dataloader to import them into Salesforce.
And that’s where the problem’s began. The procedure was to go something like this.
- Perform an upsert with the metadata CSV to create related custom object records.
- Export the newly created objects with record Ids and document Id external keys.
- Import the documents using the Ids above to attach them to the required records.
All went well until step 3 above….. “Java heap errors”. It appears I’m trying to do too much. I max out the memory allocation to the Java virtual machine used by the Dataloader and try again. Same again.
I trim out all but the first 50 records in my record set and try again. This time is appears to be working. Wait a few minutes and no errors, it looks like its going to upload. I walk away to get some coffee. Come back….more errors. Check the Salesforce org and no documents have been uploaded. Grumble…grumble. That said even if it had of worked I wouldn’t want to be uploading all our document through the above procedure
Solution…
I wrote my own app using c#.Net and the Salesforce SOAP API to do all the work for me.
The program retrieves all the required data from the previous SQL based system. Then performs upserts of new objects into the Salesforce org, returns a dictionary list of recordIds and attachment items. Finally it checks for duplicate files and then uploads the documents. Multi threading allows the progress to be logged to a form listbox and an accompanying log file. The upserts and duplicate detection allow the program to be run over the top of itself without creating additional unwanted records.