, whose site was obviously done in .Net as well. You have your choice of XML or CSV. CSV is still the fastest, but in trying to import this data, I am still having to loop through every row. The size of data is tiny! Nothing but a phone number! Granted about 750,000 of them! The thing that complicates everything is the fact that I can't write the data to a file (I'm on a webfarm), so all processing is done in memory. So, even the command line tool BCP isn't an option! XML should be easy to import with the various options OPENXML, SQLXMLBulkLoad, etc, but I had problems with every single BULK method I tried! So, either way, I'm looping through all these rows!My first version of the XML method was doing a ReadXML and it worked fine for 250,000-300,000 rows, (be sure you do a ReadXMLSchema first! Learned that the hard way! ) But I had to redo this method as well, when trying to process 750,000 rows the ReadXML method would immediately max my CPU to 100% and even after an hour or so, showed no signs of completing and this is before it even starts inserting any data! This was completely rewritten using the .Net XMLTextReader class, but yet again, loops through every node of the XML document, but at least it would process the 750,000 rows in avg of 7-10 minutes.
And as a side not I also tried to use ICSharpCode.SharpZipLib
to save the user the extra step of unzipping the file first. But it's issue is due to the uploaded zip file being written to the FileSystem first which doesn't work in a webfarm environment, so I'm still researching that one!