In the endless search for optimal performance in Bulk loading data, I have written a paper which describes how to reach top performance using the SqlBulkCopy class in .NET, trying to find the best combination of its parameter settings to load heaps, clustered table with and without indexes performing all the load in a highly parallelized environment.
In order to produce test, I have written a solution that implements a producer/consumer pattern where many producers and consumer can run in a parallel environment, that can be used to perform detailed tests by simply switching some parameters on and off.
The paper does not go as deep as the Data Loading Performance Guide from Microsoft does in describing the internals of SQL Server and bulk loading. Nevertheless, the Microsoft paper describes some exoteric hardware configurations that are pretty difficult to find in the real world, while my paper is much more closer to a real user experience. I thjink that reading both might help any ETL code writer to better understand how to boos his code performance. Mine is easier, Microsoft’s is far more complete.
If you are interested in reading it please follow
I am very interested in feedback.