# Wednesday, August 01, 2007

Hmmm, what would be the best way to process 50000 records in a database?

Bruteforce :- Just run the query through a SQLDataReader and process them one by one. Simple, easy and straightforward. Unfortunately doing something like that would tie up system resources and processing pretty badly.

ThreadPool :- Same as above but, execution would be on a seperate threadpool thread. Which usually yields processing to the OS when needed.

MSMQ :- Put all the records into MSMQ and then have another thread do the processing, since MSMQ is in charge of releasing the items for processing. Processor load would not be 100% during processing, going this route requires the presence of MSMQ on your server and also the maintainence of a new private queue.

Chunks :- Process a number of rows.. say... 500 at a time, and then take a breather of say.. 10 seconds.. and then go again, because the code takes a breather the processor load is comparatively lower since it's not CONSTANTLY in use. Unfortunately, because your code takes a breather... since you're not processing all the time, you need a lot more time to finish the job.

Hmmm.. so which method should I use?

Note that you can Post As GUEST as well.
blog comments powered by Disqus