[dba-VB] What to do, what to do?

To get back to reality, I really need to process under 6 million records, and the vast majority will 
be merging two records into one.  I will almost certainly pull about 10,000 records at a time, so I 
will be processing 5,000 people at a time.  Merge two records, merge two more, do that ~ 5000 times, 
write back to SQL Server, load the next set of 10,000 records. Do that 600 times.  I will see how 
many records can actually be loaded, perhaps 100,000 at a time would work and do that 60 times.

Maybe break the process into threads.  One thread watches the ADO recordsets in memory and keeps 
them filled, another thread performs the merge, another thread writes back to SQL Server.

In any case there is simply no need to load all 6 million records at once.

To get back to the original point however, while TSQL might be able to do what I need, I have 
serious doubts.  Even if it could, trying to debug this kind of program written in TSQL would be a 
nightmare for me.  Furthermore even TSQL could not do this on 6 million records at once.

No sane person claims that any specific language is the best for everything, and for this kind of 
thing ADO and C# is probably a good choice.

John W. Colby
www.ColbyConsulting.com


Shamil Salakhetdinov wrote:
> Hi Robert,
> 
> Yes, DataSets are mainly targeted for working with small number of records
> in memory - so the following is more a "theoretical" calculation:
>