[dba-VB] What to do, what to do?

http://arstechnica.com/hardware/news/2009/11/biography-solid-state-disk.ars

John W. Colby
www.ColbyConsulting.com


Shamil Salakhetdinov wrote:
> Hi Robert,
> 
> Yes, DataSets are mainly targeted for working with small number of records
> in memory - so the following is more a "theoretical" calculation:
> 
> - JC has 64bit PC which allow to load in memory "practically unlimited"
> volume of data - 64bit logical (process) address space could be as large as
> 16 exbibytes =  1,152,921,504,606,846,976 bytes * 16
> (http://en.wikipedia.org/wiki/Exbibyte). 
> 
> - the speed of data transfer for 64bit Intel's processor - "the 200 MHz
> McKinley bus transferred 6.4 GiB/s), and the 533 MHz Montecito bus transfers
> 17.056 GiB/s (http://en.wikipedia.org/wiki/IA-64#Memory_architecture); 
> 
> -----
> So "theoretically" with modern speedy harddisks JC can load 50 millions of
> records into memory within minutes - within half a minute in the near future
> when large enough flash-memory disks will become available and relatively
> inexpensive?
> 
> To load such large data volume it would be better to use SqlDataReader as it
> keeps loaded data in a very compact form. And 50 million records long
> SqlDataReader should be probably better split into several chunks...
> 
> Again - this is just a "theoretical" consideration - in practice JC can
> process his 50 million records long data table in chunks as he has a hash
> field (tblHashPK), which can be used to load related records into memory: I
> mean he can split tblHashPK table keeping hash values into several groups,
> e.g.:
> 
> 1 - 100,000
> 100,001 - 200,000
> ...
> 
> and process each group joining its records to source 50 million records...
> 
> --
> Shamil