[dba-VB] Thread safe list

I saw something about locking the object, then doing a deep copy and using the copy.  Have you ever 
done this?

My collections are always well under a hundred objects and the objects are just pointers (and the 
key) so it seems like a deep copy of the dictionary should be lightning fast.  Then I could use the 
ForEach on the copy and all would be right in the world.

 > (Advanced) thread-safe multi-threading is "another song" - a way more complicated than ordinary 
programming.

We have moved to using threads for every time intensive task, but also using events to signal that 
we are finished with a time intensive task.

For example we:

1) Build a database (fast) and then build a table, populate it with (tens of) millions of records 
and build a bunch of indexes (SLOW).  The table build can take minutes to complete.  We run that 
whole process on a thread and the very end of the process raises an event.  The event sets a flag 
and throws a status object up the chain for display on the form, then exits (the thread).

2) A thread then creates a chunk table, pulls a set of up to two million records into a table, 
creates indexes etc (slow - up to a minute or more) and then raises an event which sets a flag and 
throws a status object up the chain for display on the form.

3) A thread then BCPs out that chunk table (slow - up to a minute or more) and raises an event which 
sets a flag, and throws a status object up the chain for display on the form.

Steps 2 and 3 repeat until all records are BCPd to files on disk.

This all happens in a custom database class which has a collection of ProcessFile objects (each of 
which is a chunk).  A supervisor object (owns the ProcessFile objects) places ProcessFile objects 
into the collection in the database object. and the database object is responsible for getting the 
data out of SQL Server, getting the chunks, and BCPing them out.  So the supervisor and the database 
are both accessing the collection that holds the ProcessFile objects, each using its own thread.

All of that stuff is just "Stage1 - SQL Server to Staging".  We have other classes which manage 
getting these files into a VM for processing and getting the resulting (processed) files back from 
the VM and back into SQL Server.

This whole thing is morphing into a pretty well defined object model where each object has clearly 
defined responsibilities.  I say morphing because it evolved from about 50-80 user stored procedures 
out on sql server, manually executed by hand from a list written on paper.  Boy is C# / SQL Server 
faster, easier and more reliable than jwc keying stuff into a query window while reading a check 
list.  ;)

The key though is these collections where we place supervisors and ProcessFiles for processing, and 
these lists (the process file lists) are hit from two sides by independent threads - one placing 
items into the lists and the other getting them out of the lists for processing.  Fun, efficient but 
tricky.

John W. Colby
www.ColbyConsulting.com

On 12/3/2010 11:00 AM, Shamil Salakhetdinov wrote:
> Hi John --
>
> 1. Yes, static. Object ref isn't thread-safe AFAIU.
> 2. Yes, foreach isn't good. You can use
>
> int count = getCount(); // getCount should use locking
> for (int i=0; i<count; i++)
> {
>     MyClass obj = getObject(i); // getObject should use locking as well as to
> have try/catch	
> }
>
> Obvisouly while iterating that way you can loose/skip some items, which will
> be inserted/deleted by parallel threads.
>
> (Advanced) thread-safe multi-threading is "another song" - a way more
> complicated than ordinary programming.
> I'm not 100% sure my abive sample doesn't have some "hidden glitches".
>
> It' s time to master PLINQ, lamda-expressions, functional programming
> principles, RESTFul web services,.... to name a few.
>
> The above is a "to do learning list" for myself also :) - I'm not yet there.
> And I'm not sure I will find good enough time to master all that advanced
> stuff...
>
> Thank you.
>
> --
> Shamil