[dba-VB] c# lock()

jwcolby jwcolby at colbyconsulting.com
Sat Mar 26 07:44:54 CDT 2011


Well... I haven't migrated to VS2010 yet.

I tried it about 6 months ago and it was universally a pig on my systems.  I thought I would wait 
until MS polished it a bit.

John W. Colby
www.ColbyConsulting.com

On 3/26/2011 8:41 AM, Gustav Brock wrote:
> Hi John
>
> My first thought was Windows Workflow Foundation(WF) but, as Shamil mentions, it may be overkill. On the other hand it is exactly for controlling scenarios where "if task 1 is done, start task 2 and task 3, wait for task 2 and task 3 to finish, check something, then start task 4, etc.".
>
> However, that reminded me about a series of articles about the Task Parallel Library of .Net:
>
> http://www.codeproject.com/KB/cs/TPL1.aspx
>
> which I found very interesting, though I haven't had any use for it yet. Among other topics it discusses carefully canceling and error handling which I guess is quite important for your purpose.
>
> /gustav
>
>
>>>> jwcolby at colbyconsulting.com 26-03-2011 03:57>>>
> Shamil,
>
> I have processes that log results to flags.  For example, make a database (log that it was made),
> build a table (log that it was built), pull umpteen million records in sorted order (log that it was
> filled), build a chunk table (log that it was filled), bcp out (log that it was exported), build
> another chunk table (log that it was filled), BCP out (log that it was exported).  The objective is
> to be able to sustain interruptions and pick up where we left off.  These processes can take minutes
> (fill chunk table, bcp out) or a half hour (pull umpteen million records in sorted order).
>
> So each thing I do represents a step in the process and each step is logged in a field in a record
> in SQL server using a datetime.  There are so many of these flags that I am trying to standardize
> the process by building a class that can be instantiated, filled with data and log itself to SQL
> Server by one thread and be checked by another thread.
>
> These flag class instances will be checked by multiple threads, each thread trying to decide whether
> it should be doing the next step because another thread has finished it's part.  IOW if a file has
> been written to disk, then the next thread will write it to a VM for processing.  If it moved to the
> VM the next thread will watch the VM's output directory for a file to pop out and move it back to a
> directory on the server.  If the file (a couple of files actually) successfully copied back to the
> server staging then another thread will import it back into a chunk table in an input database.  If
> the file successfully imported then another thread will...  In general one thread will "own" the
> flag and use it to log its status and one other thread will be checking the status of the flag to
> determine that it can go to work on that work chunk.
>
> You get the picture.
>
> I am trying to build an entirely asynchronous highly threaded process which exports a huge table
> into multiple files, processes every file through a third party app and gets the results back into
> SQL Server.  All while logging each and every step so that no piece can possibly be dropped at any
> stage, even if the server goes down (or the VM goes down).  Eventually this process will run on my
> server 24/7.
>
> It has been working for some time but I am getting threading issues, and I need to work on the high
> level control so that all of the processes can cleanly start up and shut down and every stage can
> pick back up when the program restarts should a shutdown occur.
>
> A single database can be up to a hundred million records (the biggest so far), and the external
> program only handles roughly 2 million records.  Each "chunk" takes roughly 45 minutes to an hour
> depending on many different things so that example will take 50 chunks and could take 40 to 50 hours
> to complete.  It takes about 20 processing steps to handle each file from end to end.  It needs to
> just work, and I need to be able to view status in a meaningful way.  And I need to process that and
> a dozen other files every single month, automatically, with no manual intervention required.
>
>
>
> John W. Colby
> www.ColbyConsulting.com
>
>
> _______________________________________________
> dba-VB mailing list
> dba-VB at databaseadvisors.com
> http://databaseadvisors.com/mailman/listinfo/dba-vb
> http://www.databaseadvisors.com
>
>



More information about the dba-VB mailing list