[dba-VB] C# custom exceptions

I am trying to find info on how and when where and why to use custom exceptions and particularly how 
to log them.

I am developing a system which uses four asynchronous processes.

Stage 1) Export a SQL table or view to address validation.  I have a table that has records which 
tell stage 1 the name of the database, view to export, date to process, directory on disk and a few 
other things.

Stage 1 creates text files on disk in a staging directory, one file for each 2 million records 
exported.  This creates a record in a log table with the PK of stage 1 table, name of the database, 
view exported, directory on disk (output staging directory), file name, and some date flags to 
monitor completion of the remaining stages.  IOW one log file per text file exported, all info 
needed for the remaining stages, date fields for each remaining stage to log that it processed that 
file.

Stage 2) Move the files in the staging directory to a virtual machine.  The log files created by 
stage 1 are read and used to move the files and set a datetime flag in the log record saying that 
stage 2 completed (the file moved to VM).  The virtual machine may not be available, but if it is 
the file moves and the log record is updated. If the file goes to the VM, the VM will process the 
file and place it in an output directory.  This will take between 1/2 hour and 1 hour per file.

Stage 3) Move the completed files from the VM back to a different (input) staging directory.  Again, 
the log file is used to determine what files should be available on the VM and if found and moved, a 
datetime flag is set saying that file was moved back from the VM back to the Input Staging Directory.

Stage 4) Import the files from the input staging directory back into SQL server.  The log record for 
any file moved back in is marked with a datetime flag saying that the file was imported back in to 
SQL server.

I have about a dozen or so "data" tables which have to be processed monthly, rain or shine.  By 
placing records in the stage 1 source table, I can cause the system to process each of my data 
tables.  As the process finishes, it creates a new record adding a "number of days to next process" 
to the date so that it is queued to process again in X days.

I also have orders that I run that will write one or more records into this same table and cause 
their data to process.

So... nicely divided, asynchronous process: Sql to Staging, Staging to VM, VM to staging, Staging in 
to SQL Server.  These things must be asynchronous because the VM may or may not be available, and it 
is critical that each file is processed through each stage and tracked.

Records in the main table triggers the process (stage 1).  Records in the log table track stages 2-4 
for each file created in stage 1.

I want to run 4 threads to do these processes.  IOW, a process just runs on the server (or 
somewhere, but probably eventually on the server).  Each stage is entirely independent, and takes 
its input from a table, and writes a datetime flag as it finishes.

OK, so back to exceptions...

I am new to threads.  I wrote the code from the bottom up, with working processes for doing each of 
the 4 stages, but triggered by button clicks.  It all worked, except that it wasn't table driven (of 
course).  I even had each button click create a worker thread and that worked.

I created a supervisor class, with four methods, one for each stage.  I moved the button click code 
to these methods.  I fired each of them up, and it mostly works... each stage is running in its own 
thread and watching the tables, updating the records as each does its thing.  Except that I would 
get weird errors that were not handled by any exception handler.

I removed all traces of messagebox.show and I *think* I have caused that to go away.  Or maybe not, 
who knows.

But now I really need to get errors logged.

Again, I am new to threading, and new to custom exceptions.  New to exception logging.  Do I sound 
like a nubee?  ;)

Anyway, I do not particularly want to use the Windows error log because then the error logs would be 
scattered around on whatever machine the parts run on.  Remember that there is a user interface part 
which sets up the records in stage 1 and monitors the process.

This is potentially a nice little system but it has to be done right or I will fight it for the rest 
of my career.  So I need to learn exceptions, and I need to log the exceptions, my preference is 
into SQL Server, though there are of course some potential errors (SQL server errors) which could 
not be logged there because SQL server isn't available at the instant the log is created.

Kinda messy.

Any pointers, articles, books, or other sources of information I could go to to learn this stuff. 
Google is not my friend in this case because I am getting stuff from 2003 to the present and I 
really want modern code.

Help!

-- 
John W. Colby
www.ColbyConsulting.com