Shamil Salakhetdinov
shamil at smsconsulting.spb.ru
Sat Mar 26 19:08:10 CDT 2011
John --
I'm falling asleep here now - would that be correct assumption that you can
generalize your global task like that - you have, say:
- split input data into 50 Chunks;
- every chunk should be processed in 10 steps;
- every step's average processing time is 100 minutes.
Then in total it is needed 50*10*100 = 50,000 minutes or ~833.(3) hours to
process all your input data sequentially, chunk after chunk step after step,
say, you don't have enough computer power to process some chunks in
parallel, and you can't process the whole not split input data as it's too
large for your system...
But you have "50 small computers" and you can process every of 50 input
chunks in parallel - then you'll need just 10*100 = 1,000 minutes (~16.67
hours) to complete the job.
I assume that all the chunks processing is independent - then you can:
1) Create Scheduler class instance;
2) Schedule class creates 10 TaskQueue class instances - 1 queue for every
processing step type;
3) Scheduler class defines splitting criteria to split input data into 50
chunks;
4) Schedule defines completion criteria - when all 50 chunks get collected
in 11th FinalQueue;
5) Schedule seeds the first InputChunkSplitTaskQueue with 50
SplitTasksDescriptors;
4) From time to time - say every half a minute Scheduler class instance
creates 500 worker classes in 500 threads (500 worker classes => 50 chunks *
10 steps.), which "hunt for the job" in their dedicate Queues: if they find
Job they work on it, if not - they "die"...
5) When a worker class completes processing its job it queues its "task
results" into the next step Queue, if a worker class fails then it puts its
incomplete task descriptor back into its dedicated queue and "dies", or that
could be Scheduler's (or SchedulerAssitant's) task job to find failed
workers' "trap" and to resubmit their failed work to the proper queue
according to the custom workflow descriptors attached to every chunk...
500 worker classes is an overkill as only 50 worker classes can have job
every time but that seems to be an easy "brute force" and "lazy parallel
programming" approach - and it should work...
Or you can make worker class instances production processes smarter:
Scheduler class can start special thread for WorkerClassGenerator instance,
which will monitor all the 10 TaskQueues, and if it finds an item in a
Queque, it will pick it up, it will create corresponding Worker class in
parallel thread and it will pass WorkItem to the Worker class fro
processing...
When that described above approach will work there then you can easily(?)
scale it splitting your (constant size) input data into 100 chunks, and then
if every chunk can be processes in half time - in average (50 minutes) -
then all job can be completed in 500 minutes = ~8.33(3) hours etc...
Please correct me if I oversimplified your application business area...
Thank you.
--
Shamil
-----Original Message-----
From: dba-vb-bounces at databaseadvisors.com
[mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil
Salakhetdinov
Sent: 27 ????? 2011 ?. 1:27
To: 'Discussion concerning Visual Basic and related programming issues.'
Subject: Re: [dba-VB] c# lock()
John --
Let's define your target (application) system architecture on conceptual
level - I guess that using "micro-level locking" you're trying (?) to
achieve isn't needed...
1) you have a huge input data set;
2) you split input data set into X chunks;
3) every chunk gets processed using the same several - Y - steps;
4) on completion of step Yn input/semi-processed chunk Xn is submitted to
the Y(n+1) processor class;
5) when all X chunks are processed in Y steps they get collected into a
final "huge" output set to produce Z reports.
That's it from conceptual high level point of view?
Something important missing?
Too simplified to be true?
John, please take into account that I can't reply promptly here - it's
weekend now and I'll be out tomorrow, and it's 1:26 a.m. - time to sleep,
and then I have quite some work to do starting Monday - so I expect this
thread will be supported by other AccessD/dba-VB members...
Thank you.
--
Shamil
-----Original Message-----
From: dba-vb-bounces at databaseadvisors.com
[mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby
Sent: 27 ????? 2011 ?. 0:53
To: Discussion concerning Visual Basic and related programming issues.
Subject: Re: [dba-VB] c# lock()
Shamil,
I am trying to build a base class which contains a set of lock objects and
some standard variables.
Having defined the lock objects static per your code below, the derived
class cannot access the lock objects even though I set them protected:
protected static Object LockBln = new Object();
In the derived class when I try to access it:
lock(this.LockBln)
{
}
I get a compile error:
"Member cannot be accesses with an instance reference; Qualify it with a
type name instead."
What does this mean?
Reading on the internet it says that static properties are shared between
all instances of the class. I don't see how I can share a property between
50 instances and then use a
lock(MyStaticProperty) {}.
I'm so confused...
John W. Colby
www.ColbyConsulting.com
On 3/25/2011 5:05 PM, Shamil Salakhetdinov wrote:
> Hi John --
>
> Your sample is a correct usage of locking - just use static thisLock
> variable
>
> private static Object thisLock = new Object()
>
> as non-static variable *is not* thread safe AFAIU.
>
> Yes, such locking can be done for various operations of that class -
> just make sure you'll not get dead-locked...
> Also use *one lock object* to lock *one resource* in a class -
> otherwise "deadlock" will become your "everyday guest"...
>
> Thank you.
>
> --
> Shamil
>
> -----Original Message-----
> From: dba-vb-bounces at databaseadvisors.com
> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby
> Sent: 25 ????? 2011 ?. 22:58
> To: Discussion concerning Visual Basic and related programming issues.
> Subject: Re: [dba-VB] c# lock()
>
> Shamil,
>
> Does this allow using a single "lock object" to lock various
> operations in a class.
>
> For example I want to lock a date flag variable while using a property
> to either get or set the variable.
>
> private Object thisLock = new Object();
>
> public DateTime pFlagDte {
> get {
> lock (thisLock)
> {
> return dteFlag;
> }
> }
> set
> {
> lock (thisLock)
> {
> dteFlag = value;
> }
> }
> }
>
> The issue is not multiple threads trying to write to the date variable
> but rather one thread trying to read it while another is writing it.
>
> John W. Colby
> www.ColbyConsulting.com
_______________________________________________
dba-VB mailing list
dba-VB at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/dba-vb
http://www.databaseadvisors.com
_______________________________________________
dba-VB mailing list
dba-VB at databaseadvisors.com
http://databaseadvisors.com/mailman/listinfo/dba-vb
http://www.databaseadvisors.com