From Gustav at cactus.dk Thu Apr 3 06:34:34 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 03 Apr 2008 13:34:34 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From accessd at shaw.ca Thu Apr 3 10:59:46 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Thu, 03 Apr 2008 08:59:46 -0700 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: References: Message-ID: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From michael at ddisolutions.com.au Thu Apr 3 18:01:55 2008 From: michael at ddisolutions.com.au (Michael Maddison) Date: Fri, 4 Apr 2008 10:01:55 +1100 Subject: [dba-VB] ASP.NET, separated programming and visual design References: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D013BF515@ddi-01.DDI.local> I did an online seminar with MS about VS 2008 + WPF. They claim that WPF is pretty much designed to allow the UI designer to work seperately to the programmer. He could do it in NP as well as long as he knows his XML ;-) cheers Michael M -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Friday, 4 April 2008 3:00 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Thu Apr 3 18:53:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 03:53:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <006601c895e5$fa874e80$6401a8c0@nant> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 02:14:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 09:14:35 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From shamil at users.mns.ru Fri Apr 4 04:59:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 13:59:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <002d01c8963a$a3217e90$6501a8c0@nant> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 05:17:54 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 12:17:54 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Shamil Great! This is exactly the information I need. /gustav >>> shamil at users.mns.ru 04-04-2008 11:59 >>> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From wdhindman at dejpolsystems.com Thu Apr 10 09:11:34 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Thu, 10 Apr 2008 10:11:34 -0400 Subject: [dba-VB] ASP.NET, separated programming and visual design References: Message-ID: <002301c89b14$c8de1d30$7e6d6c4c@jisshowsbs.local> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 10 09:25:30 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 10 Apr 2008 16:25:30 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi William Fine, though it we be the other way around - I would make the mechanics first, then he would beef it up. With VSVWD8 Express as a free tool I see no reason why he should punish him self with something like Notepad. /gustav >>> wdhindman at dejpolsystems.com 10-04-2008 16:11 >>> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav From jwcolby at colbyconsulting.com Fri Apr 11 12:26:53 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:26:53 -0400 Subject: [dba-VB] Parsing strings Message-ID: <47FF9F5D.2070704@colbyconsulting.com> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Fri Apr 11 12:42:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 11 Apr 2008 19:42:18 +0200 Subject: [dba-VB] Parsing strings Message-ID: Hi John Why not use the Path class: extension = Path.GetExtension(FileName) basename = Path.GetFileNameWithoutExtension(FileName) /gustav >>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 11 12:49:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:49:18 -0400 Subject: [dba-VB] Parsing strings In-Reply-To: References: Message-ID: <47FFA49E.2020605@colbyconsulting.com> Lovely, thanks. Gustav Brock wrote: > Hi John > > Why not use the Path class: > > extension = Path.GetExtension(FileName) > basename = Path.GetFileNameWithoutExtension(FileName) > > /gustav > >>>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, -- John W. Colby www.ColbyConsulting.com From stuart at lexacorp.com.pg Fri Apr 11 18:45:55 2008 From: stuart at lexacorp.com.pg (Stuart McLachlan) Date: Sat, 12 Apr 2008 09:45:55 +1000 Subject: [dba-VB] Parsing strings In-Reply-To: <47FF9F5D.2070704@colbyconsulting.com> References: <47FF9F5D.2070704@colbyconsulting.com> Message-ID: <480084D3.12387.2763EDB7@stuart.lexacorp.com.pg> Left,Mid,Right etc are not function of the string object, they are functions in the Strings module of the Microsoft.VisualBasic namespace. i.e., fully qualified it is Microsoft.VisualBasic.Strings.Right() On 11 Apr 2008 at 13:26, jwcolby wrote: > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > From jwcolby at colbyconsulting.com Sat Apr 12 15:23:49 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 16:23:49 -0400 Subject: [dba-VB] Count a character in a string Message-ID: <48011A55.40402@colbyconsulting.com> I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 15:50:25 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 15:50:25 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From Gustav at cactus.dk Sat Apr 12 16:33:25 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 12 Apr 2008 23:33:25 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 12 17:48:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 18:48:26 -0400 Subject: [dba-VB] Count a character in a string In-Reply-To: References: Message-ID: <48013C3A.9030500@colbyconsulting.com> I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 18:00:38 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:00:38 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: Message-ID: True, don't use .Net much though. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Saturday, April 12, 2008 4:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] Count a character in a string Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From DWUTKA at Marlow.com Sat Apr 12 18:01:05 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:01:05 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48013C3A.9030500@colbyconsulting.com> Message-ID: Ya, I think I posted that solution a while back on AccessD. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 5:48 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Count a character in a string I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From shamil at users.mns.ru Sat Apr 12 21:30:24 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 06:30:24 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: <053b01c89d0e$53c595e0$6401a8c0@nant> Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:08:59 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:08:59 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> References: <48011A55.40402@colbyconsulting.com> <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <003c01c89d46$03525ca0$8119fea9@LTVM> John, You may recall the RegExp that I posted a short while ago. You can use this to get the answer. This code returns 17: Function CountRegEx() Dim strIn As String, lngCount As Long strIn = "asdf||||||qwert|zxcv|mnbv|qwert|zxcv|mnbv||||||" lngCount = (RegExpTest("\|", strIn)) MsgBox lngCount End Function Here is the original Function Function RegExpTest(patrn, strng) As Long ' Set reference to Microsoft vbscript regular expressions 5.5 ' Cheat Sheet available at http://regexlib.com/CheatSheet.aspx Dim RegEx As New VBScript_RegExp_55.RegExp Dim Match, Matches, RetStr ' Create variable. Set RegEx = New RegExp ' Create a regular expression. RegEx.Pattern = patrn ' Set pattern. RegEx.IgnoreCase = True ' Set case insensitivity. RegEx.Global = True ' Set global applicability. Set Matches = RegEx.Execute(strng) ' Execute search. RegExpTest = 0 For Each Match In Matches ' Iterate Matches collection. RetStr = RetStr & "Match found at position " RetStr = RetStr & Match.FirstIndex & ". Match Value is '" RetStr = RetStr & Match.Value & "'." & vbCrLf RegExpTest = RegExpTest + 1 Next End Function HTH Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 3:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:14:06 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:14:06 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: References: <48013C3A.9030500@colbyconsulting.com> Message-ID: <003d01c89d46$ba370a60$8119fea9@LTVM> John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max From shamil at users.mns.ru Sun Apr 13 04:26:17 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 13:26:17 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <054b01c89d48$6d543360$6401a8c0@nant> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 13 04:47:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 11:47:21 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi Shamil Neat! I'll add that to my toolbox. /gustav >>> shamil at users.mns.ru 13-04-2008 11:26 >>> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Sun Apr 13 05:08:53 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 14:08:53 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <003d01c89d46$ba370a60$8119fea9@LTVM> Message-ID: <054f01c89d4e$60f41d00$6401a8c0@nant> Hi Max, Our postings crossed - I have just posted RegEx test results... N.B.: when testing this or that method both memory consumption and execution speed should be taken into consideration: - high memory consumption giving speediest results can be neglected to solve small tasks within "tiny" applications/utilities, but - high memory consumption is the thing, which could "unexpectedly" give nasty side effects in the large application systems: One example I have got several months ago, which was so "tricky" to watch, and was not easy to find why it could ever happen in .NET: I have got a memory leakage/excessive memory consumption (in .NET!) because of "lazy loading", and that latter "lazy loading" loaded quite some data when it wasn't needed, in the cycle processing millions of records - the results were like that when I watched 'Page File Usage History' in Task Manager: /| /| /| . / | / | / | . / | / | / | . / |/ |/ |/ And in usual mode without heavy application system workload everything worked well with the same data... Recap: - Split(...) approach could result in similar to the above side effect for very large input strings; // 10 sec for 20,000,000 iterations string[] recordLine = s.Split('|'); count = recordLine.Length-1; - Replace(...) could also result in the above side effect; // 10 sec for 20,000,000 iterations string temp = s.Replace("|", Microsoft.VisualBasic.Constants.vbNullString); count = s.Length - temp.Length; - RegEx(...) seems to be the slowest - unsatisfactory slow for large input strings/heavy system workload; // ? sec (unfinished) for 20,000,000 iterations Regex rx = new Regex("|"); count = rx.Matches(s).Count; - char array iteration using char index and (XOR or char comparison) gives the fastest results and is 100% safe from memory consumption point of view.... // * using XOR: // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; // * using char comparinson... // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; Please correct me if you'll find the above results have mistakes... Any takers to find quiker code for .NET VB or C# or C++/CLI? - that would be a good weekend exercise on code optimization techniques... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Max Wanadoo Sent: Sunday, April 13, 2008 1:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Johncliviger at aol.com Sun Apr 13 13:36:20 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 14:36:20 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Gustav at cactus.dk Sun Apr 13 13:50:38 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 20:50:38 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Pick only the fields you need ... or do you wish to display 30 fields? /gustav >>> Johncliviger at aol.com 13-04-2008 20:36 >>> Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Johncliviger at aol.com Sun Apr 13 14:08:13 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 15:08:13 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Gustav at cactus.dk Mon Apr 14 08:53:40 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 14 Apr 2008 15:53:40 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Many ways to do this - depends a bit on your previous code and what features you need. A lot of examples can be found via Google for this - including Microsoft's pages. Key words could be master/child or parent/child. /gustav >>> Johncliviger at aol.com 13-04-2008 21:08 >>> Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Johncliviger at aol.com Mon Apr 14 09:23:22 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Mon, 14 Apr 2008 10:23:22 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Gustav Many thanks I've cracked it. Quite simple when you know how. When you try it for this first time your Access thinking leads you astray. Cheers Gustav john c From Gustav at cactus.dk Tue Apr 15 07:28:29 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 15 Apr 2008 14:28:29 +0200 Subject: [dba-VB] Web colours Message-ID: Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav From accessd at shaw.ca Tue Apr 15 14:03:05 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:03:05 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From accessd at shaw.ca Tue Apr 15 14:28:06 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:28:06 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: <945F14E6828D4E92AB0736A26485C259@creativesystemdesigns.com> Hi Gustav: I should have read right through your question before I, with a sandwich in hand, responded. Use any colour you want. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Tuesday, April 15, 2008 12:03 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Web colours Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 16 08:44:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 16 Apr 2008 15:44:13 +0200 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Message-ID: Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24/microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav From accessd at shaw.ca Wed Apr 16 16:00:03 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Wed, 16 Apr 2008 14:00:03 -0700 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) In-Reply-To: References: Message-ID: <2751B45E193C418490D807B02A37C79A@creativesystemdesigns.com> Thanks for your heads up Gustav. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Wednesday, April 16, 2008 6:44 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24 /microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From paul.hartland at googlemail.com Fri Apr 18 02:49:18 2008 From: paul.hartland at googlemail.com (Paul Hartland) Date: Fri, 18 Apr 2008 08:49:18 +0100 Subject: [dba-VB] OT - Free Twain ActiveX Control Message-ID: <38c884770804180049n76b46424y6d66cf4124792ced@mail.gmail.com> To all, I need to develop an application, which will involve scanning some documents etc. Does anyone know of or use a Twain ActiveX Control (preferably easy to use & most of all free, or very cheap) Thank you for any help on this..... -- Paul Hartland paul.hartland at googlemail.com From jwcolby at colbyconsulting.com Sat Apr 19 09:05:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 10:05:31 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <4809FC2B.6030602@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 19 10:08:14 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 19 Apr 2008 17:08:14 +0200 Subject: [dba-VB] NetGender - The Software Company Message-ID: Hi John One method would be to have a MDI parent form from which you open the other forms as child forms. Then you just to either keep track of what you open or to loop through the child forms collection to see if it contains a certain child form. In VS2008 the Add New wizard will happily create a new MDI parent form with basic functionality. /gustav >>> jwcolby at colbyconsulting.com 19-04-2008 16:05 >>> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 11:08:13 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 12:08:13 -0400 Subject: [dba-VB] Splitting a name Message-ID: <480A18ED.6020009@colbyconsulting.com> I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 12:27:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 13:27:25 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <480A2B7D.6000202@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sat Apr 19 13:51:12 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sat, 19 Apr 2008 14:51:12 -0400 Subject: [dba-VB] NetGender - The Software Company References: <4809FC2B.6030602@colbyconsulting.com> Message-ID: <000301c8a24e$57223eb0$60606c4c@jisshowsbs.local> ...nice looking library ...let us know what problems you run into with it ...I may well be in the market for such myself in the near future. William ----- Original Message ----- From: "jwcolby" To: "VBA" Sent: Saturday, April 19, 2008 10:05 AM Subject: [dba-VB] NetGender - The Software Company >I purchased a .Net library that does name splitting. > > http://www.softwarecompany.com/dotnet/netgender.htm > > It seems to work well just from using the little demo that you can > download. Now I am starting to actually work with it. It consists of a > single class, embedded in a dll that you reference, with properties to > set to tell it all of the controlling stuff, then properties to hand it > the name to split and read back the name pieces. > > The developer claims that it will handle between 2K and 3K names per > second. We shall see. > > So here I am trying to work with this thing. The demo form is quite > extensive and the code in the form shows how to read / write all of the > properties to populate the form. > > I have a database with (of course) about 80 million names, in a single > field, in the form LastName FirstName MiddleInitial (no commas etc) > which needs parsing. What I thought I would do (just for the moment) is > to use the existing form and just feed in the names. Thus I need to add > a couple of properties to the form such that I can feed in names and get > back the parsed pieces. I will build a switch to tell the form to > display the pieces or not, thus I could watch the process function if I > wanted, though at the cost of slowing the process down. > > Modifications to the demo form to allow me to do all that stuff is just > about complete. What I need to know is the syntax to get a pointer to a > form being opened. I know that you just do a "Someform.Show" to open > the form, but a parent form needs to get a pointer to this demo form and > then use that pointer to set / retrieve properties. How do I get a > pointer to it once opened? > > -- > John W. Colby > www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From max.wanadoo at gmail.com Sun Apr 20 07:32:53 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 20 Apr 2008 13:32:53 +0100 Subject: [dba-VB] Splitting a name In-Reply-To: <480A18ED.6020009@colbyconsulting.com> References: <480A18ED.6020009@colbyconsulting.com> Message-ID: <002101c8a2e2$a84d3580$8119fea9@LTVM> John: No doubt there are many more cleverer answers then this and this may be tooooo slow for your needs. Function jc() ' will take the 1st series up to space as Xname, right series back to space as zname and bit in the middle as yname Const conQuote As String = """" Dim sql As String sql = "update tblTest set " sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & conQuote & ")-1) ," sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") - InStr(searchname, " & conQuote & " " & conQuote & "))," sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " & conQuote & ")+1)" CurrentDb.Execute (sql) End Function Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 19, 2008 5:08 PM To: VBA Subject: [dba-VB] Splitting a name I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 20 10:29:34 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 20 Apr 2008 11:29:34 -0400 Subject: [dba-VB] Splitting a name In-Reply-To: <002101c8a2e2$a84d3580$8119fea9@LTVM> References: <480A18ED.6020009@colbyconsulting.com> <002101c8a2e2$a84d3580$8119fea9@LTVM> Message-ID: <480B615E.5040903@colbyconsulting.com> Max, The problems with names is that there are many variations and you cannot count on simple rules like that. For example Mr. Juan Miguel de la Uariate. Now you get another list with De la Uriarte Juan Miguel Sr. You get the picture I purchased a library to do the split. The library performs the split nicely, handling such variations quite well. What I need now is a strategy for reading rows or groups of rows, updating a single record at a time back to SQL Server. In the absence of experience I am going to open a chunk of records,, perhaps 10K, with a startPK and EndPK to track the chunks into an ADO recordset. I will read in 10K records (or some chunk size), then just iterate that recordset splitting each name. My first attempt will be to dynamically build up an update query to write EACH RECORD back out, one at a time. I am almost certain that this will not be fast enough but we shall see. Once I have the code I will probably move the sql into a udf and call the udf passing the name parts in to the udf which will then write one record back to the table. Also likely not fast enough but we shall see. Last will be an attempt to build an ado recordset in memory, then have ADO write the entire recordset back to SQL Server. If that is not fast enough I have run out of ideas. Max Wanadoo wrote: > John: > No doubt there are many more cleverer answers then this and this may be > tooooo slow for your needs. > > Function jc() > ' will take the 1st series up to space as Xname, right series back to > space as zname and bit in the middle as yname > Const conQuote As String = """" > Dim sql As String > sql = "update tblTest set " > sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & > conQuote & ")-1) ," > sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " > & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") > - InStr(searchname, " & conQuote & " " & conQuote & "))," > sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " > & conQuote & ")+1)" > CurrentDb.Execute (sql) > End Function > > Max > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 19, 2008 5:08 PM > To: VBA > Subject: [dba-VB] Splitting a name > > I have a table with eight million records, with a single name field, and I > have a library that splits the name into components. What I need now is a > generic method of getting the name out of a table and updating a specific > set of fields back in the table. It has to do so QUICKLY since I have to > update 80 million names. > > In this case I have a single table > > Name > > NamePrefix > LastName > FirstName > MiddleName > NameSuffix > > Name already exists and contains a name in a consistent format LName FName > MName. > > The other fields I have added to my table. > > So what I am looking for is a general strategy. I assume I will open a read > recordset with the PKID and the name field, then iterate through this > recordset performing the split. > > Once I do that should I: > > Build a dynamic SQL Update query and execute that? > > Build a stored procedure that I pass the PKID and the name parts to which > then updates the data? > > Use an ADO recordset to do this? > > some other method entirely? > > I am assuming at this point that because of the sheer size of the data I do > not want to try and do this in an ado recordset. Eight million records > sitting in data at one time does not sound fun. > > All ideas gratefully accepted. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Johncliviger at aol.com Tue Apr 22 04:40:48 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 05:40:48 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Gustav at cactus.dk Tue Apr 22 05:57:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 12:57:51 +0200 Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi John For #3 I would use a DataTableAdapter to attach the table of the database - and then feed your data to a DataTable of the DataTableAdapter. /gustav >>> Johncliviger at aol.com 22-04-2008 11:40 >>> Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Johncliviger at aol.com Tue Apr 22 06:05:04 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 07:05:04 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi Gustav Yes. TableAdapter. I'm on the right lines then, which good news. Cheers gustav From Gustav at cactus.dk Tue Apr 22 10:51:23 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 17:51:23 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 11:10:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 18:10:18 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all I located this article which will be of interest for those with an Access background: Flat-MultiColumn Combobox with Autocomplete http://www.codeproject.com/KB/architecture/MultiColumnFlatCombo.aspx /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 17:06:50 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 00:06:50 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From cfoust at infostatsystems.com Tue Apr 22 17:55:36 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Tue, 22 Apr 2008 15:55:36 -0700 Subject: [dba-VB] No flat ComboBox in .Net?? In-Reply-To: References: Message-ID: We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 23 01:26:27 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 08:26:27 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Wed Apr 23 16:31:09 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 23:31:09 +0200 Subject: [dba-VB] Custom Access 2007 ribbons with Visual Studio 2008 Message-ID: Hi all Noticed this tip from FMS: Create Custom Microsoft Office 2007 Ribbons with the New Visual Designer in Visual Studio 2008 http://www.fmsinc.com/free/NewTips/NET/CreateCustomOffice2007Ribbons.asp With the new release of Visual Studio Tools for Office (VSTO) in Visual Studio 2008, you can now use a visual designer to create your own custom Ribbons in Microsoft Access, Excel, Word, Outlook, etc. These Ribbons are encapsulated as add-ins for each of the specific Office 2007 products. If you wanted to build your own custom Ribbons, but have held back by the lack of a visual designer, take a look at Visual Studio 2008. The visual designer is available under the New Project, Office, Office 2007 dialog. More resources: http://msdn2.microsoft.com/en-us/library/bb386089.aspx /gustav From mmattys at rochester.rr.com Wed Apr 23 19:58:58 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Wed, 23 Apr 2008 20:58:58 -0400 Subject: [dba-VB] No flat ComboBox in .Net?? References: Message-ID: <01c801c8a5a6$6201d2e0$0502a8c0@Laptop> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" To: Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From Gustav at cactus.dk Thu Apr 24 02:31:43 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:31:43 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Michael That's a nice tip. Thanks. Lots of other stuff on that site ... /gustav >>> mmattys at rochester.rr.com 24-04-2008 02:58 >>> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" < Gustav at cactus.dk > To: < dba-vb at databaseadvisors.com > Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? From Gustav at cactus.dk Thu Apr 24 02:38:08 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:38:08 +0200 Subject: [dba-VB] GDI+ graphics in .Net Message-ID: Hi all This site of Bob Powell which Michael pointed out has some very good explanation and code examples on pictures, graphics and drawing in .Net: http://www.bobpowell.net/faqmain.htm He also has a very good introduction. That info can be hard to find around the web. Remember: Graphics are fun! /gustav From jwcolby at colbyconsulting.com Fri Apr 25 11:31:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 12:31:25 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: <4812075D.6000306@colbyconsulting.com> OK, I have been badgered and goaded into coming into the 21st century. Not to mention I can't figure out how to safely and effectively share development of my .Net programs on all of my machines. So I am trying to get Subversion installed and functioning. So of course I come here seeking advice or to start a user group (goad and prod you guys) if no one is using this thing. I installed VisualSVN and it told me I had to install TortoiseSVN to get full functionality. It implied that it would install Subversion as well. The install did not ask any "install" questions such as where do you want the database to go etc. My intention is to place the Subversion database on one of my servers, share the directory, then place the various clients (TortoiseSVN and VisualSVN on each machine which I develop on. So my first question, is anyone out there using this thing? Is anyone interested in using this and sharing the pain of figuring it out? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Fri Apr 25 12:25:08 2008 From: robert at webedb.com (Robert L. Stewart) Date: Fri, 25 Apr 2008 12:25:08 -0500 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <200804251729.m3PHTFbD020137@databaseadvisors.com> Been through the pain and suffering already. I much prefer Sourcesafe. This was the tool used on one of the contracts I was on. They were always having problems with it. It is open source, so there was not what you would really call technical support. Good luck. At 12:00 PM 4/25/2008, you wrote: >Date: Fri, 25 Apr 2008 12:31:25 -0400 >From: jwcolby >Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >To: VBA >Message-ID: <4812075D.6000306 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >OK, I have been badgered and goaded into coming into the 21st century. >Not to mention I can't figure out how to safely and effectively share >development of my .Net programs on all of my machines. > >So I am trying to get Subversion installed and functioning. So of >course I come here seeking advice or to start a user group (goad and >prod you guys) if no one is using this thing. > >I installed VisualSVN and it told me I had to install TortoiseSVN to get >full functionality. It implied that it would install Subversion as >well. The install did not ask any "install" questions such as where do >you want the database to go etc. > >My intention is to place the Subversion database on one of my servers, >share the directory, then place the various clients (TortoiseSVN and >VisualSVN on each machine which I develop on. > >So my first question, is anyone out there using this thing? Is anyone >interested in using this and sharing the pain of figuring it out? > >-- >John W. Colby >www.ColbyConsulting.com > > >------------------------------ > >_______________________________________________ >dba-VB mailing list >dba-VB at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-vb > > >End of dba-VB Digest, Vol 54, Issue 18 >************************************** From jwcolby at colbyconsulting.com Fri Apr 25 12:41:23 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 13:41:23 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <200804251729.m3PHTFbD020137@databaseadvisors.com> References: <200804251729.m3PHTFbD020137@databaseadvisors.com> Message-ID: <481217C3.7020105@colbyconsulting.com> Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 13:27:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Fri, 25 Apr 2008 22:27:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481217C3.7020105@colbyconsulting.com> Message-ID: <000901c8a701$f4afd8b0$6401a8c0@nant> Hi John, I did use Subversion for some time, then I abandoned it because customer didn't insist on using any source control tools, and because I'm still working mainly as one man shop... ...I do plan to restart using Subversion hopefully in the near future - I wanted to be 'in' in this discussion and participate or at least lurk it because of a lot of work here, and not that much time left for pleasure of AccessD discussions... ...I did also use TortoiseSVN... ...I do agree that MS support is usually useless... ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as well as I do plan to add to that "bundle" MS SoundCastle, NCover and some other useful tools... ...one or another bug/issues/features tracking/managing tools should be used - I do use Axosoft's OnTiome2008 now... ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS Build + NUnit working together and making automated builds and (unit) test runs - that's the only way to go for teamwork development these days... ...all that mentioned above stuff worked well here but I must admit it was not that easy to make the parts working in ensemble but when everything was tuned it happened to be relatively easy to handle stuff... ...I have had two large ASP.NET applications with probably several (1,2,3 ? I didn't count) thousands of source files and about 15 projects in two solutions to put into Subversion code base, automate compilation/building using CruiseCOntrol.NET and MS Build etc... ...the only issue I can remember (after everything was set) is that I didn't find how to easily manage the cases to synchronize central code db with local code versions when source files are moved from one folder into another within the same project - I could have missed something because of lack of time, and experience or that could be a real "bottleneck" of Subversion with everything else being very good I think... ...I'd participate in preparing and publishing on AccessD web site together with other interested AccessD members an instruction/manual and set of scripts and utilities on how to set up and effectively use together Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say how much time I will have for this work... Thank you. -- Shamil P.S. As far as I know Subversion "beats" MS Source Safe for the cases when the project development should be fork-ed, and then several branches merged back: the difference between Subversion and MS SourceSafe is conceptual here IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I can be wrong, please correct me then, thank you. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Fri Apr 25 14:35:03 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 15:35:03 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000901c8a701$f4afd8b0$6401a8c0@nant> References: <000901c8a701$f4afd8b0$6401a8c0@nant> Message-ID: <48123267.7030608@colbyconsulting.com> Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 25 15:06:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 16:06:31 -0400 Subject: [dba-VB] Reading / writing SQL Server rows with ADO.Net (or something) Message-ID: <481239C7.2030506@colbyconsulting.com> I need to read blocks of data from tables in SQL Server, use values in the rows then update those rows and write back to SQL Server. I need to do this programmatically, not in a control on a form. does anyone have any example code for setting up the various ADO.Net objects, fetching rows of data from SQL Server, updating that data and writing it back to SQL Server? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:20:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:20:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000001c8a711$bdda92c0$6401a8c0@nant> John, Everything in Subversion can be controlled by using batch files - that is the best approach (IMO) for the initial setup, bulk checking and working set checkout: if I do not miss anything then I will post in the following several e-mails under this topic the set of .bat files I used to create the test environment last autumn(three projects ProjectA, ProjectB and ProjectC). Stay tuned (after posting I will be out here until Sunday). Thanks. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 11:35 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:38:38 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:38:38 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000401c8a714$5762dae0$6401a8c0@nant> Hi John, Below is the set of .batch files I have just found, which I did make last autumn to control/direct subversion server. I must say I'm not sure this is the last working version I used (AFAIKR I did write some simple custom code in .NET to control subversion server to avoid using .bat files, which have to be made for every project you'll use - and my simple custom code was parameterized to use any source project) but as you can see the set of batch files below is the minimal set to fulfill all the source control main tasks. Watch line wraps and read subversion docs for more details - svn-book.pdf - located in Sunversion server's installation folder (I'm out until Sunday). Thanks. -- Shamil P.S. .bat files to use with Subversion !svn_admin_main.bat ===================== rem Port: 3690 rem RepositoryPath: e:\Subversion\svn_repository rem X Install svnserve to run as a Windows service rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server call make_dirs.bat new_repository call check_out.bat new_repository pause make_dirs.bat ============== rmdir tmpdir /s /q mkdir tmpdir mkdir tmpdir\projectA mkdir tmpdir\projectA\trunk mkdir tmpdir\projectA\branches mkdir tmpdir\projectA\tags mkdir tmpdir\projectB mkdir tmpdir\projectB\trunk mkdir tmpdir\projectB\branches mkdir tmpdir\projectB\tags mkdir tmpdir\projectC mkdir tmpdir\projectC\trunk mkdir tmpdir\projectC\branches mkdir tmpdir\projectC\tags rem xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E rem rmdir /s /q ..\%1 call create_fs_repository.bat %1 rem svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log call verify_fs_repository.bat %1 call svnlook_fs_repository.bat %1 rmdir tmpdir /s /q rem rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA --message svnlog.log create_fs_repository.bat ======================== svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs xcopy ..\args\conf\*.* ..\%1\conf /Y /E goto :EOF verify_fs_repository.bat ======================== svnadmin verify ../%1 GOTO :EOF svnlook_fs_repository.bat ========================= svnlook info ../%1 goto :EOF check_out.bat ============= rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S mkdir E:\Temp\SV\WORKING_SET\ /Q /S svn checkout file:///E:/Subversion/%1/projectA/trunk E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log svn checkout file:///E:/Subversion/%1/projectB/trunk E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log svn checkout file:///E:/Subversion/%1/projectC/trunk E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log commit.bat ========== svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log backup_fs_repository.bat ======================== svnadmin hotcopy ../%1 ../backups/%1 GOTO :EOF From jwcolby at colbyconsulting.com Fri Apr 25 17:15:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 18:15:27 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000401c8a714$5762dae0$6401a8c0@nant> References: <000401c8a714$5762dae0$6401a8c0@nant> Message-ID: <481257FF.30208@colbyconsulting.com> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 26 06:52:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 13:52:35 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil I would love to follow this thread. I work both at home and at the office and is a bit tired of copying files around. I located this page: Source Control with Visual Studio .NET http://www.codeproject.com/KB/dotnet/SourceControl_VSNET.aspx?fid=471385&df=90&mpp=25&noise=3&sort=Position&view=Quick&fr=26#xx0xx which, however, seems to know nothing about VS2008. It has links to these add-ins for VS: Garry Broadsword's Add-on which is updated for VS2008: http://garrys-brain.blogspot.com/2007/07/tortoisesvn-and-visual-studio.html and VisualSVN: http://www.visualsvn.com/ This site also sports VisualSVN Server: http://www.visualsvn.com/server/ VisualSVN Server is a package that contains everything you need to install, configure and manage Subversion server for your team on Windows platform. It includes Subversion, Apache and a management console. ... And last, but not least - VisualSVN Server is completely free! This must be what John is after - and myself as neither I nor any colleague have much experience with Apache. Now, big question: Does this replace or supplement Shamil's collection of batch files? As I understand it, VisualSVN Server should take care of the setup of Subversion but I must admit I have not yet grasped all the bits and pieces of this ... A final note for others that may follow this thread: Nothing of this works with the free Visual Studio Express versions. /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil From Gustav at cactus.dk Sat Apr 26 07:04:22 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 14:04:22 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil Exciting. http://www.visualsvn.com/doc/getting-started.html Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? Have never heard of Google Code. Can I use Google to store a project? http://code.google.com/ How does that work and is it only for public access project? Or how would you control access for closed source projects? /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. From jwcolby at colbyconsulting.com Sat Apr 26 07:11:54 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 08:11:54 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <48131C0A.1000302@colbyconsulting.com> the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 26 14:25:56 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 15:25:56 -0400 Subject: [dba-VB] ADO.Net Message-ID: <481381C4.7000804@colbyconsulting.com> I have been setting up an application to read a block of records, perhaps 10K or something. I designed a class that will load a block of records which consist of a name to be parsed, and a set of 6 fields which will be filled in with the parsed data returned from the name parser. Thus as I load the data only the name to parse is filled in. The supervisor iterates this set of records, pulling the name to parse, passing it off to the name parser, then reading the data back from the name parser and updating the fields in the same record. Move to the next record, wash, rinse repeat. I do have a couple of questions though. I am not very familiar with how the ADO.net stuff works. I build a connection string I build a connection object which I leave not connected (for now), but feeding it the connection string. I build a sql statement that represents the fields to pull WHERE PK between StartValue and EndValue. I build a command object which I pass SQL statement and the still unconnected connection object. I build a data adapter which I feed the command object. Notice that we are still not connected to the data store. I build a DataTable object with a table name. I call DataAdapter(DataTable).Fill Does .Net then connect correctly and proceed to fill the data table? Once I have the data table filled I start to iterate the DataTable using the for each DataRow iterator I then call the name splitter passing in the name to be split. Back comes the name pieces. I fill in the row with the name pieces and move to the next row. Once every row has been filled in I need to call an update method of the DataAdapter which is SUPPOSED to write the changes back to the table in SQL Server..... The name splitter handles ~1K names / second, NOT as speedy as I had hoped. At this point I haven't a clue how fast the data adapter will write back to the table in SQL Server. What I am trying to do is write this thing such that each instance of the data class can then be handed off to a thread so that the write back to SQL Server can begin and I can instantiate another data class to load / parse the next "chunk" of data in parallel with the write to SQL Server. I am trying to discover whether I need to close connections to the database while I do the parsing or just not worry about it. Should I just leave all the objects open and close them all down as the data class closes once the write to SQL Server is done? How will I know the write is done? Does the code "hang" at the DataAdapter.Update until the entire update (all the rows) is finished? Am I going about this the right way? Should I become a used car salesman? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <481381C4.7000804@colbyconsulting.com> Message-ID: <000501c8a7ea$a4d53080$6401a8c0@nant> Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481257FF.30208@colbyconsulting.com> Message-ID: <000401c8a7ea$a4c949a0$6401a8c0@nant> Hi John, Yes, I'm considering to purchase it also very probably... ...I will wait how well it will work for you - OK :) ? -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 2:15 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48131C0A.1000302@colbyconsulting.com> Message-ID: <000301c8a7ea$a4af0ae0$6401a8c0@nant> <<< running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever >>> John, The batches I posted here are just to work with Subversion's repository: create it, check-in, check-out, backup... ...to run builds and automate unit testing etc. CruiseControl.NET has to be used, or other similar tools... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 4:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:39:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:39:45 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <000601c8a7ee$6d9119a0$6401a8c0@nant> Hi John, Additionally to the below e-mail - you can consider using SQLDataReader to get data records batches as in this below example, then build parsed names' arrays together with ids, then Join() these arrays into strings and pass these strings as parameters to SPs, which will make actual updates by parsing strings into dynamic update sqls or something like that - T-SQL parsing functions are reasonably quick, or you can even consider managed SPs to parse the strings on server side and perform the updates... ...but that latter could be an overkill giving very little gains because .NET framework's communication with native MS SQL provider when using plain simple C#/VB.NET is very effective I have heard... ..maybe you can just process your data and generate and run update temp SPs to have update T-SQL statements batched into such temp SPs - that could give you (much) quicker updates... ...etc... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 2:13 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 18:31:32 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 19:31:32 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> References: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <4813BB54.4090203@colbyconsulting.com> Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 18:47:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 03:47:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813BB54.4090203@colbyconsulting.com> Message-ID: <000001c8a7f7$e871c530$6401a8c0@nant> <<< I am definitely interested in doing strongly typed recordsets though. >>> John, That strongly typed recordsets are imposing big overhead - you can dig into the code generated (by VS) for these recordsets... ...as for batch updates "automagically" generated by ADO.Net - you can use SQL profiler to see what happens "under the hood"... ...I'd bet that if you generate batch updates by custom code into temp SPs and use datareader to get data to build these batches then this approach will result in a way faster data processing, and you will have many options to optimize it even more... ...just my opinion but as you can find in ASP.Net apps ADO.NET typed recordsets are not recommended to use - have a look at e.g. "ASP.NET Website Programming: Problem - Design - Solution C# Edition" by Marco Bellinaso.... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 3:32 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 19:45:47 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 20:45:47 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a7f7$e871c530$6401a8c0@nant> References: <000001c8a7f7$e871c530$6401a8c0@nant> Message-ID: <4813CCBB.3060907@colbyconsulting.com> This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx There are many things to consider when I program in .net, not the least of which is simplicity. If I can get a simple method that gets me medium efficiency, it is better (for me) than a complex method that gets high efficiency simply because I may be able to implement the simple method whereas the complex method would never get done. I can always come back later and recode for a higher efficiency after I have a method working and chugging away. As long as the program does not take weeks to run, it is better to spend cpu time than John Colby time. As I mentioned, the name parser is not super fast. I am hoping to get the first database update batch happening in parallel with the second name parsing batch. I also hope to run this on a completely different computer which has more cores (and memory) so that I can use threads effectively. Which is not to say I am not interested in the most efficient method, simply that I will likely be unable to code it very quickly. Shamil Salakhetdinov wrote: > > <<< > I am definitely interested in doing strongly typed recordsets though. > John, > > That strongly typed recordsets are imposing big overhead - you can dig into > the code generated (by VS) for these recordsets... > > ...as for batch updates "automagically" generated by ADO.Net - you can use > SQL profiler to see what happens "under the hood"... > > ...I'd bet that if you generate batch updates by custom code into temp SPs > and use datareader to get data to build these batches then this approach > will result in a way faster data processing, and you will have many options > to optimize it even more... > > ...just my opinion but as you can find in ASP.Net apps ADO.NET typed > recordsets are not recommended to use - have a look at e.g. "ASP.NET Website > Programming: Problem - Design - Solution C# Edition" > by Marco Bellinaso.... > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 3:32 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Thanks Shamil, > > Unfortunately unless you are dealing with 80 million row 200 field > tables, any timing comparisons are going to be suspect. > > I considered doing (and may still do) a record by record update using > either a stored procedure or dynamic sql right from vb.net. Then I read > an article (blog) that essentially said that the batch update from an > ado dataset really did work as you would hope that it would, and this in > a blog from a member of Microsoft's ADO.Net dev team. > > He claimed that either of the other methods had network transaction > overhead as each update (when done singly) required the command to do it > as well as a result coming back, whereas if you used a batch it would > group all of the commands out and then group all the results coming back > into big packages. > > Understand that I do not know enough to even comment on the validity of > his argument however he said to definitely try it. Set the batch size > and then let ADO just send batches of updates. > > Given that the coding effort is smaller to do that I figured I would at > least get that running. > > I am definitely interested in doing strongly typed recordsets though. > > Shamil Salakhetdinov wrote: >> Hi John, >> >> Try to use SqlDataReader and custom classes as in the sample code below >> (sorry C# is used - real pleasure to use C# is also coming from the fact >> that you can copy and paste code and do not care about line warps because >> even "screwed by e-mail" code should be compiled well).... >> >> ...using ADO.NET datasets and bulk update will anyway result in a series > of >> updates on SQL server side therefore custom on-the-fly built sql updates > or >> calls to update SPs should work as quick as ADO.NET DataSets' batch >> update... >> >> ...etc... >> >> ...as you can see my sample did update 25000 records in ~10 seconds - and >> this only for starters - if that approach will work reasonably well for > you >> then the next step could be to introduce multi-threading etc... >> >> ...note also how MS SQL 2005's paging feature is used to get record's >> batches... > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 04:08:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:08:08 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813CCBB.3060907@colbyconsulting.com> Message-ID: <000001c8a846$3606c6d0$6401a8c0@nant> Hello John, OK, I will read the blog you mentioned later today... I know it's not easy to convince you to change your mind when you are "driving your way full speed" :) And I'm not trying to - I'm just musing in this coding samples I post here testing this or that approach, which I might use in my own development... <<< There are many things to consider when I program in .net, not the least of which is simplicity. >>> May I say that proposed in C# solution is simple and almost ready to use in your scenario in one thread? And also ready to be converted into a multi-threading app/utility. And you don't need to translate in manually in VB.NET - just use this free tool: http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx or this one: http://codeconverter.sharpdevelop.net/Convert.aspx and voila' the converted VB.NET is below : (watch line wraps now - this is VB.NET - it was "automagically" converted by referred above tool by copying and pasting C# code I posted yesterday + making some manual editing of several code line to make it independent of my testing/development environment) - and it seems to work even quicker than C# equivalent somehow !?. I have got additional back VB.NET -> C# conversion in the end of this message - you can use C# code copy and paste it in the tool refereed above and get VB.NET code properly formatted. Proposal: let's post C# code here for the folks who will try this code to not spend time with fixing line wraps introduced by e-mail clients): Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) ID = id Name = name End Sub Public ReadOnly Property ParsedName() As String Get ' just for testing purposes ' return real parsed name here Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry '27.04.2008 12:45:58 'Count(*) = 25045 'Counter = 25045 '27.04.2008 12:46:04 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = _ String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", _ pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list p.Update(item) count += 1 Next End While Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace Here it's converted back to C#: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB"; string dbName = "dbName"; string userName = "sa"; string password = "sapwd"; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 4:46 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx <<< tail skipped to get message through in AccessD >>> From Gustav at cactus.dk Sun Apr 27 04:33:02 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 11:33:02 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From shamil at smsconsulting.spb.ru Sun Apr 27 04:42:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:42:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: Message-ID: <000601c8a84b$0711cdc0$6401a8c0@nant> Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sun Apr 27 05:53:36 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 14:53:36 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000601c8a84b$0711cdc0$6401a8c0@nant> Message-ID: <000301c8a854$f1c57cf0$6401a8c0@nant> ...now the VB.net version of manually batched updates (watch line wraps!) It seems to work slightly quicker than similar C# version: ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 'http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx 'http://codeconverter.sharpdevelop.net/Convert.aspx Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) id = id name = name End Sub Public ReadOnly Property ParsedName() As String ' just for testing purposes ' return real parsed name here Get Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Public ReadOnly Property BatchSize() As Integer Get Return _batchSize End Get End Property Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() 'error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function Private _updateCommands As String() = Nothing Private _arrayIndex As Integer = -1 Public Sub AddUpdateCommand(ByVal item As DataItem) If _arrayIndex = -1 Then _updateCommands = DirectCast(Array.CreateInstance(GetType(String), _batchSize), String()) _arrayIndex = 0 End If Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) 'badly translated => _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql _updateCommands(_arrayIndex) = sql _arrayIndex += 1 End Sub Public Function BatchUpdate() As Integer If _arrayIndex = -1 Then Return -1 End If If _arrayIndex < _batchSize Then Array.Resize(Of String)(_updateCommands, _arrayIndex) End If Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() 'Console.WriteLine("Batch update: {0} updates", _arrayIndex) Dim sql As String = [String].Join(";", _updateCommands) _arrayIndex = -1 cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list If count > 0 AndAlso count Mod p.BatchSize = 0 Then p.BatchUpdate() End If 'p.Update(item); p.AddUpdateCommand(item) count += 1 Next End While p.BatchUpdate() Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 1:43 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From jwcolby at colbyconsulting.com Sun Apr 27 07:12:19 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 08:12:19 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a846$3606c6d0$6401a8c0@nant> References: <000001c8a846$3606c6d0$6401a8c0@nant> Message-ID: <48146DA3.1010200@colbyconsulting.com> LOL, now that I can readi it... ;-) >> and voila' the converted VB.NET is below : (watch line wraps now - this is Shamil Salakhetdinov wrote: > Hello John, > > OK, I will read the blog you mentioned later today... > > I know it's not easy to convince you to change your mind when you are > "driving your way full speed" :) And I'm not trying to - I'm just musing in > this coding samples I post here testing this or that approach, which I might > use in my own development... > > <<< > There are many things to consider when I program in .net, not the least > of which is simplicity. > May I say that proposed in C# solution is simple and almost ready to use in > your scenario in one thread? And also ready to be converted into a > multi-threading app/utility. And you don't need to translate in manually in > VB.NET - just use this free tool: > > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > > or this one: > > http://codeconverter.sharpdevelop.net/Convert.aspx > > and voila' the converted VB.NET is below : (watch line wraps now - this is > VB.NET - it was "automagically" converted by referred above tool by copying > and pasting C# code I posted yesterday + making some manual editing of > several code line to make it independent of my testing/development > environment) - and it seems to work even quicker than C# equivalent somehow > !?. I have got additional back VB.NET -> C# conversion in the end of this > message - you can use C# code copy and paste it in the tool refereed above > and get VB.NET code properly formatted. Proposal: let's post C# code here > for the folks who will try this code to not spend time with fixing line > wraps introduced by e-mail clients): > > Imports System > Imports System.Collections.Generic > Imports System.Data.SqlClient > > Namespace TestSamples > > Public Class DataItem > Public ID As Integer > Public Name As String > > Public Sub New(ByVal id As Integer, ByVal name As String) > ID = id > Name = name > End Sub > > Public ReadOnly Property ParsedName() As String > Get > ' just for testing purposes > ' return real parsed name here > Return Name > End Get > End Property > End Class > > Public Class DataProcessor > Private Const COUNT_SQL As String = "select count(*) from > zsysUserSessionTrace" > Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + > " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as > int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " > UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum > between {0} and {1}" > Private Const UPDATE_SQL As String = "update zsysUserSessionTrace > set UserBrowser = '{0}' where UserSessionTraceId = {1}" > > Private Const DEFAULT_BATCH_SIZE As Integer = 1000 > > Private _totalCount As Integer = -1 > Public ReadOnly Property TotalCount() As Integer > Get > Return _totalCount > End Get > End Property > > Private _currentBatchStartRowNum As Integer = 0 > Private _batchSize As Integer = 0 > > Private _connectionString As String > Public Sub New(ByVal connectionString As String) > initialize(connectionString, DEFAULT_BATCH_SIZE) > End Sub > Public Sub New(ByVal connectionString As String, ByVal batchSize As > Integer) > initialize(connectionString, batchSize) > End Sub > Private Sub initialize(ByVal connectionString As String, ByVal > batchSize As Integer) > > _batchSize = batchSize > _currentBatchStartRowNum = 1 > _connectionString = connectionString > > Using cnn As New > System.Data.SqlClient.SqlConnection(connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > cmd.CommandText = COUNT_SQL > cmd.CommandType = System.Data.CommandType.Text > > _totalCount = CInt(cmd.ExecuteScalar()) > End Using > End Sub > > Public ReadOnly Property HasData() As Boolean > Get > Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < > TotalCount) > End Get > End Property > > Public Function GetData() As List(Of DataItem) > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > _currentBatchStartRowNum += _batchSize + 1 > Dim rdr As System.Data.SqlClient.SqlDataReader = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) > > Dim list As New List(Of DataItem)() > While rdr.Read() > list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), > String))) > End While > Return list > End Using > End Function > > Public Function Update(ByVal item As DataItem) As Integer > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(UPDATE_SQL, > item.ParsedName, item.ID) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > Return cmd.ExecuteNonQuery() > End Using > End Function > End Class > > > Public Class A_MainEntry > '27.04.2008 12:45:58 > 'Count(*) = 25045 > 'Counter = 25045 > '27.04.2008 12:46:04 > > Public Shared Sub Main() > Dim count As Integer = 0 > Try > '+ customize - set your connection string here > Dim pcName As String = "SPB" > Dim dbName As String = "dbName" > Dim userName As String = "sa" > Dim password As String = "sapwd" > Dim connectionString As String = _ > String.Format("Data Source={0}\SQLEXPRESS;Initial > Catalog={1};User Id={2};Password={3}", _ > pcName, dbName, userName, password) > '- > > Console.WriteLine(DateTime.Now) > Dim p As New DataProcessor(connectionString) > While p.HasData > Dim list As List(Of DataItem) = p.GetData() > For Each item As DataItem In list > p.Update(item) > count += 1 > Next > End While > Console.WriteLine("Count(*) = {0}", p.TotalCount) > Console.WriteLine("Counter = {0}", count) > Console.WriteLine(DateTime.Now) > Catch ex As Exception > Console.WriteLine(ex.Message) > End Try > End Sub > > End Class > End Namespace > > Here it's converted back to C#: > > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > using System; > using System.Collections.Generic; > using System.Data.SqlClient; > > namespace TestSamples > { > > public class DataItem > { > public int ID; > public string Name; > > public DataItem(int id, string name) > { > ID = id; > Name = name; > } > > public string ParsedName > { > // just for testing purposes > // return real parsed name here > get { return Name; } > } > } > > public class DataProcessor > { > private const string COUNT_SQL = "select count(*) from > zsysUserSessionTrace"; > private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " > select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) > as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " > + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and > {1}"; > private const string UPDATE_SQL = "update zsysUserSessionTrace set > UserBrowser = '{0}' where UserSessionTraceId = {1}"; > > private const int DEFAULT_BATCH_SIZE = 1000; > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(1), (string)rdr(3))); > list.Add(new DataItem((int)rdr[1], (string)rdr[3])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(UPDATE_SQL, item.ParsedName, > item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > } > > > public class A_MainEntry > { > // VB.NET > //27.04.2008 13:01:44 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:49 > > // C# > //27.04.2008 13:00:56 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:04 > > public static void Main() > { > int count = 0; > try > { > //+ customize - set your connection string here > string pcName = "SPB"; > string dbName = "dbName"; > string userName = "sa"; > string password = "sapwd"; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > Console.WriteLine(DateTime.Now); > DataProcessor p = new DataProcessor(connectionString); > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > Console.WriteLine(DateTime.Now); > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 4:46 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > This is the blog I mentioned: > > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > <<< tail skipped to get message through in AccessD >>> > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 10:14:00 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:14:00 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48146DA3.1010200@colbyconsulting.com> Message-ID: <001401c8a879$5298bd70$6401a8c0@nant> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil From shamil at smsconsulting.spb.ru Sun Apr 27 10:24:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:24:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <000001c8a87a$cfa80270$6401a8c0@nant> ...additionally to my quoted posting here is SQL script to create test table: CREATE TABLE [dbo].[zsysUserSessionTrace]( [UserSessionTraceId] [int] IDENTITY(1,1) NOT NULL, [AspNetSessionId] [nvarchar](80) NOT NULL, [HostAddress] [nvarchar](40) NULL, [HostName] [nvarchar](40) NULL, [PageViewCount] [int] NULL, [UserName] [nvarchar](40) NOT NULL, [UserBrowser] [nvarchar](255) NOT NULL, [UpdatedBy] [nvarchar](20) NOT NULL, [UpdatedDate] [datetime] NOT NULL, [UpdatedLocation] [nvarchar](20) NOT NULL, [ParsedUserBrowserName] [nvarchar](255) NULL, CONSTRAINT [PK_zsysUserSessionTrace] PRIMARY KEY CLUSTERED ( [UserSessionTraceId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] ...and here is insert startemet to populate one row - you can run it many times: insert into zsysUserSessionTrace (AspNetSessionId, UserName, UserBrowser, UpdatedBy, UpdatedDate, UpdatedLocation) values (1,'TEST','IE7','SA','2008-04-27','SPB') ... and here are select statements to run e.g. in MS SQL Management console to see that test updates really happened: select count(*) from zsysUserSessionTrace go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId] desc) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 7:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 27 11:42:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 18:42:13 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil I'm following this thread with great interest because it touches some tasks I may soon encounter. It's a pleasure to study your work, and I really appreciate that you take your time to post it here. /gustav >>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? From jwcolby at colbyconsulting.com Sun Apr 27 11:56:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 12:56:40 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> References: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <4814B048.6070704@colbyconsulting.com> What I come hear is that using the built-in batch is as fast as any other method, so if that is what I find easier to use there is no issue using that. I will try to understand your other methods (in VB, since the language makes very little apparent speed difference) and implement that in parallel so that I can switch between them on demand. If I can do that I will do some timings on my big database, which is what I am doing the updates on. Shamil Salakhetdinov wrote: > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > I wasn't aware of MS SQL TDS feature you mentioned but as you can see from > stats it gives just slightly better results than the manual batching. Of > course playing with batch size might give better gains of this method (my > tests didn't show that gains depending on batch size) but I must say for me > it looks more complicated and less flexible that using SQLDatareader and > custom classes - that could be just my perception.... > > The C# code is below in the end of this message - its VB.NET equivalent can > be obtained using automatic conversion - > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one > code line have to be manually patched > > 'conversion issue replace the following code line => > ==================================================== > _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra > yIndex), _arrayIndex - 1)) = sql > > with this one > ============= > _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 > : > > Here are the stats: > > ==================== > C# > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:02:58 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:15 > Elapsed time: 00:00:16.7968750 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:03:15 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:23 > Elapsed time: 00:00:08.4218750 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:03:23 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:31 > Elapsed time: 00:00:07.9062500 > > ==================== > VB.NET > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:05:20 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:36 > Elapsed time: 00:00:16.7500000 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:05:36 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:45 > Elapsed time: 00:00:08.2812500 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:05:45 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:52 > Elapsed time: 00:00:07.5000000 > ==================== > > // Converters: > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > // Sql TDS batched update > //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > using System; > using System.Collections.Generic; > using System.Data; > using System.Data.SqlClient; > > namespace TestSamples > { > public sealed class Constants > { > //+ ///////////// C U S T O M I Z E ///////////////////// > public const string PcName = "NANT"; > public const string DbName = "ORS2005"; > public const string UserName = "sa"; > public const string Password = "shms"; > > public const string TABLE_NAME = "zsysUserSessionTrace"; > public const string ID_COL_NAME = "UserSessionTraceId"; > public const string NAME_COL_NAME = "UserBrowser"; > public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; > > public const int DEFAULT_BATCH_SIZE = 1000; > > public static string GetParsedName(string name) > { > // just for testing purposes - customzie > // to return your parsed name here > if (name == null) name = ""; > name = DateTime.Now.ToString() + ": " + name; > if (name.Length > 255) name = name.Substring(0, 255); > > return name; > } > //- ///////////// C U S T O M I Z E ///////////////////// > > public const int ID_COL_INDEX = 1; > public const int NAME_COL_INDEX = 2; > public const int PARSED_NAME_COL_INDEX = 3; > > public static string COUNT_SQL { get { > return String.Format("select count(*) from {0}", TABLE_NAME); } > } > > public static string PAGED_SELECT_SQL { get { > return string.Format( > "SELECT * from " + "(" + " select " + > " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as > RowNum, " + > " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", > ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, > PARSED_NAME_COL_NAME, TABLE_NAME) + > "where s.RowNum between {0} and {1}"; }} > > public static string UPDATE_SQL { get { > return string.Format("update {0} ",TABLE_NAME) + > string.Format(" set {0} = '{1}' where {2} = {3}", > PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); > }} > > public static string UPDATE_SQL_PARAMETERIZED { get { > return string.Format("update {0} set {1} = @{2} where {3} = > @{4}", > TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, > ID_COL_NAME, ID_COL_NAME); > }} > } > > public class DataItem > { > private int _id; > public int ID > { > get { return _id; } > } > > private string _name; > public string Name > { > get { return _name; } > } > > public DataItem(int id, string name) > { > _id = id; > _name = name; > } > > public string ParsedName > { > get > { > return Constants.GetParsedName(_name); > } > } > > > } > > public class DataProcessor > { > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > public int BatchSize { get { return _batchSize; } } > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = Constants.COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(Constants.ID_COL_INDEX), > (string)rdr(Constants.NAME_COL_INDEX))); > list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], > (string)rdr[Constants.NAME_COL_INDEX])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private string[] _updateCommands = null; > private int _arrayIndex = -1; > public void AddUpdateCommand(DataItem item) > { > if (_arrayIndex == -1) > { > _updateCommands = > (string[])Array.CreateInstance(typeof(string), _batchSize); > _arrayIndex = 0; > } > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > _updateCommands[_arrayIndex++] = sql; > } > > public int BatchUpdate() > { > if (_arrayIndex == -1) return -1; > if (_arrayIndex < _batchSize) > Array.Resize(ref _updateCommands, _arrayIndex); > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > > //Console.WriteLine("Batch update: {0} updates", > _arrayIndex); > > string sql = String.Join(";", _updateCommands); > _arrayIndex = -1; > > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private SqlDataAdapter _da = null; > private DataSet _ds = null; > public DataTable SQLTDSGetDataTablePaged() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > int batchSize = _batchSize; > if (_currentBatchStartRowNum + _batchSize >= _totalCount) > batchSize = _totalCount - _currentBatchStartRowNum; > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > > _currentBatchStartRowNum += _batchSize + 1; > > _da = new SqlDataAdapter(sql, cnn); > _ds = new DataSet(); > _da.Fill(_ds); > > _da.UpdateCommand = new > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); > _da.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, > Constants.PARSED_NAME_COL_NAME); > _da.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); > _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; > _da.UpdateBatchSize = batchSize; > > _da.SelectCommand.Connection = null; > _da.UpdateCommand.Connection = null; > > return _ds.Tables[0]; > } > } > > public void SQLTDSUpdate() > { > SQLTDSUpdate(false); > } > public void SQLTDSUpdate(bool forceUpdate) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > _da.SelectCommand.Connection = cnn; > _da.UpdateCommand.Connection = cnn; > if (forceUpdate) > _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; > _da.Update(_ds.Tables[0]); > > //Console.WriteLine("*** Forced test exit ***"); > //System.Environment.Exit(100); > } > } > } > > public class A_MainEntry > { > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate and execute individual update SQL > statements > /// > public static void Strategy1(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > //Console.WriteLine("{0} - rows processed...", p.BatchSize); > > } > } > > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate batched update SQL statements, and > /// execute update batch when it has all update > /// statements added > /// > public static void Strategy2(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > if (count > 0 && (count % p.BatchSize) == 0) > p.BatchUpdate(); > p.AddUpdateCommand(item); > count += 1; > } > } > p.BatchUpdate(); > } > > /// > /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, > and > /// use SQL TDS bacthed updates facility > /// > public static void Strategy3(DataProcessor p, ref int count) > { > while (p.HasData) > { > DataTable table = p.SQLTDSGetDataTablePaged(); > > foreach (DataRow row in table.Rows) > { > row.SetModified(); > row[Constants.PARSED_NAME_COL_INDEX] = > Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); > count += 1; > } > p.SQLTDSUpdate(!p.HasData); > } > } > > public static void Main() > { > try > { > //+ customize - set your connection string here > string pcName = Constants.PcName; > string dbName = Constants.DbName; > string userName = Constants.UserName; > string password = Constants.Password; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > for (int i = 1; i <= 3; i++) > { > int count = 0; > DateTime startTime, endTime; > startTime = DateTime.Now; > Console.WriteLine(""); > Console.WriteLine("Strategy {0}:", i); > Console.WriteLine("-----------"); > Console.WriteLine("Start time: {0}", startTime); > DataProcessor p = new DataProcessor(connectionString); > > switch (i) > { > case 1: > Strategy1(p, ref count); > break; > case 2: > Strategy2(p, ref count); > break; > case 3: > Strategy3(p, ref count); > break; > default: > Console.WriteLine("{0} - undefined strategy > number", i); > break; > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > endTime = DateTime.Now; > Console.WriteLine("End time: {0}", endTime); > TimeSpan ts = > TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); > Console.WriteLine("Elapsed time: {0}", ts); > } > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > Enjoy! > > -- > Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sun Apr 27 14:13:04 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sun, 27 Apr 2008 15:13:04 -0400 Subject: [dba-VB] ADO.Net References: Message-ID: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Sun Apr 27 16:02:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 01:02:11 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> Message-ID: <000401c8a8a9$f695a200$6401a8c0@nant> Here are the sources I promised to publish in my previous post: http://smsconsulting.spb.ru/samples/Stats.txt http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt Your turn, guys, to fix and improve them... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of William Hindman Sent: Sunday, April 27, 2008 11:13 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 27 22:01:58 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 23:01:58 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000401c8a8a9$f695a200$6401a8c0@nant> References: <000401c8a8a9$f695a200$6401a8c0@nant> Message-ID: <48153E26.3040803@colbyconsulting.com> Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Mon Apr 28 01:52:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 08:52:21 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all What happened? How have I been fooling around with covered eyes? If I select Flat for property FlatStyle the ComboBox changes to ... a flat design. No visible borders. Only issue is that the invisible borders of the ComboBox appears to be as wide as if the ComboBox has 3D-style, thus leaving a larger total area than an equivalent TextBox with identical font settings. If I force the size of the ComboBox to be lower, it won't display the content (not very useful). /gustav >>> Gustav at cactus.dk 23-04-2008 08:26 >>> Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Mon Apr 28 10:53:55 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 17:53:55 +0200 Subject: [dba-VB] C# event handler syntax Message-ID: Hi all Just browsed this page: http://blueonionsoftware.com/Blog.aspx?p=8afe238f-87c2-400b-9440-8231280f7576 which has a brief explanation on the syntax from the beginning to the ultra compact of version 3.5 using Lambda expression: clock.Alarm += (o, e) => Console.Write("Alarm!"); This is _not_ what IntelliSense suggests, so you are a little on your own. /gustav From shamil at smsconsulting.spb.ru Mon Apr 28 11:40:25 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 20:40:25 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48153E26.3040803@colbyconsulting.com> Message-ID: <00c301c8a94e$8f660430$6401a8c0@nant> Hi John, Thank you for your proposal to run the code against your huge db... OK, let's try this remote/offline Q&A/code musing/consulting/education in several installments published here: I must say I have quite some work to do for customers and I can't spend a lot of time for postings here/explaining what and why is done in the sample code - I will try just give refs on "concepts" I use for you to read more in many open sources... I made first adjustment to the yesterday's code - new code is here: http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt http://smsconsulting.spb.ru/samples/Stats.txt To run it (let's use VB.NET): - in VS2005 create VB.NET Console application, replace the code of the module, which VS2005 creates by default (Module1.vb) with the code from http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines Public Const PcName As String = "LOCALHOST" Public Const DbName As String = "NamesParser" Public Const UserName As String = "TESTER" Public Const Password As String = "TEST.1959" to point to your db and its credentials, push CTRL+F5, and you should see the sample app running in console window and producing something like this report: Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:13 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:19 Elapsed time: 00:00:05.5156250 Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:19 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:22 Elapsed time: 00:00:03.5468750 Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:22 Count(*) = 1000 Counter = 1009 End time: 28.04.2008 20:18:26 Elapsed time: 00:00:03.3906250 Try. If you tell me the name of your database and the name of your table I will change them here, as well as you can create username and password as above to not do any changes later for the new versions of the code. N.B: Make sure you will run the code against sample not the live db. The code assumes that the length of the target field FName is 50 chars - it uses just one this column currently to put "parsed" value - just first 50 chars of OWNERNAME field. Please have a look through the code, and ask your questions in the order you wanted them to be answered... Please everybody who wanted to participate in this discussion - you're very welcome! On debugging multi-threaded apps: that's a real PITA - yes you can debug/trace with breakpoints etc. but you have to keep in mind that as many threads as you have will cause debugger to stop on breakpoints etc. IOW better debug/trace single threaded code and them run it in multi-threaded mode, and use some indirect artefacts/techniques (as unit testing), which will show you that your app is running OK: as you can find in the new version of the code it can run both single and multi-threaded depending on the value of this constant: Public Const MULTI_THREADED_MODE As Boolean = True -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 7:02 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 11:55:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 12:55:08 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <00c301c8a94e$8f660430$6401a8c0@nant> References: <00c301c8a94e$8f660430$6401a8c0@nant> Message-ID: <4816016C.4010709@colbyconsulting.com> Thanks for the response Shamil, I know that you have lots of stuff to get done as do I. I am already back to the code I know in order to get this thing processed, however I will get your sample code running in the coming days. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > Thank you for your proposal to run the code against your huge db... > > OK, let's try this remote/offline Q&A/code musing/consulting/education in > several installments published here: I must say I have quite some work to do > for customers and I can't spend a lot of time for postings here/explaining > what and why is done in the sample code - I will try just give refs on > "concepts" I use for you to read more in many open sources... > > I made first adjustment to the yesterday's code - new code is here: > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > http://smsconsulting.spb.ru/samples/Stats.txt > > To run it (let's use VB.NET): > > - in VS2005 create VB.NET Console application, replace the code of the > module, which VS2005 creates by default (Module1.vb) with the code from > http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines > > Public Const PcName As String = "LOCALHOST" > Public Const DbName As String = "NamesParser" > Public Const UserName As String = "TESTER" > Public Const Password As String = "TEST.1959" > > to point to your db and its credentials, push CTRL+F5, and you should see > the sample app running in console window and producing something like this > report: > > Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:13 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:19 > Elapsed time: 00:00:05.5156250 > > Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:19 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:22 > Elapsed time: 00:00:03.5468750 > > Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:22 > Count(*) = 1000 > Counter = 1009 > End time: 28.04.2008 20:18:26 > Elapsed time: 00:00:03.3906250 > > Try. > > If you tell me the name of your database and the name of your table I will > change them here, as well as you can create username and password as above > to not do any changes later for the new versions of the code. > > N.B: Make sure you will run the code against sample not the live db. The > code assumes that the length of the target field FName is 50 chars - it uses > just one this column currently to put "parsed" value - just first 50 chars > of OWNERNAME field. > > Please have a look through the code, and ask your questions in the order you > wanted them to be answered... > > Please everybody who wanted to participate in this discussion - you're very > welcome! > > On debugging multi-threaded apps: that's a real PITA - yes you can > debug/trace with breakpoints etc. but you have to keep in mind that as many > threads as you have will cause debugger to stop on breakpoints etc. IOW > better debug/trace single threaded code and them run it in multi-threaded > mode, and use some indirect artefacts/techniques (as unit testing), which > will show you that your app is running OK: as you can find in the new > version of the code it can run both single and multi-threaded depending on > the value of this constant: > > Public Const MULTI_THREADED_MODE As Boolean = True > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 7:02 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Shamil, > > I would like to thank you for taking the time to do this. Now... I will > thank you in advance for taking the time to explain it to me. As > happened many years ago when you taught me WithEvents in Access, I don't > know enough to understand what the heck you are doing. > > In the absence of any instructions on how to use this thing, I cut and > pasted the entire (VB) shebang into a class in a Windows Form project. > It compiled flawlessly and the blank form opened. > > Now what? > > I hope that you can appreciate my lack of knowledge here. I have built > several projects, with (to my pitiful level of expertise) quite > extensive stuff going on but... there are a lot of concepts in your code > that I have not used. Where to start. > > I guess where to start is a good spot. What do I do to cause this to > run? Coming from Access I have always started with a form, which I > promptly place a button on so I can call some function which "starts" > things. That form will have class variables if necessary to instantiate > a supervisor class, the supervisor class will then load recordsets, load > data into data classes as required etc. So what I do not understand is > "what do I call to make this run"? I see the MainEntry at the bottom > but do I just call MainEntry from a button? Does the code belong in a > class or a plain module? I assume that if I put it in a class then I > would have to dim a variable in my form. > > Second (I am totally excited about this whole thing!!!) what are the > replaceable parameters? > > Dim connectionString As String = String.Format("Data > Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password) > > I assume here that pcName gets fed in to {0}? THAT is cool, and > demonstrates where I start from in understanding your code. > > Third (also exciting!) I have never used threads yet so here we go. How > do I debug that? Can I step through a thread in the same way I step > through any other code? Set break points etc. > > And finally (for now) it just occurred to me to ask... can you help me > (online of offline) to modify this example to actually use my own table > that I need to parse the name for. I think I can figure out how to > modify the stuff for the server, database, table etc. I have done that > already in my own code for doing this. I will also worry about the > parsing code itself, though just for a demo we can just hard code some > values. > > > My table Data: > > PKID (int32) > OWNERNAME (to be parsed) > FName (parsed) > MName (parsed) > LName (parsed) > NamePrefix (parsed) > NameSuffix (parsed) > Gender (parsed) > > If I can modify they example to actually read / write to my table then I > kill two birds with one stone, learn a bunch of cool new stuff including > getting it all happening with threads, and get my real work done. > > For my real life purposes I am going to have to have a PKStart, PKEnd > pair which tracks a block (remember I am doing 80 million rows > eventually), and increment those start/end by the chunk size to repeat > over and over (eventually). What I have found convenient in the past > (and have code already written to do) is write status log files out to > disk for each block processed. I write them as XML and include things > like the start / end PKID, status memo fields, time start / stop and the > like. As I said, I have already written that code (I'm not completely > helpless, though it sometimes appears that way) and sitting out in a > library ready to use for the logging. All I would need to do is build > the specific data class to store the log data for this project. I will > handle that. > > I will promise to absorb your example if you will promise to take the > time to help me when I get stuck. Once I have done so I will then be > able to apply it to my specific problem and give you back some real life > timings from my largish database. I can vary the chunk size from 1k to > any upward limit we find useful to test. My experience has been that > anything above 10K can get problematic, though that was using the bulk > import widget in past projects. I was getting timeouts if I went with > too large a chunk size. > > VB.Net is only a small part of my work life, though I would like to make > it the main part. I still spend most of my work life stuck in Access, > maintaining applications I wrote (or inherited) long ago. While it pays > the bills I would LOVE to get to the point where I could bang out code > like this in the short time that it probably took you. > > Again thanks for your time and effort and patience. > > Shamil Salakhetdinov wrote: >> Here are the sources I promised to publish in my previous post: >> >> http://smsconsulting.spb.ru/samples/Stats.txt >> >> http://smsconsulting.spb.ru/samples/Program.cs.txt >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> Your turn, guys, to fix and improve them... >> >> -- >> Shamil >> > From jwcolby at colbyconsulting.com Mon Apr 28 13:10:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:10:08 -0400 Subject: [dba-VB] ADO.Net data table update Message-ID: <48161300.8050807@colbyconsulting.com> I have a datatable that I opened. It contains a set of 10K records which I have updated pieces of. I now need to write it back to SQL Server. If I just call the da.update it errors saying it needs an update command. Do I have to build an update SQL statement or will the DA just figure out where the data fields came from and update them back out to SQL Server? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 13:49:38 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:49:38 -0400 Subject: [dba-VB] ADO.Net Update SQL Server Message-ID: <48161C42.1030500@colbyconsulting.com> Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Mon Apr 28 14:04:08 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Mon, 28 Apr 2008 12:04:08 -0700 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> References: <48161C42.1030500@colbyconsulting.com> Message-ID: UpdateBatch should work. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 11:50 AM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Mon Apr 28 14:09:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 23:09:11 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> Message-ID: <010001c8a963$57cd7c00$6401a8c0@nant> Hi John, The answer is in the code I published today on my site related to 'Re: [dba-VB] ADO.Net': http://smsconsulting.spb.ru/samples/Module.vb.txt - Strategy 3 - trace in single threaded mode and you'll find what to do. Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As DataSet) As DataTable ... And Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet As DataSet) .... HTH, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 10:50 PM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 14:57:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 15:57:15 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: References: <48161C42.1030500@colbyconsulting.com> Message-ID: <48162C1B.8060207@colbyconsulting.com> I hate to sound ignorant but what is the UpdateBatch a method of? I have a connection, command, dataset and datatable and AFAICT none of those things have an UpdateBatch method. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > UpdateBatch should work. > > Charlotte Foust > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 11:50 AM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 15:14:01 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 16:14:01 -0400 Subject: [dba-VB] ADO.Net Update SQL Server References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> Message-ID: <01d001c8a96c$676b5a70$0502a8c0@Laptop> It is a method of the ADO.Recordset. I was ignorant of this, also, until I looked it up in a book. Ignorant<>something you can sound like Either you know or you don't know :) Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 3:57 PM Subject: Re: [dba-VB] ADO.Net Update SQL Server >I hate to sound ignorant but what is the UpdateBatch a method of? > > I have a connection, command, dataset and datatable and AFAICT none of > those things have an UpdateBatch method. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> UpdateBatch should work. >> >> Charlotte Foust >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 11:50 AM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> >> -- >> John W. Colby >> www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 15:17:48 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:17:48 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <010001c8a963$57cd7c00$6401a8c0@nant> References: <010001c8a963$57cd7c00$6401a8c0@nant> Message-ID: <481630EC.5060600@colbyconsulting.com> When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > From jwcolby at colbyconsulting.com Mon Apr 28 15:29:52 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:29:52 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <01d001c8a96c$676b5a70$0502a8c0@Laptop> References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> <01d001c8a96c$676b5a70$0502a8c0@Laptop> Message-ID: <481633C0.9060808@colbyconsulting.com> ROTFL, Oh I think I succeed very well at sounding ignorant. 8-( And yup, I don't know. I have beaten my head against this all day with absolutely zero results. I find it difficult to believe that I am doing anything unique but I'll be damned if I can find an article or code that just says "here is how you write back the updated table to the sql server... (data store)". John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > It is a method of the ADO.Recordset. > I was ignorant of this, also, until I looked it up in a book. > Ignorant<>something you can sound like > Either you know or you don't know :) > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Monday, April 28, 2008 3:57 PM > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > >> I hate to sound ignorant but what is the UpdateBatch a method of? >> >> I have a connection, command, dataset and datatable and AFAICT none of >> those things have an UpdateBatch method. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Charlotte Foust wrote: >>> UpdateBatch should work. >>> >>> Charlotte Foust >>> >>> -----Original Message----- >>> From: dba-vb-bounces at databaseadvisors.com >>> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >>> Sent: Monday, April 28, 2008 11:50 AM >>> To: VBA >>> Subject: [dba-VB] ADO.Net Update SQL Server >>> >>> Does ANYBODY know how to write a table back out to SQL Server? >>> >>> All of my books show cute little examples of building up a data table >>> manually then BINDING it to a control or whatever. I am not finding >>> anything with code example that shows how to WRITE changed data back to >>> SQL Server. >>> >>> I have managed to load a data table, write changes to the rows and >>> now... I have to write it back out to SQL Server. The table has the PK, >>> NameToSplit, LName, FName etc etc. What am I missing? >>> >>> I desperately need help (yea, yea, not MENTAL help) and I need it >>> quickly. I have been trying to do this last piece for DAYS and I am not >>> succeeding. >>> >>> -- >>> John W. Colby >>> www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From James at fcidms.com Mon Apr 28 16:02:31 2008 From: James at fcidms.com (James Barash) Date: Mon, 28 Apr 2008 17:02:31 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> John: If you already have a data adapter that you used to fill your data table, you need to add an UpdateCommand to that data adapter and then use it to update the datastore. Below is a very quick and dirty version from an existing program I've used. It should at least give you a place to start. It assumes you already have a data adapter named da, a data table named dt and a connection named conn. You will have to change the table and field names to match yours. da.UpdateCommand = New SqlClient.SqlCommand da.UpdateCommand.Connection = conn da.UpdateCommand.CommandType = CommandType.Text da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldDisplayValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldNumber", System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.Update(dt) James Barash -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 4:18 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 17:11:07 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:11:07 -0400 Subject: [dba-VB] vb.net - Updates Message-ID: <48164B7B.2070808@colbyconsulting.com> I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 17:12:37 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:12:37 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> References: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> Message-ID: <48164BD5.60906@colbyconsulting.com> Thanks James. I also found a piece of code that shows how to use a commandbuilder object to generate those update commands automatically. At my skill level, automatic is a good thing. John W. Colby www.ColbyConsulting.com James Barash wrote: > John: > If you already have a data adapter that you used to fill your data table, > you need to add an UpdateCommand to that data adapter and then use it to > update the datastore. Below is a very quick and dirty version from an > existing program I've used. It should at least give you a place to start. It > assumes you already have a data adapter named da, a data table named dt and > a connection named conn. You will have to change the table and field names > to match yours. > > > da.UpdateCommand = New SqlClient.SqlCommand > da.UpdateCommand.Connection = conn > da.UpdateCommand.CommandType = CommandType.Text > da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, > FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", > "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldDisplayValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, > Nothing, "", "", "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldNumber", > System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, > "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", > "", "")) > da.Update(dt) > > James Barash > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 4:18 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > When I try to call the update method of the data set object I get an error: > > "Update requires a valid update command when passed datarow collection > with modified rows." > > I probably sound ungrateful but in your code you appear to set up a sql > statement for an update statement: > > adapter.UpdateCommand = New > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, > Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) > adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None > adapter.UpdateBatchSize = pageSize > > > My problem is that is just Chinese to me. I have no background > knowledge in plain English of what an update sql statement is supposed > to look like (in the ADO context) and absolutely no understanding of > even what I am trying to accomplish (expressed in plain English). I am > uneducated, and I can find no way to get educated. > > I must own 6 books on ADO.Net and VB and every single one of them > expects me to drag and drop objects out onto a form. OR they very > helpfully set everything up and show me how to do stuff RIGHT UP TO > USING IT, but never go that last step of sending updates back to the > database. Open, update, display in a grid, cleanup and close. Hmm.... > and I paid good money for these books. > > They do not discuss anything at all about what is going on at the code > level. So when I try to walk through your code it is simply > indecipherable to me. If I don't know what you are trying to accomplish > I can't understand how you code will accomplish that thing. > > Please do not take these statements as criticisms of your code, simply > that I am a freshman student still taking the basket weaving classes > looking at masters code. Thus it isn't useful to tell me to trace > through the code. > > MY code doesn't do any of that stuff. My code is pretty simple: > > Set up a connection, a data adapter, a table. Iterate the table row by > row, updating fields of the rows. > > Now... how to update the table back to SQL Server... > > Given how simple everything has been up to this point I have to believe > that I am just missing a concept and code to implement that concept. I > have called the update method and I have googled the error messages and > nothing... > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> The answer is in the code I published today on my site related to 'Re: >> [dba-VB] ADO.Net': >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> - Strategy 3 - trace in single threaded mode and you'll find what to do. >> >> Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, > ByVal >> pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As >> DataSet) As DataTable >> ... >> >> And >> >> Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet >> As DataSet) >> .... >> >> >> HTH, >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 10:50 PM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Mon Apr 28 18:31:14 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 19:31:14 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> References: <48164B7B.2070808@colbyconsulting.com> Message-ID: <48165E42.20404@colbyconsulting.com> We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. From ebarro at verizon.net Mon Apr 28 19:44:24 2008 From: ebarro at verizon.net (Eric Barro) Date: Mon, 28 Apr 2008 17:44:24 -0700 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> Message-ID: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Yeah it is a terrible thing from a DBA point of view because it opens it up to SQL injection attacks since it builds your SQL statement on the fly. I've tried it before and since it is generating it on the fly it makes it all the more harder to debug and offers the least amount of control from a DBA point of view. So...I let SQL server do all the work on the back end when it comes to batch updates by collecting the parameters and sending it to SQL server to process for the records it needs to process. Based on what you are trying to do you are building an image of the data structure in memory, manipulating the data in memory and then committing the changes to the database in memory basically sending the update command for each record to the database back end right? Or am I mistaken in thinking that all of this magically happens by sending a single update command containing several individual update commands strung together? -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 3:11 PM To: VBA Subject: [dba-VB] vb.net - Updates I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 20:08:20 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 21:08:20 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Message-ID: <48167504.6070909@colbyconsulting.com> It all happens automagically. At this point I modify about 10K records at a clip, then issue a single update and it goes. I set the batch size to 100 for this experiment but I will probably play with that to see what can be done. I tried a thousand but it timed out. I don't see any timeout period property though there might well be one somewhere. I understand the SQL Injection attack thing but you need to remember that this is a database running on my system that only I ever access. Even more importantly though, I need this kind of thing to just work, as quickly as I can make it happen. I am not a high powered DBA (nor any other powered DBA) nor a high powered .Net programmer. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Yeah it is a terrible thing from a DBA point of view because it opens it up > to SQL injection attacks since it builds your SQL statement on the fly. > > I've tried it before and since it is generating it on the fly it makes it > all the more harder to debug and offers the least amount of control from a > DBA point of view. So...I let SQL server do all the work on the back end > when it comes to batch updates by collecting the parameters and sending it > to SQL server to process for the records it needs to process. > > Based on what you are trying to do you are building an image of the data > structure in memory, manipulating the data in memory and then committing the > changes to the database in memory basically sending the update command for > each record to the database back end right? Or am I mistaken in thinking > that all of this magically happens by sending a single update command > containing several individual update commands strung together? > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 3:11 PM > To: VBA > Subject: [dba-VB] vb.net - Updates > > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece of > code that, in simple plain language, shows how to use it all, using the > simple objects I am already using. > > We shall see. > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 21:26:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 22:26:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> <48167504.6070909@colbyconsulting.com> Message-ID: <034b01c8a9a0$7b784440$0502a8c0@Laptop> These fit the bill ... How To: Protect From SQL Injection in ASP.NET http://msdn2.microsoft.com/en-us/library/ms998271.aspx Using Stored Procedures with SQLDataAdapter? http://forums.devx.com/showthread.php?t=148707 SQLDataAdapter without using SQLCommandBuilder http://www.codeproject.com/KB/database/sqldawithoutsqlcb.aspx Using Stored Procedures in Conjunction with the SqlDataAdapter http://www.code-magazine.com/Article.aspx?quickid=0303141 Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 9:08 PM Subject: Re: [dba-VB] vb.net - Updates > I understand the SQL Injection attack thing but you need to remember > that this is a database running on my system that only I ever access. From jwcolby at colbyconsulting.com Tue Apr 29 00:15:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 01:15:45 -0400 Subject: [dba-VB] Name parsing system timing Message-ID: <4816AF01.8090500@colbyconsulting.com> OK, it is running in a loop now, processing 10K records every 19 seconds. It will take 50 hours at the current rate to parse 96.4 million records, or about 2 million records / hour. the SQL Server update portion is taking about 5 seconds of each 19 second cycle, 14 seconds to parse, 5 seconds to write back to the database, virtually nothing to load. At least it is running now, making progress. Thanks for all the help from everyone getting this thing to happen. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 08:53:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 09:53:26 -0400 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: <48172856.9080502@colbyconsulting.com> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Tue Apr 29 09:39:45 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 16:39:45 +0200 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: Hi John That is what the DataTableAdapter is for. If you use the wizard for creating these to attach your tables, it will do all the plumbing and create a (giant) class which wraps all this for you taking care of relations, not null, max lengths, strong types, error handling etc. etc. not to say provide the basis for creating all the queries you need including ready-made basic queries for CRUD as well as giving you the choice to use stored procedures, pass-through or normal queries. All of this is controlled from a graphic layout like a relations view. If that isn't enough it also lets you carry on with Data Access Layers and business rules. Nothing is hidden or secret. You can browse the class should you have the patience and wish so. This is a snippet: private void InitClass() { this.columnKundeNr = new global::System.Data.DataColumn("KundeNr", typeof(int), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnKundeNr); this.columnNavn = new global::System.Data.DataColumn("Navn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnNavn); this.columnAdr1 = new global::System.Data.DataColumn("Adr1", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr1); this.columnAdr2 = new global::System.Data.DataColumn("Adr2", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr2); this.columnPostNr = new global::System.Data.DataColumn("PostNr", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnPostNr); this.columnBynavn = new global::System.Data.DataColumn("Bynavn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnBynavn); this.columnTlf = new global::System.Data.DataColumn("Tlf", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnTlf); this.Constraints.Add(new global::System.Data.UniqueConstraint("Constraint1", new global::System.Data.DataColumn[] { this.columnKundeNr}, true)); this.columnKundeNr.AllowDBNull = false; this.columnKundeNr.Unique = true; this.columnNavn.AllowDBNull = false; this.columnNavn.MaxLength = 50; this.columnAdr1.MaxLength = 50; this.columnAdr2.MaxLength = 50; this.columnPostNr.MaxLength = 50; this.columnBynavn.MaxLength = 50; this.columnTlf.MaxLength = 50; } If you redesign the table structure from within Visual Studio, it will track the changes and adjust the class accordingly. It is __so clever__ that I wonder why most are still moving on with SQL builders, hand built command strings and the like - pure waste of time in my opinion - except for programmers with special needs and knowledge like Shamil who knows what he is doing. William, I think, will amen this. I must admit that because of VS2008 I'm ready to forgive MS much of their bad doings. /gustav >>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 09:46:24 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 09:46:24 -0500 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: References: Message-ID: <200804291451.m3TEpqu6010208@databaseadvisors.com> Yes, that is correct. He is ending up loading the records into memory. Then passing individual updates back for each row of data that he changed. This method is not designed for doing large datasets like he is doing. It should be done on the SQL Server side. John, Just what are you updating...the UPDATE statement you would use to do what you are doing. Robert At 09:40 AM 4/29/2008, you wrote: >Date: Mon, 28 Apr 2008 17:44:24 -0700 >From: "Eric Barro" >Subject: Re: [dba-VB] vb.net - Updates >To: "'Discussion concerning Visual Basic and related programming > issues.'" >Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >Content-Type: text/plain; charset=us-ascii > >Yeah it is a terrible thing from a DBA point of view because it opens it up >to SQL injection attacks since it builds your SQL statement on the fly. > >I've tried it before and since it is generating it on the fly it makes it >all the more harder to debug and offers the least amount of control from a >DBA point of view. So...I let SQL server do all the work on the back end >when it comes to batch updates by collecting the parameters and sending it >to SQL server to process for the records it needs to process. > >Based on what you are trying to do you are building an image of the data >structure in memory, manipulating the data in memory and then committing the >changes to the database in memory basically sending the update command for >each record to the database back end right? Or am I mistaken in thinking >that all of this magically happens by sending a single update command >containing several individual update commands strung together? From wdhindman at dejpolsystems.com Tue Apr 29 10:26:31 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Tue, 29 Apr 2008 11:26:31 -0400 Subject: [dba-VB] Update gets string or binary truncation error References: Message-ID: ...sigh ...I look at some of the stuff Shamil posts and haven't a clue what he's doing, much less why ...waaaaaaaay over my head ...so much so I get a bad headache every time I try to read some of it ...but you definitely get an AMEN! ...and then some ...on VS2K8 ...when MS finally gets something right, its RIGHT!!! :) William -------------------------------------------------- From: "Gustav Brock" Sent: Tuesday, April 29, 2008 10:39 AM To: Subject: Re: [dba-VB] Update gets string or binary truncation error > Hi John > > That is what the DataTableAdapter is for. If you use the wizard for > creating these to attach your tables, it will do all the plumbing and > create a (giant) class which wraps all this for you taking care of > relations, not null, max lengths, strong types, error handling etc. etc. > not to say provide the basis for creating all the queries you need > including ready-made basic queries for CRUD as well as giving you the > choice to use stored procedures, pass-through or normal queries. > > All of this is controlled from a graphic layout like a relations view. If > that isn't enough it also lets you carry on with Data Access Layers and > business rules. > > Nothing is hidden or secret. You can browse the class should you have the > patience and wish so. This is a snippet: > > private void InitClass() { > this.columnKundeNr = new > global::System.Data.DataColumn("KundeNr", typeof(int), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnKundeNr); > this.columnNavn = new > global::System.Data.DataColumn("Navn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnNavn); > this.columnAdr1 = new > global::System.Data.DataColumn("Adr1", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr1); > this.columnAdr2 = new > global::System.Data.DataColumn("Adr2", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr2); > this.columnPostNr = new > global::System.Data.DataColumn("PostNr", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnPostNr); > this.columnBynavn = new > global::System.Data.DataColumn("Bynavn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnBynavn); > this.columnTlf = new global::System.Data.DataColumn("Tlf", > typeof(string), null, global::System.Data.MappingType.Element); > base.Columns.Add(this.columnTlf); > this.Constraints.Add(new > global::System.Data.UniqueConstraint("Constraint1", new > global::System.Data.DataColumn[] { > this.columnKundeNr}, true)); > this.columnKundeNr.AllowDBNull = false; > this.columnKundeNr.Unique = true; > this.columnNavn.AllowDBNull = false; > this.columnNavn.MaxLength = 50; > this.columnAdr1.MaxLength = 50; > this.columnAdr2.MaxLength = 50; > this.columnPostNr.MaxLength = 50; > this.columnBynavn.MaxLength = 50; > this.columnTlf.MaxLength = 50; > } > > If you redesign the table structure from within Visual Studio, it will > track the changes and adjust the class accordingly. > > It is __so clever__ that I wonder why most are still moving on with SQL > builders, hand built command strings and the like - pure waste of time in > my opinion - except for programmers with special needs and knowledge like > Shamil who knows what he is doing. > > William, I think, will amen this. > > I must admit that because of VS2008 I'm ready to forgive MS much of their > bad doings. > > /gustav > >>>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> > I am doing the update thing, updating data back to SQL Server. I have > been updating an ADO recordset in memory and when I try to update it > back to SQL Server it throws an error that the update failed because it > would cause a string value to be truncated. > > How is this possible. I thought that the table structure in memory > mirrored the structure in the data store (SQL Server in this instance). > Doesn't the read of the data out of SQL Service discover the field > size and type and build a similar structure in memory? > > Apparently not, but if not how do I handle a case like this? > -- > John W. Colby > www.ColbyConsulting.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 10:43:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 11:43:45 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <200804291451.m3TEpqu6010208@databaseadvisors.com> References: <200804291451.m3TEpqu6010208@databaseadvisors.com> Message-ID: <48174231.4070501@colbyconsulting.com> > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 12:13:02 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 12:13:02 -0500 (CDT) Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 Message-ID: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Trying to understand the scenario... 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Questions: 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? From: jwcolby Date: 2008/04/29 Tue AM 10:43:45 CDT > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? From jwcolby at colbyconsulting.com Tue Apr 29 12:48:28 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 13:48:28 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> References: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Message-ID: <48175F6C.5050804@colbyconsulting.com> > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? Correct. I have a 96 million record table, 155 fields. Of that I pull the PKID, NameToParse (not called that obviously) and then 6 "parsed name pieces" fields, lname, fname etc (which I added to the table to hold the pieces). > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Not quite correct. The vb.Net results are due to the fact that I am using a 3rd party library which does the parsing of the name. Name parsing is non-trivial to say the least. That library is a .Net object (DLL actually) which you set up parameters in and then pass in a name to parse and back comes 5 to 8 "pieces". In fact fluid results are the least of my worries in SQL Server, ANY results are more like it. > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? In fact I do but that is not necessary. The name parser came with a demo form and I just modified that to display every 1000th record (which I pass to the form every 1000th record in the table), just so I could "watch it". The form is not a necessity and I may or may not get rid of it in a later iteration (probably will). I have a form (I created) which has a ServerName, DatabaseName, TableName (being parsed), PKFldName, ParseFldName, StartID, EndID, and button to trigger the parse. I fill in the pieces and push the button. Since I have control of the tables, I "hard code" the "parsed parts" field names and they will (at least for now) always be those names. I have to add those fields to the source table anyway. > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? Correct. I grab X records into an ADO.Net DataTable, iterate that table row by row, passing the Unparsed name field off to the parser. I then write the resulting "Pieces" that the parser returns (properties of the parser object) into the matching fields in the ADO.Net DataTable. The name parser handles roughly 800 names / second and is the real bottle neck. I have a StartPK / EndPK which I then manipulate in a while loop, as I finish a "chunk" I write it to SQL Server, update the StartPK / EndPK and grab another chunk to process. I have to do this for 96 million names, however I also receive weekly updates to this table which I will also have to process in a similar manner before I append them to the table. The results of this will be used to send through Accuzip for Address validation / NCOA processing. Address validation can be done just on an address, but NCOA is change of address which is about a person at an address. Once that is done... there is more but you get the picture. And in fact there is another different table from the same data source that I will have to perform this same process on. It is highly likely that I will get other lists in the future with single name fields. The world is full of idiots. ;-) John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Trying to understand the scenario... > > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? > > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. > > Questions: > > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? > > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? > > > From: jwcolby > Date: 2008/04/29 Tue AM 10:43:45 CDT > > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > > Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > > ROTFL. That is a definite NSS. > > Unfortunately I am not a DBA and with the responses I get from the SQL > group I am not even going there. I can do this purely in VB.Net (it is > cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > > No idea. I thought I could find out from the DA after the command > builder was finished building the update statement but I am getting an > error if I try to access the update property. > > John W. Colby > www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 13:13:49 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 13:13:49 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804291817.m3TIH41W019987@databaseadvisors.com> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert At 12:00 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 11:43:45 -0400 >From: jwcolby >Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >To: "Discussion concerning Visual Basic and related programming > issues." >Message-ID: <48174231.4070501 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > >Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > >ROTFL. That is a definite NSS. > >Unfortunately I am not a DBA and with the responses I get from the SQL >group I am not even going there. I can do this purely in VB.Net (it is >cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > >No idea. I thought I could find out from the DA after the command >builder was finished building the update statement but I am getting an >error if I try to access the update property. > >John W. Colby From Gustav at cactus.dk Tue Apr 29 13:24:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 20:24:51 +0200 Subject: [dba-VB] SQL VS VB Message-ID: Hi Robert If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? /gustav >>> robert at webedb.com 29-04-2008 20:13 >>> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert From jwcolby at colbyconsulting.com Tue Apr 29 13:33:22 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:33:22 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804291817.m3TIH41W019987@databaseadvisors.com> References: <200804291817.m3TIH41W019987@databaseadvisors.com> Message-ID: <481769F2.2090303@colbyconsulting.com> It is going to take 50 hours because an external program which does the name parsing will take at least 40 hours just to parse the data. Writing to the database is only about 25% of the total time. If you managed to completely eliminate my database update time it would still take about 40 hours to complete. Thanks for the offer though. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > At 12:00 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 11:43:45 -0400 >> From: jwcolby >> Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >> To: "Discussion concerning Visual Basic and related programming >> issues." >> Message-ID: <48174231.4070501 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> > Yes, that is correct. He is ending up loading the records into memory. >> > Then passing individual updates back for each row of data that he >> > changed. >> >> Yes, and most records have updates. >> >> > This method is not designed for doing large datasets like he is doing. >> > >> > It should be done on the SQL Server side. >> >> ROTFL. That is a definite NSS. >> >> Unfortunately I am not a DBA and with the responses I get from the SQL >> group I am not even going there. I can do this purely in VB.Net (it is >> cranking as we speak) so I will. >> >> > Just what are you updating...the UPDATE statement you would use to >> > do what you are doing. >> >> No idea. I thought I could find out from the DA after the command >> builder was finished building the update statement but I am getting an >> error if I try to access the update property. >> >> John W. Colby > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 13:44:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:44:40 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <48176C98.8030201@colbyconsulting.com> > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? Precisely Gustav. What I WOULD like to do is spin off a thread to do the SQL Server update so the parser could go back to work on the next chunk. IOW put the update in parallel with the parser instead of in series with it. In fact if I had this running on the 4 core system I could probably call the parser DLL in several threads as well. In fact if I ran the VB.Net application on a separate machine, then the SQL Server machine could be just running updates and another machine running the application. What I REALLY need is a thousand blade mini-super that I could just run the entire thing in parallel. It would only take a few minutes then. ;-) John W. Colby www.ColbyConsulting.com Gustav Brock wrote: > Hi Robert > > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? > > /gustav > >>>> robert at webedb.com 29-04-2008 20:13 >>> > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <022401c8aa2b$786352f0$6401a8c0@nant> <<< Please do not take these statements as criticisms of your code >>> Hi John, No problem with criticism - in fact it's very welcome! :) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 12:18 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48165E42.20404@colbyconsulting.com> Message-ID: <022801c8aa2b$787b20b0$6401a8c0@nant> <<< ...it is definitely going to be desirable to feed the update process off to a thread... >>> Hi John, I was lucky with multi-threaded batch updates' samples I posted here the day before yesterday - and the new tests show that batch updates in multi-threading environments result in MS SQL deadlocks: I should have thought about those issues before... Recap: forget batch updates - one of the scenarios of your task could be: - many readers; - many parsers; - but just one updater making batch updates, or many updaters but updating just one row in one cycle... HTH to save your time, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 3:31 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 14:11:36 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 15:11:36 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022801c8aa2b$787b20b0$6401a8c0@nant> References: <022801c8aa2b$787b20b0$6401a8c0@nant> Message-ID: <481772E8.3050505@colbyconsulting.com> Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 14:43:53 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:43:53 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <30862132.1249691209498233814.JavaMail.root@vms124.mailsrvcs.net> I believe that Shamil was referring to multi-threaded updates. If you used the multi-threaded approach you could potentially end up slowing the process more than you could ever imagine because of the way SQL server locks pages of records when you run an update. From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:51:42 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:51:42 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <481772E8.3050505@colbyconsulting.com> Message-ID: <022c01c8aa32$7269c440$6401a8c0@nant> John, I'm not sure yet what happens< I will try to find out more details in the coming days - here is the message I'm currently getting with new Strategy 4 , which I'm making similar to your way of coding this task but running in multi-threading environment: +++ System.Data.SqlClient.SqlException: Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. --- Just wanted to warn you - it might happen you'll never see this message but if you'll see it then you may find there is no workaround except abandoning the idea of batch updates running in several threads - and I mean parallel batch updates: of course, as I noted in my previous e-mail parallel reading and parsing could be done but batch updates very probably can't be run in parallel threads because they might result in deadlocks... But it looks like (from your recent posting here) that you don't need such parallel batch updates at all? Thanks. Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 11:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 14:59:51 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:59:51 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 15:02:10 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 15:02:10 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <29420937.173771209499330766.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 15:13:29 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:13:29 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022c01c8aa32$7269c440$6401a8c0@nant> References: <022c01c8aa32$7269c440$6401a8c0@nant> Message-ID: <48178169.70002@colbyconsulting.com> Shamil, I have just moved the whole project to my quad core x64 server and am bringing it up now. If I can get that code running over on another machine (especially IF it is a faster machine) then the update part will be less of an issue. In any event the parse WAS taking about 15 seconds for 10K records on the dual core x32 machine which was also running SQL Server. I have designed this thing as a set of classes. There is a "data class" which is opened by a class that parses the names. The NameParser just writes the data directly into the ADO recordset in the data class. I MIGHT be able to just shuffle the class holding the open ado recordsets off into a collection. Another thread could pull those class instances out of the collection and call the update method which would start the update. As it finished the update, it would destroy that data class instance and pull the next one out of the collection. This would effectively separate the parsing and the updating, and each could proceed at it's own speed. I have not investigated yet but I might be able to have multiple threads doing the name parsing, and storing the resulting data class in that collection. It depends on whether the dll is reentrant I suppose and I really don't have a clue yet.. There would only be a single thread processing the dataset classes "update". I would need to watch that I didn't get too many open data classes at once time I suppose, though this machine has Windows x64 and 8 gigs of RAM so it has a little more breathing room than the Windows x32 machine I was running it on. Of course all of this is just "play". At least it is running now. I was also thinking about running two threads in "opposite directions", one processing data PK high to low, and the other processing data PK Low to High. Until they met in the middle that MIGHT avoid the deadlock thing. Again though, that is just play. NOT TO MENTION that I have never run a thread anyway so i have to figure THAT out. ;-) John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > I'm not sure yet what happens< I will try to find out more details in the > coming days - here is the message I'm currently getting with new Strategy 4 > , which I'm making similar to your way of coding this task but running in > multi-threading environment: > > +++ > System.Data.SqlClient.SqlException: Transaction (Process ID > 57) was deadlocked on lock resources with another process and has been > chosen as the deadlock victim. Rerun the transaction. > --- > > Just wanted to warn you - it might happen you'll never see this message but > if you'll see it then you may find there is no workaround except abandoning > the idea of batch updates running in several threads - and I mean parallel > batch updates: of course, as I noted in my previous e-mail parallel reading > and parsing could be done but batch updates very probably can't be run in > parallel threads because they might result in deadlocks... > > But it looks like (from your recent posting here) that you don't need such > parallel batch updates at all? > > Thanks. > > Shamil From jwcolby at colbyconsulting.com Tue Apr 29 15:18:50 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:18:50 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> References: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> Message-ID: <481782AA.6030204@colbyconsulting.com> Eric, I will read it. Unfortunately that puts me back in the SQL forum and you know how I ... uhh... well, let's just say I would rather muddle along in VB.Net. That is something that I can get better at and will spend enough time at to pay back my efforts. Besides... I come from the "just throw more hardware at it" school of Engineering. ;-) Seriously though, of course I'd LOVE to just pass the ADO recordset over and let SQL Server do it. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > John, > > This article might be of use in your scenario... > > http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 > > It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... > > "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " > > --Eric > > From: jwcolby > Date: 2008/04/29 Tue PM 02:11:36 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > Interesting. Do you think that you would get deadlocks for different > blocks of records? In my case there is not "a million users" scenario. > This process (and its threads) are the only things modifying data. > Each "chunk" is a defined block of records. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> <<< >> ...it is definitely going to be desirable to feed the update >> process off to a thread... >> Hi John, >> >> I was lucky with multi-threaded batch updates' samples I posted here the day >> before yesterday - and the new tests show that batch updates in >> multi-threading environments result in MS SQL deadlocks: I should have >> thought about those issues before... >> >> Recap: forget batch updates - one of the scenarios of your task could be: >> >> - many readers; >> - many parsers; >> - but just one updater making batch updates, or many updaters but updating >> just one row in one cycle... >> >> HTH to save your time, >> Shamil >> >> >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Tuesday, April 29, 2008 3:31 AM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> We have ignition folks. I just updated the first 10K records in the >> database. I had to set the batch size to 100, as I was getting a >> timeout with a batch size as small as 1000 records. I haven't done any >> timings but it is definitely going to be desirable to feed the update >> process off to a thread if I can figure out how to do that. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> jwcolby wrote: >>> I THINK I have found my solution, called the SQLCommandBuilder. I found >>> reference to the command build saying how terrible it is, then a piece >>> of code that, in simple plain language, shows how to use it all, using >>> the simple objects I am already using. >>> >>> We shall see. >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From robert at webedb.com Tue Apr 29 16:21:31 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 16:21:31 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804292127.m3TLRIWT012647@databaseadvisors.com> His original estimate was 50 hours. (not mine) But, he would have to do a bit of fancy coding in SQL Server to use the same DLL he is using. That bit of information was not in the first post. At 02:02 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 20:24:51 +0200 >From: "Gustav Brock" >Subject: Re: [dba-VB] SQL VS VB >To: >Message-ID: >Content-Type: text/plain; charset=US-ASCII > >Hi Robert > >If the parser dll runs at about 800 records/s, 93 mio. records will >need 33 hours as a minimum. Or? > >/gustav From jwcolby at colbyconsulting.com Tue Apr 29 16:28:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:28:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48178169.70002@colbyconsulting.com> References: <022c01c8aa32$7269c440$6401a8c0@nant> <48178169.70002@colbyconsulting.com> Message-ID: <4817930D.3080400@colbyconsulting.com> Well I shoulda done that sooner. With the process running on Azul (the quad core x64) machine and Stonehenge (dual core x32) just processing the SQL Server thing, the parse time dropped dramatically! I am now processing about 2800 - 2900 records / second. Interestingly the update time is about the same, ~5 seconds, but that is for ~80K records. I am wondering if I am timing what i think I am timing on the update thing. 8~/ PERHAPS there is some kind of "connection" time that is the majority of the time At any rate I will finish MUCH sooner, and that with no thread stuff yet. John W. Colby www.ColbyConsulting.com jwcolby wrote: > Shamil, > > I have just moved the whole project to my quad core x64 server and am > bringing it up now. If I can get that code running over on another > machine (especially IF it is a faster machine) then the update part will > be less of an issue. In any event the parse WAS taking about 15 seconds > for 10K records on the dual core x32 machine which was also running SQL > Server. > > I have designed this thing as a set of classes. There is a "data class" > which is opened by a class that parses the names. The NameParser just > writes the data directly into the ADO recordset in the data class. I > MIGHT be able to just shuffle the class holding the open ado recordsets > off into a collection. Another thread could pull those class instances > out of the collection and call the update method which would start the > update. As it finished the update, it would destroy that data class > instance and pull the next one out of the collection. > > This would effectively separate the parsing and the updating, and each > could proceed at it's own speed. > > I have not investigated yet but I might be able to have multiple threads > doing the name parsing, and storing the resulting data class in that > collection. It depends on whether the dll is reentrant I suppose and I > really don't have a clue yet.. > > There would only be a single thread processing the dataset classes "update". > > I would need to watch that I didn't get too many open data classes at > once time I suppose, though this machine has Windows x64 and 8 gigs of > RAM so it has a little more breathing room than the Windows x32 machine > I was running it on. > > Of course all of this is just "play". At least it is running now. > > I was also thinking about running two threads in "opposite directions", > one processing data PK high to low, and the other processing data PK Low > to High. Until they met in the middle that MIGHT avoid the deadlock > thing. Again though, that is just play. > > NOT TO MENTION that I have never run a thread anyway so i have to figure > THAT out. ;-) > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> John, >> >> I'm not sure yet what happens< I will try to find out more details in the >> coming days - here is the message I'm currently getting with new Strategy 4 >> , which I'm making similar to your way of coding this task but running in >> multi-threading environment: >> >> +++ >> System.Data.SqlClient.SqlException: Transaction (Process ID >> 57) was deadlocked on lock resources with another process and has been >> chosen as the deadlock victim. Rerun the transaction. >> --- >> >> Just wanted to warn you - it might happen you'll never see this message but >> if you'll see it then you may find there is no workaround except abandoning >> the idea of batch updates running in several threads - and I mean parallel >> batch updates: of course, as I noted in my previous e-mail parallel reading >> and parsing could be done but batch updates very probably can't be run in >> parallel threads because they might result in deadlocks... >> >> But it looks like (from your recent posting here) that you don't need such >> parallel batch updates at all? >> >> Thanks. >> >> Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 16:45:10 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:45:10 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804292127.m3TLRIWT012647@databaseadvisors.com> References: <200804292127.m3TLRIWT012647@databaseadvisors.com> Message-ID: <481796E6.7060300@colbyconsulting.com> That was indeed my original estimate. The PARSER was running at ~800 names / second. The UPDATE occurs after the parser finishes its thing., and is thus "in series with" (time wise) the parser. I just moved the VB.Net off to a more powerful machine and it is now running the parser at just over 3K records / second. I am in fact not parsing every record, I am actually processing ~80% of the records. Now I am off to see if I can figure out how to use threads. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > His original estimate was 50 hours. (not mine) > > But, he would have to do a bit of fancy coding in > SQL Server to use the same DLL he is using. That > bit of information was not in the first post. > > > > At 02:02 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 20:24:51 +0200 >> From: "Gustav Brock" >> Subject: Re: [dba-VB] SQL VS VB >> To: >> Message-ID: >> Content-Type: text/plain; charset=US-ASCII >> >> Hi Robert >> >> If the parser dll runs at about 800 records/s, 93 mio. records will >> need 33 hours as a minimum. Or? >> >> /gustav > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 19:03:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 20:03:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> Message-ID: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> I've fallen behind quite a bit, I see. Well, I'll catch up. Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 20:13:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 21:13:25 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> <030f01c8aa55$aa6384d0$0502a8c0@Laptop> Message-ID: <4817C7B5.6020005@colbyconsulting.com> > I've fallen behind quite a bit, I see. eh? John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > I've fallen behind quite a bit, I see. > Well, I'll catch up. > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 20:23:54 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 21:23:54 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> Message-ID: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Hi John, I was looking over Gustav's posts of Feb 20 with Charlotte and posts of this and nearby threads. I've been doing stuff with maps that didn't require use of large databases. The difference between an sqladapter and a tableadapter were unknown to me until today. C#, SQL Server 200X, ASP.Net All very compelling! Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Tuesday, April 29, 2008 9:13 PM Subject: Re: [dba-VB] vb.net - Updates > > I've fallen behind quite a bit, I see. > > eh? > > John W. Colby > www.ColbyConsulting.com > > > Michael R Mattys wrote: >> I've fallen behind quite a bit, I see. >> Well, I'll catch up. >> >> Michael R. Mattys >> MapPoint & Access Dev >> www.mattysconsulting.com >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 21:14:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 22:14:27 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Message-ID: <4817D603.9050308@colbyconsulting.com> Michael, > C#, SQL Server 200X, ASP.Net > All very compelling! Indeed they are. So much to know, so little time. I am slowly learning some of this, painfully slowly. John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > Hi John, > > I was looking over Gustav's posts of Feb 20 > with Charlotte and posts of this and nearby threads. > > I've been doing stuff with maps that didn't require use > of large databases. The difference between an sqladapter > and a tableadapter were unknown to me until today. > > C#, SQL Server 200X, ASP.Net > All very compelling! > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Tuesday, April 29, 2008 9:13 PM > Subject: Re: [dba-VB] vb.net - Updates > > >>> I've fallen behind quite a bit, I see. >> eh? >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Michael R Mattys wrote: >>> I've fallen behind quite a bit, I see. >>> Well, I'll catch up. >>> >>> Michael R. Mattys >>> MapPoint & Access Dev >>> www.mattysconsulting.com >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 23:54:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 00:54:18 -0400 Subject: [dba-VB] Worker threads and class global variables Message-ID: <4817FB7A.4000002@colbyconsulting.com> Can a backgroundworker access a class global variable in the class that dimensions the background worker. I have a process that will be creating data classes. these classes will fill ADO data sets, modify them and then place them in a collection global to the class. This same class would then dim a backgroundworker object. The way I envision this working is that the backgroundworker is initialized and started. It simply sits in a loop checking the class global collection for data class instances deposited into the collection by the main thread. If a class instance is found in the thread it pulls it out of the collection and processes it. This backgroundworker thread's job is to run the SQL Server update code in the data class (call a method of that class). Somehow that backgroundworker has to be told that the main thread is done parsing names, ran through the entire table and is done. So I thought the main thread could set a class global boolean done flag. Or something. Can I do this at all the way I am thinking? My problem is that there is no way to predict which process will be faster, name parse or SQL Server update. My idea was simply to use a collection to act as an interface between the two. The name parser instantiates a data class which loads the data to be parsed into an ADO.Net datatable. The name parser updates the data in the the data class. When it is done updating the data it drops the class into the collection. the backgroundworker watches the collection and when it finds a data class in the collection, it pulls it out and calls the update method of that class to process the data. when it is done it cleans up the ADO objects and stores the class in another "history" collection up in the parse class. The data class will have start / stop times, start / end PKIDs, record count etc information which will be of interest to me. I have never used Background worker objects so I don't know whether it can interface in the manner described, using collections to pass objects. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Wed Apr 30 08:09:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 09:09:45 -0400 Subject: [dba-VB] Background worker threads Message-ID: <48186F99.6090401@colbyconsulting.com> I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 10:19:03 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 08:19:03 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <48186F99.6090401@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> Message-ID: You can create your own events with whatever parameters you want and raise those. Just Dim Public Event Whatever(parameter) Then raise it as needed. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 6:10 AM To: VBA Subject: [dba-VB] Background worker threads I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 11:05:17 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 12:05:17 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> Message-ID: <481898BD.2050301@colbyconsulting.com> Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust From shamil at smsconsulting.spb.ru Wed Apr 30 11:07:47 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:07:47 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4817D603.9050308@colbyconsulting.com> Message-ID: <02cc01c8aadc$55aeb860$6401a8c0@nant> Hi John, May I ask you to run this query (e.g. in MS SQL Management Studio) against your huge db table? --- cut here --- SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID, OWNERNAME, FName, MName, LName, NamePrefix, NameSuffix, Gender from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] --- cut here --- M.B. [Names] have to be substituted with the name of your table - what is it BTW? This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... I'm curious to know how much time approx. this query will take to execute... Thank you. -- Shamil P.S. BTW, here is a useful for your case article on multi-threading (practical solution) with good introduction - http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik Sorry, I can't react promptly on your messages in this discussion thread: a lot of custom work here, long weekends and our time zones difference but I do continue to work in this direction and if I get some useful results I will post them here... P.P.S. This book could be also useful: C# Threading Handbook by Tobin Titus et al. ISBN:1861008295 APress, LLC C 2004 (288 pages) This book addresses the fundamental units of Windows and .NET programming-threads. Coverage includes how .NET applications are executed, the life cycle of a thread in .NET, how the .NET Framework uses threads, and more. From shamil at smsconsulting.spb.ru Wed Apr 30 11:25:15 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:25:15 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> Message-ID: <02d901c8aade$c66c3120$6401a8c0@nant> John, Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that the fact that "The basic code WAS working before I carved it out into a worker." doesn't mean that the same code (copied & pasted) into multi-threading environment will work OK - very often the opposite will be the truth - it could produce (very) faulty solution in multi-threaded environment, which will not be easy to sort out into properly running multi-threaded code... I'd dare to advise you to prepare and test your code in VS console application (or .net console app as test-bed app and .net classlib(s) keeping all your non-UI code), and when that will work OK then add some more code to run your classlibs with your .NET WinForms application: you have a lot to do, to learn and to test - multi-threading and .NET programming can't be "captured by attack" - it's a painful and lengthy natural process, which usually takes a lot of time - I know that from my own experience and experience of others - and better not "read all that books" - find just one but good and then experiment a lot on your own - then you'll get it working properly and you'll teach all of us here - I'm quite sure about that latter! Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 8:05 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:17:04 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:17:04 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: <02d901c8aade$c66c3120$6401a8c0@nant> References: <02d901c8aade$c66c3120$6401a8c0@nant> Message-ID: <4818A990.2020306@colbyconsulting.com> Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 12:28:02 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 10:28:02 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: I'm sorry, John, I missed the contextt of your question. We don't use background workers that way, so I don't have an answer for you. Charlotte -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:05 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:32:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:32:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AD3D.6030003@colbyconsulting.com> It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:36:02 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:36:02 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: <4818AE02.2080601@colbyconsulting.com> Bueno, thanks! John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > I'm sorry, John, I missed the contextt of your question. We don't use > background workers that way, so I don't have an answer for you. > > Charlotte > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:05 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> You can create your own events with whatever parameters you want and >> raise those. Just Dim Public Event Whatever(parameter) Then raise it > >> as needed. >> >> Charlotte Foust > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:41:16 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:41:16 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AF3C.3040601@colbyconsulting.com> Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 13:06:29 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:06:29 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <4818A990.2020306@colbyconsulting.com> Message-ID: <02f601c8aaec$ea01f620$6401a8c0@nant> John, Please read the article I referred already: "Simple and Efficient Multi-threading with Blocking Queues" http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik I think it is "what doctor ordered" for your task, and it has code, C# but you can convert it if needed using free converters.... As for collections and variables - they are not thread safe by default. Read these two articles (they have sample code): "Static & Thread Safety: Part I and II" http://www.odetocode.com/Articles/313.aspx http://www.odetocode.com/Articles/314.aspx I believe the only book you need now is the one I have also mentioned already: "C# Threading Handbook by Tobin Titus et al.", ISBN:1861008295 APress, 2004 - it has a lot of code samples (C# sorry) and very good .NET multi-threading features coverage And this is an excerpt from MSDN on collections (MSDN is our best friend!): +++ Collection Object (Visual Basic) Caution Iterating through a Visual Basic Collection is not a thread-safe procedure. Even if the collection is synchronized, other threads can still modify the collection, causing the enumerator to throw an exception. To guarantee thread safety during enumeration, either lock the collection or catch the exceptions resulting from changes made by other threads. For more information on locking a programming element, see SyncLock Statement. ---- The referred above as "what doctor ordered for your task" article uses .NET's Queue class - here is the info on it from MSDN on thread safety: +++ Queue Class Thread Safety Public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. To guarantee the thread safety of the Queue, all operations must be done through the wrapper returned by the Synchronized method. Enumerating through a collection is intrinsically not a thread safe procedure. Even when a collection is synchronized, other threads can still modify the collection, which causes the enumerator to throw an exception. To guarantee thread safety during enumeration, you can either lock the collection during the entire enumeration or catch the exceptions resulting from changes made by other threads. ---- The "Simple and Efficient Multi-threading with Blocking Queues" article has sample code, which shows what simple coding is needed to implement thread-safe collections and queues... HTH, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:17 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:22:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:22:08 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AD3D.6030003@colbyconsulting.com> Message-ID: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:28:58 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:28:58 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AF3C.3040601@colbyconsulting.com> Message-ID: <02fb01c8aaf0$0e512110$6401a8c0@nant> Thank you, John, You have a very good PC, lucky man! :) As I wrote already I (I'm sorry) did post incorrect query - I wanted to run this one: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] It will also return 96K rows for your huge table, but it's expected (here) it will get first results instantly, and it will finish within just several (3?) seconds... Please prove am I wrong with that bright above expectations on the execution time of the fixed version of the query?... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Wed Apr 30 13:50:38 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 13:50:38 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 14:31:28 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 23:31:28 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> Message-ID: <030a01c8aaf8$c971d130$6401a8c0@nant> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead of ~3 seconds.... SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc] (nolock)) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] (nolock) BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of useful advices including all kinds of locking - the first time I see so detailed advices and samples on that subject (but I must say I didn't read that much computer books in the past)... Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Wednesday, April 30, 2008 10:51 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil From jwcolby at colbyconsulting.com Wed Apr 30 14:55:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:55:15 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> References: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> Message-ID: <4818CEA3.5000704@colbyconsulting.com> Well, I do not know what you want me to do with the result. To just open a query window in SSMS and execute it in there I get pretty much the same result. First results in 14 seconds and ten minutes and counting. I don't have "just the PK" indexed, at least not intentionally and visibly. I am now using cover indexes that index the PKID and one or more fields that I am interested in. Whether there is a PKID index behind the scenes I don't know. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:33 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > It started returning results in about 12-13 seconds. You are asking for > 96 thousand result rows you realize. This database is 96 million rows, > so every thousandth row is 96K rows. > > It is still processing at 9 minutes. I'll email the final execution > time whenever the query finishes. > > This is fascinating because I pull just the fields into an ADO.Net table > for 100K rows in just a second or two. Of course these are consecutive > rows, it is not scanning the entire table. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 14:59:44 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:59:44 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030a01c8aaf8$c971d130$6401a8c0@nant> References: <030a01c8aaf8$c971d130$6401a8c0@nant> Message-ID: <4818CFB0.4020402@colbyconsulting.com> I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 15:19:00 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:19:00 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fb01c8aaf0$0e512110$6401a8c0@nant> References: <02fb01c8aaf0$0e512110$6401a8c0@nant> Message-ID: <4818D434.4090104@colbyconsulting.com> Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Wed Apr 30 15:24:32 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 15:24:32 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 15:32:43 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:32:43 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> References: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <4818D76B.7000602@colbyconsulting.com> Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <031501c8ab05$422d2460$6401a8c0@nant> <<< I see the UNION ALL and ORDER BY as bottlenecks. >>> Eric, But union all is executed with two row sets: - first having every 1000th record, and - second having ONLY ONE record... When query execution plan is analyzed then I see that first operation for both parts of UNION ALL query is index scan IOW UNION ALL and ORDER BY do not look as a bottleneck because UNION ALL merges (without any attempts to suppress full duplicate) a set of records with just one record, and because ORDER by of clustered index is 0% cost operation? Thank you -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Thursday, May 01, 2008 12:25 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D434.4090104@colbyconsulting.com> Message-ID: <031601c8ab05$42476320$6401a8c0@nant> <<< Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? >>> Yes, it was, but I expected you'll get results much quicker... Could that be that you do not have (clustered)index on your PK field? Can you: Right-Click on table name -> Script Table as -> CREATE to -> New Query Window => I'm getting the following for your simulated table. CREATE TABLE [dbo].[Names]( [PKID] [int] IDENTITY(1,1) NOT NULL, [OWNERNAME] [nvarchar](255) NOT NULL, [FName] [nvarchar](50) NULL, [MName] [nvarchar](50) NULL, [LName] [nvarchar](50) NULL, [NamePrefix] [nvarchar](50) NULL, [NameSuffix] [nvarchar](50) NULL, [Gender] [nvarchar](50) NULL, [ParsedName] [nvarchar](255) NULL, CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED ( [PKID] ASC ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] Also if you select query's text in SSMS query window, and then Ctrl+L then you'll get query execution plan - here as the last step with 99% cost I'm getting: "Clustered Index Scan" IOW the query doesn't scan table's data at all - it just scans the index... Thanks, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:19 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:07:55 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:07:55 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D76B.7000602@colbyconsulting.com> Message-ID: <032201c8ab06$42a01500$6401a8c0@nant> John, This ORDER BY used here to get ROW_NUMBER is/was expected as very speedy "virtual" zero cost operation because I thought/expected your PK (PKID) was a (clustered) index... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:33 AM To: ebarro at verizon.net; Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 16:16:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 17:16:08 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <031601c8ab05$42476320$6401a8c0@nant> References: <031601c8ab05$42476320$6401a8c0@nant> Message-ID: <4818E198.30501@colbyconsulting.com> I do have the clustered stuff exactly as you show: > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] Likewise I am getting the same 99% cost clustered index scan. I have to guess it is something else. There is a lot of empty space in the database file which I will compact out tonight. It takes hours because of the size and when it is doing that I cannot use the db. Perhaps I have severe fragmentation or something. The database size is 300 GIGS. And finally just remember that when all is said and done it is still a hundred times larger than your database. No matter how you slice it, it has to process more physical disk sectors etc. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > > <<< > Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. > Was that your intention? > Yes, it was, but I expected you'll get results much quicker... > > Could that be that you do not have (clustered)index on your PK field? > > Can you: Right-Click on table name -> Script Table as -> CREATE to -> New > Query Window => I'm getting the following for your simulated table. > > CREATE TABLE [dbo].[Names]( > [PKID] [int] IDENTITY(1,1) NOT NULL, > [OWNERNAME] [nvarchar](255) NOT NULL, > [FName] [nvarchar](50) NULL, > [MName] [nvarchar](50) NULL, > [LName] [nvarchar](50) NULL, > [NamePrefix] [nvarchar](50) NULL, > [NameSuffix] [nvarchar](50) NULL, > [Gender] [nvarchar](50) NULL, > [ParsedName] [nvarchar](255) NULL, > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] > > Also if you select query's text in SSMS query window, and then Ctrl+L then > you'll get query execution plan - here as the last step with 99% cost I'm > getting: "Clustered Index Scan" IOW the query doesn't scan table's data at > all - it just scans the index... > > Thanks, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Thursday, May 01, 2008 12:19 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > > You have a very good PC, lucky man! :) > > As I mentioned elsewhere, I come from the "throw more hardware at it" > school of Engineering. ;-) > > I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 > gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster > in many cases. > > I do not have this large database over on that machine, and it would > take me hours to get it over there so i can't test this query on that, > at least today. that database has not been compacted recently and is > about 300 Gigs with 136 gigs free space. I will compact it tonight and > then copy the smaller file to Azul. > > As for your new query, with the NoLocks in place it returned first > results in only 8 seconds as opposed to 14 before, but it still took > 13:59 to pull the entire result set. Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. Was that your > intention? > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Thank you, John, >> >> You have a very good PC, lucky man! :) >> >> As I wrote already I (I'm sorry) did post incorrect query - I wanted to > run >> this one: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> It will also return 96K rows for your huge table, but it's expected (here) >> it will get first results instantly, and it will finish within just > several >> (3?) seconds... >> >> Please prove am I wrong with that bright above expectations on the > execution >> time of the fixed version of the query?... >> >> Thank you. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Wednesday, April 30, 2008 9:41 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> Shamil, >> >> It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server >> x32) 13:51 to pull 96,417 rows for the result set. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Shamil Salakhetdinov wrote: >>> Hi John, >>> >>> May I ask you to run this query (e.g. in MS SQL Management Studio) > against >>> your huge db table? >>> >>> --- cut here --- >>> >>> SELECT RowNum, PKID from ( select >>> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >>> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >>> NameSuffix, Gender from [Names]) s >>> where (s.RowNum % 1000) = 1 >>> union all >>> Select NULL as RowNum, Max(PKID) as PKID from [Names] >>> >>> --- cut here --- >>> >>> M.B. [Names] have to be substituted with the name of your table - what is >> it >>> BTW? >>> >>> This query returns PKID of every 1000th row sorted by PKID + > MAX(PKID).... >>> I'm curious to know how much time approx. this query will take to >> execute... >>> Thank you. >>> >>> -- >>> Shamil >>> >>> P.S. BTW, here is a useful for your case article on multi-threading >>> (practical solution) with good introduction - >>> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >>> cik >>> >>> Sorry, I can't react promptly on your messages in this discussion thread: >> a >>> lot of custom work here, long weekends and our time zones difference but > I >>> do continue to work in this direction and if I get some useful results I >>> will post them here... >>> >>> P.P.S. This book could be also useful: >>> >>> C# Threading Handbook >>> by Tobin Titus et al. >>> >>> ISBN:1861008295 >>> >>> APress, LLC C 2004 (288 pages) >>> >>> This book addresses the fundamental units of Windows and .NET >>> programming-threads. Coverage includes how .NET applications are > executed, >>> the life cycle of a thread in .NET, how the .NET Framework uses threads, >> and >>> more. >>> >>> >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 3 06:34:34 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 03 Apr 2008 13:34:34 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From accessd at shaw.ca Thu Apr 3 10:59:46 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Thu, 03 Apr 2008 08:59:46 -0700 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: References: Message-ID: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From michael at ddisolutions.com.au Thu Apr 3 18:01:55 2008 From: michael at ddisolutions.com.au (Michael Maddison) Date: Fri, 4 Apr 2008 10:01:55 +1100 Subject: [dba-VB] ASP.NET, separated programming and visual design References: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D013BF515@ddi-01.DDI.local> I did an online seminar with MS about VS 2008 + WPF. They claim that WPF is pretty much designed to allow the UI designer to work seperately to the programmer. He could do it in NP as well as long as he knows his XML ;-) cheers Michael M -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Friday, 4 April 2008 3:00 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Thu Apr 3 18:53:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 03:53:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <006601c895e5$fa874e80$6401a8c0@nant> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 02:14:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 09:14:35 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From shamil at users.mns.ru Fri Apr 4 04:59:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 13:59:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <002d01c8963a$a3217e90$6501a8c0@nant> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 05:17:54 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 12:17:54 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Shamil Great! This is exactly the information I need. /gustav >>> shamil at users.mns.ru 04-04-2008 11:59 >>> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From wdhindman at dejpolsystems.com Thu Apr 10 09:11:34 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Thu, 10 Apr 2008 10:11:34 -0400 Subject: [dba-VB] ASP.NET, separated programming and visual design References: Message-ID: <002301c89b14$c8de1d30$7e6d6c4c@jisshowsbs.local> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 10 09:25:30 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 10 Apr 2008 16:25:30 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi William Fine, though it we be the other way around - I would make the mechanics first, then he would beef it up. With VSVWD8 Express as a free tool I see no reason why he should punish him self with something like Notepad. /gustav >>> wdhindman at dejpolsystems.com 10-04-2008 16:11 >>> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav From jwcolby at colbyconsulting.com Fri Apr 11 12:26:53 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:26:53 -0400 Subject: [dba-VB] Parsing strings Message-ID: <47FF9F5D.2070704@colbyconsulting.com> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Fri Apr 11 12:42:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 11 Apr 2008 19:42:18 +0200 Subject: [dba-VB] Parsing strings Message-ID: Hi John Why not use the Path class: extension = Path.GetExtension(FileName) basename = Path.GetFileNameWithoutExtension(FileName) /gustav >>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 11 12:49:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:49:18 -0400 Subject: [dba-VB] Parsing strings In-Reply-To: References: Message-ID: <47FFA49E.2020605@colbyconsulting.com> Lovely, thanks. Gustav Brock wrote: > Hi John > > Why not use the Path class: > > extension = Path.GetExtension(FileName) > basename = Path.GetFileNameWithoutExtension(FileName) > > /gustav > >>>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, -- John W. Colby www.ColbyConsulting.com From stuart at lexacorp.com.pg Fri Apr 11 18:45:55 2008 From: stuart at lexacorp.com.pg (Stuart McLachlan) Date: Sat, 12 Apr 2008 09:45:55 +1000 Subject: [dba-VB] Parsing strings In-Reply-To: <47FF9F5D.2070704@colbyconsulting.com> References: <47FF9F5D.2070704@colbyconsulting.com> Message-ID: <480084D3.12387.2763EDB7@stuart.lexacorp.com.pg> Left,Mid,Right etc are not function of the string object, they are functions in the Strings module of the Microsoft.VisualBasic namespace. i.e., fully qualified it is Microsoft.VisualBasic.Strings.Right() On 11 Apr 2008 at 13:26, jwcolby wrote: > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > From jwcolby at colbyconsulting.com Sat Apr 12 15:23:49 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 16:23:49 -0400 Subject: [dba-VB] Count a character in a string Message-ID: <48011A55.40402@colbyconsulting.com> I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 15:50:25 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 15:50:25 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From Gustav at cactus.dk Sat Apr 12 16:33:25 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 12 Apr 2008 23:33:25 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 12 17:48:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 18:48:26 -0400 Subject: [dba-VB] Count a character in a string In-Reply-To: References: Message-ID: <48013C3A.9030500@colbyconsulting.com> I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 18:00:38 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:00:38 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: Message-ID: True, don't use .Net much though. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Saturday, April 12, 2008 4:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] Count a character in a string Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From DWUTKA at Marlow.com Sat Apr 12 18:01:05 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:01:05 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48013C3A.9030500@colbyconsulting.com> Message-ID: Ya, I think I posted that solution a while back on AccessD. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 5:48 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Count a character in a string I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From shamil at users.mns.ru Sat Apr 12 21:30:24 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 06:30:24 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: <053b01c89d0e$53c595e0$6401a8c0@nant> Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:08:59 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:08:59 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> References: <48011A55.40402@colbyconsulting.com> <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <003c01c89d46$03525ca0$8119fea9@LTVM> John, You may recall the RegExp that I posted a short while ago. You can use this to get the answer. This code returns 17: Function CountRegEx() Dim strIn As String, lngCount As Long strIn = "asdf||||||qwert|zxcv|mnbv|qwert|zxcv|mnbv||||||" lngCount = (RegExpTest("\|", strIn)) MsgBox lngCount End Function Here is the original Function Function RegExpTest(patrn, strng) As Long ' Set reference to Microsoft vbscript regular expressions 5.5 ' Cheat Sheet available at http://regexlib.com/CheatSheet.aspx Dim RegEx As New VBScript_RegExp_55.RegExp Dim Match, Matches, RetStr ' Create variable. Set RegEx = New RegExp ' Create a regular expression. RegEx.Pattern = patrn ' Set pattern. RegEx.IgnoreCase = True ' Set case insensitivity. RegEx.Global = True ' Set global applicability. Set Matches = RegEx.Execute(strng) ' Execute search. RegExpTest = 0 For Each Match In Matches ' Iterate Matches collection. RetStr = RetStr & "Match found at position " RetStr = RetStr & Match.FirstIndex & ". Match Value is '" RetStr = RetStr & Match.Value & "'." & vbCrLf RegExpTest = RegExpTest + 1 Next End Function HTH Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 3:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:14:06 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:14:06 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: References: <48013C3A.9030500@colbyconsulting.com> Message-ID: <003d01c89d46$ba370a60$8119fea9@LTVM> John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max From shamil at users.mns.ru Sun Apr 13 04:26:17 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 13:26:17 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <054b01c89d48$6d543360$6401a8c0@nant> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 13 04:47:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 11:47:21 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi Shamil Neat! I'll add that to my toolbox. /gustav >>> shamil at users.mns.ru 13-04-2008 11:26 >>> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Sun Apr 13 05:08:53 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 14:08:53 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <003d01c89d46$ba370a60$8119fea9@LTVM> Message-ID: <054f01c89d4e$60f41d00$6401a8c0@nant> Hi Max, Our postings crossed - I have just posted RegEx test results... N.B.: when testing this or that method both memory consumption and execution speed should be taken into consideration: - high memory consumption giving speediest results can be neglected to solve small tasks within "tiny" applications/utilities, but - high memory consumption is the thing, which could "unexpectedly" give nasty side effects in the large application systems: One example I have got several months ago, which was so "tricky" to watch, and was not easy to find why it could ever happen in .NET: I have got a memory leakage/excessive memory consumption (in .NET!) because of "lazy loading", and that latter "lazy loading" loaded quite some data when it wasn't needed, in the cycle processing millions of records - the results were like that when I watched 'Page File Usage History' in Task Manager: /| /| /| . / | / | / | . / | / | / | . / |/ |/ |/ And in usual mode without heavy application system workload everything worked well with the same data... Recap: - Split(...) approach could result in similar to the above side effect for very large input strings; // 10 sec for 20,000,000 iterations string[] recordLine = s.Split('|'); count = recordLine.Length-1; - Replace(...) could also result in the above side effect; // 10 sec for 20,000,000 iterations string temp = s.Replace("|", Microsoft.VisualBasic.Constants.vbNullString); count = s.Length - temp.Length; - RegEx(...) seems to be the slowest - unsatisfactory slow for large input strings/heavy system workload; // ? sec (unfinished) for 20,000,000 iterations Regex rx = new Regex("|"); count = rx.Matches(s).Count; - char array iteration using char index and (XOR or char comparison) gives the fastest results and is 100% safe from memory consumption point of view.... // * using XOR: // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; // * using char comparinson... // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; Please correct me if you'll find the above results have mistakes... Any takers to find quiker code for .NET VB or C# or C++/CLI? - that would be a good weekend exercise on code optimization techniques... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Max Wanadoo Sent: Sunday, April 13, 2008 1:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Johncliviger at aol.com Sun Apr 13 13:36:20 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 14:36:20 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Gustav at cactus.dk Sun Apr 13 13:50:38 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 20:50:38 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Pick only the fields you need ... or do you wish to display 30 fields? /gustav >>> Johncliviger at aol.com 13-04-2008 20:36 >>> Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Johncliviger at aol.com Sun Apr 13 14:08:13 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 15:08:13 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Gustav at cactus.dk Mon Apr 14 08:53:40 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 14 Apr 2008 15:53:40 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Many ways to do this - depends a bit on your previous code and what features you need. A lot of examples can be found via Google for this - including Microsoft's pages. Key words could be master/child or parent/child. /gustav >>> Johncliviger at aol.com 13-04-2008 21:08 >>> Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Johncliviger at aol.com Mon Apr 14 09:23:22 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Mon, 14 Apr 2008 10:23:22 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Gustav Many thanks I've cracked it. Quite simple when you know how. When you try it for this first time your Access thinking leads you astray. Cheers Gustav john c From Gustav at cactus.dk Tue Apr 15 07:28:29 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 15 Apr 2008 14:28:29 +0200 Subject: [dba-VB] Web colours Message-ID: Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav From accessd at shaw.ca Tue Apr 15 14:03:05 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:03:05 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From accessd at shaw.ca Tue Apr 15 14:28:06 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:28:06 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: <945F14E6828D4E92AB0736A26485C259@creativesystemdesigns.com> Hi Gustav: I should have read right through your question before I, with a sandwich in hand, responded. Use any colour you want. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Tuesday, April 15, 2008 12:03 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Web colours Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 16 08:44:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 16 Apr 2008 15:44:13 +0200 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Message-ID: Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24/microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav From accessd at shaw.ca Wed Apr 16 16:00:03 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Wed, 16 Apr 2008 14:00:03 -0700 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) In-Reply-To: References: Message-ID: <2751B45E193C418490D807B02A37C79A@creativesystemdesigns.com> Thanks for your heads up Gustav. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Wednesday, April 16, 2008 6:44 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24 /microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From paul.hartland at googlemail.com Fri Apr 18 02:49:18 2008 From: paul.hartland at googlemail.com (Paul Hartland) Date: Fri, 18 Apr 2008 08:49:18 +0100 Subject: [dba-VB] OT - Free Twain ActiveX Control Message-ID: <38c884770804180049n76b46424y6d66cf4124792ced@mail.gmail.com> To all, I need to develop an application, which will involve scanning some documents etc. Does anyone know of or use a Twain ActiveX Control (preferably easy to use & most of all free, or very cheap) Thank you for any help on this..... -- Paul Hartland paul.hartland at googlemail.com From jwcolby at colbyconsulting.com Sat Apr 19 09:05:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 10:05:31 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <4809FC2B.6030602@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 19 10:08:14 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 19 Apr 2008 17:08:14 +0200 Subject: [dba-VB] NetGender - The Software Company Message-ID: Hi John One method would be to have a MDI parent form from which you open the other forms as child forms. Then you just to either keep track of what you open or to loop through the child forms collection to see if it contains a certain child form. In VS2008 the Add New wizard will happily create a new MDI parent form with basic functionality. /gustav >>> jwcolby at colbyconsulting.com 19-04-2008 16:05 >>> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 11:08:13 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 12:08:13 -0400 Subject: [dba-VB] Splitting a name Message-ID: <480A18ED.6020009@colbyconsulting.com> I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 12:27:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 13:27:25 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <480A2B7D.6000202@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sat Apr 19 13:51:12 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sat, 19 Apr 2008 14:51:12 -0400 Subject: [dba-VB] NetGender - The Software Company References: <4809FC2B.6030602@colbyconsulting.com> Message-ID: <000301c8a24e$57223eb0$60606c4c@jisshowsbs.local> ...nice looking library ...let us know what problems you run into with it ...I may well be in the market for such myself in the near future. William ----- Original Message ----- From: "jwcolby" To: "VBA" Sent: Saturday, April 19, 2008 10:05 AM Subject: [dba-VB] NetGender - The Software Company >I purchased a .Net library that does name splitting. > > http://www.softwarecompany.com/dotnet/netgender.htm > > It seems to work well just from using the little demo that you can > download. Now I am starting to actually work with it. It consists of a > single class, embedded in a dll that you reference, with properties to > set to tell it all of the controlling stuff, then properties to hand it > the name to split and read back the name pieces. > > The developer claims that it will handle between 2K and 3K names per > second. We shall see. > > So here I am trying to work with this thing. The demo form is quite > extensive and the code in the form shows how to read / write all of the > properties to populate the form. > > I have a database with (of course) about 80 million names, in a single > field, in the form LastName FirstName MiddleInitial (no commas etc) > which needs parsing. What I thought I would do (just for the moment) is > to use the existing form and just feed in the names. Thus I need to add > a couple of properties to the form such that I can feed in names and get > back the parsed pieces. I will build a switch to tell the form to > display the pieces or not, thus I could watch the process function if I > wanted, though at the cost of slowing the process down. > > Modifications to the demo form to allow me to do all that stuff is just > about complete. What I need to know is the syntax to get a pointer to a > form being opened. I know that you just do a "Someform.Show" to open > the form, but a parent form needs to get a pointer to this demo form and > then use that pointer to set / retrieve properties. How do I get a > pointer to it once opened? > > -- > John W. Colby > www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From max.wanadoo at gmail.com Sun Apr 20 07:32:53 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 20 Apr 2008 13:32:53 +0100 Subject: [dba-VB] Splitting a name In-Reply-To: <480A18ED.6020009@colbyconsulting.com> References: <480A18ED.6020009@colbyconsulting.com> Message-ID: <002101c8a2e2$a84d3580$8119fea9@LTVM> John: No doubt there are many more cleverer answers then this and this may be tooooo slow for your needs. Function jc() ' will take the 1st series up to space as Xname, right series back to space as zname and bit in the middle as yname Const conQuote As String = """" Dim sql As String sql = "update tblTest set " sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & conQuote & ")-1) ," sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") - InStr(searchname, " & conQuote & " " & conQuote & "))," sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " & conQuote & ")+1)" CurrentDb.Execute (sql) End Function Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 19, 2008 5:08 PM To: VBA Subject: [dba-VB] Splitting a name I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 20 10:29:34 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 20 Apr 2008 11:29:34 -0400 Subject: [dba-VB] Splitting a name In-Reply-To: <002101c8a2e2$a84d3580$8119fea9@LTVM> References: <480A18ED.6020009@colbyconsulting.com> <002101c8a2e2$a84d3580$8119fea9@LTVM> Message-ID: <480B615E.5040903@colbyconsulting.com> Max, The problems with names is that there are many variations and you cannot count on simple rules like that. For example Mr. Juan Miguel de la Uariate. Now you get another list with De la Uriarte Juan Miguel Sr. You get the picture I purchased a library to do the split. The library performs the split nicely, handling such variations quite well. What I need now is a strategy for reading rows or groups of rows, updating a single record at a time back to SQL Server. In the absence of experience I am going to open a chunk of records,, perhaps 10K, with a startPK and EndPK to track the chunks into an ADO recordset. I will read in 10K records (or some chunk size), then just iterate that recordset splitting each name. My first attempt will be to dynamically build up an update query to write EACH RECORD back out, one at a time. I am almost certain that this will not be fast enough but we shall see. Once I have the code I will probably move the sql into a udf and call the udf passing the name parts in to the udf which will then write one record back to the table. Also likely not fast enough but we shall see. Last will be an attempt to build an ado recordset in memory, then have ADO write the entire recordset back to SQL Server. If that is not fast enough I have run out of ideas. Max Wanadoo wrote: > John: > No doubt there are many more cleverer answers then this and this may be > tooooo slow for your needs. > > Function jc() > ' will take the 1st series up to space as Xname, right series back to > space as zname and bit in the middle as yname > Const conQuote As String = """" > Dim sql As String > sql = "update tblTest set " > sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & > conQuote & ")-1) ," > sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " > & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") > - InStr(searchname, " & conQuote & " " & conQuote & "))," > sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " > & conQuote & ")+1)" > CurrentDb.Execute (sql) > End Function > > Max > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 19, 2008 5:08 PM > To: VBA > Subject: [dba-VB] Splitting a name > > I have a table with eight million records, with a single name field, and I > have a library that splits the name into components. What I need now is a > generic method of getting the name out of a table and updating a specific > set of fields back in the table. It has to do so QUICKLY since I have to > update 80 million names. > > In this case I have a single table > > Name > > NamePrefix > LastName > FirstName > MiddleName > NameSuffix > > Name already exists and contains a name in a consistent format LName FName > MName. > > The other fields I have added to my table. > > So what I am looking for is a general strategy. I assume I will open a read > recordset with the PKID and the name field, then iterate through this > recordset performing the split. > > Once I do that should I: > > Build a dynamic SQL Update query and execute that? > > Build a stored procedure that I pass the PKID and the name parts to which > then updates the data? > > Use an ADO recordset to do this? > > some other method entirely? > > I am assuming at this point that because of the sheer size of the data I do > not want to try and do this in an ado recordset. Eight million records > sitting in data at one time does not sound fun. > > All ideas gratefully accepted. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Johncliviger at aol.com Tue Apr 22 04:40:48 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 05:40:48 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Gustav at cactus.dk Tue Apr 22 05:57:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 12:57:51 +0200 Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi John For #3 I would use a DataTableAdapter to attach the table of the database - and then feed your data to a DataTable of the DataTableAdapter. /gustav >>> Johncliviger at aol.com 22-04-2008 11:40 >>> Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Johncliviger at aol.com Tue Apr 22 06:05:04 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 07:05:04 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi Gustav Yes. TableAdapter. I'm on the right lines then, which good news. Cheers gustav From Gustav at cactus.dk Tue Apr 22 10:51:23 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 17:51:23 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 11:10:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 18:10:18 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all I located this article which will be of interest for those with an Access background: Flat-MultiColumn Combobox with Autocomplete http://www.codeproject.com/KB/architecture/MultiColumnFlatCombo.aspx /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 17:06:50 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 00:06:50 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From cfoust at infostatsystems.com Tue Apr 22 17:55:36 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Tue, 22 Apr 2008 15:55:36 -0700 Subject: [dba-VB] No flat ComboBox in .Net?? In-Reply-To: References: Message-ID: We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 23 01:26:27 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 08:26:27 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Wed Apr 23 16:31:09 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 23:31:09 +0200 Subject: [dba-VB] Custom Access 2007 ribbons with Visual Studio 2008 Message-ID: Hi all Noticed this tip from FMS: Create Custom Microsoft Office 2007 Ribbons with the New Visual Designer in Visual Studio 2008 http://www.fmsinc.com/free/NewTips/NET/CreateCustomOffice2007Ribbons.asp With the new release of Visual Studio Tools for Office (VSTO) in Visual Studio 2008, you can now use a visual designer to create your own custom Ribbons in Microsoft Access, Excel, Word, Outlook, etc. These Ribbons are encapsulated as add-ins for each of the specific Office 2007 products. If you wanted to build your own custom Ribbons, but have held back by the lack of a visual designer, take a look at Visual Studio 2008. The visual designer is available under the New Project, Office, Office 2007 dialog. More resources: http://msdn2.microsoft.com/en-us/library/bb386089.aspx /gustav From mmattys at rochester.rr.com Wed Apr 23 19:58:58 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Wed, 23 Apr 2008 20:58:58 -0400 Subject: [dba-VB] No flat ComboBox in .Net?? References: Message-ID: <01c801c8a5a6$6201d2e0$0502a8c0@Laptop> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" To: Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From Gustav at cactus.dk Thu Apr 24 02:31:43 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:31:43 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Michael That's a nice tip. Thanks. Lots of other stuff on that site ... /gustav >>> mmattys at rochester.rr.com 24-04-2008 02:58 >>> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" < Gustav at cactus.dk > To: < dba-vb at databaseadvisors.com > Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? From Gustav at cactus.dk Thu Apr 24 02:38:08 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:38:08 +0200 Subject: [dba-VB] GDI+ graphics in .Net Message-ID: Hi all This site of Bob Powell which Michael pointed out has some very good explanation and code examples on pictures, graphics and drawing in .Net: http://www.bobpowell.net/faqmain.htm He also has a very good introduction. That info can be hard to find around the web. Remember: Graphics are fun! /gustav From jwcolby at colbyconsulting.com Fri Apr 25 11:31:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 12:31:25 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: <4812075D.6000306@colbyconsulting.com> OK, I have been badgered and goaded into coming into the 21st century. Not to mention I can't figure out how to safely and effectively share development of my .Net programs on all of my machines. So I am trying to get Subversion installed and functioning. So of course I come here seeking advice or to start a user group (goad and prod you guys) if no one is using this thing. I installed VisualSVN and it told me I had to install TortoiseSVN to get full functionality. It implied that it would install Subversion as well. The install did not ask any "install" questions such as where do you want the database to go etc. My intention is to place the Subversion database on one of my servers, share the directory, then place the various clients (TortoiseSVN and VisualSVN on each machine which I develop on. So my first question, is anyone out there using this thing? Is anyone interested in using this and sharing the pain of figuring it out? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Fri Apr 25 12:25:08 2008 From: robert at webedb.com (Robert L. Stewart) Date: Fri, 25 Apr 2008 12:25:08 -0500 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <200804251729.m3PHTFbD020137@databaseadvisors.com> Been through the pain and suffering already. I much prefer Sourcesafe. This was the tool used on one of the contracts I was on. They were always having problems with it. It is open source, so there was not what you would really call technical support. Good luck. At 12:00 PM 4/25/2008, you wrote: >Date: Fri, 25 Apr 2008 12:31:25 -0400 >From: jwcolby >Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >To: VBA >Message-ID: <4812075D.6000306 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >OK, I have been badgered and goaded into coming into the 21st century. >Not to mention I can't figure out how to safely and effectively share >development of my .Net programs on all of my machines. > >So I am trying to get Subversion installed and functioning. So of >course I come here seeking advice or to start a user group (goad and >prod you guys) if no one is using this thing. > >I installed VisualSVN and it told me I had to install TortoiseSVN to get >full functionality. It implied that it would install Subversion as >well. The install did not ask any "install" questions such as where do >you want the database to go etc. > >My intention is to place the Subversion database on one of my servers, >share the directory, then place the various clients (TortoiseSVN and >VisualSVN on each machine which I develop on. > >So my first question, is anyone out there using this thing? Is anyone >interested in using this and sharing the pain of figuring it out? > >-- >John W. Colby >www.ColbyConsulting.com > > >------------------------------ > >_______________________________________________ >dba-VB mailing list >dba-VB at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-vb > > >End of dba-VB Digest, Vol 54, Issue 18 >************************************** From jwcolby at colbyconsulting.com Fri Apr 25 12:41:23 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 13:41:23 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <200804251729.m3PHTFbD020137@databaseadvisors.com> References: <200804251729.m3PHTFbD020137@databaseadvisors.com> Message-ID: <481217C3.7020105@colbyconsulting.com> Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 13:27:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Fri, 25 Apr 2008 22:27:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481217C3.7020105@colbyconsulting.com> Message-ID: <000901c8a701$f4afd8b0$6401a8c0@nant> Hi John, I did use Subversion for some time, then I abandoned it because customer didn't insist on using any source control tools, and because I'm still working mainly as one man shop... ...I do plan to restart using Subversion hopefully in the near future - I wanted to be 'in' in this discussion and participate or at least lurk it because of a lot of work here, and not that much time left for pleasure of AccessD discussions... ...I did also use TortoiseSVN... ...I do agree that MS support is usually useless... ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as well as I do plan to add to that "bundle" MS SoundCastle, NCover and some other useful tools... ...one or another bug/issues/features tracking/managing tools should be used - I do use Axosoft's OnTiome2008 now... ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS Build + NUnit working together and making automated builds and (unit) test runs - that's the only way to go for teamwork development these days... ...all that mentioned above stuff worked well here but I must admit it was not that easy to make the parts working in ensemble but when everything was tuned it happened to be relatively easy to handle stuff... ...I have had two large ASP.NET applications with probably several (1,2,3 ? I didn't count) thousands of source files and about 15 projects in two solutions to put into Subversion code base, automate compilation/building using CruiseCOntrol.NET and MS Build etc... ...the only issue I can remember (after everything was set) is that I didn't find how to easily manage the cases to synchronize central code db with local code versions when source files are moved from one folder into another within the same project - I could have missed something because of lack of time, and experience or that could be a real "bottleneck" of Subversion with everything else being very good I think... ...I'd participate in preparing and publishing on AccessD web site together with other interested AccessD members an instruction/manual and set of scripts and utilities on how to set up and effectively use together Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say how much time I will have for this work... Thank you. -- Shamil P.S. As far as I know Subversion "beats" MS Source Safe for the cases when the project development should be fork-ed, and then several branches merged back: the difference between Subversion and MS SourceSafe is conceptual here IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I can be wrong, please correct me then, thank you. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Fri Apr 25 14:35:03 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 15:35:03 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000901c8a701$f4afd8b0$6401a8c0@nant> References: <000901c8a701$f4afd8b0$6401a8c0@nant> Message-ID: <48123267.7030608@colbyconsulting.com> Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 25 15:06:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 16:06:31 -0400 Subject: [dba-VB] Reading / writing SQL Server rows with ADO.Net (or something) Message-ID: <481239C7.2030506@colbyconsulting.com> I need to read blocks of data from tables in SQL Server, use values in the rows then update those rows and write back to SQL Server. I need to do this programmatically, not in a control on a form. does anyone have any example code for setting up the various ADO.Net objects, fetching rows of data from SQL Server, updating that data and writing it back to SQL Server? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:20:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:20:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000001c8a711$bdda92c0$6401a8c0@nant> John, Everything in Subversion can be controlled by using batch files - that is the best approach (IMO) for the initial setup, bulk checking and working set checkout: if I do not miss anything then I will post in the following several e-mails under this topic the set of .bat files I used to create the test environment last autumn(three projects ProjectA, ProjectB and ProjectC). Stay tuned (after posting I will be out here until Sunday). Thanks. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 11:35 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:38:38 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:38:38 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000401c8a714$5762dae0$6401a8c0@nant> Hi John, Below is the set of .batch files I have just found, which I did make last autumn to control/direct subversion server. I must say I'm not sure this is the last working version I used (AFAIKR I did write some simple custom code in .NET to control subversion server to avoid using .bat files, which have to be made for every project you'll use - and my simple custom code was parameterized to use any source project) but as you can see the set of batch files below is the minimal set to fulfill all the source control main tasks. Watch line wraps and read subversion docs for more details - svn-book.pdf - located in Sunversion server's installation folder (I'm out until Sunday). Thanks. -- Shamil P.S. .bat files to use with Subversion !svn_admin_main.bat ===================== rem Port: 3690 rem RepositoryPath: e:\Subversion\svn_repository rem X Install svnserve to run as a Windows service rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server call make_dirs.bat new_repository call check_out.bat new_repository pause make_dirs.bat ============== rmdir tmpdir /s /q mkdir tmpdir mkdir tmpdir\projectA mkdir tmpdir\projectA\trunk mkdir tmpdir\projectA\branches mkdir tmpdir\projectA\tags mkdir tmpdir\projectB mkdir tmpdir\projectB\trunk mkdir tmpdir\projectB\branches mkdir tmpdir\projectB\tags mkdir tmpdir\projectC mkdir tmpdir\projectC\trunk mkdir tmpdir\projectC\branches mkdir tmpdir\projectC\tags rem xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E rem rmdir /s /q ..\%1 call create_fs_repository.bat %1 rem svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log call verify_fs_repository.bat %1 call svnlook_fs_repository.bat %1 rmdir tmpdir /s /q rem rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA --message svnlog.log create_fs_repository.bat ======================== svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs xcopy ..\args\conf\*.* ..\%1\conf /Y /E goto :EOF verify_fs_repository.bat ======================== svnadmin verify ../%1 GOTO :EOF svnlook_fs_repository.bat ========================= svnlook info ../%1 goto :EOF check_out.bat ============= rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S mkdir E:\Temp\SV\WORKING_SET\ /Q /S svn checkout file:///E:/Subversion/%1/projectA/trunk E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log svn checkout file:///E:/Subversion/%1/projectB/trunk E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log svn checkout file:///E:/Subversion/%1/projectC/trunk E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log commit.bat ========== svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log backup_fs_repository.bat ======================== svnadmin hotcopy ../%1 ../backups/%1 GOTO :EOF From jwcolby at colbyconsulting.com Fri Apr 25 17:15:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 18:15:27 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000401c8a714$5762dae0$6401a8c0@nant> References: <000401c8a714$5762dae0$6401a8c0@nant> Message-ID: <481257FF.30208@colbyconsulting.com> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 26 06:52:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 13:52:35 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil I would love to follow this thread. I work both at home and at the office and is a bit tired of copying files around. I located this page: Source Control with Visual Studio .NET http://www.codeproject.com/KB/dotnet/SourceControl_VSNET.aspx?fid=471385&df=90&mpp=25&noise=3&sort=Position&view=Quick&fr=26#xx0xx which, however, seems to know nothing about VS2008. It has links to these add-ins for VS: Garry Broadsword's Add-on which is updated for VS2008: http://garrys-brain.blogspot.com/2007/07/tortoisesvn-and-visual-studio.html and VisualSVN: http://www.visualsvn.com/ This site also sports VisualSVN Server: http://www.visualsvn.com/server/ VisualSVN Server is a package that contains everything you need to install, configure and manage Subversion server for your team on Windows platform. It includes Subversion, Apache and a management console. ... And last, but not least - VisualSVN Server is completely free! This must be what John is after - and myself as neither I nor any colleague have much experience with Apache. Now, big question: Does this replace or supplement Shamil's collection of batch files? As I understand it, VisualSVN Server should take care of the setup of Subversion but I must admit I have not yet grasped all the bits and pieces of this ... A final note for others that may follow this thread: Nothing of this works with the free Visual Studio Express versions. /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil From Gustav at cactus.dk Sat Apr 26 07:04:22 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 14:04:22 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil Exciting. http://www.visualsvn.com/doc/getting-started.html Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? Have never heard of Google Code. Can I use Google to store a project? http://code.google.com/ How does that work and is it only for public access project? Or how would you control access for closed source projects? /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. From jwcolby at colbyconsulting.com Sat Apr 26 07:11:54 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 08:11:54 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <48131C0A.1000302@colbyconsulting.com> the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 26 14:25:56 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 15:25:56 -0400 Subject: [dba-VB] ADO.Net Message-ID: <481381C4.7000804@colbyconsulting.com> I have been setting up an application to read a block of records, perhaps 10K or something. I designed a class that will load a block of records which consist of a name to be parsed, and a set of 6 fields which will be filled in with the parsed data returned from the name parser. Thus as I load the data only the name to parse is filled in. The supervisor iterates this set of records, pulling the name to parse, passing it off to the name parser, then reading the data back from the name parser and updating the fields in the same record. Move to the next record, wash, rinse repeat. I do have a couple of questions though. I am not very familiar with how the ADO.net stuff works. I build a connection string I build a connection object which I leave not connected (for now), but feeding it the connection string. I build a sql statement that represents the fields to pull WHERE PK between StartValue and EndValue. I build a command object which I pass SQL statement and the still unconnected connection object. I build a data adapter which I feed the command object. Notice that we are still not connected to the data store. I build a DataTable object with a table name. I call DataAdapter(DataTable).Fill Does .Net then connect correctly and proceed to fill the data table? Once I have the data table filled I start to iterate the DataTable using the for each DataRow iterator I then call the name splitter passing in the name to be split. Back comes the name pieces. I fill in the row with the name pieces and move to the next row. Once every row has been filled in I need to call an update method of the DataAdapter which is SUPPOSED to write the changes back to the table in SQL Server..... The name splitter handles ~1K names / second, NOT as speedy as I had hoped. At this point I haven't a clue how fast the data adapter will write back to the table in SQL Server. What I am trying to do is write this thing such that each instance of the data class can then be handed off to a thread so that the write back to SQL Server can begin and I can instantiate another data class to load / parse the next "chunk" of data in parallel with the write to SQL Server. I am trying to discover whether I need to close connections to the database while I do the parsing or just not worry about it. Should I just leave all the objects open and close them all down as the data class closes once the write to SQL Server is done? How will I know the write is done? Does the code "hang" at the DataAdapter.Update until the entire update (all the rows) is finished? Am I going about this the right way? Should I become a used car salesman? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <481381C4.7000804@colbyconsulting.com> Message-ID: <000501c8a7ea$a4d53080$6401a8c0@nant> Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481257FF.30208@colbyconsulting.com> Message-ID: <000401c8a7ea$a4c949a0$6401a8c0@nant> Hi John, Yes, I'm considering to purchase it also very probably... ...I will wait how well it will work for you - OK :) ? -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 2:15 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48131C0A.1000302@colbyconsulting.com> Message-ID: <000301c8a7ea$a4af0ae0$6401a8c0@nant> <<< running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever >>> John, The batches I posted here are just to work with Subversion's repository: create it, check-in, check-out, backup... ...to run builds and automate unit testing etc. CruiseControl.NET has to be used, or other similar tools... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 4:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:39:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:39:45 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <000601c8a7ee$6d9119a0$6401a8c0@nant> Hi John, Additionally to the below e-mail - you can consider using SQLDataReader to get data records batches as in this below example, then build parsed names' arrays together with ids, then Join() these arrays into strings and pass these strings as parameters to SPs, which will make actual updates by parsing strings into dynamic update sqls or something like that - T-SQL parsing functions are reasonably quick, or you can even consider managed SPs to parse the strings on server side and perform the updates... ...but that latter could be an overkill giving very little gains because .NET framework's communication with native MS SQL provider when using plain simple C#/VB.NET is very effective I have heard... ..maybe you can just process your data and generate and run update temp SPs to have update T-SQL statements batched into such temp SPs - that could give you (much) quicker updates... ...etc... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 2:13 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 18:31:32 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 19:31:32 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> References: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <4813BB54.4090203@colbyconsulting.com> Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 18:47:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 03:47:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813BB54.4090203@colbyconsulting.com> Message-ID: <000001c8a7f7$e871c530$6401a8c0@nant> <<< I am definitely interested in doing strongly typed recordsets though. >>> John, That strongly typed recordsets are imposing big overhead - you can dig into the code generated (by VS) for these recordsets... ...as for batch updates "automagically" generated by ADO.Net - you can use SQL profiler to see what happens "under the hood"... ...I'd bet that if you generate batch updates by custom code into temp SPs and use datareader to get data to build these batches then this approach will result in a way faster data processing, and you will have many options to optimize it even more... ...just my opinion but as you can find in ASP.Net apps ADO.NET typed recordsets are not recommended to use - have a look at e.g. "ASP.NET Website Programming: Problem - Design - Solution C# Edition" by Marco Bellinaso.... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 3:32 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 19:45:47 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 20:45:47 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a7f7$e871c530$6401a8c0@nant> References: <000001c8a7f7$e871c530$6401a8c0@nant> Message-ID: <4813CCBB.3060907@colbyconsulting.com> This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx There are many things to consider when I program in .net, not the least of which is simplicity. If I can get a simple method that gets me medium efficiency, it is better (for me) than a complex method that gets high efficiency simply because I may be able to implement the simple method whereas the complex method would never get done. I can always come back later and recode for a higher efficiency after I have a method working and chugging away. As long as the program does not take weeks to run, it is better to spend cpu time than John Colby time. As I mentioned, the name parser is not super fast. I am hoping to get the first database update batch happening in parallel with the second name parsing batch. I also hope to run this on a completely different computer which has more cores (and memory) so that I can use threads effectively. Which is not to say I am not interested in the most efficient method, simply that I will likely be unable to code it very quickly. Shamil Salakhetdinov wrote: > > <<< > I am definitely interested in doing strongly typed recordsets though. > John, > > That strongly typed recordsets are imposing big overhead - you can dig into > the code generated (by VS) for these recordsets... > > ...as for batch updates "automagically" generated by ADO.Net - you can use > SQL profiler to see what happens "under the hood"... > > ...I'd bet that if you generate batch updates by custom code into temp SPs > and use datareader to get data to build these batches then this approach > will result in a way faster data processing, and you will have many options > to optimize it even more... > > ...just my opinion but as you can find in ASP.Net apps ADO.NET typed > recordsets are not recommended to use - have a look at e.g. "ASP.NET Website > Programming: Problem - Design - Solution C# Edition" > by Marco Bellinaso.... > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 3:32 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Thanks Shamil, > > Unfortunately unless you are dealing with 80 million row 200 field > tables, any timing comparisons are going to be suspect. > > I considered doing (and may still do) a record by record update using > either a stored procedure or dynamic sql right from vb.net. Then I read > an article (blog) that essentially said that the batch update from an > ado dataset really did work as you would hope that it would, and this in > a blog from a member of Microsoft's ADO.Net dev team. > > He claimed that either of the other methods had network transaction > overhead as each update (when done singly) required the command to do it > as well as a result coming back, whereas if you used a batch it would > group all of the commands out and then group all the results coming back > into big packages. > > Understand that I do not know enough to even comment on the validity of > his argument however he said to definitely try it. Set the batch size > and then let ADO just send batches of updates. > > Given that the coding effort is smaller to do that I figured I would at > least get that running. > > I am definitely interested in doing strongly typed recordsets though. > > Shamil Salakhetdinov wrote: >> Hi John, >> >> Try to use SqlDataReader and custom classes as in the sample code below >> (sorry C# is used - real pleasure to use C# is also coming from the fact >> that you can copy and paste code and do not care about line warps because >> even "screwed by e-mail" code should be compiled well).... >> >> ...using ADO.NET datasets and bulk update will anyway result in a series > of >> updates on SQL server side therefore custom on-the-fly built sql updates > or >> calls to update SPs should work as quick as ADO.NET DataSets' batch >> update... >> >> ...etc... >> >> ...as you can see my sample did update 25000 records in ~10 seconds - and >> this only for starters - if that approach will work reasonably well for > you >> then the next step could be to introduce multi-threading etc... >> >> ...note also how MS SQL 2005's paging feature is used to get record's >> batches... > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 04:08:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:08:08 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813CCBB.3060907@colbyconsulting.com> Message-ID: <000001c8a846$3606c6d0$6401a8c0@nant> Hello John, OK, I will read the blog you mentioned later today... I know it's not easy to convince you to change your mind when you are "driving your way full speed" :) And I'm not trying to - I'm just musing in this coding samples I post here testing this or that approach, which I might use in my own development... <<< There are many things to consider when I program in .net, not the least of which is simplicity. >>> May I say that proposed in C# solution is simple and almost ready to use in your scenario in one thread? And also ready to be converted into a multi-threading app/utility. And you don't need to translate in manually in VB.NET - just use this free tool: http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx or this one: http://codeconverter.sharpdevelop.net/Convert.aspx and voila' the converted VB.NET is below : (watch line wraps now - this is VB.NET - it was "automagically" converted by referred above tool by copying and pasting C# code I posted yesterday + making some manual editing of several code line to make it independent of my testing/development environment) - and it seems to work even quicker than C# equivalent somehow !?. I have got additional back VB.NET -> C# conversion in the end of this message - you can use C# code copy and paste it in the tool refereed above and get VB.NET code properly formatted. Proposal: let's post C# code here for the folks who will try this code to not spend time with fixing line wraps introduced by e-mail clients): Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) ID = id Name = name End Sub Public ReadOnly Property ParsedName() As String Get ' just for testing purposes ' return real parsed name here Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry '27.04.2008 12:45:58 'Count(*) = 25045 'Counter = 25045 '27.04.2008 12:46:04 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = _ String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", _ pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list p.Update(item) count += 1 Next End While Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace Here it's converted back to C#: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB"; string dbName = "dbName"; string userName = "sa"; string password = "sapwd"; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 4:46 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx <<< tail skipped to get message through in AccessD >>> From Gustav at cactus.dk Sun Apr 27 04:33:02 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 11:33:02 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From shamil at smsconsulting.spb.ru Sun Apr 27 04:42:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:42:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: Message-ID: <000601c8a84b$0711cdc0$6401a8c0@nant> Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sun Apr 27 05:53:36 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 14:53:36 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000601c8a84b$0711cdc0$6401a8c0@nant> Message-ID: <000301c8a854$f1c57cf0$6401a8c0@nant> ...now the VB.net version of manually batched updates (watch line wraps!) It seems to work slightly quicker than similar C# version: ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 'http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx 'http://codeconverter.sharpdevelop.net/Convert.aspx Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) id = id name = name End Sub Public ReadOnly Property ParsedName() As String ' just for testing purposes ' return real parsed name here Get Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Public ReadOnly Property BatchSize() As Integer Get Return _batchSize End Get End Property Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() 'error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function Private _updateCommands As String() = Nothing Private _arrayIndex As Integer = -1 Public Sub AddUpdateCommand(ByVal item As DataItem) If _arrayIndex = -1 Then _updateCommands = DirectCast(Array.CreateInstance(GetType(String), _batchSize), String()) _arrayIndex = 0 End If Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) 'badly translated => _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql _updateCommands(_arrayIndex) = sql _arrayIndex += 1 End Sub Public Function BatchUpdate() As Integer If _arrayIndex = -1 Then Return -1 End If If _arrayIndex < _batchSize Then Array.Resize(Of String)(_updateCommands, _arrayIndex) End If Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() 'Console.WriteLine("Batch update: {0} updates", _arrayIndex) Dim sql As String = [String].Join(";", _updateCommands) _arrayIndex = -1 cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list If count > 0 AndAlso count Mod p.BatchSize = 0 Then p.BatchUpdate() End If 'p.Update(item); p.AddUpdateCommand(item) count += 1 Next End While p.BatchUpdate() Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 1:43 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From jwcolby at colbyconsulting.com Sun Apr 27 07:12:19 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 08:12:19 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a846$3606c6d0$6401a8c0@nant> References: <000001c8a846$3606c6d0$6401a8c0@nant> Message-ID: <48146DA3.1010200@colbyconsulting.com> LOL, now that I can readi it... ;-) >> and voila' the converted VB.NET is below : (watch line wraps now - this is Shamil Salakhetdinov wrote: > Hello John, > > OK, I will read the blog you mentioned later today... > > I know it's not easy to convince you to change your mind when you are > "driving your way full speed" :) And I'm not trying to - I'm just musing in > this coding samples I post here testing this or that approach, which I might > use in my own development... > > <<< > There are many things to consider when I program in .net, not the least > of which is simplicity. > May I say that proposed in C# solution is simple and almost ready to use in > your scenario in one thread? And also ready to be converted into a > multi-threading app/utility. And you don't need to translate in manually in > VB.NET - just use this free tool: > > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > > or this one: > > http://codeconverter.sharpdevelop.net/Convert.aspx > > and voila' the converted VB.NET is below : (watch line wraps now - this is > VB.NET - it was "automagically" converted by referred above tool by copying > and pasting C# code I posted yesterday + making some manual editing of > several code line to make it independent of my testing/development > environment) - and it seems to work even quicker than C# equivalent somehow > !?. I have got additional back VB.NET -> C# conversion in the end of this > message - you can use C# code copy and paste it in the tool refereed above > and get VB.NET code properly formatted. Proposal: let's post C# code here > for the folks who will try this code to not spend time with fixing line > wraps introduced by e-mail clients): > > Imports System > Imports System.Collections.Generic > Imports System.Data.SqlClient > > Namespace TestSamples > > Public Class DataItem > Public ID As Integer > Public Name As String > > Public Sub New(ByVal id As Integer, ByVal name As String) > ID = id > Name = name > End Sub > > Public ReadOnly Property ParsedName() As String > Get > ' just for testing purposes > ' return real parsed name here > Return Name > End Get > End Property > End Class > > Public Class DataProcessor > Private Const COUNT_SQL As String = "select count(*) from > zsysUserSessionTrace" > Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + > " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as > int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " > UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum > between {0} and {1}" > Private Const UPDATE_SQL As String = "update zsysUserSessionTrace > set UserBrowser = '{0}' where UserSessionTraceId = {1}" > > Private Const DEFAULT_BATCH_SIZE As Integer = 1000 > > Private _totalCount As Integer = -1 > Public ReadOnly Property TotalCount() As Integer > Get > Return _totalCount > End Get > End Property > > Private _currentBatchStartRowNum As Integer = 0 > Private _batchSize As Integer = 0 > > Private _connectionString As String > Public Sub New(ByVal connectionString As String) > initialize(connectionString, DEFAULT_BATCH_SIZE) > End Sub > Public Sub New(ByVal connectionString As String, ByVal batchSize As > Integer) > initialize(connectionString, batchSize) > End Sub > Private Sub initialize(ByVal connectionString As String, ByVal > batchSize As Integer) > > _batchSize = batchSize > _currentBatchStartRowNum = 1 > _connectionString = connectionString > > Using cnn As New > System.Data.SqlClient.SqlConnection(connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > cmd.CommandText = COUNT_SQL > cmd.CommandType = System.Data.CommandType.Text > > _totalCount = CInt(cmd.ExecuteScalar()) > End Using > End Sub > > Public ReadOnly Property HasData() As Boolean > Get > Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < > TotalCount) > End Get > End Property > > Public Function GetData() As List(Of DataItem) > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > _currentBatchStartRowNum += _batchSize + 1 > Dim rdr As System.Data.SqlClient.SqlDataReader = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) > > Dim list As New List(Of DataItem)() > While rdr.Read() > list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), > String))) > End While > Return list > End Using > End Function > > Public Function Update(ByVal item As DataItem) As Integer > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(UPDATE_SQL, > item.ParsedName, item.ID) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > Return cmd.ExecuteNonQuery() > End Using > End Function > End Class > > > Public Class A_MainEntry > '27.04.2008 12:45:58 > 'Count(*) = 25045 > 'Counter = 25045 > '27.04.2008 12:46:04 > > Public Shared Sub Main() > Dim count As Integer = 0 > Try > '+ customize - set your connection string here > Dim pcName As String = "SPB" > Dim dbName As String = "dbName" > Dim userName As String = "sa" > Dim password As String = "sapwd" > Dim connectionString As String = _ > String.Format("Data Source={0}\SQLEXPRESS;Initial > Catalog={1};User Id={2};Password={3}", _ > pcName, dbName, userName, password) > '- > > Console.WriteLine(DateTime.Now) > Dim p As New DataProcessor(connectionString) > While p.HasData > Dim list As List(Of DataItem) = p.GetData() > For Each item As DataItem In list > p.Update(item) > count += 1 > Next > End While > Console.WriteLine("Count(*) = {0}", p.TotalCount) > Console.WriteLine("Counter = {0}", count) > Console.WriteLine(DateTime.Now) > Catch ex As Exception > Console.WriteLine(ex.Message) > End Try > End Sub > > End Class > End Namespace > > Here it's converted back to C#: > > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > using System; > using System.Collections.Generic; > using System.Data.SqlClient; > > namespace TestSamples > { > > public class DataItem > { > public int ID; > public string Name; > > public DataItem(int id, string name) > { > ID = id; > Name = name; > } > > public string ParsedName > { > // just for testing purposes > // return real parsed name here > get { return Name; } > } > } > > public class DataProcessor > { > private const string COUNT_SQL = "select count(*) from > zsysUserSessionTrace"; > private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " > select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) > as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " > + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and > {1}"; > private const string UPDATE_SQL = "update zsysUserSessionTrace set > UserBrowser = '{0}' where UserSessionTraceId = {1}"; > > private const int DEFAULT_BATCH_SIZE = 1000; > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(1), (string)rdr(3))); > list.Add(new DataItem((int)rdr[1], (string)rdr[3])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(UPDATE_SQL, item.ParsedName, > item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > } > > > public class A_MainEntry > { > // VB.NET > //27.04.2008 13:01:44 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:49 > > // C# > //27.04.2008 13:00:56 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:04 > > public static void Main() > { > int count = 0; > try > { > //+ customize - set your connection string here > string pcName = "SPB"; > string dbName = "dbName"; > string userName = "sa"; > string password = "sapwd"; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > Console.WriteLine(DateTime.Now); > DataProcessor p = new DataProcessor(connectionString); > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > Console.WriteLine(DateTime.Now); > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 4:46 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > This is the blog I mentioned: > > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > <<< tail skipped to get message through in AccessD >>> > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 10:14:00 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:14:00 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48146DA3.1010200@colbyconsulting.com> Message-ID: <001401c8a879$5298bd70$6401a8c0@nant> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil From shamil at smsconsulting.spb.ru Sun Apr 27 10:24:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:24:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <000001c8a87a$cfa80270$6401a8c0@nant> ...additionally to my quoted posting here is SQL script to create test table: CREATE TABLE [dbo].[zsysUserSessionTrace]( [UserSessionTraceId] [int] IDENTITY(1,1) NOT NULL, [AspNetSessionId] [nvarchar](80) NOT NULL, [HostAddress] [nvarchar](40) NULL, [HostName] [nvarchar](40) NULL, [PageViewCount] [int] NULL, [UserName] [nvarchar](40) NOT NULL, [UserBrowser] [nvarchar](255) NOT NULL, [UpdatedBy] [nvarchar](20) NOT NULL, [UpdatedDate] [datetime] NOT NULL, [UpdatedLocation] [nvarchar](20) NOT NULL, [ParsedUserBrowserName] [nvarchar](255) NULL, CONSTRAINT [PK_zsysUserSessionTrace] PRIMARY KEY CLUSTERED ( [UserSessionTraceId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] ...and here is insert startemet to populate one row - you can run it many times: insert into zsysUserSessionTrace (AspNetSessionId, UserName, UserBrowser, UpdatedBy, UpdatedDate, UpdatedLocation) values (1,'TEST','IE7','SA','2008-04-27','SPB') ... and here are select statements to run e.g. in MS SQL Management console to see that test updates really happened: select count(*) from zsysUserSessionTrace go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId] desc) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 7:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 27 11:42:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 18:42:13 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil I'm following this thread with great interest because it touches some tasks I may soon encounter. It's a pleasure to study your work, and I really appreciate that you take your time to post it here. /gustav >>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? From jwcolby at colbyconsulting.com Sun Apr 27 11:56:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 12:56:40 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> References: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <4814B048.6070704@colbyconsulting.com> What I come hear is that using the built-in batch is as fast as any other method, so if that is what I find easier to use there is no issue using that. I will try to understand your other methods (in VB, since the language makes very little apparent speed difference) and implement that in parallel so that I can switch between them on demand. If I can do that I will do some timings on my big database, which is what I am doing the updates on. Shamil Salakhetdinov wrote: > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > I wasn't aware of MS SQL TDS feature you mentioned but as you can see from > stats it gives just slightly better results than the manual batching. Of > course playing with batch size might give better gains of this method (my > tests didn't show that gains depending on batch size) but I must say for me > it looks more complicated and less flexible that using SQLDatareader and > custom classes - that could be just my perception.... > > The C# code is below in the end of this message - its VB.NET equivalent can > be obtained using automatic conversion - > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one > code line have to be manually patched > > 'conversion issue replace the following code line => > ==================================================== > _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra > yIndex), _arrayIndex - 1)) = sql > > with this one > ============= > _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 > : > > Here are the stats: > > ==================== > C# > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:02:58 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:15 > Elapsed time: 00:00:16.7968750 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:03:15 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:23 > Elapsed time: 00:00:08.4218750 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:03:23 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:31 > Elapsed time: 00:00:07.9062500 > > ==================== > VB.NET > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:05:20 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:36 > Elapsed time: 00:00:16.7500000 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:05:36 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:45 > Elapsed time: 00:00:08.2812500 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:05:45 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:52 > Elapsed time: 00:00:07.5000000 > ==================== > > // Converters: > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > // Sql TDS batched update > //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > using System; > using System.Collections.Generic; > using System.Data; > using System.Data.SqlClient; > > namespace TestSamples > { > public sealed class Constants > { > //+ ///////////// C U S T O M I Z E ///////////////////// > public const string PcName = "NANT"; > public const string DbName = "ORS2005"; > public const string UserName = "sa"; > public const string Password = "shms"; > > public const string TABLE_NAME = "zsysUserSessionTrace"; > public const string ID_COL_NAME = "UserSessionTraceId"; > public const string NAME_COL_NAME = "UserBrowser"; > public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; > > public const int DEFAULT_BATCH_SIZE = 1000; > > public static string GetParsedName(string name) > { > // just for testing purposes - customzie > // to return your parsed name here > if (name == null) name = ""; > name = DateTime.Now.ToString() + ": " + name; > if (name.Length > 255) name = name.Substring(0, 255); > > return name; > } > //- ///////////// C U S T O M I Z E ///////////////////// > > public const int ID_COL_INDEX = 1; > public const int NAME_COL_INDEX = 2; > public const int PARSED_NAME_COL_INDEX = 3; > > public static string COUNT_SQL { get { > return String.Format("select count(*) from {0}", TABLE_NAME); } > } > > public static string PAGED_SELECT_SQL { get { > return string.Format( > "SELECT * from " + "(" + " select " + > " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as > RowNum, " + > " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", > ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, > PARSED_NAME_COL_NAME, TABLE_NAME) + > "where s.RowNum between {0} and {1}"; }} > > public static string UPDATE_SQL { get { > return string.Format("update {0} ",TABLE_NAME) + > string.Format(" set {0} = '{1}' where {2} = {3}", > PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); > }} > > public static string UPDATE_SQL_PARAMETERIZED { get { > return string.Format("update {0} set {1} = @{2} where {3} = > @{4}", > TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, > ID_COL_NAME, ID_COL_NAME); > }} > } > > public class DataItem > { > private int _id; > public int ID > { > get { return _id; } > } > > private string _name; > public string Name > { > get { return _name; } > } > > public DataItem(int id, string name) > { > _id = id; > _name = name; > } > > public string ParsedName > { > get > { > return Constants.GetParsedName(_name); > } > } > > > } > > public class DataProcessor > { > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > public int BatchSize { get { return _batchSize; } } > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = Constants.COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(Constants.ID_COL_INDEX), > (string)rdr(Constants.NAME_COL_INDEX))); > list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], > (string)rdr[Constants.NAME_COL_INDEX])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private string[] _updateCommands = null; > private int _arrayIndex = -1; > public void AddUpdateCommand(DataItem item) > { > if (_arrayIndex == -1) > { > _updateCommands = > (string[])Array.CreateInstance(typeof(string), _batchSize); > _arrayIndex = 0; > } > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > _updateCommands[_arrayIndex++] = sql; > } > > public int BatchUpdate() > { > if (_arrayIndex == -1) return -1; > if (_arrayIndex < _batchSize) > Array.Resize(ref _updateCommands, _arrayIndex); > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > > //Console.WriteLine("Batch update: {0} updates", > _arrayIndex); > > string sql = String.Join(";", _updateCommands); > _arrayIndex = -1; > > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private SqlDataAdapter _da = null; > private DataSet _ds = null; > public DataTable SQLTDSGetDataTablePaged() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > int batchSize = _batchSize; > if (_currentBatchStartRowNum + _batchSize >= _totalCount) > batchSize = _totalCount - _currentBatchStartRowNum; > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > > _currentBatchStartRowNum += _batchSize + 1; > > _da = new SqlDataAdapter(sql, cnn); > _ds = new DataSet(); > _da.Fill(_ds); > > _da.UpdateCommand = new > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); > _da.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, > Constants.PARSED_NAME_COL_NAME); > _da.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); > _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; > _da.UpdateBatchSize = batchSize; > > _da.SelectCommand.Connection = null; > _da.UpdateCommand.Connection = null; > > return _ds.Tables[0]; > } > } > > public void SQLTDSUpdate() > { > SQLTDSUpdate(false); > } > public void SQLTDSUpdate(bool forceUpdate) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > _da.SelectCommand.Connection = cnn; > _da.UpdateCommand.Connection = cnn; > if (forceUpdate) > _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; > _da.Update(_ds.Tables[0]); > > //Console.WriteLine("*** Forced test exit ***"); > //System.Environment.Exit(100); > } > } > } > > public class A_MainEntry > { > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate and execute individual update SQL > statements > /// > public static void Strategy1(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > //Console.WriteLine("{0} - rows processed...", p.BatchSize); > > } > } > > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate batched update SQL statements, and > /// execute update batch when it has all update > /// statements added > /// > public static void Strategy2(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > if (count > 0 && (count % p.BatchSize) == 0) > p.BatchUpdate(); > p.AddUpdateCommand(item); > count += 1; > } > } > p.BatchUpdate(); > } > > /// > /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, > and > /// use SQL TDS bacthed updates facility > /// > public static void Strategy3(DataProcessor p, ref int count) > { > while (p.HasData) > { > DataTable table = p.SQLTDSGetDataTablePaged(); > > foreach (DataRow row in table.Rows) > { > row.SetModified(); > row[Constants.PARSED_NAME_COL_INDEX] = > Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); > count += 1; > } > p.SQLTDSUpdate(!p.HasData); > } > } > > public static void Main() > { > try > { > //+ customize - set your connection string here > string pcName = Constants.PcName; > string dbName = Constants.DbName; > string userName = Constants.UserName; > string password = Constants.Password; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > for (int i = 1; i <= 3; i++) > { > int count = 0; > DateTime startTime, endTime; > startTime = DateTime.Now; > Console.WriteLine(""); > Console.WriteLine("Strategy {0}:", i); > Console.WriteLine("-----------"); > Console.WriteLine("Start time: {0}", startTime); > DataProcessor p = new DataProcessor(connectionString); > > switch (i) > { > case 1: > Strategy1(p, ref count); > break; > case 2: > Strategy2(p, ref count); > break; > case 3: > Strategy3(p, ref count); > break; > default: > Console.WriteLine("{0} - undefined strategy > number", i); > break; > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > endTime = DateTime.Now; > Console.WriteLine("End time: {0}", endTime); > TimeSpan ts = > TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); > Console.WriteLine("Elapsed time: {0}", ts); > } > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > Enjoy! > > -- > Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sun Apr 27 14:13:04 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sun, 27 Apr 2008 15:13:04 -0400 Subject: [dba-VB] ADO.Net References: Message-ID: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Sun Apr 27 16:02:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 01:02:11 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> Message-ID: <000401c8a8a9$f695a200$6401a8c0@nant> Here are the sources I promised to publish in my previous post: http://smsconsulting.spb.ru/samples/Stats.txt http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt Your turn, guys, to fix and improve them... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of William Hindman Sent: Sunday, April 27, 2008 11:13 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 27 22:01:58 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 23:01:58 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000401c8a8a9$f695a200$6401a8c0@nant> References: <000401c8a8a9$f695a200$6401a8c0@nant> Message-ID: <48153E26.3040803@colbyconsulting.com> Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Mon Apr 28 01:52:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 08:52:21 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all What happened? How have I been fooling around with covered eyes? If I select Flat for property FlatStyle the ComboBox changes to ... a flat design. No visible borders. Only issue is that the invisible borders of the ComboBox appears to be as wide as if the ComboBox has 3D-style, thus leaving a larger total area than an equivalent TextBox with identical font settings. If I force the size of the ComboBox to be lower, it won't display the content (not very useful). /gustav >>> Gustav at cactus.dk 23-04-2008 08:26 >>> Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Mon Apr 28 10:53:55 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 17:53:55 +0200 Subject: [dba-VB] C# event handler syntax Message-ID: Hi all Just browsed this page: http://blueonionsoftware.com/Blog.aspx?p=8afe238f-87c2-400b-9440-8231280f7576 which has a brief explanation on the syntax from the beginning to the ultra compact of version 3.5 using Lambda expression: clock.Alarm += (o, e) => Console.Write("Alarm!"); This is _not_ what IntelliSense suggests, so you are a little on your own. /gustav From shamil at smsconsulting.spb.ru Mon Apr 28 11:40:25 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 20:40:25 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48153E26.3040803@colbyconsulting.com> Message-ID: <00c301c8a94e$8f660430$6401a8c0@nant> Hi John, Thank you for your proposal to run the code against your huge db... OK, let's try this remote/offline Q&A/code musing/consulting/education in several installments published here: I must say I have quite some work to do for customers and I can't spend a lot of time for postings here/explaining what and why is done in the sample code - I will try just give refs on "concepts" I use for you to read more in many open sources... I made first adjustment to the yesterday's code - new code is here: http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt http://smsconsulting.spb.ru/samples/Stats.txt To run it (let's use VB.NET): - in VS2005 create VB.NET Console application, replace the code of the module, which VS2005 creates by default (Module1.vb) with the code from http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines Public Const PcName As String = "LOCALHOST" Public Const DbName As String = "NamesParser" Public Const UserName As String = "TESTER" Public Const Password As String = "TEST.1959" to point to your db and its credentials, push CTRL+F5, and you should see the sample app running in console window and producing something like this report: Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:13 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:19 Elapsed time: 00:00:05.5156250 Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:19 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:22 Elapsed time: 00:00:03.5468750 Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:22 Count(*) = 1000 Counter = 1009 End time: 28.04.2008 20:18:26 Elapsed time: 00:00:03.3906250 Try. If you tell me the name of your database and the name of your table I will change them here, as well as you can create username and password as above to not do any changes later for the new versions of the code. N.B: Make sure you will run the code against sample not the live db. The code assumes that the length of the target field FName is 50 chars - it uses just one this column currently to put "parsed" value - just first 50 chars of OWNERNAME field. Please have a look through the code, and ask your questions in the order you wanted them to be answered... Please everybody who wanted to participate in this discussion - you're very welcome! On debugging multi-threaded apps: that's a real PITA - yes you can debug/trace with breakpoints etc. but you have to keep in mind that as many threads as you have will cause debugger to stop on breakpoints etc. IOW better debug/trace single threaded code and them run it in multi-threaded mode, and use some indirect artefacts/techniques (as unit testing), which will show you that your app is running OK: as you can find in the new version of the code it can run both single and multi-threaded depending on the value of this constant: Public Const MULTI_THREADED_MODE As Boolean = True -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 7:02 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 11:55:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 12:55:08 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <00c301c8a94e$8f660430$6401a8c0@nant> References: <00c301c8a94e$8f660430$6401a8c0@nant> Message-ID: <4816016C.4010709@colbyconsulting.com> Thanks for the response Shamil, I know that you have lots of stuff to get done as do I. I am already back to the code I know in order to get this thing processed, however I will get your sample code running in the coming days. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > Thank you for your proposal to run the code against your huge db... > > OK, let's try this remote/offline Q&A/code musing/consulting/education in > several installments published here: I must say I have quite some work to do > for customers and I can't spend a lot of time for postings here/explaining > what and why is done in the sample code - I will try just give refs on > "concepts" I use for you to read more in many open sources... > > I made first adjustment to the yesterday's code - new code is here: > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > http://smsconsulting.spb.ru/samples/Stats.txt > > To run it (let's use VB.NET): > > - in VS2005 create VB.NET Console application, replace the code of the > module, which VS2005 creates by default (Module1.vb) with the code from > http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines > > Public Const PcName As String = "LOCALHOST" > Public Const DbName As String = "NamesParser" > Public Const UserName As String = "TESTER" > Public Const Password As String = "TEST.1959" > > to point to your db and its credentials, push CTRL+F5, and you should see > the sample app running in console window and producing something like this > report: > > Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:13 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:19 > Elapsed time: 00:00:05.5156250 > > Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:19 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:22 > Elapsed time: 00:00:03.5468750 > > Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:22 > Count(*) = 1000 > Counter = 1009 > End time: 28.04.2008 20:18:26 > Elapsed time: 00:00:03.3906250 > > Try. > > If you tell me the name of your database and the name of your table I will > change them here, as well as you can create username and password as above > to not do any changes later for the new versions of the code. > > N.B: Make sure you will run the code against sample not the live db. The > code assumes that the length of the target field FName is 50 chars - it uses > just one this column currently to put "parsed" value - just first 50 chars > of OWNERNAME field. > > Please have a look through the code, and ask your questions in the order you > wanted them to be answered... > > Please everybody who wanted to participate in this discussion - you're very > welcome! > > On debugging multi-threaded apps: that's a real PITA - yes you can > debug/trace with breakpoints etc. but you have to keep in mind that as many > threads as you have will cause debugger to stop on breakpoints etc. IOW > better debug/trace single threaded code and them run it in multi-threaded > mode, and use some indirect artefacts/techniques (as unit testing), which > will show you that your app is running OK: as you can find in the new > version of the code it can run both single and multi-threaded depending on > the value of this constant: > > Public Const MULTI_THREADED_MODE As Boolean = True > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 7:02 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Shamil, > > I would like to thank you for taking the time to do this. Now... I will > thank you in advance for taking the time to explain it to me. As > happened many years ago when you taught me WithEvents in Access, I don't > know enough to understand what the heck you are doing. > > In the absence of any instructions on how to use this thing, I cut and > pasted the entire (VB) shebang into a class in a Windows Form project. > It compiled flawlessly and the blank form opened. > > Now what? > > I hope that you can appreciate my lack of knowledge here. I have built > several projects, with (to my pitiful level of expertise) quite > extensive stuff going on but... there are a lot of concepts in your code > that I have not used. Where to start. > > I guess where to start is a good spot. What do I do to cause this to > run? Coming from Access I have always started with a form, which I > promptly place a button on so I can call some function which "starts" > things. That form will have class variables if necessary to instantiate > a supervisor class, the supervisor class will then load recordsets, load > data into data classes as required etc. So what I do not understand is > "what do I call to make this run"? I see the MainEntry at the bottom > but do I just call MainEntry from a button? Does the code belong in a > class or a plain module? I assume that if I put it in a class then I > would have to dim a variable in my form. > > Second (I am totally excited about this whole thing!!!) what are the > replaceable parameters? > > Dim connectionString As String = String.Format("Data > Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password) > > I assume here that pcName gets fed in to {0}? THAT is cool, and > demonstrates where I start from in understanding your code. > > Third (also exciting!) I have never used threads yet so here we go. How > do I debug that? Can I step through a thread in the same way I step > through any other code? Set break points etc. > > And finally (for now) it just occurred to me to ask... can you help me > (online of offline) to modify this example to actually use my own table > that I need to parse the name for. I think I can figure out how to > modify the stuff for the server, database, table etc. I have done that > already in my own code for doing this. I will also worry about the > parsing code itself, though just for a demo we can just hard code some > values. > > > My table Data: > > PKID (int32) > OWNERNAME (to be parsed) > FName (parsed) > MName (parsed) > LName (parsed) > NamePrefix (parsed) > NameSuffix (parsed) > Gender (parsed) > > If I can modify they example to actually read / write to my table then I > kill two birds with one stone, learn a bunch of cool new stuff including > getting it all happening with threads, and get my real work done. > > For my real life purposes I am going to have to have a PKStart, PKEnd > pair which tracks a block (remember I am doing 80 million rows > eventually), and increment those start/end by the chunk size to repeat > over and over (eventually). What I have found convenient in the past > (and have code already written to do) is write status log files out to > disk for each block processed. I write them as XML and include things > like the start / end PKID, status memo fields, time start / stop and the > like. As I said, I have already written that code (I'm not completely > helpless, though it sometimes appears that way) and sitting out in a > library ready to use for the logging. All I would need to do is build > the specific data class to store the log data for this project. I will > handle that. > > I will promise to absorb your example if you will promise to take the > time to help me when I get stuck. Once I have done so I will then be > able to apply it to my specific problem and give you back some real life > timings from my largish database. I can vary the chunk size from 1k to > any upward limit we find useful to test. My experience has been that > anything above 10K can get problematic, though that was using the bulk > import widget in past projects. I was getting timeouts if I went with > too large a chunk size. > > VB.Net is only a small part of my work life, though I would like to make > it the main part. I still spend most of my work life stuck in Access, > maintaining applications I wrote (or inherited) long ago. While it pays > the bills I would LOVE to get to the point where I could bang out code > like this in the short time that it probably took you. > > Again thanks for your time and effort and patience. > > Shamil Salakhetdinov wrote: >> Here are the sources I promised to publish in my previous post: >> >> http://smsconsulting.spb.ru/samples/Stats.txt >> >> http://smsconsulting.spb.ru/samples/Program.cs.txt >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> Your turn, guys, to fix and improve them... >> >> -- >> Shamil >> > From jwcolby at colbyconsulting.com Mon Apr 28 13:10:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:10:08 -0400 Subject: [dba-VB] ADO.Net data table update Message-ID: <48161300.8050807@colbyconsulting.com> I have a datatable that I opened. It contains a set of 10K records which I have updated pieces of. I now need to write it back to SQL Server. If I just call the da.update it errors saying it needs an update command. Do I have to build an update SQL statement or will the DA just figure out where the data fields came from and update them back out to SQL Server? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 13:49:38 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:49:38 -0400 Subject: [dba-VB] ADO.Net Update SQL Server Message-ID: <48161C42.1030500@colbyconsulting.com> Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Mon Apr 28 14:04:08 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Mon, 28 Apr 2008 12:04:08 -0700 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> References: <48161C42.1030500@colbyconsulting.com> Message-ID: UpdateBatch should work. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 11:50 AM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Mon Apr 28 14:09:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 23:09:11 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> Message-ID: <010001c8a963$57cd7c00$6401a8c0@nant> Hi John, The answer is in the code I published today on my site related to 'Re: [dba-VB] ADO.Net': http://smsconsulting.spb.ru/samples/Module.vb.txt - Strategy 3 - trace in single threaded mode and you'll find what to do. Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As DataSet) As DataTable ... And Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet As DataSet) .... HTH, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 10:50 PM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 14:57:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 15:57:15 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: References: <48161C42.1030500@colbyconsulting.com> Message-ID: <48162C1B.8060207@colbyconsulting.com> I hate to sound ignorant but what is the UpdateBatch a method of? I have a connection, command, dataset and datatable and AFAICT none of those things have an UpdateBatch method. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > UpdateBatch should work. > > Charlotte Foust > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 11:50 AM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 15:14:01 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 16:14:01 -0400 Subject: [dba-VB] ADO.Net Update SQL Server References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> Message-ID: <01d001c8a96c$676b5a70$0502a8c0@Laptop> It is a method of the ADO.Recordset. I was ignorant of this, also, until I looked it up in a book. Ignorant<>something you can sound like Either you know or you don't know :) Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 3:57 PM Subject: Re: [dba-VB] ADO.Net Update SQL Server >I hate to sound ignorant but what is the UpdateBatch a method of? > > I have a connection, command, dataset and datatable and AFAICT none of > those things have an UpdateBatch method. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> UpdateBatch should work. >> >> Charlotte Foust >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 11:50 AM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> >> -- >> John W. Colby >> www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 15:17:48 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:17:48 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <010001c8a963$57cd7c00$6401a8c0@nant> References: <010001c8a963$57cd7c00$6401a8c0@nant> Message-ID: <481630EC.5060600@colbyconsulting.com> When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > From jwcolby at colbyconsulting.com Mon Apr 28 15:29:52 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:29:52 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <01d001c8a96c$676b5a70$0502a8c0@Laptop> References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> <01d001c8a96c$676b5a70$0502a8c0@Laptop> Message-ID: <481633C0.9060808@colbyconsulting.com> ROTFL, Oh I think I succeed very well at sounding ignorant. 8-( And yup, I don't know. I have beaten my head against this all day with absolutely zero results. I find it difficult to believe that I am doing anything unique but I'll be damned if I can find an article or code that just says "here is how you write back the updated table to the sql server... (data store)". John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > It is a method of the ADO.Recordset. > I was ignorant of this, also, until I looked it up in a book. > Ignorant<>something you can sound like > Either you know or you don't know :) > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Monday, April 28, 2008 3:57 PM > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > >> I hate to sound ignorant but what is the UpdateBatch a method of? >> >> I have a connection, command, dataset and datatable and AFAICT none of >> those things have an UpdateBatch method. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Charlotte Foust wrote: >>> UpdateBatch should work. >>> >>> Charlotte Foust >>> >>> -----Original Message----- >>> From: dba-vb-bounces at databaseadvisors.com >>> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >>> Sent: Monday, April 28, 2008 11:50 AM >>> To: VBA >>> Subject: [dba-VB] ADO.Net Update SQL Server >>> >>> Does ANYBODY know how to write a table back out to SQL Server? >>> >>> All of my books show cute little examples of building up a data table >>> manually then BINDING it to a control or whatever. I am not finding >>> anything with code example that shows how to WRITE changed data back to >>> SQL Server. >>> >>> I have managed to load a data table, write changes to the rows and >>> now... I have to write it back out to SQL Server. The table has the PK, >>> NameToSplit, LName, FName etc etc. What am I missing? >>> >>> I desperately need help (yea, yea, not MENTAL help) and I need it >>> quickly. I have been trying to do this last piece for DAYS and I am not >>> succeeding. >>> >>> -- >>> John W. Colby >>> www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From James at fcidms.com Mon Apr 28 16:02:31 2008 From: James at fcidms.com (James Barash) Date: Mon, 28 Apr 2008 17:02:31 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> John: If you already have a data adapter that you used to fill your data table, you need to add an UpdateCommand to that data adapter and then use it to update the datastore. Below is a very quick and dirty version from an existing program I've used. It should at least give you a place to start. It assumes you already have a data adapter named da, a data table named dt and a connection named conn. You will have to change the table and field names to match yours. da.UpdateCommand = New SqlClient.SqlCommand da.UpdateCommand.Connection = conn da.UpdateCommand.CommandType = CommandType.Text da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldDisplayValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldNumber", System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.Update(dt) James Barash -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 4:18 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 17:11:07 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:11:07 -0400 Subject: [dba-VB] vb.net - Updates Message-ID: <48164B7B.2070808@colbyconsulting.com> I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 17:12:37 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:12:37 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> References: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> Message-ID: <48164BD5.60906@colbyconsulting.com> Thanks James. I also found a piece of code that shows how to use a commandbuilder object to generate those update commands automatically. At my skill level, automatic is a good thing. John W. Colby www.ColbyConsulting.com James Barash wrote: > John: > If you already have a data adapter that you used to fill your data table, > you need to add an UpdateCommand to that data adapter and then use it to > update the datastore. Below is a very quick and dirty version from an > existing program I've used. It should at least give you a place to start. It > assumes you already have a data adapter named da, a data table named dt and > a connection named conn. You will have to change the table and field names > to match yours. > > > da.UpdateCommand = New SqlClient.SqlCommand > da.UpdateCommand.Connection = conn > da.UpdateCommand.CommandType = CommandType.Text > da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, > FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", > "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldDisplayValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, > Nothing, "", "", "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldNumber", > System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, > "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", > "", "")) > da.Update(dt) > > James Barash > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 4:18 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > When I try to call the update method of the data set object I get an error: > > "Update requires a valid update command when passed datarow collection > with modified rows." > > I probably sound ungrateful but in your code you appear to set up a sql > statement for an update statement: > > adapter.UpdateCommand = New > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, > Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) > adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None > adapter.UpdateBatchSize = pageSize > > > My problem is that is just Chinese to me. I have no background > knowledge in plain English of what an update sql statement is supposed > to look like (in the ADO context) and absolutely no understanding of > even what I am trying to accomplish (expressed in plain English). I am > uneducated, and I can find no way to get educated. > > I must own 6 books on ADO.Net and VB and every single one of them > expects me to drag and drop objects out onto a form. OR they very > helpfully set everything up and show me how to do stuff RIGHT UP TO > USING IT, but never go that last step of sending updates back to the > database. Open, update, display in a grid, cleanup and close. Hmm.... > and I paid good money for these books. > > They do not discuss anything at all about what is going on at the code > level. So when I try to walk through your code it is simply > indecipherable to me. If I don't know what you are trying to accomplish > I can't understand how you code will accomplish that thing. > > Please do not take these statements as criticisms of your code, simply > that I am a freshman student still taking the basket weaving classes > looking at masters code. Thus it isn't useful to tell me to trace > through the code. > > MY code doesn't do any of that stuff. My code is pretty simple: > > Set up a connection, a data adapter, a table. Iterate the table row by > row, updating fields of the rows. > > Now... how to update the table back to SQL Server... > > Given how simple everything has been up to this point I have to believe > that I am just missing a concept and code to implement that concept. I > have called the update method and I have googled the error messages and > nothing... > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> The answer is in the code I published today on my site related to 'Re: >> [dba-VB] ADO.Net': >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> - Strategy 3 - trace in single threaded mode and you'll find what to do. >> >> Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, > ByVal >> pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As >> DataSet) As DataTable >> ... >> >> And >> >> Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet >> As DataSet) >> .... >> >> >> HTH, >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 10:50 PM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Mon Apr 28 18:31:14 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 19:31:14 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> References: <48164B7B.2070808@colbyconsulting.com> Message-ID: <48165E42.20404@colbyconsulting.com> We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. From ebarro at verizon.net Mon Apr 28 19:44:24 2008 From: ebarro at verizon.net (Eric Barro) Date: Mon, 28 Apr 2008 17:44:24 -0700 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> Message-ID: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Yeah it is a terrible thing from a DBA point of view because it opens it up to SQL injection attacks since it builds your SQL statement on the fly. I've tried it before and since it is generating it on the fly it makes it all the more harder to debug and offers the least amount of control from a DBA point of view. So...I let SQL server do all the work on the back end when it comes to batch updates by collecting the parameters and sending it to SQL server to process for the records it needs to process. Based on what you are trying to do you are building an image of the data structure in memory, manipulating the data in memory and then committing the changes to the database in memory basically sending the update command for each record to the database back end right? Or am I mistaken in thinking that all of this magically happens by sending a single update command containing several individual update commands strung together? -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 3:11 PM To: VBA Subject: [dba-VB] vb.net - Updates I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 20:08:20 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 21:08:20 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Message-ID: <48167504.6070909@colbyconsulting.com> It all happens automagically. At this point I modify about 10K records at a clip, then issue a single update and it goes. I set the batch size to 100 for this experiment but I will probably play with that to see what can be done. I tried a thousand but it timed out. I don't see any timeout period property though there might well be one somewhere. I understand the SQL Injection attack thing but you need to remember that this is a database running on my system that only I ever access. Even more importantly though, I need this kind of thing to just work, as quickly as I can make it happen. I am not a high powered DBA (nor any other powered DBA) nor a high powered .Net programmer. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Yeah it is a terrible thing from a DBA point of view because it opens it up > to SQL injection attacks since it builds your SQL statement on the fly. > > I've tried it before and since it is generating it on the fly it makes it > all the more harder to debug and offers the least amount of control from a > DBA point of view. So...I let SQL server do all the work on the back end > when it comes to batch updates by collecting the parameters and sending it > to SQL server to process for the records it needs to process. > > Based on what you are trying to do you are building an image of the data > structure in memory, manipulating the data in memory and then committing the > changes to the database in memory basically sending the update command for > each record to the database back end right? Or am I mistaken in thinking > that all of this magically happens by sending a single update command > containing several individual update commands strung together? > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 3:11 PM > To: VBA > Subject: [dba-VB] vb.net - Updates > > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece of > code that, in simple plain language, shows how to use it all, using the > simple objects I am already using. > > We shall see. > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 21:26:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 22:26:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> <48167504.6070909@colbyconsulting.com> Message-ID: <034b01c8a9a0$7b784440$0502a8c0@Laptop> These fit the bill ... How To: Protect From SQL Injection in ASP.NET http://msdn2.microsoft.com/en-us/library/ms998271.aspx Using Stored Procedures with SQLDataAdapter? http://forums.devx.com/showthread.php?t=148707 SQLDataAdapter without using SQLCommandBuilder http://www.codeproject.com/KB/database/sqldawithoutsqlcb.aspx Using Stored Procedures in Conjunction with the SqlDataAdapter http://www.code-magazine.com/Article.aspx?quickid=0303141 Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 9:08 PM Subject: Re: [dba-VB] vb.net - Updates > I understand the SQL Injection attack thing but you need to remember > that this is a database running on my system that only I ever access. From jwcolby at colbyconsulting.com Tue Apr 29 00:15:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 01:15:45 -0400 Subject: [dba-VB] Name parsing system timing Message-ID: <4816AF01.8090500@colbyconsulting.com> OK, it is running in a loop now, processing 10K records every 19 seconds. It will take 50 hours at the current rate to parse 96.4 million records, or about 2 million records / hour. the SQL Server update portion is taking about 5 seconds of each 19 second cycle, 14 seconds to parse, 5 seconds to write back to the database, virtually nothing to load. At least it is running now, making progress. Thanks for all the help from everyone getting this thing to happen. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 08:53:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 09:53:26 -0400 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: <48172856.9080502@colbyconsulting.com> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Tue Apr 29 09:39:45 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 16:39:45 +0200 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: Hi John That is what the DataTableAdapter is for. If you use the wizard for creating these to attach your tables, it will do all the plumbing and create a (giant) class which wraps all this for you taking care of relations, not null, max lengths, strong types, error handling etc. etc. not to say provide the basis for creating all the queries you need including ready-made basic queries for CRUD as well as giving you the choice to use stored procedures, pass-through or normal queries. All of this is controlled from a graphic layout like a relations view. If that isn't enough it also lets you carry on with Data Access Layers and business rules. Nothing is hidden or secret. You can browse the class should you have the patience and wish so. This is a snippet: private void InitClass() { this.columnKundeNr = new global::System.Data.DataColumn("KundeNr", typeof(int), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnKundeNr); this.columnNavn = new global::System.Data.DataColumn("Navn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnNavn); this.columnAdr1 = new global::System.Data.DataColumn("Adr1", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr1); this.columnAdr2 = new global::System.Data.DataColumn("Adr2", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr2); this.columnPostNr = new global::System.Data.DataColumn("PostNr", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnPostNr); this.columnBynavn = new global::System.Data.DataColumn("Bynavn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnBynavn); this.columnTlf = new global::System.Data.DataColumn("Tlf", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnTlf); this.Constraints.Add(new global::System.Data.UniqueConstraint("Constraint1", new global::System.Data.DataColumn[] { this.columnKundeNr}, true)); this.columnKundeNr.AllowDBNull = false; this.columnKundeNr.Unique = true; this.columnNavn.AllowDBNull = false; this.columnNavn.MaxLength = 50; this.columnAdr1.MaxLength = 50; this.columnAdr2.MaxLength = 50; this.columnPostNr.MaxLength = 50; this.columnBynavn.MaxLength = 50; this.columnTlf.MaxLength = 50; } If you redesign the table structure from within Visual Studio, it will track the changes and adjust the class accordingly. It is __so clever__ that I wonder why most are still moving on with SQL builders, hand built command strings and the like - pure waste of time in my opinion - except for programmers with special needs and knowledge like Shamil who knows what he is doing. William, I think, will amen this. I must admit that because of VS2008 I'm ready to forgive MS much of their bad doings. /gustav >>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 09:46:24 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 09:46:24 -0500 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: References: Message-ID: <200804291451.m3TEpqu6010208@databaseadvisors.com> Yes, that is correct. He is ending up loading the records into memory. Then passing individual updates back for each row of data that he changed. This method is not designed for doing large datasets like he is doing. It should be done on the SQL Server side. John, Just what are you updating...the UPDATE statement you would use to do what you are doing. Robert At 09:40 AM 4/29/2008, you wrote: >Date: Mon, 28 Apr 2008 17:44:24 -0700 >From: "Eric Barro" >Subject: Re: [dba-VB] vb.net - Updates >To: "'Discussion concerning Visual Basic and related programming > issues.'" >Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >Content-Type: text/plain; charset=us-ascii > >Yeah it is a terrible thing from a DBA point of view because it opens it up >to SQL injection attacks since it builds your SQL statement on the fly. > >I've tried it before and since it is generating it on the fly it makes it >all the more harder to debug and offers the least amount of control from a >DBA point of view. So...I let SQL server do all the work on the back end >when it comes to batch updates by collecting the parameters and sending it >to SQL server to process for the records it needs to process. > >Based on what you are trying to do you are building an image of the data >structure in memory, manipulating the data in memory and then committing the >changes to the database in memory basically sending the update command for >each record to the database back end right? Or am I mistaken in thinking >that all of this magically happens by sending a single update command >containing several individual update commands strung together? From wdhindman at dejpolsystems.com Tue Apr 29 10:26:31 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Tue, 29 Apr 2008 11:26:31 -0400 Subject: [dba-VB] Update gets string or binary truncation error References: Message-ID: ...sigh ...I look at some of the stuff Shamil posts and haven't a clue what he's doing, much less why ...waaaaaaaay over my head ...so much so I get a bad headache every time I try to read some of it ...but you definitely get an AMEN! ...and then some ...on VS2K8 ...when MS finally gets something right, its RIGHT!!! :) William -------------------------------------------------- From: "Gustav Brock" Sent: Tuesday, April 29, 2008 10:39 AM To: Subject: Re: [dba-VB] Update gets string or binary truncation error > Hi John > > That is what the DataTableAdapter is for. If you use the wizard for > creating these to attach your tables, it will do all the plumbing and > create a (giant) class which wraps all this for you taking care of > relations, not null, max lengths, strong types, error handling etc. etc. > not to say provide the basis for creating all the queries you need > including ready-made basic queries for CRUD as well as giving you the > choice to use stored procedures, pass-through or normal queries. > > All of this is controlled from a graphic layout like a relations view. If > that isn't enough it also lets you carry on with Data Access Layers and > business rules. > > Nothing is hidden or secret. You can browse the class should you have the > patience and wish so. This is a snippet: > > private void InitClass() { > this.columnKundeNr = new > global::System.Data.DataColumn("KundeNr", typeof(int), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnKundeNr); > this.columnNavn = new > global::System.Data.DataColumn("Navn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnNavn); > this.columnAdr1 = new > global::System.Data.DataColumn("Adr1", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr1); > this.columnAdr2 = new > global::System.Data.DataColumn("Adr2", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr2); > this.columnPostNr = new > global::System.Data.DataColumn("PostNr", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnPostNr); > this.columnBynavn = new > global::System.Data.DataColumn("Bynavn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnBynavn); > this.columnTlf = new global::System.Data.DataColumn("Tlf", > typeof(string), null, global::System.Data.MappingType.Element); > base.Columns.Add(this.columnTlf); > this.Constraints.Add(new > global::System.Data.UniqueConstraint("Constraint1", new > global::System.Data.DataColumn[] { > this.columnKundeNr}, true)); > this.columnKundeNr.AllowDBNull = false; > this.columnKundeNr.Unique = true; > this.columnNavn.AllowDBNull = false; > this.columnNavn.MaxLength = 50; > this.columnAdr1.MaxLength = 50; > this.columnAdr2.MaxLength = 50; > this.columnPostNr.MaxLength = 50; > this.columnBynavn.MaxLength = 50; > this.columnTlf.MaxLength = 50; > } > > If you redesign the table structure from within Visual Studio, it will > track the changes and adjust the class accordingly. > > It is __so clever__ that I wonder why most are still moving on with SQL > builders, hand built command strings and the like - pure waste of time in > my opinion - except for programmers with special needs and knowledge like > Shamil who knows what he is doing. > > William, I think, will amen this. > > I must admit that because of VS2008 I'm ready to forgive MS much of their > bad doings. > > /gustav > >>>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> > I am doing the update thing, updating data back to SQL Server. I have > been updating an ADO recordset in memory and when I try to update it > back to SQL Server it throws an error that the update failed because it > would cause a string value to be truncated. > > How is this possible. I thought that the table structure in memory > mirrored the structure in the data store (SQL Server in this instance). > Doesn't the read of the data out of SQL Service discover the field > size and type and build a similar structure in memory? > > Apparently not, but if not how do I handle a case like this? > -- > John W. Colby > www.ColbyConsulting.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 10:43:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 11:43:45 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <200804291451.m3TEpqu6010208@databaseadvisors.com> References: <200804291451.m3TEpqu6010208@databaseadvisors.com> Message-ID: <48174231.4070501@colbyconsulting.com> > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 12:13:02 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 12:13:02 -0500 (CDT) Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 Message-ID: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Trying to understand the scenario... 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Questions: 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? From: jwcolby Date: 2008/04/29 Tue AM 10:43:45 CDT > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? From jwcolby at colbyconsulting.com Tue Apr 29 12:48:28 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 13:48:28 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> References: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Message-ID: <48175F6C.5050804@colbyconsulting.com> > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? Correct. I have a 96 million record table, 155 fields. Of that I pull the PKID, NameToParse (not called that obviously) and then 6 "parsed name pieces" fields, lname, fname etc (which I added to the table to hold the pieces). > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Not quite correct. The vb.Net results are due to the fact that I am using a 3rd party library which does the parsing of the name. Name parsing is non-trivial to say the least. That library is a .Net object (DLL actually) which you set up parameters in and then pass in a name to parse and back comes 5 to 8 "pieces". In fact fluid results are the least of my worries in SQL Server, ANY results are more like it. > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? In fact I do but that is not necessary. The name parser came with a demo form and I just modified that to display every 1000th record (which I pass to the form every 1000th record in the table), just so I could "watch it". The form is not a necessity and I may or may not get rid of it in a later iteration (probably will). I have a form (I created) which has a ServerName, DatabaseName, TableName (being parsed), PKFldName, ParseFldName, StartID, EndID, and button to trigger the parse. I fill in the pieces and push the button. Since I have control of the tables, I "hard code" the "parsed parts" field names and they will (at least for now) always be those names. I have to add those fields to the source table anyway. > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? Correct. I grab X records into an ADO.Net DataTable, iterate that table row by row, passing the Unparsed name field off to the parser. I then write the resulting "Pieces" that the parser returns (properties of the parser object) into the matching fields in the ADO.Net DataTable. The name parser handles roughly 800 names / second and is the real bottle neck. I have a StartPK / EndPK which I then manipulate in a while loop, as I finish a "chunk" I write it to SQL Server, update the StartPK / EndPK and grab another chunk to process. I have to do this for 96 million names, however I also receive weekly updates to this table which I will also have to process in a similar manner before I append them to the table. The results of this will be used to send through Accuzip for Address validation / NCOA processing. Address validation can be done just on an address, but NCOA is change of address which is about a person at an address. Once that is done... there is more but you get the picture. And in fact there is another different table from the same data source that I will have to perform this same process on. It is highly likely that I will get other lists in the future with single name fields. The world is full of idiots. ;-) John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Trying to understand the scenario... > > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? > > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. > > Questions: > > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? > > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? > > > From: jwcolby > Date: 2008/04/29 Tue AM 10:43:45 CDT > > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > > Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > > ROTFL. That is a definite NSS. > > Unfortunately I am not a DBA and with the responses I get from the SQL > group I am not even going there. I can do this purely in VB.Net (it is > cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > > No idea. I thought I could find out from the DA after the command > builder was finished building the update statement but I am getting an > error if I try to access the update property. > > John W. Colby > www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 13:13:49 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 13:13:49 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804291817.m3TIH41W019987@databaseadvisors.com> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert At 12:00 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 11:43:45 -0400 >From: jwcolby >Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >To: "Discussion concerning Visual Basic and related programming > issues." >Message-ID: <48174231.4070501 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > >Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > >ROTFL. That is a definite NSS. > >Unfortunately I am not a DBA and with the responses I get from the SQL >group I am not even going there. I can do this purely in VB.Net (it is >cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > >No idea. I thought I could find out from the DA after the command >builder was finished building the update statement but I am getting an >error if I try to access the update property. > >John W. Colby From Gustav at cactus.dk Tue Apr 29 13:24:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 20:24:51 +0200 Subject: [dba-VB] SQL VS VB Message-ID: Hi Robert If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? /gustav >>> robert at webedb.com 29-04-2008 20:13 >>> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert From jwcolby at colbyconsulting.com Tue Apr 29 13:33:22 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:33:22 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804291817.m3TIH41W019987@databaseadvisors.com> References: <200804291817.m3TIH41W019987@databaseadvisors.com> Message-ID: <481769F2.2090303@colbyconsulting.com> It is going to take 50 hours because an external program which does the name parsing will take at least 40 hours just to parse the data. Writing to the database is only about 25% of the total time. If you managed to completely eliminate my database update time it would still take about 40 hours to complete. Thanks for the offer though. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > At 12:00 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 11:43:45 -0400 >> From: jwcolby >> Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >> To: "Discussion concerning Visual Basic and related programming >> issues." >> Message-ID: <48174231.4070501 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> > Yes, that is correct. He is ending up loading the records into memory. >> > Then passing individual updates back for each row of data that he >> > changed. >> >> Yes, and most records have updates. >> >> > This method is not designed for doing large datasets like he is doing. >> > >> > It should be done on the SQL Server side. >> >> ROTFL. That is a definite NSS. >> >> Unfortunately I am not a DBA and with the responses I get from the SQL >> group I am not even going there. I can do this purely in VB.Net (it is >> cranking as we speak) so I will. >> >> > Just what are you updating...the UPDATE statement you would use to >> > do what you are doing. >> >> No idea. I thought I could find out from the DA after the command >> builder was finished building the update statement but I am getting an >> error if I try to access the update property. >> >> John W. Colby > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 13:44:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:44:40 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <48176C98.8030201@colbyconsulting.com> > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? Precisely Gustav. What I WOULD like to do is spin off a thread to do the SQL Server update so the parser could go back to work on the next chunk. IOW put the update in parallel with the parser instead of in series with it. In fact if I had this running on the 4 core system I could probably call the parser DLL in several threads as well. In fact if I ran the VB.Net application on a separate machine, then the SQL Server machine could be just running updates and another machine running the application. What I REALLY need is a thousand blade mini-super that I could just run the entire thing in parallel. It would only take a few minutes then. ;-) John W. Colby www.ColbyConsulting.com Gustav Brock wrote: > Hi Robert > > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? > > /gustav > >>>> robert at webedb.com 29-04-2008 20:13 >>> > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <022401c8aa2b$786352f0$6401a8c0@nant> <<< Please do not take these statements as criticisms of your code >>> Hi John, No problem with criticism - in fact it's very welcome! :) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 12:18 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48165E42.20404@colbyconsulting.com> Message-ID: <022801c8aa2b$787b20b0$6401a8c0@nant> <<< ...it is definitely going to be desirable to feed the update process off to a thread... >>> Hi John, I was lucky with multi-threaded batch updates' samples I posted here the day before yesterday - and the new tests show that batch updates in multi-threading environments result in MS SQL deadlocks: I should have thought about those issues before... Recap: forget batch updates - one of the scenarios of your task could be: - many readers; - many parsers; - but just one updater making batch updates, or many updaters but updating just one row in one cycle... HTH to save your time, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 3:31 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 14:11:36 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 15:11:36 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022801c8aa2b$787b20b0$6401a8c0@nant> References: <022801c8aa2b$787b20b0$6401a8c0@nant> Message-ID: <481772E8.3050505@colbyconsulting.com> Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 14:43:53 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:43:53 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <30862132.1249691209498233814.JavaMail.root@vms124.mailsrvcs.net> I believe that Shamil was referring to multi-threaded updates. If you used the multi-threaded approach you could potentially end up slowing the process more than you could ever imagine because of the way SQL server locks pages of records when you run an update. From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:51:42 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:51:42 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <481772E8.3050505@colbyconsulting.com> Message-ID: <022c01c8aa32$7269c440$6401a8c0@nant> John, I'm not sure yet what happens< I will try to find out more details in the coming days - here is the message I'm currently getting with new Strategy 4 , which I'm making similar to your way of coding this task but running in multi-threading environment: +++ System.Data.SqlClient.SqlException: Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. --- Just wanted to warn you - it might happen you'll never see this message but if you'll see it then you may find there is no workaround except abandoning the idea of batch updates running in several threads - and I mean parallel batch updates: of course, as I noted in my previous e-mail parallel reading and parsing could be done but batch updates very probably can't be run in parallel threads because they might result in deadlocks... But it looks like (from your recent posting here) that you don't need such parallel batch updates at all? Thanks. Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 11:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 14:59:51 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:59:51 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 15:02:10 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 15:02:10 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <29420937.173771209499330766.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 15:13:29 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:13:29 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022c01c8aa32$7269c440$6401a8c0@nant> References: <022c01c8aa32$7269c440$6401a8c0@nant> Message-ID: <48178169.70002@colbyconsulting.com> Shamil, I have just moved the whole project to my quad core x64 server and am bringing it up now. If I can get that code running over on another machine (especially IF it is a faster machine) then the update part will be less of an issue. In any event the parse WAS taking about 15 seconds for 10K records on the dual core x32 machine which was also running SQL Server. I have designed this thing as a set of classes. There is a "data class" which is opened by a class that parses the names. The NameParser just writes the data directly into the ADO recordset in the data class. I MIGHT be able to just shuffle the class holding the open ado recordsets off into a collection. Another thread could pull those class instances out of the collection and call the update method which would start the update. As it finished the update, it would destroy that data class instance and pull the next one out of the collection. This would effectively separate the parsing and the updating, and each could proceed at it's own speed. I have not investigated yet but I might be able to have multiple threads doing the name parsing, and storing the resulting data class in that collection. It depends on whether the dll is reentrant I suppose and I really don't have a clue yet.. There would only be a single thread processing the dataset classes "update". I would need to watch that I didn't get too many open data classes at once time I suppose, though this machine has Windows x64 and 8 gigs of RAM so it has a little more breathing room than the Windows x32 machine I was running it on. Of course all of this is just "play". At least it is running now. I was also thinking about running two threads in "opposite directions", one processing data PK high to low, and the other processing data PK Low to High. Until they met in the middle that MIGHT avoid the deadlock thing. Again though, that is just play. NOT TO MENTION that I have never run a thread anyway so i have to figure THAT out. ;-) John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > I'm not sure yet what happens< I will try to find out more details in the > coming days - here is the message I'm currently getting with new Strategy 4 > , which I'm making similar to your way of coding this task but running in > multi-threading environment: > > +++ > System.Data.SqlClient.SqlException: Transaction (Process ID > 57) was deadlocked on lock resources with another process and has been > chosen as the deadlock victim. Rerun the transaction. > --- > > Just wanted to warn you - it might happen you'll never see this message but > if you'll see it then you may find there is no workaround except abandoning > the idea of batch updates running in several threads - and I mean parallel > batch updates: of course, as I noted in my previous e-mail parallel reading > and parsing could be done but batch updates very probably can't be run in > parallel threads because they might result in deadlocks... > > But it looks like (from your recent posting here) that you don't need such > parallel batch updates at all? > > Thanks. > > Shamil From jwcolby at colbyconsulting.com Tue Apr 29 15:18:50 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:18:50 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> References: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> Message-ID: <481782AA.6030204@colbyconsulting.com> Eric, I will read it. Unfortunately that puts me back in the SQL forum and you know how I ... uhh... well, let's just say I would rather muddle along in VB.Net. That is something that I can get better at and will spend enough time at to pay back my efforts. Besides... I come from the "just throw more hardware at it" school of Engineering. ;-) Seriously though, of course I'd LOVE to just pass the ADO recordset over and let SQL Server do it. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > John, > > This article might be of use in your scenario... > > http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 > > It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... > > "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " > > --Eric > > From: jwcolby > Date: 2008/04/29 Tue PM 02:11:36 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > Interesting. Do you think that you would get deadlocks for different > blocks of records? In my case there is not "a million users" scenario. > This process (and its threads) are the only things modifying data. > Each "chunk" is a defined block of records. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> <<< >> ...it is definitely going to be desirable to feed the update >> process off to a thread... >> Hi John, >> >> I was lucky with multi-threaded batch updates' samples I posted here the day >> before yesterday - and the new tests show that batch updates in >> multi-threading environments result in MS SQL deadlocks: I should have >> thought about those issues before... >> >> Recap: forget batch updates - one of the scenarios of your task could be: >> >> - many readers; >> - many parsers; >> - but just one updater making batch updates, or many updaters but updating >> just one row in one cycle... >> >> HTH to save your time, >> Shamil >> >> >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Tuesday, April 29, 2008 3:31 AM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> We have ignition folks. I just updated the first 10K records in the >> database. I had to set the batch size to 100, as I was getting a >> timeout with a batch size as small as 1000 records. I haven't done any >> timings but it is definitely going to be desirable to feed the update >> process off to a thread if I can figure out how to do that. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> jwcolby wrote: >>> I THINK I have found my solution, called the SQLCommandBuilder. I found >>> reference to the command build saying how terrible it is, then a piece >>> of code that, in simple plain language, shows how to use it all, using >>> the simple objects I am already using. >>> >>> We shall see. >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From robert at webedb.com Tue Apr 29 16:21:31 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 16:21:31 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804292127.m3TLRIWT012647@databaseadvisors.com> His original estimate was 50 hours. (not mine) But, he would have to do a bit of fancy coding in SQL Server to use the same DLL he is using. That bit of information was not in the first post. At 02:02 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 20:24:51 +0200 >From: "Gustav Brock" >Subject: Re: [dba-VB] SQL VS VB >To: >Message-ID: >Content-Type: text/plain; charset=US-ASCII > >Hi Robert > >If the parser dll runs at about 800 records/s, 93 mio. records will >need 33 hours as a minimum. Or? > >/gustav From jwcolby at colbyconsulting.com Tue Apr 29 16:28:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:28:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48178169.70002@colbyconsulting.com> References: <022c01c8aa32$7269c440$6401a8c0@nant> <48178169.70002@colbyconsulting.com> Message-ID: <4817930D.3080400@colbyconsulting.com> Well I shoulda done that sooner. With the process running on Azul (the quad core x64) machine and Stonehenge (dual core x32) just processing the SQL Server thing, the parse time dropped dramatically! I am now processing about 2800 - 2900 records / second. Interestingly the update time is about the same, ~5 seconds, but that is for ~80K records. I am wondering if I am timing what i think I am timing on the update thing. 8~/ PERHAPS there is some kind of "connection" time that is the majority of the time At any rate I will finish MUCH sooner, and that with no thread stuff yet. John W. Colby www.ColbyConsulting.com jwcolby wrote: > Shamil, > > I have just moved the whole project to my quad core x64 server and am > bringing it up now. If I can get that code running over on another > machine (especially IF it is a faster machine) then the update part will > be less of an issue. In any event the parse WAS taking about 15 seconds > for 10K records on the dual core x32 machine which was also running SQL > Server. > > I have designed this thing as a set of classes. There is a "data class" > which is opened by a class that parses the names. The NameParser just > writes the data directly into the ADO recordset in the data class. I > MIGHT be able to just shuffle the class holding the open ado recordsets > off into a collection. Another thread could pull those class instances > out of the collection and call the update method which would start the > update. As it finished the update, it would destroy that data class > instance and pull the next one out of the collection. > > This would effectively separate the parsing and the updating, and each > could proceed at it's own speed. > > I have not investigated yet but I might be able to have multiple threads > doing the name parsing, and storing the resulting data class in that > collection. It depends on whether the dll is reentrant I suppose and I > really don't have a clue yet.. > > There would only be a single thread processing the dataset classes "update". > > I would need to watch that I didn't get too many open data classes at > once time I suppose, though this machine has Windows x64 and 8 gigs of > RAM so it has a little more breathing room than the Windows x32 machine > I was running it on. > > Of course all of this is just "play". At least it is running now. > > I was also thinking about running two threads in "opposite directions", > one processing data PK high to low, and the other processing data PK Low > to High. Until they met in the middle that MIGHT avoid the deadlock > thing. Again though, that is just play. > > NOT TO MENTION that I have never run a thread anyway so i have to figure > THAT out. ;-) > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> John, >> >> I'm not sure yet what happens< I will try to find out more details in the >> coming days - here is the message I'm currently getting with new Strategy 4 >> , which I'm making similar to your way of coding this task but running in >> multi-threading environment: >> >> +++ >> System.Data.SqlClient.SqlException: Transaction (Process ID >> 57) was deadlocked on lock resources with another process and has been >> chosen as the deadlock victim. Rerun the transaction. >> --- >> >> Just wanted to warn you - it might happen you'll never see this message but >> if you'll see it then you may find there is no workaround except abandoning >> the idea of batch updates running in several threads - and I mean parallel >> batch updates: of course, as I noted in my previous e-mail parallel reading >> and parsing could be done but batch updates very probably can't be run in >> parallel threads because they might result in deadlocks... >> >> But it looks like (from your recent posting here) that you don't need such >> parallel batch updates at all? >> >> Thanks. >> >> Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 16:45:10 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:45:10 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804292127.m3TLRIWT012647@databaseadvisors.com> References: <200804292127.m3TLRIWT012647@databaseadvisors.com> Message-ID: <481796E6.7060300@colbyconsulting.com> That was indeed my original estimate. The PARSER was running at ~800 names / second. The UPDATE occurs after the parser finishes its thing., and is thus "in series with" (time wise) the parser. I just moved the VB.Net off to a more powerful machine and it is now running the parser at just over 3K records / second. I am in fact not parsing every record, I am actually processing ~80% of the records. Now I am off to see if I can figure out how to use threads. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > His original estimate was 50 hours. (not mine) > > But, he would have to do a bit of fancy coding in > SQL Server to use the same DLL he is using. That > bit of information was not in the first post. > > > > At 02:02 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 20:24:51 +0200 >> From: "Gustav Brock" >> Subject: Re: [dba-VB] SQL VS VB >> To: >> Message-ID: >> Content-Type: text/plain; charset=US-ASCII >> >> Hi Robert >> >> If the parser dll runs at about 800 records/s, 93 mio. records will >> need 33 hours as a minimum. Or? >> >> /gustav > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 19:03:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 20:03:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> Message-ID: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> I've fallen behind quite a bit, I see. Well, I'll catch up. Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 20:13:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 21:13:25 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> <030f01c8aa55$aa6384d0$0502a8c0@Laptop> Message-ID: <4817C7B5.6020005@colbyconsulting.com> > I've fallen behind quite a bit, I see. eh? John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > I've fallen behind quite a bit, I see. > Well, I'll catch up. > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 20:23:54 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 21:23:54 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> Message-ID: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Hi John, I was looking over Gustav's posts of Feb 20 with Charlotte and posts of this and nearby threads. I've been doing stuff with maps that didn't require use of large databases. The difference between an sqladapter and a tableadapter were unknown to me until today. C#, SQL Server 200X, ASP.Net All very compelling! Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Tuesday, April 29, 2008 9:13 PM Subject: Re: [dba-VB] vb.net - Updates > > I've fallen behind quite a bit, I see. > > eh? > > John W. Colby > www.ColbyConsulting.com > > > Michael R Mattys wrote: >> I've fallen behind quite a bit, I see. >> Well, I'll catch up. >> >> Michael R. Mattys >> MapPoint & Access Dev >> www.mattysconsulting.com >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 21:14:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 22:14:27 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Message-ID: <4817D603.9050308@colbyconsulting.com> Michael, > C#, SQL Server 200X, ASP.Net > All very compelling! Indeed they are. So much to know, so little time. I am slowly learning some of this, painfully slowly. John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > Hi John, > > I was looking over Gustav's posts of Feb 20 > with Charlotte and posts of this and nearby threads. > > I've been doing stuff with maps that didn't require use > of large databases. The difference between an sqladapter > and a tableadapter were unknown to me until today. > > C#, SQL Server 200X, ASP.Net > All very compelling! > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Tuesday, April 29, 2008 9:13 PM > Subject: Re: [dba-VB] vb.net - Updates > > >>> I've fallen behind quite a bit, I see. >> eh? >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Michael R Mattys wrote: >>> I've fallen behind quite a bit, I see. >>> Well, I'll catch up. >>> >>> Michael R. Mattys >>> MapPoint & Access Dev >>> www.mattysconsulting.com >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 23:54:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 00:54:18 -0400 Subject: [dba-VB] Worker threads and class global variables Message-ID: <4817FB7A.4000002@colbyconsulting.com> Can a backgroundworker access a class global variable in the class that dimensions the background worker. I have a process that will be creating data classes. these classes will fill ADO data sets, modify them and then place them in a collection global to the class. This same class would then dim a backgroundworker object. The way I envision this working is that the backgroundworker is initialized and started. It simply sits in a loop checking the class global collection for data class instances deposited into the collection by the main thread. If a class instance is found in the thread it pulls it out of the collection and processes it. This backgroundworker thread's job is to run the SQL Server update code in the data class (call a method of that class). Somehow that backgroundworker has to be told that the main thread is done parsing names, ran through the entire table and is done. So I thought the main thread could set a class global boolean done flag. Or something. Can I do this at all the way I am thinking? My problem is that there is no way to predict which process will be faster, name parse or SQL Server update. My idea was simply to use a collection to act as an interface between the two. The name parser instantiates a data class which loads the data to be parsed into an ADO.Net datatable. The name parser updates the data in the the data class. When it is done updating the data it drops the class into the collection. the backgroundworker watches the collection and when it finds a data class in the collection, it pulls it out and calls the update method of that class to process the data. when it is done it cleans up the ADO objects and stores the class in another "history" collection up in the parse class. The data class will have start / stop times, start / end PKIDs, record count etc information which will be of interest to me. I have never used Background worker objects so I don't know whether it can interface in the manner described, using collections to pass objects. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Wed Apr 30 08:09:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 09:09:45 -0400 Subject: [dba-VB] Background worker threads Message-ID: <48186F99.6090401@colbyconsulting.com> I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 10:19:03 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 08:19:03 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <48186F99.6090401@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> Message-ID: You can create your own events with whatever parameters you want and raise those. Just Dim Public Event Whatever(parameter) Then raise it as needed. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 6:10 AM To: VBA Subject: [dba-VB] Background worker threads I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 11:05:17 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 12:05:17 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> Message-ID: <481898BD.2050301@colbyconsulting.com> Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust From shamil at smsconsulting.spb.ru Wed Apr 30 11:07:47 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:07:47 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4817D603.9050308@colbyconsulting.com> Message-ID: <02cc01c8aadc$55aeb860$6401a8c0@nant> Hi John, May I ask you to run this query (e.g. in MS SQL Management Studio) against your huge db table? --- cut here --- SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID, OWNERNAME, FName, MName, LName, NamePrefix, NameSuffix, Gender from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] --- cut here --- M.B. [Names] have to be substituted with the name of your table - what is it BTW? This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... I'm curious to know how much time approx. this query will take to execute... Thank you. -- Shamil P.S. BTW, here is a useful for your case article on multi-threading (practical solution) with good introduction - http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik Sorry, I can't react promptly on your messages in this discussion thread: a lot of custom work here, long weekends and our time zones difference but I do continue to work in this direction and if I get some useful results I will post them here... P.P.S. This book could be also useful: C# Threading Handbook by Tobin Titus et al. ISBN:1861008295 APress, LLC C 2004 (288 pages) This book addresses the fundamental units of Windows and .NET programming-threads. Coverage includes how .NET applications are executed, the life cycle of a thread in .NET, how the .NET Framework uses threads, and more. From shamil at smsconsulting.spb.ru Wed Apr 30 11:25:15 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:25:15 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> Message-ID: <02d901c8aade$c66c3120$6401a8c0@nant> John, Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that the fact that "The basic code WAS working before I carved it out into a worker." doesn't mean that the same code (copied & pasted) into multi-threading environment will work OK - very often the opposite will be the truth - it could produce (very) faulty solution in multi-threaded environment, which will not be easy to sort out into properly running multi-threaded code... I'd dare to advise you to prepare and test your code in VS console application (or .net console app as test-bed app and .net classlib(s) keeping all your non-UI code), and when that will work OK then add some more code to run your classlibs with your .NET WinForms application: you have a lot to do, to learn and to test - multi-threading and .NET programming can't be "captured by attack" - it's a painful and lengthy natural process, which usually takes a lot of time - I know that from my own experience and experience of others - and better not "read all that books" - find just one but good and then experiment a lot on your own - then you'll get it working properly and you'll teach all of us here - I'm quite sure about that latter! Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 8:05 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:17:04 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:17:04 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: <02d901c8aade$c66c3120$6401a8c0@nant> References: <02d901c8aade$c66c3120$6401a8c0@nant> Message-ID: <4818A990.2020306@colbyconsulting.com> Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 12:28:02 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 10:28:02 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: I'm sorry, John, I missed the contextt of your question. We don't use background workers that way, so I don't have an answer for you. Charlotte -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:05 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:32:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:32:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AD3D.6030003@colbyconsulting.com> It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:36:02 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:36:02 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: <4818AE02.2080601@colbyconsulting.com> Bueno, thanks! John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > I'm sorry, John, I missed the contextt of your question. We don't use > background workers that way, so I don't have an answer for you. > > Charlotte > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:05 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> You can create your own events with whatever parameters you want and >> raise those. Just Dim Public Event Whatever(parameter) Then raise it > >> as needed. >> >> Charlotte Foust > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:41:16 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:41:16 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AF3C.3040601@colbyconsulting.com> Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 13:06:29 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:06:29 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <4818A990.2020306@colbyconsulting.com> Message-ID: <02f601c8aaec$ea01f620$6401a8c0@nant> John, Please read the article I referred already: "Simple and Efficient Multi-threading with Blocking Queues" http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik I think it is "what doctor ordered" for your task, and it has code, C# but you can convert it if needed using free converters.... As for collections and variables - they are not thread safe by default. Read these two articles (they have sample code): "Static & Thread Safety: Part I and II" http://www.odetocode.com/Articles/313.aspx http://www.odetocode.com/Articles/314.aspx I believe the only book you need now is the one I have also mentioned already: "C# Threading Handbook by Tobin Titus et al.", ISBN:1861008295 APress, 2004 - it has a lot of code samples (C# sorry) and very good .NET multi-threading features coverage And this is an excerpt from MSDN on collections (MSDN is our best friend!): +++ Collection Object (Visual Basic) Caution Iterating through a Visual Basic Collection is not a thread-safe procedure. Even if the collection is synchronized, other threads can still modify the collection, causing the enumerator to throw an exception. To guarantee thread safety during enumeration, either lock the collection or catch the exceptions resulting from changes made by other threads. For more information on locking a programming element, see SyncLock Statement. ---- The referred above as "what doctor ordered for your task" article uses .NET's Queue class - here is the info on it from MSDN on thread safety: +++ Queue Class Thread Safety Public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. To guarantee the thread safety of the Queue, all operations must be done through the wrapper returned by the Synchronized method. Enumerating through a collection is intrinsically not a thread safe procedure. Even when a collection is synchronized, other threads can still modify the collection, which causes the enumerator to throw an exception. To guarantee thread safety during enumeration, you can either lock the collection during the entire enumeration or catch the exceptions resulting from changes made by other threads. ---- The "Simple and Efficient Multi-threading with Blocking Queues" article has sample code, which shows what simple coding is needed to implement thread-safe collections and queues... HTH, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:17 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:22:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:22:08 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AD3D.6030003@colbyconsulting.com> Message-ID: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:28:58 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:28:58 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AF3C.3040601@colbyconsulting.com> Message-ID: <02fb01c8aaf0$0e512110$6401a8c0@nant> Thank you, John, You have a very good PC, lucky man! :) As I wrote already I (I'm sorry) did post incorrect query - I wanted to run this one: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] It will also return 96K rows for your huge table, but it's expected (here) it will get first results instantly, and it will finish within just several (3?) seconds... Please prove am I wrong with that bright above expectations on the execution time of the fixed version of the query?... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Wed Apr 30 13:50:38 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 13:50:38 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 14:31:28 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 23:31:28 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> Message-ID: <030a01c8aaf8$c971d130$6401a8c0@nant> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead of ~3 seconds.... SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc] (nolock)) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] (nolock) BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of useful advices including all kinds of locking - the first time I see so detailed advices and samples on that subject (but I must say I didn't read that much computer books in the past)... Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Wednesday, April 30, 2008 10:51 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil From jwcolby at colbyconsulting.com Wed Apr 30 14:55:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:55:15 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> References: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> Message-ID: <4818CEA3.5000704@colbyconsulting.com> Well, I do not know what you want me to do with the result. To just open a query window in SSMS and execute it in there I get pretty much the same result. First results in 14 seconds and ten minutes and counting. I don't have "just the PK" indexed, at least not intentionally and visibly. I am now using cover indexes that index the PKID and one or more fields that I am interested in. Whether there is a PKID index behind the scenes I don't know. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:33 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > It started returning results in about 12-13 seconds. You are asking for > 96 thousand result rows you realize. This database is 96 million rows, > so every thousandth row is 96K rows. > > It is still processing at 9 minutes. I'll email the final execution > time whenever the query finishes. > > This is fascinating because I pull just the fields into an ADO.Net table > for 100K rows in just a second or two. Of course these are consecutive > rows, it is not scanning the entire table. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 14:59:44 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:59:44 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030a01c8aaf8$c971d130$6401a8c0@nant> References: <030a01c8aaf8$c971d130$6401a8c0@nant> Message-ID: <4818CFB0.4020402@colbyconsulting.com> I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 15:19:00 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:19:00 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fb01c8aaf0$0e512110$6401a8c0@nant> References: <02fb01c8aaf0$0e512110$6401a8c0@nant> Message-ID: <4818D434.4090104@colbyconsulting.com> Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Wed Apr 30 15:24:32 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 15:24:32 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 15:32:43 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:32:43 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> References: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <4818D76B.7000602@colbyconsulting.com> Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <031501c8ab05$422d2460$6401a8c0@nant> <<< I see the UNION ALL and ORDER BY as bottlenecks. >>> Eric, But union all is executed with two row sets: - first having every 1000th record, and - second having ONLY ONE record... When query execution plan is analyzed then I see that first operation for both parts of UNION ALL query is index scan IOW UNION ALL and ORDER BY do not look as a bottleneck because UNION ALL merges (without any attempts to suppress full duplicate) a set of records with just one record, and because ORDER by of clustered index is 0% cost operation? Thank you -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Thursday, May 01, 2008 12:25 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D434.4090104@colbyconsulting.com> Message-ID: <031601c8ab05$42476320$6401a8c0@nant> <<< Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? >>> Yes, it was, but I expected you'll get results much quicker... Could that be that you do not have (clustered)index on your PK field? Can you: Right-Click on table name -> Script Table as -> CREATE to -> New Query Window => I'm getting the following for your simulated table. CREATE TABLE [dbo].[Names]( [PKID] [int] IDENTITY(1,1) NOT NULL, [OWNERNAME] [nvarchar](255) NOT NULL, [FName] [nvarchar](50) NULL, [MName] [nvarchar](50) NULL, [LName] [nvarchar](50) NULL, [NamePrefix] [nvarchar](50) NULL, [NameSuffix] [nvarchar](50) NULL, [Gender] [nvarchar](50) NULL, [ParsedName] [nvarchar](255) NULL, CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED ( [PKID] ASC ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] Also if you select query's text in SSMS query window, and then Ctrl+L then you'll get query execution plan - here as the last step with 99% cost I'm getting: "Clustered Index Scan" IOW the query doesn't scan table's data at all - it just scans the index... Thanks, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:19 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:07:55 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:07:55 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D76B.7000602@colbyconsulting.com> Message-ID: <032201c8ab06$42a01500$6401a8c0@nant> John, This ORDER BY used here to get ROW_NUMBER is/was expected as very speedy "virtual" zero cost operation because I thought/expected your PK (PKID) was a (clustered) index... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:33 AM To: ebarro at verizon.net; Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 16:16:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 17:16:08 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <031601c8ab05$42476320$6401a8c0@nant> References: <031601c8ab05$42476320$6401a8c0@nant> Message-ID: <4818E198.30501@colbyconsulting.com> I do have the clustered stuff exactly as you show: > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] Likewise I am getting the same 99% cost clustered index scan. I have to guess it is something else. There is a lot of empty space in the database file which I will compact out tonight. It takes hours because of the size and when it is doing that I cannot use the db. Perhaps I have severe fragmentation or something. The database size is 300 GIGS. And finally just remember that when all is said and done it is still a hundred times larger than your database. No matter how you slice it, it has to process more physical disk sectors etc. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > > <<< > Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. > Was that your intention? > Yes, it was, but I expected you'll get results much quicker... > > Could that be that you do not have (clustered)index on your PK field? > > Can you: Right-Click on table name -> Script Table as -> CREATE to -> New > Query Window => I'm getting the following for your simulated table. > > CREATE TABLE [dbo].[Names]( > [PKID] [int] IDENTITY(1,1) NOT NULL, > [OWNERNAME] [nvarchar](255) NOT NULL, > [FName] [nvarchar](50) NULL, > [MName] [nvarchar](50) NULL, > [LName] [nvarchar](50) NULL, > [NamePrefix] [nvarchar](50) NULL, > [NameSuffix] [nvarchar](50) NULL, > [Gender] [nvarchar](50) NULL, > [ParsedName] [nvarchar](255) NULL, > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] > > Also if you select query's text in SSMS query window, and then Ctrl+L then > you'll get query execution plan - here as the last step with 99% cost I'm > getting: "Clustered Index Scan" IOW the query doesn't scan table's data at > all - it just scans the index... > > Thanks, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Thursday, May 01, 2008 12:19 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > > You have a very good PC, lucky man! :) > > As I mentioned elsewhere, I come from the "throw more hardware at it" > school of Engineering. ;-) > > I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 > gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster > in many cases. > > I do not have this large database over on that machine, and it would > take me hours to get it over there so i can't test this query on that, > at least today. that database has not been compacted recently and is > about 300 Gigs with 136 gigs free space. I will compact it tonight and > then copy the smaller file to Azul. > > As for your new query, with the NoLocks in place it returned first > results in only 8 seconds as opposed to 14 before, but it still took > 13:59 to pull the entire result set. Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. Was that your > intention? > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Thank you, John, >> >> You have a very good PC, lucky man! :) >> >> As I wrote already I (I'm sorry) did post incorrect query - I wanted to > run >> this one: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> It will also return 96K rows for your huge table, but it's expected (here) >> it will get first results instantly, and it will finish within just > several >> (3?) seconds... >> >> Please prove am I wrong with that bright above expectations on the > execution >> time of the fixed version of the query?... >> >> Thank you. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Wednesday, April 30, 2008 9:41 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> Shamil, >> >> It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server >> x32) 13:51 to pull 96,417 rows for the result set. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Shamil Salakhetdinov wrote: >>> Hi John, >>> >>> May I ask you to run this query (e.g. in MS SQL Management Studio) > against >>> your huge db table? >>> >>> --- cut here --- >>> >>> SELECT RowNum, PKID from ( select >>> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >>> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >>> NameSuffix, Gender from [Names]) s >>> where (s.RowNum % 1000) = 1 >>> union all >>> Select NULL as RowNum, Max(PKID) as PKID from [Names] >>> >>> --- cut here --- >>> >>> M.B. [Names] have to be substituted with the name of your table - what is >> it >>> BTW? >>> >>> This query returns PKID of every 1000th row sorted by PKID + > MAX(PKID).... >>> I'm curious to know how much time approx. this query will take to >> execute... >>> Thank you. >>> >>> -- >>> Shamil >>> >>> P.S. BTW, here is a useful for your case article on multi-threading >>> (practical solution) with good introduction - >>> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >>> cik >>> >>> Sorry, I can't react promptly on your messages in this discussion thread: >> a >>> lot of custom work here, long weekends and our time zones difference but > I >>> do continue to work in this direction and if I get some useful results I >>> will post them here... >>> >>> P.P.S. This book could be also useful: >>> >>> C# Threading Handbook >>> by Tobin Titus et al. >>> >>> ISBN:1861008295 >>> >>> APress, LLC C 2004 (288 pages) >>> >>> This book addresses the fundamental units of Windows and .NET >>> programming-threads. Coverage includes how .NET applications are > executed, >>> the life cycle of a thread in .NET, how the .NET Framework uses threads, >> and >>> more. >>> >>> >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 3 06:34:34 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 03 Apr 2008 13:34:34 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From accessd at shaw.ca Thu Apr 3 10:59:46 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Thu, 03 Apr 2008 08:59:46 -0700 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: References: Message-ID: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From michael at ddisolutions.com.au Thu Apr 3 18:01:55 2008 From: michael at ddisolutions.com.au (Michael Maddison) Date: Fri, 4 Apr 2008 10:01:55 +1100 Subject: [dba-VB] ASP.NET, separated programming and visual design References: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D013BF515@ddi-01.DDI.local> I did an online seminar with MS about VS 2008 + WPF. They claim that WPF is pretty much designed to allow the UI designer to work seperately to the programmer. He could do it in NP as well as long as he knows his XML ;-) cheers Michael M -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Friday, 4 April 2008 3:00 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Thu Apr 3 18:53:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 03:53:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <006601c895e5$fa874e80$6401a8c0@nant> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 02:14:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 09:14:35 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From shamil at users.mns.ru Fri Apr 4 04:59:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 13:59:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <002d01c8963a$a3217e90$6501a8c0@nant> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 05:17:54 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 12:17:54 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Shamil Great! This is exactly the information I need. /gustav >>> shamil at users.mns.ru 04-04-2008 11:59 >>> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From wdhindman at dejpolsystems.com Thu Apr 10 09:11:34 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Thu, 10 Apr 2008 10:11:34 -0400 Subject: [dba-VB] ASP.NET, separated programming and visual design References: Message-ID: <002301c89b14$c8de1d30$7e6d6c4c@jisshowsbs.local> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 10 09:25:30 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 10 Apr 2008 16:25:30 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi William Fine, though it we be the other way around - I would make the mechanics first, then he would beef it up. With VSVWD8 Express as a free tool I see no reason why he should punish him self with something like Notepad. /gustav >>> wdhindman at dejpolsystems.com 10-04-2008 16:11 >>> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav From jwcolby at colbyconsulting.com Fri Apr 11 12:26:53 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:26:53 -0400 Subject: [dba-VB] Parsing strings Message-ID: <47FF9F5D.2070704@colbyconsulting.com> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Fri Apr 11 12:42:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 11 Apr 2008 19:42:18 +0200 Subject: [dba-VB] Parsing strings Message-ID: Hi John Why not use the Path class: extension = Path.GetExtension(FileName) basename = Path.GetFileNameWithoutExtension(FileName) /gustav >>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 11 12:49:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:49:18 -0400 Subject: [dba-VB] Parsing strings In-Reply-To: References: Message-ID: <47FFA49E.2020605@colbyconsulting.com> Lovely, thanks. Gustav Brock wrote: > Hi John > > Why not use the Path class: > > extension = Path.GetExtension(FileName) > basename = Path.GetFileNameWithoutExtension(FileName) > > /gustav > >>>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, -- John W. Colby www.ColbyConsulting.com From stuart at lexacorp.com.pg Fri Apr 11 18:45:55 2008 From: stuart at lexacorp.com.pg (Stuart McLachlan) Date: Sat, 12 Apr 2008 09:45:55 +1000 Subject: [dba-VB] Parsing strings In-Reply-To: <47FF9F5D.2070704@colbyconsulting.com> References: <47FF9F5D.2070704@colbyconsulting.com> Message-ID: <480084D3.12387.2763EDB7@stuart.lexacorp.com.pg> Left,Mid,Right etc are not function of the string object, they are functions in the Strings module of the Microsoft.VisualBasic namespace. i.e., fully qualified it is Microsoft.VisualBasic.Strings.Right() On 11 Apr 2008 at 13:26, jwcolby wrote: > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > From jwcolby at colbyconsulting.com Sat Apr 12 15:23:49 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 16:23:49 -0400 Subject: [dba-VB] Count a character in a string Message-ID: <48011A55.40402@colbyconsulting.com> I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 15:50:25 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 15:50:25 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From Gustav at cactus.dk Sat Apr 12 16:33:25 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 12 Apr 2008 23:33:25 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 12 17:48:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 18:48:26 -0400 Subject: [dba-VB] Count a character in a string In-Reply-To: References: Message-ID: <48013C3A.9030500@colbyconsulting.com> I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 18:00:38 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:00:38 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: Message-ID: True, don't use .Net much though. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Saturday, April 12, 2008 4:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] Count a character in a string Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From DWUTKA at Marlow.com Sat Apr 12 18:01:05 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:01:05 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48013C3A.9030500@colbyconsulting.com> Message-ID: Ya, I think I posted that solution a while back on AccessD. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 5:48 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Count a character in a string I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From shamil at users.mns.ru Sat Apr 12 21:30:24 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 06:30:24 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: <053b01c89d0e$53c595e0$6401a8c0@nant> Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:08:59 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:08:59 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> References: <48011A55.40402@colbyconsulting.com> <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <003c01c89d46$03525ca0$8119fea9@LTVM> John, You may recall the RegExp that I posted a short while ago. You can use this to get the answer. This code returns 17: Function CountRegEx() Dim strIn As String, lngCount As Long strIn = "asdf||||||qwert|zxcv|mnbv|qwert|zxcv|mnbv||||||" lngCount = (RegExpTest("\|", strIn)) MsgBox lngCount End Function Here is the original Function Function RegExpTest(patrn, strng) As Long ' Set reference to Microsoft vbscript regular expressions 5.5 ' Cheat Sheet available at http://regexlib.com/CheatSheet.aspx Dim RegEx As New VBScript_RegExp_55.RegExp Dim Match, Matches, RetStr ' Create variable. Set RegEx = New RegExp ' Create a regular expression. RegEx.Pattern = patrn ' Set pattern. RegEx.IgnoreCase = True ' Set case insensitivity. RegEx.Global = True ' Set global applicability. Set Matches = RegEx.Execute(strng) ' Execute search. RegExpTest = 0 For Each Match In Matches ' Iterate Matches collection. RetStr = RetStr & "Match found at position " RetStr = RetStr & Match.FirstIndex & ". Match Value is '" RetStr = RetStr & Match.Value & "'." & vbCrLf RegExpTest = RegExpTest + 1 Next End Function HTH Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 3:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:14:06 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:14:06 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: References: <48013C3A.9030500@colbyconsulting.com> Message-ID: <003d01c89d46$ba370a60$8119fea9@LTVM> John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max From shamil at users.mns.ru Sun Apr 13 04:26:17 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 13:26:17 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <054b01c89d48$6d543360$6401a8c0@nant> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 13 04:47:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 11:47:21 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi Shamil Neat! I'll add that to my toolbox. /gustav >>> shamil at users.mns.ru 13-04-2008 11:26 >>> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Sun Apr 13 05:08:53 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 14:08:53 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <003d01c89d46$ba370a60$8119fea9@LTVM> Message-ID: <054f01c89d4e$60f41d00$6401a8c0@nant> Hi Max, Our postings crossed - I have just posted RegEx test results... N.B.: when testing this or that method both memory consumption and execution speed should be taken into consideration: - high memory consumption giving speediest results can be neglected to solve small tasks within "tiny" applications/utilities, but - high memory consumption is the thing, which could "unexpectedly" give nasty side effects in the large application systems: One example I have got several months ago, which was so "tricky" to watch, and was not easy to find why it could ever happen in .NET: I have got a memory leakage/excessive memory consumption (in .NET!) because of "lazy loading", and that latter "lazy loading" loaded quite some data when it wasn't needed, in the cycle processing millions of records - the results were like that when I watched 'Page File Usage History' in Task Manager: /| /| /| . / | / | / | . / | / | / | . / |/ |/ |/ And in usual mode without heavy application system workload everything worked well with the same data... Recap: - Split(...) approach could result in similar to the above side effect for very large input strings; // 10 sec for 20,000,000 iterations string[] recordLine = s.Split('|'); count = recordLine.Length-1; - Replace(...) could also result in the above side effect; // 10 sec for 20,000,000 iterations string temp = s.Replace("|", Microsoft.VisualBasic.Constants.vbNullString); count = s.Length - temp.Length; - RegEx(...) seems to be the slowest - unsatisfactory slow for large input strings/heavy system workload; // ? sec (unfinished) for 20,000,000 iterations Regex rx = new Regex("|"); count = rx.Matches(s).Count; - char array iteration using char index and (XOR or char comparison) gives the fastest results and is 100% safe from memory consumption point of view.... // * using XOR: // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; // * using char comparinson... // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; Please correct me if you'll find the above results have mistakes... Any takers to find quiker code for .NET VB or C# or C++/CLI? - that would be a good weekend exercise on code optimization techniques... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Max Wanadoo Sent: Sunday, April 13, 2008 1:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Johncliviger at aol.com Sun Apr 13 13:36:20 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 14:36:20 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Gustav at cactus.dk Sun Apr 13 13:50:38 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 20:50:38 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Pick only the fields you need ... or do you wish to display 30 fields? /gustav >>> Johncliviger at aol.com 13-04-2008 20:36 >>> Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Johncliviger at aol.com Sun Apr 13 14:08:13 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 15:08:13 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Gustav at cactus.dk Mon Apr 14 08:53:40 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 14 Apr 2008 15:53:40 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Many ways to do this - depends a bit on your previous code and what features you need. A lot of examples can be found via Google for this - including Microsoft's pages. Key words could be master/child or parent/child. /gustav >>> Johncliviger at aol.com 13-04-2008 21:08 >>> Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Johncliviger at aol.com Mon Apr 14 09:23:22 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Mon, 14 Apr 2008 10:23:22 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Gustav Many thanks I've cracked it. Quite simple when you know how. When you try it for this first time your Access thinking leads you astray. Cheers Gustav john c From Gustav at cactus.dk Tue Apr 15 07:28:29 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 15 Apr 2008 14:28:29 +0200 Subject: [dba-VB] Web colours Message-ID: Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav From accessd at shaw.ca Tue Apr 15 14:03:05 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:03:05 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From accessd at shaw.ca Tue Apr 15 14:28:06 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:28:06 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: <945F14E6828D4E92AB0736A26485C259@creativesystemdesigns.com> Hi Gustav: I should have read right through your question before I, with a sandwich in hand, responded. Use any colour you want. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Tuesday, April 15, 2008 12:03 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Web colours Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 16 08:44:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 16 Apr 2008 15:44:13 +0200 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Message-ID: Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24/microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav From accessd at shaw.ca Wed Apr 16 16:00:03 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Wed, 16 Apr 2008 14:00:03 -0700 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) In-Reply-To: References: Message-ID: <2751B45E193C418490D807B02A37C79A@creativesystemdesigns.com> Thanks for your heads up Gustav. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Wednesday, April 16, 2008 6:44 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24 /microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From paul.hartland at googlemail.com Fri Apr 18 02:49:18 2008 From: paul.hartland at googlemail.com (Paul Hartland) Date: Fri, 18 Apr 2008 08:49:18 +0100 Subject: [dba-VB] OT - Free Twain ActiveX Control Message-ID: <38c884770804180049n76b46424y6d66cf4124792ced@mail.gmail.com> To all, I need to develop an application, which will involve scanning some documents etc. Does anyone know of or use a Twain ActiveX Control (preferably easy to use & most of all free, or very cheap) Thank you for any help on this..... -- Paul Hartland paul.hartland at googlemail.com From jwcolby at colbyconsulting.com Sat Apr 19 09:05:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 10:05:31 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <4809FC2B.6030602@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 19 10:08:14 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 19 Apr 2008 17:08:14 +0200 Subject: [dba-VB] NetGender - The Software Company Message-ID: Hi John One method would be to have a MDI parent form from which you open the other forms as child forms. Then you just to either keep track of what you open or to loop through the child forms collection to see if it contains a certain child form. In VS2008 the Add New wizard will happily create a new MDI parent form with basic functionality. /gustav >>> jwcolby at colbyconsulting.com 19-04-2008 16:05 >>> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 11:08:13 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 12:08:13 -0400 Subject: [dba-VB] Splitting a name Message-ID: <480A18ED.6020009@colbyconsulting.com> I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 12:27:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 13:27:25 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <480A2B7D.6000202@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sat Apr 19 13:51:12 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sat, 19 Apr 2008 14:51:12 -0400 Subject: [dba-VB] NetGender - The Software Company References: <4809FC2B.6030602@colbyconsulting.com> Message-ID: <000301c8a24e$57223eb0$60606c4c@jisshowsbs.local> ...nice looking library ...let us know what problems you run into with it ...I may well be in the market for such myself in the near future. William ----- Original Message ----- From: "jwcolby" To: "VBA" Sent: Saturday, April 19, 2008 10:05 AM Subject: [dba-VB] NetGender - The Software Company >I purchased a .Net library that does name splitting. > > http://www.softwarecompany.com/dotnet/netgender.htm > > It seems to work well just from using the little demo that you can > download. Now I am starting to actually work with it. It consists of a > single class, embedded in a dll that you reference, with properties to > set to tell it all of the controlling stuff, then properties to hand it > the name to split and read back the name pieces. > > The developer claims that it will handle between 2K and 3K names per > second. We shall see. > > So here I am trying to work with this thing. The demo form is quite > extensive and the code in the form shows how to read / write all of the > properties to populate the form. > > I have a database with (of course) about 80 million names, in a single > field, in the form LastName FirstName MiddleInitial (no commas etc) > which needs parsing. What I thought I would do (just for the moment) is > to use the existing form and just feed in the names. Thus I need to add > a couple of properties to the form such that I can feed in names and get > back the parsed pieces. I will build a switch to tell the form to > display the pieces or not, thus I could watch the process function if I > wanted, though at the cost of slowing the process down. > > Modifications to the demo form to allow me to do all that stuff is just > about complete. What I need to know is the syntax to get a pointer to a > form being opened. I know that you just do a "Someform.Show" to open > the form, but a parent form needs to get a pointer to this demo form and > then use that pointer to set / retrieve properties. How do I get a > pointer to it once opened? > > -- > John W. Colby > www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From max.wanadoo at gmail.com Sun Apr 20 07:32:53 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 20 Apr 2008 13:32:53 +0100 Subject: [dba-VB] Splitting a name In-Reply-To: <480A18ED.6020009@colbyconsulting.com> References: <480A18ED.6020009@colbyconsulting.com> Message-ID: <002101c8a2e2$a84d3580$8119fea9@LTVM> John: No doubt there are many more cleverer answers then this and this may be tooooo slow for your needs. Function jc() ' will take the 1st series up to space as Xname, right series back to space as zname and bit in the middle as yname Const conQuote As String = """" Dim sql As String sql = "update tblTest set " sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & conQuote & ")-1) ," sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") - InStr(searchname, " & conQuote & " " & conQuote & "))," sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " & conQuote & ")+1)" CurrentDb.Execute (sql) End Function Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 19, 2008 5:08 PM To: VBA Subject: [dba-VB] Splitting a name I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 20 10:29:34 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 20 Apr 2008 11:29:34 -0400 Subject: [dba-VB] Splitting a name In-Reply-To: <002101c8a2e2$a84d3580$8119fea9@LTVM> References: <480A18ED.6020009@colbyconsulting.com> <002101c8a2e2$a84d3580$8119fea9@LTVM> Message-ID: <480B615E.5040903@colbyconsulting.com> Max, The problems with names is that there are many variations and you cannot count on simple rules like that. For example Mr. Juan Miguel de la Uariate. Now you get another list with De la Uriarte Juan Miguel Sr. You get the picture I purchased a library to do the split. The library performs the split nicely, handling such variations quite well. What I need now is a strategy for reading rows or groups of rows, updating a single record at a time back to SQL Server. In the absence of experience I am going to open a chunk of records,, perhaps 10K, with a startPK and EndPK to track the chunks into an ADO recordset. I will read in 10K records (or some chunk size), then just iterate that recordset splitting each name. My first attempt will be to dynamically build up an update query to write EACH RECORD back out, one at a time. I am almost certain that this will not be fast enough but we shall see. Once I have the code I will probably move the sql into a udf and call the udf passing the name parts in to the udf which will then write one record back to the table. Also likely not fast enough but we shall see. Last will be an attempt to build an ado recordset in memory, then have ADO write the entire recordset back to SQL Server. If that is not fast enough I have run out of ideas. Max Wanadoo wrote: > John: > No doubt there are many more cleverer answers then this and this may be > tooooo slow for your needs. > > Function jc() > ' will take the 1st series up to space as Xname, right series back to > space as zname and bit in the middle as yname > Const conQuote As String = """" > Dim sql As String > sql = "update tblTest set " > sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & > conQuote & ")-1) ," > sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " > & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") > - InStr(searchname, " & conQuote & " " & conQuote & "))," > sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " > & conQuote & ")+1)" > CurrentDb.Execute (sql) > End Function > > Max > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 19, 2008 5:08 PM > To: VBA > Subject: [dba-VB] Splitting a name > > I have a table with eight million records, with a single name field, and I > have a library that splits the name into components. What I need now is a > generic method of getting the name out of a table and updating a specific > set of fields back in the table. It has to do so QUICKLY since I have to > update 80 million names. > > In this case I have a single table > > Name > > NamePrefix > LastName > FirstName > MiddleName > NameSuffix > > Name already exists and contains a name in a consistent format LName FName > MName. > > The other fields I have added to my table. > > So what I am looking for is a general strategy. I assume I will open a read > recordset with the PKID and the name field, then iterate through this > recordset performing the split. > > Once I do that should I: > > Build a dynamic SQL Update query and execute that? > > Build a stored procedure that I pass the PKID and the name parts to which > then updates the data? > > Use an ADO recordset to do this? > > some other method entirely? > > I am assuming at this point that because of the sheer size of the data I do > not want to try and do this in an ado recordset. Eight million records > sitting in data at one time does not sound fun. > > All ideas gratefully accepted. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Johncliviger at aol.com Tue Apr 22 04:40:48 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 05:40:48 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Gustav at cactus.dk Tue Apr 22 05:57:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 12:57:51 +0200 Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi John For #3 I would use a DataTableAdapter to attach the table of the database - and then feed your data to a DataTable of the DataTableAdapter. /gustav >>> Johncliviger at aol.com 22-04-2008 11:40 >>> Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Johncliviger at aol.com Tue Apr 22 06:05:04 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 07:05:04 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi Gustav Yes. TableAdapter. I'm on the right lines then, which good news. Cheers gustav From Gustav at cactus.dk Tue Apr 22 10:51:23 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 17:51:23 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 11:10:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 18:10:18 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all I located this article which will be of interest for those with an Access background: Flat-MultiColumn Combobox with Autocomplete http://www.codeproject.com/KB/architecture/MultiColumnFlatCombo.aspx /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 17:06:50 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 00:06:50 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From cfoust at infostatsystems.com Tue Apr 22 17:55:36 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Tue, 22 Apr 2008 15:55:36 -0700 Subject: [dba-VB] No flat ComboBox in .Net?? In-Reply-To: References: Message-ID: We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 23 01:26:27 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 08:26:27 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Wed Apr 23 16:31:09 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 23:31:09 +0200 Subject: [dba-VB] Custom Access 2007 ribbons with Visual Studio 2008 Message-ID: Hi all Noticed this tip from FMS: Create Custom Microsoft Office 2007 Ribbons with the New Visual Designer in Visual Studio 2008 http://www.fmsinc.com/free/NewTips/NET/CreateCustomOffice2007Ribbons.asp With the new release of Visual Studio Tools for Office (VSTO) in Visual Studio 2008, you can now use a visual designer to create your own custom Ribbons in Microsoft Access, Excel, Word, Outlook, etc. These Ribbons are encapsulated as add-ins for each of the specific Office 2007 products. If you wanted to build your own custom Ribbons, but have held back by the lack of a visual designer, take a look at Visual Studio 2008. The visual designer is available under the New Project, Office, Office 2007 dialog. More resources: http://msdn2.microsoft.com/en-us/library/bb386089.aspx /gustav From mmattys at rochester.rr.com Wed Apr 23 19:58:58 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Wed, 23 Apr 2008 20:58:58 -0400 Subject: [dba-VB] No flat ComboBox in .Net?? References: Message-ID: <01c801c8a5a6$6201d2e0$0502a8c0@Laptop> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" To: Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From Gustav at cactus.dk Thu Apr 24 02:31:43 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:31:43 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Michael That's a nice tip. Thanks. Lots of other stuff on that site ... /gustav >>> mmattys at rochester.rr.com 24-04-2008 02:58 >>> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" < Gustav at cactus.dk > To: < dba-vb at databaseadvisors.com > Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? From Gustav at cactus.dk Thu Apr 24 02:38:08 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:38:08 +0200 Subject: [dba-VB] GDI+ graphics in .Net Message-ID: Hi all This site of Bob Powell which Michael pointed out has some very good explanation and code examples on pictures, graphics and drawing in .Net: http://www.bobpowell.net/faqmain.htm He also has a very good introduction. That info can be hard to find around the web. Remember: Graphics are fun! /gustav From jwcolby at colbyconsulting.com Fri Apr 25 11:31:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 12:31:25 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: <4812075D.6000306@colbyconsulting.com> OK, I have been badgered and goaded into coming into the 21st century. Not to mention I can't figure out how to safely and effectively share development of my .Net programs on all of my machines. So I am trying to get Subversion installed and functioning. So of course I come here seeking advice or to start a user group (goad and prod you guys) if no one is using this thing. I installed VisualSVN and it told me I had to install TortoiseSVN to get full functionality. It implied that it would install Subversion as well. The install did not ask any "install" questions such as where do you want the database to go etc. My intention is to place the Subversion database on one of my servers, share the directory, then place the various clients (TortoiseSVN and VisualSVN on each machine which I develop on. So my first question, is anyone out there using this thing? Is anyone interested in using this and sharing the pain of figuring it out? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Fri Apr 25 12:25:08 2008 From: robert at webedb.com (Robert L. Stewart) Date: Fri, 25 Apr 2008 12:25:08 -0500 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <200804251729.m3PHTFbD020137@databaseadvisors.com> Been through the pain and suffering already. I much prefer Sourcesafe. This was the tool used on one of the contracts I was on. They were always having problems with it. It is open source, so there was not what you would really call technical support. Good luck. At 12:00 PM 4/25/2008, you wrote: >Date: Fri, 25 Apr 2008 12:31:25 -0400 >From: jwcolby >Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >To: VBA >Message-ID: <4812075D.6000306 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >OK, I have been badgered and goaded into coming into the 21st century. >Not to mention I can't figure out how to safely and effectively share >development of my .Net programs on all of my machines. > >So I am trying to get Subversion installed and functioning. So of >course I come here seeking advice or to start a user group (goad and >prod you guys) if no one is using this thing. > >I installed VisualSVN and it told me I had to install TortoiseSVN to get >full functionality. It implied that it would install Subversion as >well. The install did not ask any "install" questions such as where do >you want the database to go etc. > >My intention is to place the Subversion database on one of my servers, >share the directory, then place the various clients (TortoiseSVN and >VisualSVN on each machine which I develop on. > >So my first question, is anyone out there using this thing? Is anyone >interested in using this and sharing the pain of figuring it out? > >-- >John W. Colby >www.ColbyConsulting.com > > >------------------------------ > >_______________________________________________ >dba-VB mailing list >dba-VB at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-vb > > >End of dba-VB Digest, Vol 54, Issue 18 >************************************** From jwcolby at colbyconsulting.com Fri Apr 25 12:41:23 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 13:41:23 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <200804251729.m3PHTFbD020137@databaseadvisors.com> References: <200804251729.m3PHTFbD020137@databaseadvisors.com> Message-ID: <481217C3.7020105@colbyconsulting.com> Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 13:27:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Fri, 25 Apr 2008 22:27:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481217C3.7020105@colbyconsulting.com> Message-ID: <000901c8a701$f4afd8b0$6401a8c0@nant> Hi John, I did use Subversion for some time, then I abandoned it because customer didn't insist on using any source control tools, and because I'm still working mainly as one man shop... ...I do plan to restart using Subversion hopefully in the near future - I wanted to be 'in' in this discussion and participate or at least lurk it because of a lot of work here, and not that much time left for pleasure of AccessD discussions... ...I did also use TortoiseSVN... ...I do agree that MS support is usually useless... ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as well as I do plan to add to that "bundle" MS SoundCastle, NCover and some other useful tools... ...one or another bug/issues/features tracking/managing tools should be used - I do use Axosoft's OnTiome2008 now... ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS Build + NUnit working together and making automated builds and (unit) test runs - that's the only way to go for teamwork development these days... ...all that mentioned above stuff worked well here but I must admit it was not that easy to make the parts working in ensemble but when everything was tuned it happened to be relatively easy to handle stuff... ...I have had two large ASP.NET applications with probably several (1,2,3 ? I didn't count) thousands of source files and about 15 projects in two solutions to put into Subversion code base, automate compilation/building using CruiseCOntrol.NET and MS Build etc... ...the only issue I can remember (after everything was set) is that I didn't find how to easily manage the cases to synchronize central code db with local code versions when source files are moved from one folder into another within the same project - I could have missed something because of lack of time, and experience or that could be a real "bottleneck" of Subversion with everything else being very good I think... ...I'd participate in preparing and publishing on AccessD web site together with other interested AccessD members an instruction/manual and set of scripts and utilities on how to set up and effectively use together Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say how much time I will have for this work... Thank you. -- Shamil P.S. As far as I know Subversion "beats" MS Source Safe for the cases when the project development should be fork-ed, and then several branches merged back: the difference between Subversion and MS SourceSafe is conceptual here IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I can be wrong, please correct me then, thank you. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Fri Apr 25 14:35:03 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 15:35:03 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000901c8a701$f4afd8b0$6401a8c0@nant> References: <000901c8a701$f4afd8b0$6401a8c0@nant> Message-ID: <48123267.7030608@colbyconsulting.com> Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 25 15:06:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 16:06:31 -0400 Subject: [dba-VB] Reading / writing SQL Server rows with ADO.Net (or something) Message-ID: <481239C7.2030506@colbyconsulting.com> I need to read blocks of data from tables in SQL Server, use values in the rows then update those rows and write back to SQL Server. I need to do this programmatically, not in a control on a form. does anyone have any example code for setting up the various ADO.Net objects, fetching rows of data from SQL Server, updating that data and writing it back to SQL Server? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:20:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:20:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000001c8a711$bdda92c0$6401a8c0@nant> John, Everything in Subversion can be controlled by using batch files - that is the best approach (IMO) for the initial setup, bulk checking and working set checkout: if I do not miss anything then I will post in the following several e-mails under this topic the set of .bat files I used to create the test environment last autumn(three projects ProjectA, ProjectB and ProjectC). Stay tuned (after posting I will be out here until Sunday). Thanks. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 11:35 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:38:38 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:38:38 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000401c8a714$5762dae0$6401a8c0@nant> Hi John, Below is the set of .batch files I have just found, which I did make last autumn to control/direct subversion server. I must say I'm not sure this is the last working version I used (AFAIKR I did write some simple custom code in .NET to control subversion server to avoid using .bat files, which have to be made for every project you'll use - and my simple custom code was parameterized to use any source project) but as you can see the set of batch files below is the minimal set to fulfill all the source control main tasks. Watch line wraps and read subversion docs for more details - svn-book.pdf - located in Sunversion server's installation folder (I'm out until Sunday). Thanks. -- Shamil P.S. .bat files to use with Subversion !svn_admin_main.bat ===================== rem Port: 3690 rem RepositoryPath: e:\Subversion\svn_repository rem X Install svnserve to run as a Windows service rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server call make_dirs.bat new_repository call check_out.bat new_repository pause make_dirs.bat ============== rmdir tmpdir /s /q mkdir tmpdir mkdir tmpdir\projectA mkdir tmpdir\projectA\trunk mkdir tmpdir\projectA\branches mkdir tmpdir\projectA\tags mkdir tmpdir\projectB mkdir tmpdir\projectB\trunk mkdir tmpdir\projectB\branches mkdir tmpdir\projectB\tags mkdir tmpdir\projectC mkdir tmpdir\projectC\trunk mkdir tmpdir\projectC\branches mkdir tmpdir\projectC\tags rem xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E rem rmdir /s /q ..\%1 call create_fs_repository.bat %1 rem svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log call verify_fs_repository.bat %1 call svnlook_fs_repository.bat %1 rmdir tmpdir /s /q rem rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA --message svnlog.log create_fs_repository.bat ======================== svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs xcopy ..\args\conf\*.* ..\%1\conf /Y /E goto :EOF verify_fs_repository.bat ======================== svnadmin verify ../%1 GOTO :EOF svnlook_fs_repository.bat ========================= svnlook info ../%1 goto :EOF check_out.bat ============= rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S mkdir E:\Temp\SV\WORKING_SET\ /Q /S svn checkout file:///E:/Subversion/%1/projectA/trunk E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log svn checkout file:///E:/Subversion/%1/projectB/trunk E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log svn checkout file:///E:/Subversion/%1/projectC/trunk E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log commit.bat ========== svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log backup_fs_repository.bat ======================== svnadmin hotcopy ../%1 ../backups/%1 GOTO :EOF From jwcolby at colbyconsulting.com Fri Apr 25 17:15:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 18:15:27 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000401c8a714$5762dae0$6401a8c0@nant> References: <000401c8a714$5762dae0$6401a8c0@nant> Message-ID: <481257FF.30208@colbyconsulting.com> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 26 06:52:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 13:52:35 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil I would love to follow this thread. I work both at home and at the office and is a bit tired of copying files around. I located this page: Source Control with Visual Studio .NET http://www.codeproject.com/KB/dotnet/SourceControl_VSNET.aspx?fid=471385&df=90&mpp=25&noise=3&sort=Position&view=Quick&fr=26#xx0xx which, however, seems to know nothing about VS2008. It has links to these add-ins for VS: Garry Broadsword's Add-on which is updated for VS2008: http://garrys-brain.blogspot.com/2007/07/tortoisesvn-and-visual-studio.html and VisualSVN: http://www.visualsvn.com/ This site also sports VisualSVN Server: http://www.visualsvn.com/server/ VisualSVN Server is a package that contains everything you need to install, configure and manage Subversion server for your team on Windows platform. It includes Subversion, Apache and a management console. ... And last, but not least - VisualSVN Server is completely free! This must be what John is after - and myself as neither I nor any colleague have much experience with Apache. Now, big question: Does this replace or supplement Shamil's collection of batch files? As I understand it, VisualSVN Server should take care of the setup of Subversion but I must admit I have not yet grasped all the bits and pieces of this ... A final note for others that may follow this thread: Nothing of this works with the free Visual Studio Express versions. /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil From Gustav at cactus.dk Sat Apr 26 07:04:22 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 14:04:22 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil Exciting. http://www.visualsvn.com/doc/getting-started.html Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? Have never heard of Google Code. Can I use Google to store a project? http://code.google.com/ How does that work and is it only for public access project? Or how would you control access for closed source projects? /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. From jwcolby at colbyconsulting.com Sat Apr 26 07:11:54 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 08:11:54 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <48131C0A.1000302@colbyconsulting.com> the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 26 14:25:56 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 15:25:56 -0400 Subject: [dba-VB] ADO.Net Message-ID: <481381C4.7000804@colbyconsulting.com> I have been setting up an application to read a block of records, perhaps 10K or something. I designed a class that will load a block of records which consist of a name to be parsed, and a set of 6 fields which will be filled in with the parsed data returned from the name parser. Thus as I load the data only the name to parse is filled in. The supervisor iterates this set of records, pulling the name to parse, passing it off to the name parser, then reading the data back from the name parser and updating the fields in the same record. Move to the next record, wash, rinse repeat. I do have a couple of questions though. I am not very familiar with how the ADO.net stuff works. I build a connection string I build a connection object which I leave not connected (for now), but feeding it the connection string. I build a sql statement that represents the fields to pull WHERE PK between StartValue and EndValue. I build a command object which I pass SQL statement and the still unconnected connection object. I build a data adapter which I feed the command object. Notice that we are still not connected to the data store. I build a DataTable object with a table name. I call DataAdapter(DataTable).Fill Does .Net then connect correctly and proceed to fill the data table? Once I have the data table filled I start to iterate the DataTable using the for each DataRow iterator I then call the name splitter passing in the name to be split. Back comes the name pieces. I fill in the row with the name pieces and move to the next row. Once every row has been filled in I need to call an update method of the DataAdapter which is SUPPOSED to write the changes back to the table in SQL Server..... The name splitter handles ~1K names / second, NOT as speedy as I had hoped. At this point I haven't a clue how fast the data adapter will write back to the table in SQL Server. What I am trying to do is write this thing such that each instance of the data class can then be handed off to a thread so that the write back to SQL Server can begin and I can instantiate another data class to load / parse the next "chunk" of data in parallel with the write to SQL Server. I am trying to discover whether I need to close connections to the database while I do the parsing or just not worry about it. Should I just leave all the objects open and close them all down as the data class closes once the write to SQL Server is done? How will I know the write is done? Does the code "hang" at the DataAdapter.Update until the entire update (all the rows) is finished? Am I going about this the right way? Should I become a used car salesman? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <481381C4.7000804@colbyconsulting.com> Message-ID: <000501c8a7ea$a4d53080$6401a8c0@nant> Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481257FF.30208@colbyconsulting.com> Message-ID: <000401c8a7ea$a4c949a0$6401a8c0@nant> Hi John, Yes, I'm considering to purchase it also very probably... ...I will wait how well it will work for you - OK :) ? -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 2:15 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48131C0A.1000302@colbyconsulting.com> Message-ID: <000301c8a7ea$a4af0ae0$6401a8c0@nant> <<< running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever >>> John, The batches I posted here are just to work with Subversion's repository: create it, check-in, check-out, backup... ...to run builds and automate unit testing etc. CruiseControl.NET has to be used, or other similar tools... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 4:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:39:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:39:45 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <000601c8a7ee$6d9119a0$6401a8c0@nant> Hi John, Additionally to the below e-mail - you can consider using SQLDataReader to get data records batches as in this below example, then build parsed names' arrays together with ids, then Join() these arrays into strings and pass these strings as parameters to SPs, which will make actual updates by parsing strings into dynamic update sqls or something like that - T-SQL parsing functions are reasonably quick, or you can even consider managed SPs to parse the strings on server side and perform the updates... ...but that latter could be an overkill giving very little gains because .NET framework's communication with native MS SQL provider when using plain simple C#/VB.NET is very effective I have heard... ..maybe you can just process your data and generate and run update temp SPs to have update T-SQL statements batched into such temp SPs - that could give you (much) quicker updates... ...etc... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 2:13 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 18:31:32 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 19:31:32 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> References: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <4813BB54.4090203@colbyconsulting.com> Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 18:47:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 03:47:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813BB54.4090203@colbyconsulting.com> Message-ID: <000001c8a7f7$e871c530$6401a8c0@nant> <<< I am definitely interested in doing strongly typed recordsets though. >>> John, That strongly typed recordsets are imposing big overhead - you can dig into the code generated (by VS) for these recordsets... ...as for batch updates "automagically" generated by ADO.Net - you can use SQL profiler to see what happens "under the hood"... ...I'd bet that if you generate batch updates by custom code into temp SPs and use datareader to get data to build these batches then this approach will result in a way faster data processing, and you will have many options to optimize it even more... ...just my opinion but as you can find in ASP.Net apps ADO.NET typed recordsets are not recommended to use - have a look at e.g. "ASP.NET Website Programming: Problem - Design - Solution C# Edition" by Marco Bellinaso.... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 3:32 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 19:45:47 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 20:45:47 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a7f7$e871c530$6401a8c0@nant> References: <000001c8a7f7$e871c530$6401a8c0@nant> Message-ID: <4813CCBB.3060907@colbyconsulting.com> This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx There are many things to consider when I program in .net, not the least of which is simplicity. If I can get a simple method that gets me medium efficiency, it is better (for me) than a complex method that gets high efficiency simply because I may be able to implement the simple method whereas the complex method would never get done. I can always come back later and recode for a higher efficiency after I have a method working and chugging away. As long as the program does not take weeks to run, it is better to spend cpu time than John Colby time. As I mentioned, the name parser is not super fast. I am hoping to get the first database update batch happening in parallel with the second name parsing batch. I also hope to run this on a completely different computer which has more cores (and memory) so that I can use threads effectively. Which is not to say I am not interested in the most efficient method, simply that I will likely be unable to code it very quickly. Shamil Salakhetdinov wrote: > > <<< > I am definitely interested in doing strongly typed recordsets though. > John, > > That strongly typed recordsets are imposing big overhead - you can dig into > the code generated (by VS) for these recordsets... > > ...as for batch updates "automagically" generated by ADO.Net - you can use > SQL profiler to see what happens "under the hood"... > > ...I'd bet that if you generate batch updates by custom code into temp SPs > and use datareader to get data to build these batches then this approach > will result in a way faster data processing, and you will have many options > to optimize it even more... > > ...just my opinion but as you can find in ASP.Net apps ADO.NET typed > recordsets are not recommended to use - have a look at e.g. "ASP.NET Website > Programming: Problem - Design - Solution C# Edition" > by Marco Bellinaso.... > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 3:32 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Thanks Shamil, > > Unfortunately unless you are dealing with 80 million row 200 field > tables, any timing comparisons are going to be suspect. > > I considered doing (and may still do) a record by record update using > either a stored procedure or dynamic sql right from vb.net. Then I read > an article (blog) that essentially said that the batch update from an > ado dataset really did work as you would hope that it would, and this in > a blog from a member of Microsoft's ADO.Net dev team. > > He claimed that either of the other methods had network transaction > overhead as each update (when done singly) required the command to do it > as well as a result coming back, whereas if you used a batch it would > group all of the commands out and then group all the results coming back > into big packages. > > Understand that I do not know enough to even comment on the validity of > his argument however he said to definitely try it. Set the batch size > and then let ADO just send batches of updates. > > Given that the coding effort is smaller to do that I figured I would at > least get that running. > > I am definitely interested in doing strongly typed recordsets though. > > Shamil Salakhetdinov wrote: >> Hi John, >> >> Try to use SqlDataReader and custom classes as in the sample code below >> (sorry C# is used - real pleasure to use C# is also coming from the fact >> that you can copy and paste code and do not care about line warps because >> even "screwed by e-mail" code should be compiled well).... >> >> ...using ADO.NET datasets and bulk update will anyway result in a series > of >> updates on SQL server side therefore custom on-the-fly built sql updates > or >> calls to update SPs should work as quick as ADO.NET DataSets' batch >> update... >> >> ...etc... >> >> ...as you can see my sample did update 25000 records in ~10 seconds - and >> this only for starters - if that approach will work reasonably well for > you >> then the next step could be to introduce multi-threading etc... >> >> ...note also how MS SQL 2005's paging feature is used to get record's >> batches... > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 04:08:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:08:08 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813CCBB.3060907@colbyconsulting.com> Message-ID: <000001c8a846$3606c6d0$6401a8c0@nant> Hello John, OK, I will read the blog you mentioned later today... I know it's not easy to convince you to change your mind when you are "driving your way full speed" :) And I'm not trying to - I'm just musing in this coding samples I post here testing this or that approach, which I might use in my own development... <<< There are many things to consider when I program in .net, not the least of which is simplicity. >>> May I say that proposed in C# solution is simple and almost ready to use in your scenario in one thread? And also ready to be converted into a multi-threading app/utility. And you don't need to translate in manually in VB.NET - just use this free tool: http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx or this one: http://codeconverter.sharpdevelop.net/Convert.aspx and voila' the converted VB.NET is below : (watch line wraps now - this is VB.NET - it was "automagically" converted by referred above tool by copying and pasting C# code I posted yesterday + making some manual editing of several code line to make it independent of my testing/development environment) - and it seems to work even quicker than C# equivalent somehow !?. I have got additional back VB.NET -> C# conversion in the end of this message - you can use C# code copy and paste it in the tool refereed above and get VB.NET code properly formatted. Proposal: let's post C# code here for the folks who will try this code to not spend time with fixing line wraps introduced by e-mail clients): Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) ID = id Name = name End Sub Public ReadOnly Property ParsedName() As String Get ' just for testing purposes ' return real parsed name here Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry '27.04.2008 12:45:58 'Count(*) = 25045 'Counter = 25045 '27.04.2008 12:46:04 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = _ String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", _ pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list p.Update(item) count += 1 Next End While Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace Here it's converted back to C#: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB"; string dbName = "dbName"; string userName = "sa"; string password = "sapwd"; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 4:46 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx <<< tail skipped to get message through in AccessD >>> From Gustav at cactus.dk Sun Apr 27 04:33:02 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 11:33:02 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From shamil at smsconsulting.spb.ru Sun Apr 27 04:42:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:42:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: Message-ID: <000601c8a84b$0711cdc0$6401a8c0@nant> Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sun Apr 27 05:53:36 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 14:53:36 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000601c8a84b$0711cdc0$6401a8c0@nant> Message-ID: <000301c8a854$f1c57cf0$6401a8c0@nant> ...now the VB.net version of manually batched updates (watch line wraps!) It seems to work slightly quicker than similar C# version: ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 'http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx 'http://codeconverter.sharpdevelop.net/Convert.aspx Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) id = id name = name End Sub Public ReadOnly Property ParsedName() As String ' just for testing purposes ' return real parsed name here Get Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Public ReadOnly Property BatchSize() As Integer Get Return _batchSize End Get End Property Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() 'error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function Private _updateCommands As String() = Nothing Private _arrayIndex As Integer = -1 Public Sub AddUpdateCommand(ByVal item As DataItem) If _arrayIndex = -1 Then _updateCommands = DirectCast(Array.CreateInstance(GetType(String), _batchSize), String()) _arrayIndex = 0 End If Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) 'badly translated => _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql _updateCommands(_arrayIndex) = sql _arrayIndex += 1 End Sub Public Function BatchUpdate() As Integer If _arrayIndex = -1 Then Return -1 End If If _arrayIndex < _batchSize Then Array.Resize(Of String)(_updateCommands, _arrayIndex) End If Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() 'Console.WriteLine("Batch update: {0} updates", _arrayIndex) Dim sql As String = [String].Join(";", _updateCommands) _arrayIndex = -1 cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list If count > 0 AndAlso count Mod p.BatchSize = 0 Then p.BatchUpdate() End If 'p.Update(item); p.AddUpdateCommand(item) count += 1 Next End While p.BatchUpdate() Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 1:43 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From jwcolby at colbyconsulting.com Sun Apr 27 07:12:19 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 08:12:19 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a846$3606c6d0$6401a8c0@nant> References: <000001c8a846$3606c6d0$6401a8c0@nant> Message-ID: <48146DA3.1010200@colbyconsulting.com> LOL, now that I can readi it... ;-) >> and voila' the converted VB.NET is below : (watch line wraps now - this is Shamil Salakhetdinov wrote: > Hello John, > > OK, I will read the blog you mentioned later today... > > I know it's not easy to convince you to change your mind when you are > "driving your way full speed" :) And I'm not trying to - I'm just musing in > this coding samples I post here testing this or that approach, which I might > use in my own development... > > <<< > There are many things to consider when I program in .net, not the least > of which is simplicity. > May I say that proposed in C# solution is simple and almost ready to use in > your scenario in one thread? And also ready to be converted into a > multi-threading app/utility. And you don't need to translate in manually in > VB.NET - just use this free tool: > > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > > or this one: > > http://codeconverter.sharpdevelop.net/Convert.aspx > > and voila' the converted VB.NET is below : (watch line wraps now - this is > VB.NET - it was "automagically" converted by referred above tool by copying > and pasting C# code I posted yesterday + making some manual editing of > several code line to make it independent of my testing/development > environment) - and it seems to work even quicker than C# equivalent somehow > !?. I have got additional back VB.NET -> C# conversion in the end of this > message - you can use C# code copy and paste it in the tool refereed above > and get VB.NET code properly formatted. Proposal: let's post C# code here > for the folks who will try this code to not spend time with fixing line > wraps introduced by e-mail clients): > > Imports System > Imports System.Collections.Generic > Imports System.Data.SqlClient > > Namespace TestSamples > > Public Class DataItem > Public ID As Integer > Public Name As String > > Public Sub New(ByVal id As Integer, ByVal name As String) > ID = id > Name = name > End Sub > > Public ReadOnly Property ParsedName() As String > Get > ' just for testing purposes > ' return real parsed name here > Return Name > End Get > End Property > End Class > > Public Class DataProcessor > Private Const COUNT_SQL As String = "select count(*) from > zsysUserSessionTrace" > Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + > " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as > int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " > UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum > between {0} and {1}" > Private Const UPDATE_SQL As String = "update zsysUserSessionTrace > set UserBrowser = '{0}' where UserSessionTraceId = {1}" > > Private Const DEFAULT_BATCH_SIZE As Integer = 1000 > > Private _totalCount As Integer = -1 > Public ReadOnly Property TotalCount() As Integer > Get > Return _totalCount > End Get > End Property > > Private _currentBatchStartRowNum As Integer = 0 > Private _batchSize As Integer = 0 > > Private _connectionString As String > Public Sub New(ByVal connectionString As String) > initialize(connectionString, DEFAULT_BATCH_SIZE) > End Sub > Public Sub New(ByVal connectionString As String, ByVal batchSize As > Integer) > initialize(connectionString, batchSize) > End Sub > Private Sub initialize(ByVal connectionString As String, ByVal > batchSize As Integer) > > _batchSize = batchSize > _currentBatchStartRowNum = 1 > _connectionString = connectionString > > Using cnn As New > System.Data.SqlClient.SqlConnection(connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > cmd.CommandText = COUNT_SQL > cmd.CommandType = System.Data.CommandType.Text > > _totalCount = CInt(cmd.ExecuteScalar()) > End Using > End Sub > > Public ReadOnly Property HasData() As Boolean > Get > Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < > TotalCount) > End Get > End Property > > Public Function GetData() As List(Of DataItem) > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > _currentBatchStartRowNum += _batchSize + 1 > Dim rdr As System.Data.SqlClient.SqlDataReader = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) > > Dim list As New List(Of DataItem)() > While rdr.Read() > list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), > String))) > End While > Return list > End Using > End Function > > Public Function Update(ByVal item As DataItem) As Integer > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(UPDATE_SQL, > item.ParsedName, item.ID) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > Return cmd.ExecuteNonQuery() > End Using > End Function > End Class > > > Public Class A_MainEntry > '27.04.2008 12:45:58 > 'Count(*) = 25045 > 'Counter = 25045 > '27.04.2008 12:46:04 > > Public Shared Sub Main() > Dim count As Integer = 0 > Try > '+ customize - set your connection string here > Dim pcName As String = "SPB" > Dim dbName As String = "dbName" > Dim userName As String = "sa" > Dim password As String = "sapwd" > Dim connectionString As String = _ > String.Format("Data Source={0}\SQLEXPRESS;Initial > Catalog={1};User Id={2};Password={3}", _ > pcName, dbName, userName, password) > '- > > Console.WriteLine(DateTime.Now) > Dim p As New DataProcessor(connectionString) > While p.HasData > Dim list As List(Of DataItem) = p.GetData() > For Each item As DataItem In list > p.Update(item) > count += 1 > Next > End While > Console.WriteLine("Count(*) = {0}", p.TotalCount) > Console.WriteLine("Counter = {0}", count) > Console.WriteLine(DateTime.Now) > Catch ex As Exception > Console.WriteLine(ex.Message) > End Try > End Sub > > End Class > End Namespace > > Here it's converted back to C#: > > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > using System; > using System.Collections.Generic; > using System.Data.SqlClient; > > namespace TestSamples > { > > public class DataItem > { > public int ID; > public string Name; > > public DataItem(int id, string name) > { > ID = id; > Name = name; > } > > public string ParsedName > { > // just for testing purposes > // return real parsed name here > get { return Name; } > } > } > > public class DataProcessor > { > private const string COUNT_SQL = "select count(*) from > zsysUserSessionTrace"; > private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " > select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) > as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " > + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and > {1}"; > private const string UPDATE_SQL = "update zsysUserSessionTrace set > UserBrowser = '{0}' where UserSessionTraceId = {1}"; > > private const int DEFAULT_BATCH_SIZE = 1000; > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(1), (string)rdr(3))); > list.Add(new DataItem((int)rdr[1], (string)rdr[3])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(UPDATE_SQL, item.ParsedName, > item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > } > > > public class A_MainEntry > { > // VB.NET > //27.04.2008 13:01:44 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:49 > > // C# > //27.04.2008 13:00:56 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:04 > > public static void Main() > { > int count = 0; > try > { > //+ customize - set your connection string here > string pcName = "SPB"; > string dbName = "dbName"; > string userName = "sa"; > string password = "sapwd"; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > Console.WriteLine(DateTime.Now); > DataProcessor p = new DataProcessor(connectionString); > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > Console.WriteLine(DateTime.Now); > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 4:46 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > This is the blog I mentioned: > > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > <<< tail skipped to get message through in AccessD >>> > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 10:14:00 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:14:00 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48146DA3.1010200@colbyconsulting.com> Message-ID: <001401c8a879$5298bd70$6401a8c0@nant> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil From shamil at smsconsulting.spb.ru Sun Apr 27 10:24:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:24:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <000001c8a87a$cfa80270$6401a8c0@nant> ...additionally to my quoted posting here is SQL script to create test table: CREATE TABLE [dbo].[zsysUserSessionTrace]( [UserSessionTraceId] [int] IDENTITY(1,1) NOT NULL, [AspNetSessionId] [nvarchar](80) NOT NULL, [HostAddress] [nvarchar](40) NULL, [HostName] [nvarchar](40) NULL, [PageViewCount] [int] NULL, [UserName] [nvarchar](40) NOT NULL, [UserBrowser] [nvarchar](255) NOT NULL, [UpdatedBy] [nvarchar](20) NOT NULL, [UpdatedDate] [datetime] NOT NULL, [UpdatedLocation] [nvarchar](20) NOT NULL, [ParsedUserBrowserName] [nvarchar](255) NULL, CONSTRAINT [PK_zsysUserSessionTrace] PRIMARY KEY CLUSTERED ( [UserSessionTraceId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] ...and here is insert startemet to populate one row - you can run it many times: insert into zsysUserSessionTrace (AspNetSessionId, UserName, UserBrowser, UpdatedBy, UpdatedDate, UpdatedLocation) values (1,'TEST','IE7','SA','2008-04-27','SPB') ... and here are select statements to run e.g. in MS SQL Management console to see that test updates really happened: select count(*) from zsysUserSessionTrace go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId] desc) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 7:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 27 11:42:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 18:42:13 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil I'm following this thread with great interest because it touches some tasks I may soon encounter. It's a pleasure to study your work, and I really appreciate that you take your time to post it here. /gustav >>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? From jwcolby at colbyconsulting.com Sun Apr 27 11:56:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 12:56:40 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> References: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <4814B048.6070704@colbyconsulting.com> What I come hear is that using the built-in batch is as fast as any other method, so if that is what I find easier to use there is no issue using that. I will try to understand your other methods (in VB, since the language makes very little apparent speed difference) and implement that in parallel so that I can switch between them on demand. If I can do that I will do some timings on my big database, which is what I am doing the updates on. Shamil Salakhetdinov wrote: > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > I wasn't aware of MS SQL TDS feature you mentioned but as you can see from > stats it gives just slightly better results than the manual batching. Of > course playing with batch size might give better gains of this method (my > tests didn't show that gains depending on batch size) but I must say for me > it looks more complicated and less flexible that using SQLDatareader and > custom classes - that could be just my perception.... > > The C# code is below in the end of this message - its VB.NET equivalent can > be obtained using automatic conversion - > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one > code line have to be manually patched > > 'conversion issue replace the following code line => > ==================================================== > _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra > yIndex), _arrayIndex - 1)) = sql > > with this one > ============= > _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 > : > > Here are the stats: > > ==================== > C# > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:02:58 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:15 > Elapsed time: 00:00:16.7968750 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:03:15 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:23 > Elapsed time: 00:00:08.4218750 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:03:23 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:31 > Elapsed time: 00:00:07.9062500 > > ==================== > VB.NET > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:05:20 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:36 > Elapsed time: 00:00:16.7500000 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:05:36 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:45 > Elapsed time: 00:00:08.2812500 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:05:45 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:52 > Elapsed time: 00:00:07.5000000 > ==================== > > // Converters: > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > // Sql TDS batched update > //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > using System; > using System.Collections.Generic; > using System.Data; > using System.Data.SqlClient; > > namespace TestSamples > { > public sealed class Constants > { > //+ ///////////// C U S T O M I Z E ///////////////////// > public const string PcName = "NANT"; > public const string DbName = "ORS2005"; > public const string UserName = "sa"; > public const string Password = "shms"; > > public const string TABLE_NAME = "zsysUserSessionTrace"; > public const string ID_COL_NAME = "UserSessionTraceId"; > public const string NAME_COL_NAME = "UserBrowser"; > public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; > > public const int DEFAULT_BATCH_SIZE = 1000; > > public static string GetParsedName(string name) > { > // just for testing purposes - customzie > // to return your parsed name here > if (name == null) name = ""; > name = DateTime.Now.ToString() + ": " + name; > if (name.Length > 255) name = name.Substring(0, 255); > > return name; > } > //- ///////////// C U S T O M I Z E ///////////////////// > > public const int ID_COL_INDEX = 1; > public const int NAME_COL_INDEX = 2; > public const int PARSED_NAME_COL_INDEX = 3; > > public static string COUNT_SQL { get { > return String.Format("select count(*) from {0}", TABLE_NAME); } > } > > public static string PAGED_SELECT_SQL { get { > return string.Format( > "SELECT * from " + "(" + " select " + > " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as > RowNum, " + > " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", > ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, > PARSED_NAME_COL_NAME, TABLE_NAME) + > "where s.RowNum between {0} and {1}"; }} > > public static string UPDATE_SQL { get { > return string.Format("update {0} ",TABLE_NAME) + > string.Format(" set {0} = '{1}' where {2} = {3}", > PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); > }} > > public static string UPDATE_SQL_PARAMETERIZED { get { > return string.Format("update {0} set {1} = @{2} where {3} = > @{4}", > TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, > ID_COL_NAME, ID_COL_NAME); > }} > } > > public class DataItem > { > private int _id; > public int ID > { > get { return _id; } > } > > private string _name; > public string Name > { > get { return _name; } > } > > public DataItem(int id, string name) > { > _id = id; > _name = name; > } > > public string ParsedName > { > get > { > return Constants.GetParsedName(_name); > } > } > > > } > > public class DataProcessor > { > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > public int BatchSize { get { return _batchSize; } } > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = Constants.COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(Constants.ID_COL_INDEX), > (string)rdr(Constants.NAME_COL_INDEX))); > list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], > (string)rdr[Constants.NAME_COL_INDEX])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private string[] _updateCommands = null; > private int _arrayIndex = -1; > public void AddUpdateCommand(DataItem item) > { > if (_arrayIndex == -1) > { > _updateCommands = > (string[])Array.CreateInstance(typeof(string), _batchSize); > _arrayIndex = 0; > } > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > _updateCommands[_arrayIndex++] = sql; > } > > public int BatchUpdate() > { > if (_arrayIndex == -1) return -1; > if (_arrayIndex < _batchSize) > Array.Resize(ref _updateCommands, _arrayIndex); > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > > //Console.WriteLine("Batch update: {0} updates", > _arrayIndex); > > string sql = String.Join(";", _updateCommands); > _arrayIndex = -1; > > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private SqlDataAdapter _da = null; > private DataSet _ds = null; > public DataTable SQLTDSGetDataTablePaged() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > int batchSize = _batchSize; > if (_currentBatchStartRowNum + _batchSize >= _totalCount) > batchSize = _totalCount - _currentBatchStartRowNum; > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > > _currentBatchStartRowNum += _batchSize + 1; > > _da = new SqlDataAdapter(sql, cnn); > _ds = new DataSet(); > _da.Fill(_ds); > > _da.UpdateCommand = new > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); > _da.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, > Constants.PARSED_NAME_COL_NAME); > _da.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); > _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; > _da.UpdateBatchSize = batchSize; > > _da.SelectCommand.Connection = null; > _da.UpdateCommand.Connection = null; > > return _ds.Tables[0]; > } > } > > public void SQLTDSUpdate() > { > SQLTDSUpdate(false); > } > public void SQLTDSUpdate(bool forceUpdate) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > _da.SelectCommand.Connection = cnn; > _da.UpdateCommand.Connection = cnn; > if (forceUpdate) > _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; > _da.Update(_ds.Tables[0]); > > //Console.WriteLine("*** Forced test exit ***"); > //System.Environment.Exit(100); > } > } > } > > public class A_MainEntry > { > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate and execute individual update SQL > statements > /// > public static void Strategy1(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > //Console.WriteLine("{0} - rows processed...", p.BatchSize); > > } > } > > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate batched update SQL statements, and > /// execute update batch when it has all update > /// statements added > /// > public static void Strategy2(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > if (count > 0 && (count % p.BatchSize) == 0) > p.BatchUpdate(); > p.AddUpdateCommand(item); > count += 1; > } > } > p.BatchUpdate(); > } > > /// > /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, > and > /// use SQL TDS bacthed updates facility > /// > public static void Strategy3(DataProcessor p, ref int count) > { > while (p.HasData) > { > DataTable table = p.SQLTDSGetDataTablePaged(); > > foreach (DataRow row in table.Rows) > { > row.SetModified(); > row[Constants.PARSED_NAME_COL_INDEX] = > Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); > count += 1; > } > p.SQLTDSUpdate(!p.HasData); > } > } > > public static void Main() > { > try > { > //+ customize - set your connection string here > string pcName = Constants.PcName; > string dbName = Constants.DbName; > string userName = Constants.UserName; > string password = Constants.Password; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > for (int i = 1; i <= 3; i++) > { > int count = 0; > DateTime startTime, endTime; > startTime = DateTime.Now; > Console.WriteLine(""); > Console.WriteLine("Strategy {0}:", i); > Console.WriteLine("-----------"); > Console.WriteLine("Start time: {0}", startTime); > DataProcessor p = new DataProcessor(connectionString); > > switch (i) > { > case 1: > Strategy1(p, ref count); > break; > case 2: > Strategy2(p, ref count); > break; > case 3: > Strategy3(p, ref count); > break; > default: > Console.WriteLine("{0} - undefined strategy > number", i); > break; > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > endTime = DateTime.Now; > Console.WriteLine("End time: {0}", endTime); > TimeSpan ts = > TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); > Console.WriteLine("Elapsed time: {0}", ts); > } > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > Enjoy! > > -- > Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sun Apr 27 14:13:04 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sun, 27 Apr 2008 15:13:04 -0400 Subject: [dba-VB] ADO.Net References: Message-ID: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Sun Apr 27 16:02:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 01:02:11 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> Message-ID: <000401c8a8a9$f695a200$6401a8c0@nant> Here are the sources I promised to publish in my previous post: http://smsconsulting.spb.ru/samples/Stats.txt http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt Your turn, guys, to fix and improve them... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of William Hindman Sent: Sunday, April 27, 2008 11:13 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 27 22:01:58 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 23:01:58 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000401c8a8a9$f695a200$6401a8c0@nant> References: <000401c8a8a9$f695a200$6401a8c0@nant> Message-ID: <48153E26.3040803@colbyconsulting.com> Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Mon Apr 28 01:52:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 08:52:21 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all What happened? How have I been fooling around with covered eyes? If I select Flat for property FlatStyle the ComboBox changes to ... a flat design. No visible borders. Only issue is that the invisible borders of the ComboBox appears to be as wide as if the ComboBox has 3D-style, thus leaving a larger total area than an equivalent TextBox with identical font settings. If I force the size of the ComboBox to be lower, it won't display the content (not very useful). /gustav >>> Gustav at cactus.dk 23-04-2008 08:26 >>> Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Mon Apr 28 10:53:55 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 17:53:55 +0200 Subject: [dba-VB] C# event handler syntax Message-ID: Hi all Just browsed this page: http://blueonionsoftware.com/Blog.aspx?p=8afe238f-87c2-400b-9440-8231280f7576 which has a brief explanation on the syntax from the beginning to the ultra compact of version 3.5 using Lambda expression: clock.Alarm += (o, e) => Console.Write("Alarm!"); This is _not_ what IntelliSense suggests, so you are a little on your own. /gustav From shamil at smsconsulting.spb.ru Mon Apr 28 11:40:25 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 20:40:25 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48153E26.3040803@colbyconsulting.com> Message-ID: <00c301c8a94e$8f660430$6401a8c0@nant> Hi John, Thank you for your proposal to run the code against your huge db... OK, let's try this remote/offline Q&A/code musing/consulting/education in several installments published here: I must say I have quite some work to do for customers and I can't spend a lot of time for postings here/explaining what and why is done in the sample code - I will try just give refs on "concepts" I use for you to read more in many open sources... I made first adjustment to the yesterday's code - new code is here: http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt http://smsconsulting.spb.ru/samples/Stats.txt To run it (let's use VB.NET): - in VS2005 create VB.NET Console application, replace the code of the module, which VS2005 creates by default (Module1.vb) with the code from http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines Public Const PcName As String = "LOCALHOST" Public Const DbName As String = "NamesParser" Public Const UserName As String = "TESTER" Public Const Password As String = "TEST.1959" to point to your db and its credentials, push CTRL+F5, and you should see the sample app running in console window and producing something like this report: Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:13 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:19 Elapsed time: 00:00:05.5156250 Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:19 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:22 Elapsed time: 00:00:03.5468750 Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:22 Count(*) = 1000 Counter = 1009 End time: 28.04.2008 20:18:26 Elapsed time: 00:00:03.3906250 Try. If you tell me the name of your database and the name of your table I will change them here, as well as you can create username and password as above to not do any changes later for the new versions of the code. N.B: Make sure you will run the code against sample not the live db. The code assumes that the length of the target field FName is 50 chars - it uses just one this column currently to put "parsed" value - just first 50 chars of OWNERNAME field. Please have a look through the code, and ask your questions in the order you wanted them to be answered... Please everybody who wanted to participate in this discussion - you're very welcome! On debugging multi-threaded apps: that's a real PITA - yes you can debug/trace with breakpoints etc. but you have to keep in mind that as many threads as you have will cause debugger to stop on breakpoints etc. IOW better debug/trace single threaded code and them run it in multi-threaded mode, and use some indirect artefacts/techniques (as unit testing), which will show you that your app is running OK: as you can find in the new version of the code it can run both single and multi-threaded depending on the value of this constant: Public Const MULTI_THREADED_MODE As Boolean = True -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 7:02 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 11:55:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 12:55:08 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <00c301c8a94e$8f660430$6401a8c0@nant> References: <00c301c8a94e$8f660430$6401a8c0@nant> Message-ID: <4816016C.4010709@colbyconsulting.com> Thanks for the response Shamil, I know that you have lots of stuff to get done as do I. I am already back to the code I know in order to get this thing processed, however I will get your sample code running in the coming days. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > Thank you for your proposal to run the code against your huge db... > > OK, let's try this remote/offline Q&A/code musing/consulting/education in > several installments published here: I must say I have quite some work to do > for customers and I can't spend a lot of time for postings here/explaining > what and why is done in the sample code - I will try just give refs on > "concepts" I use for you to read more in many open sources... > > I made first adjustment to the yesterday's code - new code is here: > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > http://smsconsulting.spb.ru/samples/Stats.txt > > To run it (let's use VB.NET): > > - in VS2005 create VB.NET Console application, replace the code of the > module, which VS2005 creates by default (Module1.vb) with the code from > http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines > > Public Const PcName As String = "LOCALHOST" > Public Const DbName As String = "NamesParser" > Public Const UserName As String = "TESTER" > Public Const Password As String = "TEST.1959" > > to point to your db and its credentials, push CTRL+F5, and you should see > the sample app running in console window and producing something like this > report: > > Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:13 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:19 > Elapsed time: 00:00:05.5156250 > > Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:19 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:22 > Elapsed time: 00:00:03.5468750 > > Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:22 > Count(*) = 1000 > Counter = 1009 > End time: 28.04.2008 20:18:26 > Elapsed time: 00:00:03.3906250 > > Try. > > If you tell me the name of your database and the name of your table I will > change them here, as well as you can create username and password as above > to not do any changes later for the new versions of the code. > > N.B: Make sure you will run the code against sample not the live db. The > code assumes that the length of the target field FName is 50 chars - it uses > just one this column currently to put "parsed" value - just first 50 chars > of OWNERNAME field. > > Please have a look through the code, and ask your questions in the order you > wanted them to be answered... > > Please everybody who wanted to participate in this discussion - you're very > welcome! > > On debugging multi-threaded apps: that's a real PITA - yes you can > debug/trace with breakpoints etc. but you have to keep in mind that as many > threads as you have will cause debugger to stop on breakpoints etc. IOW > better debug/trace single threaded code and them run it in multi-threaded > mode, and use some indirect artefacts/techniques (as unit testing), which > will show you that your app is running OK: as you can find in the new > version of the code it can run both single and multi-threaded depending on > the value of this constant: > > Public Const MULTI_THREADED_MODE As Boolean = True > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 7:02 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Shamil, > > I would like to thank you for taking the time to do this. Now... I will > thank you in advance for taking the time to explain it to me. As > happened many years ago when you taught me WithEvents in Access, I don't > know enough to understand what the heck you are doing. > > In the absence of any instructions on how to use this thing, I cut and > pasted the entire (VB) shebang into a class in a Windows Form project. > It compiled flawlessly and the blank form opened. > > Now what? > > I hope that you can appreciate my lack of knowledge here. I have built > several projects, with (to my pitiful level of expertise) quite > extensive stuff going on but... there are a lot of concepts in your code > that I have not used. Where to start. > > I guess where to start is a good spot. What do I do to cause this to > run? Coming from Access I have always started with a form, which I > promptly place a button on so I can call some function which "starts" > things. That form will have class variables if necessary to instantiate > a supervisor class, the supervisor class will then load recordsets, load > data into data classes as required etc. So what I do not understand is > "what do I call to make this run"? I see the MainEntry at the bottom > but do I just call MainEntry from a button? Does the code belong in a > class or a plain module? I assume that if I put it in a class then I > would have to dim a variable in my form. > > Second (I am totally excited about this whole thing!!!) what are the > replaceable parameters? > > Dim connectionString As String = String.Format("Data > Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password) > > I assume here that pcName gets fed in to {0}? THAT is cool, and > demonstrates where I start from in understanding your code. > > Third (also exciting!) I have never used threads yet so here we go. How > do I debug that? Can I step through a thread in the same way I step > through any other code? Set break points etc. > > And finally (for now) it just occurred to me to ask... can you help me > (online of offline) to modify this example to actually use my own table > that I need to parse the name for. I think I can figure out how to > modify the stuff for the server, database, table etc. I have done that > already in my own code for doing this. I will also worry about the > parsing code itself, though just for a demo we can just hard code some > values. > > > My table Data: > > PKID (int32) > OWNERNAME (to be parsed) > FName (parsed) > MName (parsed) > LName (parsed) > NamePrefix (parsed) > NameSuffix (parsed) > Gender (parsed) > > If I can modify they example to actually read / write to my table then I > kill two birds with one stone, learn a bunch of cool new stuff including > getting it all happening with threads, and get my real work done. > > For my real life purposes I am going to have to have a PKStart, PKEnd > pair which tracks a block (remember I am doing 80 million rows > eventually), and increment those start/end by the chunk size to repeat > over and over (eventually). What I have found convenient in the past > (and have code already written to do) is write status log files out to > disk for each block processed. I write them as XML and include things > like the start / end PKID, status memo fields, time start / stop and the > like. As I said, I have already written that code (I'm not completely > helpless, though it sometimes appears that way) and sitting out in a > library ready to use for the logging. All I would need to do is build > the specific data class to store the log data for this project. I will > handle that. > > I will promise to absorb your example if you will promise to take the > time to help me when I get stuck. Once I have done so I will then be > able to apply it to my specific problem and give you back some real life > timings from my largish database. I can vary the chunk size from 1k to > any upward limit we find useful to test. My experience has been that > anything above 10K can get problematic, though that was using the bulk > import widget in past projects. I was getting timeouts if I went with > too large a chunk size. > > VB.Net is only a small part of my work life, though I would like to make > it the main part. I still spend most of my work life stuck in Access, > maintaining applications I wrote (or inherited) long ago. While it pays > the bills I would LOVE to get to the point where I could bang out code > like this in the short time that it probably took you. > > Again thanks for your time and effort and patience. > > Shamil Salakhetdinov wrote: >> Here are the sources I promised to publish in my previous post: >> >> http://smsconsulting.spb.ru/samples/Stats.txt >> >> http://smsconsulting.spb.ru/samples/Program.cs.txt >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> Your turn, guys, to fix and improve them... >> >> -- >> Shamil >> > From jwcolby at colbyconsulting.com Mon Apr 28 13:10:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:10:08 -0400 Subject: [dba-VB] ADO.Net data table update Message-ID: <48161300.8050807@colbyconsulting.com> I have a datatable that I opened. It contains a set of 10K records which I have updated pieces of. I now need to write it back to SQL Server. If I just call the da.update it errors saying it needs an update command. Do I have to build an update SQL statement or will the DA just figure out where the data fields came from and update them back out to SQL Server? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 13:49:38 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:49:38 -0400 Subject: [dba-VB] ADO.Net Update SQL Server Message-ID: <48161C42.1030500@colbyconsulting.com> Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Mon Apr 28 14:04:08 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Mon, 28 Apr 2008 12:04:08 -0700 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> References: <48161C42.1030500@colbyconsulting.com> Message-ID: UpdateBatch should work. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 11:50 AM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Mon Apr 28 14:09:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 23:09:11 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> Message-ID: <010001c8a963$57cd7c00$6401a8c0@nant> Hi John, The answer is in the code I published today on my site related to 'Re: [dba-VB] ADO.Net': http://smsconsulting.spb.ru/samples/Module.vb.txt - Strategy 3 - trace in single threaded mode and you'll find what to do. Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As DataSet) As DataTable ... And Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet As DataSet) .... HTH, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 10:50 PM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 14:57:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 15:57:15 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: References: <48161C42.1030500@colbyconsulting.com> Message-ID: <48162C1B.8060207@colbyconsulting.com> I hate to sound ignorant but what is the UpdateBatch a method of? I have a connection, command, dataset and datatable and AFAICT none of those things have an UpdateBatch method. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > UpdateBatch should work. > > Charlotte Foust > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 11:50 AM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 15:14:01 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 16:14:01 -0400 Subject: [dba-VB] ADO.Net Update SQL Server References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> Message-ID: <01d001c8a96c$676b5a70$0502a8c0@Laptop> It is a method of the ADO.Recordset. I was ignorant of this, also, until I looked it up in a book. Ignorant<>something you can sound like Either you know or you don't know :) Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 3:57 PM Subject: Re: [dba-VB] ADO.Net Update SQL Server >I hate to sound ignorant but what is the UpdateBatch a method of? > > I have a connection, command, dataset and datatable and AFAICT none of > those things have an UpdateBatch method. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> UpdateBatch should work. >> >> Charlotte Foust >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 11:50 AM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> >> -- >> John W. Colby >> www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 15:17:48 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:17:48 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <010001c8a963$57cd7c00$6401a8c0@nant> References: <010001c8a963$57cd7c00$6401a8c0@nant> Message-ID: <481630EC.5060600@colbyconsulting.com> When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > From jwcolby at colbyconsulting.com Mon Apr 28 15:29:52 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:29:52 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <01d001c8a96c$676b5a70$0502a8c0@Laptop> References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> <01d001c8a96c$676b5a70$0502a8c0@Laptop> Message-ID: <481633C0.9060808@colbyconsulting.com> ROTFL, Oh I think I succeed very well at sounding ignorant. 8-( And yup, I don't know. I have beaten my head against this all day with absolutely zero results. I find it difficult to believe that I am doing anything unique but I'll be damned if I can find an article or code that just says "here is how you write back the updated table to the sql server... (data store)". John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > It is a method of the ADO.Recordset. > I was ignorant of this, also, until I looked it up in a book. > Ignorant<>something you can sound like > Either you know or you don't know :) > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Monday, April 28, 2008 3:57 PM > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > >> I hate to sound ignorant but what is the UpdateBatch a method of? >> >> I have a connection, command, dataset and datatable and AFAICT none of >> those things have an UpdateBatch method. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Charlotte Foust wrote: >>> UpdateBatch should work. >>> >>> Charlotte Foust >>> >>> -----Original Message----- >>> From: dba-vb-bounces at databaseadvisors.com >>> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >>> Sent: Monday, April 28, 2008 11:50 AM >>> To: VBA >>> Subject: [dba-VB] ADO.Net Update SQL Server >>> >>> Does ANYBODY know how to write a table back out to SQL Server? >>> >>> All of my books show cute little examples of building up a data table >>> manually then BINDING it to a control or whatever. I am not finding >>> anything with code example that shows how to WRITE changed data back to >>> SQL Server. >>> >>> I have managed to load a data table, write changes to the rows and >>> now... I have to write it back out to SQL Server. The table has the PK, >>> NameToSplit, LName, FName etc etc. What am I missing? >>> >>> I desperately need help (yea, yea, not MENTAL help) and I need it >>> quickly. I have been trying to do this last piece for DAYS and I am not >>> succeeding. >>> >>> -- >>> John W. Colby >>> www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From James at fcidms.com Mon Apr 28 16:02:31 2008 From: James at fcidms.com (James Barash) Date: Mon, 28 Apr 2008 17:02:31 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> John: If you already have a data adapter that you used to fill your data table, you need to add an UpdateCommand to that data adapter and then use it to update the datastore. Below is a very quick and dirty version from an existing program I've used. It should at least give you a place to start. It assumes you already have a data adapter named da, a data table named dt and a connection named conn. You will have to change the table and field names to match yours. da.UpdateCommand = New SqlClient.SqlCommand da.UpdateCommand.Connection = conn da.UpdateCommand.CommandType = CommandType.Text da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldDisplayValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldNumber", System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.Update(dt) James Barash -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 4:18 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 17:11:07 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:11:07 -0400 Subject: [dba-VB] vb.net - Updates Message-ID: <48164B7B.2070808@colbyconsulting.com> I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 17:12:37 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:12:37 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> References: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> Message-ID: <48164BD5.60906@colbyconsulting.com> Thanks James. I also found a piece of code that shows how to use a commandbuilder object to generate those update commands automatically. At my skill level, automatic is a good thing. John W. Colby www.ColbyConsulting.com James Barash wrote: > John: > If you already have a data adapter that you used to fill your data table, > you need to add an UpdateCommand to that data adapter and then use it to > update the datastore. Below is a very quick and dirty version from an > existing program I've used. It should at least give you a place to start. It > assumes you already have a data adapter named da, a data table named dt and > a connection named conn. You will have to change the table and field names > to match yours. > > > da.UpdateCommand = New SqlClient.SqlCommand > da.UpdateCommand.Connection = conn > da.UpdateCommand.CommandType = CommandType.Text > da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, > FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", > "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldDisplayValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, > Nothing, "", "", "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldNumber", > System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, > "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", > "", "")) > da.Update(dt) > > James Barash > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 4:18 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > When I try to call the update method of the data set object I get an error: > > "Update requires a valid update command when passed datarow collection > with modified rows." > > I probably sound ungrateful but in your code you appear to set up a sql > statement for an update statement: > > adapter.UpdateCommand = New > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, > Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) > adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None > adapter.UpdateBatchSize = pageSize > > > My problem is that is just Chinese to me. I have no background > knowledge in plain English of what an update sql statement is supposed > to look like (in the ADO context) and absolutely no understanding of > even what I am trying to accomplish (expressed in plain English). I am > uneducated, and I can find no way to get educated. > > I must own 6 books on ADO.Net and VB and every single one of them > expects me to drag and drop objects out onto a form. OR they very > helpfully set everything up and show me how to do stuff RIGHT UP TO > USING IT, but never go that last step of sending updates back to the > database. Open, update, display in a grid, cleanup and close. Hmm.... > and I paid good money for these books. > > They do not discuss anything at all about what is going on at the code > level. So when I try to walk through your code it is simply > indecipherable to me. If I don't know what you are trying to accomplish > I can't understand how you code will accomplish that thing. > > Please do not take these statements as criticisms of your code, simply > that I am a freshman student still taking the basket weaving classes > looking at masters code. Thus it isn't useful to tell me to trace > through the code. > > MY code doesn't do any of that stuff. My code is pretty simple: > > Set up a connection, a data adapter, a table. Iterate the table row by > row, updating fields of the rows. > > Now... how to update the table back to SQL Server... > > Given how simple everything has been up to this point I have to believe > that I am just missing a concept and code to implement that concept. I > have called the update method and I have googled the error messages and > nothing... > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> The answer is in the code I published today on my site related to 'Re: >> [dba-VB] ADO.Net': >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> - Strategy 3 - trace in single threaded mode and you'll find what to do. >> >> Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, > ByVal >> pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As >> DataSet) As DataTable >> ... >> >> And >> >> Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet >> As DataSet) >> .... >> >> >> HTH, >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 10:50 PM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Mon Apr 28 18:31:14 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 19:31:14 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> References: <48164B7B.2070808@colbyconsulting.com> Message-ID: <48165E42.20404@colbyconsulting.com> We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. From ebarro at verizon.net Mon Apr 28 19:44:24 2008 From: ebarro at verizon.net (Eric Barro) Date: Mon, 28 Apr 2008 17:44:24 -0700 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> Message-ID: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Yeah it is a terrible thing from a DBA point of view because it opens it up to SQL injection attacks since it builds your SQL statement on the fly. I've tried it before and since it is generating it on the fly it makes it all the more harder to debug and offers the least amount of control from a DBA point of view. So...I let SQL server do all the work on the back end when it comes to batch updates by collecting the parameters and sending it to SQL server to process for the records it needs to process. Based on what you are trying to do you are building an image of the data structure in memory, manipulating the data in memory and then committing the changes to the database in memory basically sending the update command for each record to the database back end right? Or am I mistaken in thinking that all of this magically happens by sending a single update command containing several individual update commands strung together? -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 3:11 PM To: VBA Subject: [dba-VB] vb.net - Updates I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 20:08:20 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 21:08:20 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Message-ID: <48167504.6070909@colbyconsulting.com> It all happens automagically. At this point I modify about 10K records at a clip, then issue a single update and it goes. I set the batch size to 100 for this experiment but I will probably play with that to see what can be done. I tried a thousand but it timed out. I don't see any timeout period property though there might well be one somewhere. I understand the SQL Injection attack thing but you need to remember that this is a database running on my system that only I ever access. Even more importantly though, I need this kind of thing to just work, as quickly as I can make it happen. I am not a high powered DBA (nor any other powered DBA) nor a high powered .Net programmer. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Yeah it is a terrible thing from a DBA point of view because it opens it up > to SQL injection attacks since it builds your SQL statement on the fly. > > I've tried it before and since it is generating it on the fly it makes it > all the more harder to debug and offers the least amount of control from a > DBA point of view. So...I let SQL server do all the work on the back end > when it comes to batch updates by collecting the parameters and sending it > to SQL server to process for the records it needs to process. > > Based on what you are trying to do you are building an image of the data > structure in memory, manipulating the data in memory and then committing the > changes to the database in memory basically sending the update command for > each record to the database back end right? Or am I mistaken in thinking > that all of this magically happens by sending a single update command > containing several individual update commands strung together? > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 3:11 PM > To: VBA > Subject: [dba-VB] vb.net - Updates > > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece of > code that, in simple plain language, shows how to use it all, using the > simple objects I am already using. > > We shall see. > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 21:26:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 22:26:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> <48167504.6070909@colbyconsulting.com> Message-ID: <034b01c8a9a0$7b784440$0502a8c0@Laptop> These fit the bill ... How To: Protect From SQL Injection in ASP.NET http://msdn2.microsoft.com/en-us/library/ms998271.aspx Using Stored Procedures with SQLDataAdapter? http://forums.devx.com/showthread.php?t=148707 SQLDataAdapter without using SQLCommandBuilder http://www.codeproject.com/KB/database/sqldawithoutsqlcb.aspx Using Stored Procedures in Conjunction with the SqlDataAdapter http://www.code-magazine.com/Article.aspx?quickid=0303141 Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 9:08 PM Subject: Re: [dba-VB] vb.net - Updates > I understand the SQL Injection attack thing but you need to remember > that this is a database running on my system that only I ever access. From jwcolby at colbyconsulting.com Tue Apr 29 00:15:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 01:15:45 -0400 Subject: [dba-VB] Name parsing system timing Message-ID: <4816AF01.8090500@colbyconsulting.com> OK, it is running in a loop now, processing 10K records every 19 seconds. It will take 50 hours at the current rate to parse 96.4 million records, or about 2 million records / hour. the SQL Server update portion is taking about 5 seconds of each 19 second cycle, 14 seconds to parse, 5 seconds to write back to the database, virtually nothing to load. At least it is running now, making progress. Thanks for all the help from everyone getting this thing to happen. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 08:53:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 09:53:26 -0400 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: <48172856.9080502@colbyconsulting.com> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Tue Apr 29 09:39:45 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 16:39:45 +0200 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: Hi John That is what the DataTableAdapter is for. If you use the wizard for creating these to attach your tables, it will do all the plumbing and create a (giant) class which wraps all this for you taking care of relations, not null, max lengths, strong types, error handling etc. etc. not to say provide the basis for creating all the queries you need including ready-made basic queries for CRUD as well as giving you the choice to use stored procedures, pass-through or normal queries. All of this is controlled from a graphic layout like a relations view. If that isn't enough it also lets you carry on with Data Access Layers and business rules. Nothing is hidden or secret. You can browse the class should you have the patience and wish so. This is a snippet: private void InitClass() { this.columnKundeNr = new global::System.Data.DataColumn("KundeNr", typeof(int), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnKundeNr); this.columnNavn = new global::System.Data.DataColumn("Navn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnNavn); this.columnAdr1 = new global::System.Data.DataColumn("Adr1", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr1); this.columnAdr2 = new global::System.Data.DataColumn("Adr2", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr2); this.columnPostNr = new global::System.Data.DataColumn("PostNr", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnPostNr); this.columnBynavn = new global::System.Data.DataColumn("Bynavn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnBynavn); this.columnTlf = new global::System.Data.DataColumn("Tlf", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnTlf); this.Constraints.Add(new global::System.Data.UniqueConstraint("Constraint1", new global::System.Data.DataColumn[] { this.columnKundeNr}, true)); this.columnKundeNr.AllowDBNull = false; this.columnKundeNr.Unique = true; this.columnNavn.AllowDBNull = false; this.columnNavn.MaxLength = 50; this.columnAdr1.MaxLength = 50; this.columnAdr2.MaxLength = 50; this.columnPostNr.MaxLength = 50; this.columnBynavn.MaxLength = 50; this.columnTlf.MaxLength = 50; } If you redesign the table structure from within Visual Studio, it will track the changes and adjust the class accordingly. It is __so clever__ that I wonder why most are still moving on with SQL builders, hand built command strings and the like - pure waste of time in my opinion - except for programmers with special needs and knowledge like Shamil who knows what he is doing. William, I think, will amen this. I must admit that because of VS2008 I'm ready to forgive MS much of their bad doings. /gustav >>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 09:46:24 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 09:46:24 -0500 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: References: Message-ID: <200804291451.m3TEpqu6010208@databaseadvisors.com> Yes, that is correct. He is ending up loading the records into memory. Then passing individual updates back for each row of data that he changed. This method is not designed for doing large datasets like he is doing. It should be done on the SQL Server side. John, Just what are you updating...the UPDATE statement you would use to do what you are doing. Robert At 09:40 AM 4/29/2008, you wrote: >Date: Mon, 28 Apr 2008 17:44:24 -0700 >From: "Eric Barro" >Subject: Re: [dba-VB] vb.net - Updates >To: "'Discussion concerning Visual Basic and related programming > issues.'" >Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >Content-Type: text/plain; charset=us-ascii > >Yeah it is a terrible thing from a DBA point of view because it opens it up >to SQL injection attacks since it builds your SQL statement on the fly. > >I've tried it before and since it is generating it on the fly it makes it >all the more harder to debug and offers the least amount of control from a >DBA point of view. So...I let SQL server do all the work on the back end >when it comes to batch updates by collecting the parameters and sending it >to SQL server to process for the records it needs to process. > >Based on what you are trying to do you are building an image of the data >structure in memory, manipulating the data in memory and then committing the >changes to the database in memory basically sending the update command for >each record to the database back end right? Or am I mistaken in thinking >that all of this magically happens by sending a single update command >containing several individual update commands strung together? From wdhindman at dejpolsystems.com Tue Apr 29 10:26:31 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Tue, 29 Apr 2008 11:26:31 -0400 Subject: [dba-VB] Update gets string or binary truncation error References: Message-ID: ...sigh ...I look at some of the stuff Shamil posts and haven't a clue what he's doing, much less why ...waaaaaaaay over my head ...so much so I get a bad headache every time I try to read some of it ...but you definitely get an AMEN! ...and then some ...on VS2K8 ...when MS finally gets something right, its RIGHT!!! :) William -------------------------------------------------- From: "Gustav Brock" Sent: Tuesday, April 29, 2008 10:39 AM To: Subject: Re: [dba-VB] Update gets string or binary truncation error > Hi John > > That is what the DataTableAdapter is for. If you use the wizard for > creating these to attach your tables, it will do all the plumbing and > create a (giant) class which wraps all this for you taking care of > relations, not null, max lengths, strong types, error handling etc. etc. > not to say provide the basis for creating all the queries you need > including ready-made basic queries for CRUD as well as giving you the > choice to use stored procedures, pass-through or normal queries. > > All of this is controlled from a graphic layout like a relations view. If > that isn't enough it also lets you carry on with Data Access Layers and > business rules. > > Nothing is hidden or secret. You can browse the class should you have the > patience and wish so. This is a snippet: > > private void InitClass() { > this.columnKundeNr = new > global::System.Data.DataColumn("KundeNr", typeof(int), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnKundeNr); > this.columnNavn = new > global::System.Data.DataColumn("Navn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnNavn); > this.columnAdr1 = new > global::System.Data.DataColumn("Adr1", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr1); > this.columnAdr2 = new > global::System.Data.DataColumn("Adr2", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr2); > this.columnPostNr = new > global::System.Data.DataColumn("PostNr", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnPostNr); > this.columnBynavn = new > global::System.Data.DataColumn("Bynavn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnBynavn); > this.columnTlf = new global::System.Data.DataColumn("Tlf", > typeof(string), null, global::System.Data.MappingType.Element); > base.Columns.Add(this.columnTlf); > this.Constraints.Add(new > global::System.Data.UniqueConstraint("Constraint1", new > global::System.Data.DataColumn[] { > this.columnKundeNr}, true)); > this.columnKundeNr.AllowDBNull = false; > this.columnKundeNr.Unique = true; > this.columnNavn.AllowDBNull = false; > this.columnNavn.MaxLength = 50; > this.columnAdr1.MaxLength = 50; > this.columnAdr2.MaxLength = 50; > this.columnPostNr.MaxLength = 50; > this.columnBynavn.MaxLength = 50; > this.columnTlf.MaxLength = 50; > } > > If you redesign the table structure from within Visual Studio, it will > track the changes and adjust the class accordingly. > > It is __so clever__ that I wonder why most are still moving on with SQL > builders, hand built command strings and the like - pure waste of time in > my opinion - except for programmers with special needs and knowledge like > Shamil who knows what he is doing. > > William, I think, will amen this. > > I must admit that because of VS2008 I'm ready to forgive MS much of their > bad doings. > > /gustav > >>>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> > I am doing the update thing, updating data back to SQL Server. I have > been updating an ADO recordset in memory and when I try to update it > back to SQL Server it throws an error that the update failed because it > would cause a string value to be truncated. > > How is this possible. I thought that the table structure in memory > mirrored the structure in the data store (SQL Server in this instance). > Doesn't the read of the data out of SQL Service discover the field > size and type and build a similar structure in memory? > > Apparently not, but if not how do I handle a case like this? > -- > John W. Colby > www.ColbyConsulting.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 10:43:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 11:43:45 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <200804291451.m3TEpqu6010208@databaseadvisors.com> References: <200804291451.m3TEpqu6010208@databaseadvisors.com> Message-ID: <48174231.4070501@colbyconsulting.com> > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 12:13:02 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 12:13:02 -0500 (CDT) Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 Message-ID: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Trying to understand the scenario... 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Questions: 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? From: jwcolby Date: 2008/04/29 Tue AM 10:43:45 CDT > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? From jwcolby at colbyconsulting.com Tue Apr 29 12:48:28 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 13:48:28 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> References: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Message-ID: <48175F6C.5050804@colbyconsulting.com> > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? Correct. I have a 96 million record table, 155 fields. Of that I pull the PKID, NameToParse (not called that obviously) and then 6 "parsed name pieces" fields, lname, fname etc (which I added to the table to hold the pieces). > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Not quite correct. The vb.Net results are due to the fact that I am using a 3rd party library which does the parsing of the name. Name parsing is non-trivial to say the least. That library is a .Net object (DLL actually) which you set up parameters in and then pass in a name to parse and back comes 5 to 8 "pieces". In fact fluid results are the least of my worries in SQL Server, ANY results are more like it. > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? In fact I do but that is not necessary. The name parser came with a demo form and I just modified that to display every 1000th record (which I pass to the form every 1000th record in the table), just so I could "watch it". The form is not a necessity and I may or may not get rid of it in a later iteration (probably will). I have a form (I created) which has a ServerName, DatabaseName, TableName (being parsed), PKFldName, ParseFldName, StartID, EndID, and button to trigger the parse. I fill in the pieces and push the button. Since I have control of the tables, I "hard code" the "parsed parts" field names and they will (at least for now) always be those names. I have to add those fields to the source table anyway. > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? Correct. I grab X records into an ADO.Net DataTable, iterate that table row by row, passing the Unparsed name field off to the parser. I then write the resulting "Pieces" that the parser returns (properties of the parser object) into the matching fields in the ADO.Net DataTable. The name parser handles roughly 800 names / second and is the real bottle neck. I have a StartPK / EndPK which I then manipulate in a while loop, as I finish a "chunk" I write it to SQL Server, update the StartPK / EndPK and grab another chunk to process. I have to do this for 96 million names, however I also receive weekly updates to this table which I will also have to process in a similar manner before I append them to the table. The results of this will be used to send through Accuzip for Address validation / NCOA processing. Address validation can be done just on an address, but NCOA is change of address which is about a person at an address. Once that is done... there is more but you get the picture. And in fact there is another different table from the same data source that I will have to perform this same process on. It is highly likely that I will get other lists in the future with single name fields. The world is full of idiots. ;-) John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Trying to understand the scenario... > > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? > > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. > > Questions: > > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? > > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? > > > From: jwcolby > Date: 2008/04/29 Tue AM 10:43:45 CDT > > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > > Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > > ROTFL. That is a definite NSS. > > Unfortunately I am not a DBA and with the responses I get from the SQL > group I am not even going there. I can do this purely in VB.Net (it is > cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > > No idea. I thought I could find out from the DA after the command > builder was finished building the update statement but I am getting an > error if I try to access the update property. > > John W. Colby > www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 13:13:49 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 13:13:49 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804291817.m3TIH41W019987@databaseadvisors.com> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert At 12:00 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 11:43:45 -0400 >From: jwcolby >Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >To: "Discussion concerning Visual Basic and related programming > issues." >Message-ID: <48174231.4070501 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > >Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > >ROTFL. That is a definite NSS. > >Unfortunately I am not a DBA and with the responses I get from the SQL >group I am not even going there. I can do this purely in VB.Net (it is >cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > >No idea. I thought I could find out from the DA after the command >builder was finished building the update statement but I am getting an >error if I try to access the update property. > >John W. Colby From Gustav at cactus.dk Tue Apr 29 13:24:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 20:24:51 +0200 Subject: [dba-VB] SQL VS VB Message-ID: Hi Robert If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? /gustav >>> robert at webedb.com 29-04-2008 20:13 >>> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert From jwcolby at colbyconsulting.com Tue Apr 29 13:33:22 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:33:22 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804291817.m3TIH41W019987@databaseadvisors.com> References: <200804291817.m3TIH41W019987@databaseadvisors.com> Message-ID: <481769F2.2090303@colbyconsulting.com> It is going to take 50 hours because an external program which does the name parsing will take at least 40 hours just to parse the data. Writing to the database is only about 25% of the total time. If you managed to completely eliminate my database update time it would still take about 40 hours to complete. Thanks for the offer though. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > At 12:00 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 11:43:45 -0400 >> From: jwcolby >> Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >> To: "Discussion concerning Visual Basic and related programming >> issues." >> Message-ID: <48174231.4070501 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> > Yes, that is correct. He is ending up loading the records into memory. >> > Then passing individual updates back for each row of data that he >> > changed. >> >> Yes, and most records have updates. >> >> > This method is not designed for doing large datasets like he is doing. >> > >> > It should be done on the SQL Server side. >> >> ROTFL. That is a definite NSS. >> >> Unfortunately I am not a DBA and with the responses I get from the SQL >> group I am not even going there. I can do this purely in VB.Net (it is >> cranking as we speak) so I will. >> >> > Just what are you updating...the UPDATE statement you would use to >> > do what you are doing. >> >> No idea. I thought I could find out from the DA after the command >> builder was finished building the update statement but I am getting an >> error if I try to access the update property. >> >> John W. Colby > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 13:44:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:44:40 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <48176C98.8030201@colbyconsulting.com> > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? Precisely Gustav. What I WOULD like to do is spin off a thread to do the SQL Server update so the parser could go back to work on the next chunk. IOW put the update in parallel with the parser instead of in series with it. In fact if I had this running on the 4 core system I could probably call the parser DLL in several threads as well. In fact if I ran the VB.Net application on a separate machine, then the SQL Server machine could be just running updates and another machine running the application. What I REALLY need is a thousand blade mini-super that I could just run the entire thing in parallel. It would only take a few minutes then. ;-) John W. Colby www.ColbyConsulting.com Gustav Brock wrote: > Hi Robert > > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? > > /gustav > >>>> robert at webedb.com 29-04-2008 20:13 >>> > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <022401c8aa2b$786352f0$6401a8c0@nant> <<< Please do not take these statements as criticisms of your code >>> Hi John, No problem with criticism - in fact it's very welcome! :) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 12:18 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48165E42.20404@colbyconsulting.com> Message-ID: <022801c8aa2b$787b20b0$6401a8c0@nant> <<< ...it is definitely going to be desirable to feed the update process off to a thread... >>> Hi John, I was lucky with multi-threaded batch updates' samples I posted here the day before yesterday - and the new tests show that batch updates in multi-threading environments result in MS SQL deadlocks: I should have thought about those issues before... Recap: forget batch updates - one of the scenarios of your task could be: - many readers; - many parsers; - but just one updater making batch updates, or many updaters but updating just one row in one cycle... HTH to save your time, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 3:31 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 14:11:36 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 15:11:36 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022801c8aa2b$787b20b0$6401a8c0@nant> References: <022801c8aa2b$787b20b0$6401a8c0@nant> Message-ID: <481772E8.3050505@colbyconsulting.com> Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 14:43:53 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:43:53 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <30862132.1249691209498233814.JavaMail.root@vms124.mailsrvcs.net> I believe that Shamil was referring to multi-threaded updates. If you used the multi-threaded approach you could potentially end up slowing the process more than you could ever imagine because of the way SQL server locks pages of records when you run an update. From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:51:42 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:51:42 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <481772E8.3050505@colbyconsulting.com> Message-ID: <022c01c8aa32$7269c440$6401a8c0@nant> John, I'm not sure yet what happens< I will try to find out more details in the coming days - here is the message I'm currently getting with new Strategy 4 , which I'm making similar to your way of coding this task but running in multi-threading environment: +++ System.Data.SqlClient.SqlException: Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. --- Just wanted to warn you - it might happen you'll never see this message but if you'll see it then you may find there is no workaround except abandoning the idea of batch updates running in several threads - and I mean parallel batch updates: of course, as I noted in my previous e-mail parallel reading and parsing could be done but batch updates very probably can't be run in parallel threads because they might result in deadlocks... But it looks like (from your recent posting here) that you don't need such parallel batch updates at all? Thanks. Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 11:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 14:59:51 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:59:51 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 15:02:10 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 15:02:10 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <29420937.173771209499330766.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 15:13:29 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:13:29 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022c01c8aa32$7269c440$6401a8c0@nant> References: <022c01c8aa32$7269c440$6401a8c0@nant> Message-ID: <48178169.70002@colbyconsulting.com> Shamil, I have just moved the whole project to my quad core x64 server and am bringing it up now. If I can get that code running over on another machine (especially IF it is a faster machine) then the update part will be less of an issue. In any event the parse WAS taking about 15 seconds for 10K records on the dual core x32 machine which was also running SQL Server. I have designed this thing as a set of classes. There is a "data class" which is opened by a class that parses the names. The NameParser just writes the data directly into the ADO recordset in the data class. I MIGHT be able to just shuffle the class holding the open ado recordsets off into a collection. Another thread could pull those class instances out of the collection and call the update method which would start the update. As it finished the update, it would destroy that data class instance and pull the next one out of the collection. This would effectively separate the parsing and the updating, and each could proceed at it's own speed. I have not investigated yet but I might be able to have multiple threads doing the name parsing, and storing the resulting data class in that collection. It depends on whether the dll is reentrant I suppose and I really don't have a clue yet.. There would only be a single thread processing the dataset classes "update". I would need to watch that I didn't get too many open data classes at once time I suppose, though this machine has Windows x64 and 8 gigs of RAM so it has a little more breathing room than the Windows x32 machine I was running it on. Of course all of this is just "play". At least it is running now. I was also thinking about running two threads in "opposite directions", one processing data PK high to low, and the other processing data PK Low to High. Until they met in the middle that MIGHT avoid the deadlock thing. Again though, that is just play. NOT TO MENTION that I have never run a thread anyway so i have to figure THAT out. ;-) John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > I'm not sure yet what happens< I will try to find out more details in the > coming days - here is the message I'm currently getting with new Strategy 4 > , which I'm making similar to your way of coding this task but running in > multi-threading environment: > > +++ > System.Data.SqlClient.SqlException: Transaction (Process ID > 57) was deadlocked on lock resources with another process and has been > chosen as the deadlock victim. Rerun the transaction. > --- > > Just wanted to warn you - it might happen you'll never see this message but > if you'll see it then you may find there is no workaround except abandoning > the idea of batch updates running in several threads - and I mean parallel > batch updates: of course, as I noted in my previous e-mail parallel reading > and parsing could be done but batch updates very probably can't be run in > parallel threads because they might result in deadlocks... > > But it looks like (from your recent posting here) that you don't need such > parallel batch updates at all? > > Thanks. > > Shamil From jwcolby at colbyconsulting.com Tue Apr 29 15:18:50 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:18:50 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> References: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> Message-ID: <481782AA.6030204@colbyconsulting.com> Eric, I will read it. Unfortunately that puts me back in the SQL forum and you know how I ... uhh... well, let's just say I would rather muddle along in VB.Net. That is something that I can get better at and will spend enough time at to pay back my efforts. Besides... I come from the "just throw more hardware at it" school of Engineering. ;-) Seriously though, of course I'd LOVE to just pass the ADO recordset over and let SQL Server do it. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > John, > > This article might be of use in your scenario... > > http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 > > It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... > > "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " > > --Eric > > From: jwcolby > Date: 2008/04/29 Tue PM 02:11:36 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > Interesting. Do you think that you would get deadlocks for different > blocks of records? In my case there is not "a million users" scenario. > This process (and its threads) are the only things modifying data. > Each "chunk" is a defined block of records. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> <<< >> ...it is definitely going to be desirable to feed the update >> process off to a thread... >> Hi John, >> >> I was lucky with multi-threaded batch updates' samples I posted here the day >> before yesterday - and the new tests show that batch updates in >> multi-threading environments result in MS SQL deadlocks: I should have >> thought about those issues before... >> >> Recap: forget batch updates - one of the scenarios of your task could be: >> >> - many readers; >> - many parsers; >> - but just one updater making batch updates, or many updaters but updating >> just one row in one cycle... >> >> HTH to save your time, >> Shamil >> >> >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Tuesday, April 29, 2008 3:31 AM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> We have ignition folks. I just updated the first 10K records in the >> database. I had to set the batch size to 100, as I was getting a >> timeout with a batch size as small as 1000 records. I haven't done any >> timings but it is definitely going to be desirable to feed the update >> process off to a thread if I can figure out how to do that. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> jwcolby wrote: >>> I THINK I have found my solution, called the SQLCommandBuilder. I found >>> reference to the command build saying how terrible it is, then a piece >>> of code that, in simple plain language, shows how to use it all, using >>> the simple objects I am already using. >>> >>> We shall see. >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From robert at webedb.com Tue Apr 29 16:21:31 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 16:21:31 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804292127.m3TLRIWT012647@databaseadvisors.com> His original estimate was 50 hours. (not mine) But, he would have to do a bit of fancy coding in SQL Server to use the same DLL he is using. That bit of information was not in the first post. At 02:02 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 20:24:51 +0200 >From: "Gustav Brock" >Subject: Re: [dba-VB] SQL VS VB >To: >Message-ID: >Content-Type: text/plain; charset=US-ASCII > >Hi Robert > >If the parser dll runs at about 800 records/s, 93 mio. records will >need 33 hours as a minimum. Or? > >/gustav From jwcolby at colbyconsulting.com Tue Apr 29 16:28:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:28:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48178169.70002@colbyconsulting.com> References: <022c01c8aa32$7269c440$6401a8c0@nant> <48178169.70002@colbyconsulting.com> Message-ID: <4817930D.3080400@colbyconsulting.com> Well I shoulda done that sooner. With the process running on Azul (the quad core x64) machine and Stonehenge (dual core x32) just processing the SQL Server thing, the parse time dropped dramatically! I am now processing about 2800 - 2900 records / second. Interestingly the update time is about the same, ~5 seconds, but that is for ~80K records. I am wondering if I am timing what i think I am timing on the update thing. 8~/ PERHAPS there is some kind of "connection" time that is the majority of the time At any rate I will finish MUCH sooner, and that with no thread stuff yet. John W. Colby www.ColbyConsulting.com jwcolby wrote: > Shamil, > > I have just moved the whole project to my quad core x64 server and am > bringing it up now. If I can get that code running over on another > machine (especially IF it is a faster machine) then the update part will > be less of an issue. In any event the parse WAS taking about 15 seconds > for 10K records on the dual core x32 machine which was also running SQL > Server. > > I have designed this thing as a set of classes. There is a "data class" > which is opened by a class that parses the names. The NameParser just > writes the data directly into the ADO recordset in the data class. I > MIGHT be able to just shuffle the class holding the open ado recordsets > off into a collection. Another thread could pull those class instances > out of the collection and call the update method which would start the > update. As it finished the update, it would destroy that data class > instance and pull the next one out of the collection. > > This would effectively separate the parsing and the updating, and each > could proceed at it's own speed. > > I have not investigated yet but I might be able to have multiple threads > doing the name parsing, and storing the resulting data class in that > collection. It depends on whether the dll is reentrant I suppose and I > really don't have a clue yet.. > > There would only be a single thread processing the dataset classes "update". > > I would need to watch that I didn't get too many open data classes at > once time I suppose, though this machine has Windows x64 and 8 gigs of > RAM so it has a little more breathing room than the Windows x32 machine > I was running it on. > > Of course all of this is just "play". At least it is running now. > > I was also thinking about running two threads in "opposite directions", > one processing data PK high to low, and the other processing data PK Low > to High. Until they met in the middle that MIGHT avoid the deadlock > thing. Again though, that is just play. > > NOT TO MENTION that I have never run a thread anyway so i have to figure > THAT out. ;-) > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> John, >> >> I'm not sure yet what happens< I will try to find out more details in the >> coming days - here is the message I'm currently getting with new Strategy 4 >> , which I'm making similar to your way of coding this task but running in >> multi-threading environment: >> >> +++ >> System.Data.SqlClient.SqlException: Transaction (Process ID >> 57) was deadlocked on lock resources with another process and has been >> chosen as the deadlock victim. Rerun the transaction. >> --- >> >> Just wanted to warn you - it might happen you'll never see this message but >> if you'll see it then you may find there is no workaround except abandoning >> the idea of batch updates running in several threads - and I mean parallel >> batch updates: of course, as I noted in my previous e-mail parallel reading >> and parsing could be done but batch updates very probably can't be run in >> parallel threads because they might result in deadlocks... >> >> But it looks like (from your recent posting here) that you don't need such >> parallel batch updates at all? >> >> Thanks. >> >> Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 16:45:10 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:45:10 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804292127.m3TLRIWT012647@databaseadvisors.com> References: <200804292127.m3TLRIWT012647@databaseadvisors.com> Message-ID: <481796E6.7060300@colbyconsulting.com> That was indeed my original estimate. The PARSER was running at ~800 names / second. The UPDATE occurs after the parser finishes its thing., and is thus "in series with" (time wise) the parser. I just moved the VB.Net off to a more powerful machine and it is now running the parser at just over 3K records / second. I am in fact not parsing every record, I am actually processing ~80% of the records. Now I am off to see if I can figure out how to use threads. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > His original estimate was 50 hours. (not mine) > > But, he would have to do a bit of fancy coding in > SQL Server to use the same DLL he is using. That > bit of information was not in the first post. > > > > At 02:02 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 20:24:51 +0200 >> From: "Gustav Brock" >> Subject: Re: [dba-VB] SQL VS VB >> To: >> Message-ID: >> Content-Type: text/plain; charset=US-ASCII >> >> Hi Robert >> >> If the parser dll runs at about 800 records/s, 93 mio. records will >> need 33 hours as a minimum. Or? >> >> /gustav > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 19:03:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 20:03:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> Message-ID: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> I've fallen behind quite a bit, I see. Well, I'll catch up. Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 20:13:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 21:13:25 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> <030f01c8aa55$aa6384d0$0502a8c0@Laptop> Message-ID: <4817C7B5.6020005@colbyconsulting.com> > I've fallen behind quite a bit, I see. eh? John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > I've fallen behind quite a bit, I see. > Well, I'll catch up. > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 20:23:54 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 21:23:54 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> Message-ID: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Hi John, I was looking over Gustav's posts of Feb 20 with Charlotte and posts of this and nearby threads. I've been doing stuff with maps that didn't require use of large databases. The difference between an sqladapter and a tableadapter were unknown to me until today. C#, SQL Server 200X, ASP.Net All very compelling! Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Tuesday, April 29, 2008 9:13 PM Subject: Re: [dba-VB] vb.net - Updates > > I've fallen behind quite a bit, I see. > > eh? > > John W. Colby > www.ColbyConsulting.com > > > Michael R Mattys wrote: >> I've fallen behind quite a bit, I see. >> Well, I'll catch up. >> >> Michael R. Mattys >> MapPoint & Access Dev >> www.mattysconsulting.com >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 21:14:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 22:14:27 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Message-ID: <4817D603.9050308@colbyconsulting.com> Michael, > C#, SQL Server 200X, ASP.Net > All very compelling! Indeed they are. So much to know, so little time. I am slowly learning some of this, painfully slowly. John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > Hi John, > > I was looking over Gustav's posts of Feb 20 > with Charlotte and posts of this and nearby threads. > > I've been doing stuff with maps that didn't require use > of large databases. The difference between an sqladapter > and a tableadapter were unknown to me until today. > > C#, SQL Server 200X, ASP.Net > All very compelling! > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Tuesday, April 29, 2008 9:13 PM > Subject: Re: [dba-VB] vb.net - Updates > > >>> I've fallen behind quite a bit, I see. >> eh? >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Michael R Mattys wrote: >>> I've fallen behind quite a bit, I see. >>> Well, I'll catch up. >>> >>> Michael R. Mattys >>> MapPoint & Access Dev >>> www.mattysconsulting.com >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 23:54:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 00:54:18 -0400 Subject: [dba-VB] Worker threads and class global variables Message-ID: <4817FB7A.4000002@colbyconsulting.com> Can a backgroundworker access a class global variable in the class that dimensions the background worker. I have a process that will be creating data classes. these classes will fill ADO data sets, modify them and then place them in a collection global to the class. This same class would then dim a backgroundworker object. The way I envision this working is that the backgroundworker is initialized and started. It simply sits in a loop checking the class global collection for data class instances deposited into the collection by the main thread. If a class instance is found in the thread it pulls it out of the collection and processes it. This backgroundworker thread's job is to run the SQL Server update code in the data class (call a method of that class). Somehow that backgroundworker has to be told that the main thread is done parsing names, ran through the entire table and is done. So I thought the main thread could set a class global boolean done flag. Or something. Can I do this at all the way I am thinking? My problem is that there is no way to predict which process will be faster, name parse or SQL Server update. My idea was simply to use a collection to act as an interface between the two. The name parser instantiates a data class which loads the data to be parsed into an ADO.Net datatable. The name parser updates the data in the the data class. When it is done updating the data it drops the class into the collection. the backgroundworker watches the collection and when it finds a data class in the collection, it pulls it out and calls the update method of that class to process the data. when it is done it cleans up the ADO objects and stores the class in another "history" collection up in the parse class. The data class will have start / stop times, start / end PKIDs, record count etc information which will be of interest to me. I have never used Background worker objects so I don't know whether it can interface in the manner described, using collections to pass objects. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Wed Apr 30 08:09:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 09:09:45 -0400 Subject: [dba-VB] Background worker threads Message-ID: <48186F99.6090401@colbyconsulting.com> I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 10:19:03 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 08:19:03 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <48186F99.6090401@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> Message-ID: You can create your own events with whatever parameters you want and raise those. Just Dim Public Event Whatever(parameter) Then raise it as needed. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 6:10 AM To: VBA Subject: [dba-VB] Background worker threads I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 11:05:17 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 12:05:17 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> Message-ID: <481898BD.2050301@colbyconsulting.com> Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust From shamil at smsconsulting.spb.ru Wed Apr 30 11:07:47 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:07:47 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4817D603.9050308@colbyconsulting.com> Message-ID: <02cc01c8aadc$55aeb860$6401a8c0@nant> Hi John, May I ask you to run this query (e.g. in MS SQL Management Studio) against your huge db table? --- cut here --- SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID, OWNERNAME, FName, MName, LName, NamePrefix, NameSuffix, Gender from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] --- cut here --- M.B. [Names] have to be substituted with the name of your table - what is it BTW? This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... I'm curious to know how much time approx. this query will take to execute... Thank you. -- Shamil P.S. BTW, here is a useful for your case article on multi-threading (practical solution) with good introduction - http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik Sorry, I can't react promptly on your messages in this discussion thread: a lot of custom work here, long weekends and our time zones difference but I do continue to work in this direction and if I get some useful results I will post them here... P.P.S. This book could be also useful: C# Threading Handbook by Tobin Titus et al. ISBN:1861008295 APress, LLC C 2004 (288 pages) This book addresses the fundamental units of Windows and .NET programming-threads. Coverage includes how .NET applications are executed, the life cycle of a thread in .NET, how the .NET Framework uses threads, and more. From shamil at smsconsulting.spb.ru Wed Apr 30 11:25:15 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:25:15 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> Message-ID: <02d901c8aade$c66c3120$6401a8c0@nant> John, Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that the fact that "The basic code WAS working before I carved it out into a worker." doesn't mean that the same code (copied & pasted) into multi-threading environment will work OK - very often the opposite will be the truth - it could produce (very) faulty solution in multi-threaded environment, which will not be easy to sort out into properly running multi-threaded code... I'd dare to advise you to prepare and test your code in VS console application (or .net console app as test-bed app and .net classlib(s) keeping all your non-UI code), and when that will work OK then add some more code to run your classlibs with your .NET WinForms application: you have a lot to do, to learn and to test - multi-threading and .NET programming can't be "captured by attack" - it's a painful and lengthy natural process, which usually takes a lot of time - I know that from my own experience and experience of others - and better not "read all that books" - find just one but good and then experiment a lot on your own - then you'll get it working properly and you'll teach all of us here - I'm quite sure about that latter! Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 8:05 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:17:04 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:17:04 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: <02d901c8aade$c66c3120$6401a8c0@nant> References: <02d901c8aade$c66c3120$6401a8c0@nant> Message-ID: <4818A990.2020306@colbyconsulting.com> Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 12:28:02 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 10:28:02 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: I'm sorry, John, I missed the contextt of your question. We don't use background workers that way, so I don't have an answer for you. Charlotte -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:05 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:32:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:32:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AD3D.6030003@colbyconsulting.com> It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:36:02 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:36:02 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: <4818AE02.2080601@colbyconsulting.com> Bueno, thanks! John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > I'm sorry, John, I missed the contextt of your question. We don't use > background workers that way, so I don't have an answer for you. > > Charlotte > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:05 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> You can create your own events with whatever parameters you want and >> raise those. Just Dim Public Event Whatever(parameter) Then raise it > >> as needed. >> >> Charlotte Foust > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:41:16 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:41:16 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AF3C.3040601@colbyconsulting.com> Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 13:06:29 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:06:29 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <4818A990.2020306@colbyconsulting.com> Message-ID: <02f601c8aaec$ea01f620$6401a8c0@nant> John, Please read the article I referred already: "Simple and Efficient Multi-threading with Blocking Queues" http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik I think it is "what doctor ordered" for your task, and it has code, C# but you can convert it if needed using free converters.... As for collections and variables - they are not thread safe by default. Read these two articles (they have sample code): "Static & Thread Safety: Part I and II" http://www.odetocode.com/Articles/313.aspx http://www.odetocode.com/Articles/314.aspx I believe the only book you need now is the one I have also mentioned already: "C# Threading Handbook by Tobin Titus et al.", ISBN:1861008295 APress, 2004 - it has a lot of code samples (C# sorry) and very good .NET multi-threading features coverage And this is an excerpt from MSDN on collections (MSDN is our best friend!): +++ Collection Object (Visual Basic) Caution Iterating through a Visual Basic Collection is not a thread-safe procedure. Even if the collection is synchronized, other threads can still modify the collection, causing the enumerator to throw an exception. To guarantee thread safety during enumeration, either lock the collection or catch the exceptions resulting from changes made by other threads. For more information on locking a programming element, see SyncLock Statement. ---- The referred above as "what doctor ordered for your task" article uses .NET's Queue class - here is the info on it from MSDN on thread safety: +++ Queue Class Thread Safety Public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. To guarantee the thread safety of the Queue, all operations must be done through the wrapper returned by the Synchronized method. Enumerating through a collection is intrinsically not a thread safe procedure. Even when a collection is synchronized, other threads can still modify the collection, which causes the enumerator to throw an exception. To guarantee thread safety during enumeration, you can either lock the collection during the entire enumeration or catch the exceptions resulting from changes made by other threads. ---- The "Simple and Efficient Multi-threading with Blocking Queues" article has sample code, which shows what simple coding is needed to implement thread-safe collections and queues... HTH, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:17 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:22:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:22:08 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AD3D.6030003@colbyconsulting.com> Message-ID: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:28:58 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:28:58 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AF3C.3040601@colbyconsulting.com> Message-ID: <02fb01c8aaf0$0e512110$6401a8c0@nant> Thank you, John, You have a very good PC, lucky man! :) As I wrote already I (I'm sorry) did post incorrect query - I wanted to run this one: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] It will also return 96K rows for your huge table, but it's expected (here) it will get first results instantly, and it will finish within just several (3?) seconds... Please prove am I wrong with that bright above expectations on the execution time of the fixed version of the query?... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Wed Apr 30 13:50:38 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 13:50:38 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 14:31:28 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 23:31:28 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> Message-ID: <030a01c8aaf8$c971d130$6401a8c0@nant> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead of ~3 seconds.... SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc] (nolock)) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] (nolock) BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of useful advices including all kinds of locking - the first time I see so detailed advices and samples on that subject (but I must say I didn't read that much computer books in the past)... Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Wednesday, April 30, 2008 10:51 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil From jwcolby at colbyconsulting.com Wed Apr 30 14:55:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:55:15 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> References: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> Message-ID: <4818CEA3.5000704@colbyconsulting.com> Well, I do not know what you want me to do with the result. To just open a query window in SSMS and execute it in there I get pretty much the same result. First results in 14 seconds and ten minutes and counting. I don't have "just the PK" indexed, at least not intentionally and visibly. I am now using cover indexes that index the PKID and one or more fields that I am interested in. Whether there is a PKID index behind the scenes I don't know. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:33 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > It started returning results in about 12-13 seconds. You are asking for > 96 thousand result rows you realize. This database is 96 million rows, > so every thousandth row is 96K rows. > > It is still processing at 9 minutes. I'll email the final execution > time whenever the query finishes. > > This is fascinating because I pull just the fields into an ADO.Net table > for 100K rows in just a second or two. Of course these are consecutive > rows, it is not scanning the entire table. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 14:59:44 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:59:44 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030a01c8aaf8$c971d130$6401a8c0@nant> References: <030a01c8aaf8$c971d130$6401a8c0@nant> Message-ID: <4818CFB0.4020402@colbyconsulting.com> I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 15:19:00 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:19:00 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fb01c8aaf0$0e512110$6401a8c0@nant> References: <02fb01c8aaf0$0e512110$6401a8c0@nant> Message-ID: <4818D434.4090104@colbyconsulting.com> Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Wed Apr 30 15:24:32 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 15:24:32 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 15:32:43 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:32:43 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> References: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <4818D76B.7000602@colbyconsulting.com> Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <031501c8ab05$422d2460$6401a8c0@nant> <<< I see the UNION ALL and ORDER BY as bottlenecks. >>> Eric, But union all is executed with two row sets: - first having every 1000th record, and - second having ONLY ONE record... When query execution plan is analyzed then I see that first operation for both parts of UNION ALL query is index scan IOW UNION ALL and ORDER BY do not look as a bottleneck because UNION ALL merges (without any attempts to suppress full duplicate) a set of records with just one record, and because ORDER by of clustered index is 0% cost operation? Thank you -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Thursday, May 01, 2008 12:25 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D434.4090104@colbyconsulting.com> Message-ID: <031601c8ab05$42476320$6401a8c0@nant> <<< Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? >>> Yes, it was, but I expected you'll get results much quicker... Could that be that you do not have (clustered)index on your PK field? Can you: Right-Click on table name -> Script Table as -> CREATE to -> New Query Window => I'm getting the following for your simulated table. CREATE TABLE [dbo].[Names]( [PKID] [int] IDENTITY(1,1) NOT NULL, [OWNERNAME] [nvarchar](255) NOT NULL, [FName] [nvarchar](50) NULL, [MName] [nvarchar](50) NULL, [LName] [nvarchar](50) NULL, [NamePrefix] [nvarchar](50) NULL, [NameSuffix] [nvarchar](50) NULL, [Gender] [nvarchar](50) NULL, [ParsedName] [nvarchar](255) NULL, CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED ( [PKID] ASC ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] Also if you select query's text in SSMS query window, and then Ctrl+L then you'll get query execution plan - here as the last step with 99% cost I'm getting: "Clustered Index Scan" IOW the query doesn't scan table's data at all - it just scans the index... Thanks, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:19 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:07:55 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:07:55 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D76B.7000602@colbyconsulting.com> Message-ID: <032201c8ab06$42a01500$6401a8c0@nant> John, This ORDER BY used here to get ROW_NUMBER is/was expected as very speedy "virtual" zero cost operation because I thought/expected your PK (PKID) was a (clustered) index... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:33 AM To: ebarro at verizon.net; Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 16:16:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 17:16:08 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <031601c8ab05$42476320$6401a8c0@nant> References: <031601c8ab05$42476320$6401a8c0@nant> Message-ID: <4818E198.30501@colbyconsulting.com> I do have the clustered stuff exactly as you show: > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] Likewise I am getting the same 99% cost clustered index scan. I have to guess it is something else. There is a lot of empty space in the database file which I will compact out tonight. It takes hours because of the size and when it is doing that I cannot use the db. Perhaps I have severe fragmentation or something. The database size is 300 GIGS. And finally just remember that when all is said and done it is still a hundred times larger than your database. No matter how you slice it, it has to process more physical disk sectors etc. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > > <<< > Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. > Was that your intention? > Yes, it was, but I expected you'll get results much quicker... > > Could that be that you do not have (clustered)index on your PK field? > > Can you: Right-Click on table name -> Script Table as -> CREATE to -> New > Query Window => I'm getting the following for your simulated table. > > CREATE TABLE [dbo].[Names]( > [PKID] [int] IDENTITY(1,1) NOT NULL, > [OWNERNAME] [nvarchar](255) NOT NULL, > [FName] [nvarchar](50) NULL, > [MName] [nvarchar](50) NULL, > [LName] [nvarchar](50) NULL, > [NamePrefix] [nvarchar](50) NULL, > [NameSuffix] [nvarchar](50) NULL, > [Gender] [nvarchar](50) NULL, > [ParsedName] [nvarchar](255) NULL, > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] > > Also if you select query's text in SSMS query window, and then Ctrl+L then > you'll get query execution plan - here as the last step with 99% cost I'm > getting: "Clustered Index Scan" IOW the query doesn't scan table's data at > all - it just scans the index... > > Thanks, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Thursday, May 01, 2008 12:19 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > > You have a very good PC, lucky man! :) > > As I mentioned elsewhere, I come from the "throw more hardware at it" > school of Engineering. ;-) > > I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 > gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster > in many cases. > > I do not have this large database over on that machine, and it would > take me hours to get it over there so i can't test this query on that, > at least today. that database has not been compacted recently and is > about 300 Gigs with 136 gigs free space. I will compact it tonight and > then copy the smaller file to Azul. > > As for your new query, with the NoLocks in place it returned first > results in only 8 seconds as opposed to 14 before, but it still took > 13:59 to pull the entire result set. Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. Was that your > intention? > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Thank you, John, >> >> You have a very good PC, lucky man! :) >> >> As I wrote already I (I'm sorry) did post incorrect query - I wanted to > run >> this one: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> It will also return 96K rows for your huge table, but it's expected (here) >> it will get first results instantly, and it will finish within just > several >> (3?) seconds... >> >> Please prove am I wrong with that bright above expectations on the > execution >> time of the fixed version of the query?... >> >> Thank you. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Wednesday, April 30, 2008 9:41 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> Shamil, >> >> It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server >> x32) 13:51 to pull 96,417 rows for the result set. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Shamil Salakhetdinov wrote: >>> Hi John, >>> >>> May I ask you to run this query (e.g. in MS SQL Management Studio) > against >>> your huge db table? >>> >>> --- cut here --- >>> >>> SELECT RowNum, PKID from ( select >>> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >>> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >>> NameSuffix, Gender from [Names]) s >>> where (s.RowNum % 1000) = 1 >>> union all >>> Select NULL as RowNum, Max(PKID) as PKID from [Names] >>> >>> --- cut here --- >>> >>> M.B. [Names] have to be substituted with the name of your table - what is >> it >>> BTW? >>> >>> This query returns PKID of every 1000th row sorted by PKID + > MAX(PKID).... >>> I'm curious to know how much time approx. this query will take to >> execute... >>> Thank you. >>> >>> -- >>> Shamil >>> >>> P.S. BTW, here is a useful for your case article on multi-threading >>> (practical solution) with good introduction - >>> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >>> cik >>> >>> Sorry, I can't react promptly on your messages in this discussion thread: >> a >>> lot of custom work here, long weekends and our time zones difference but > I >>> do continue to work in this direction and if I get some useful results I >>> will post them here... >>> >>> P.P.S. This book could be also useful: >>> >>> C# Threading Handbook >>> by Tobin Titus et al. >>> >>> ISBN:1861008295 >>> >>> APress, LLC C 2004 (288 pages) >>> >>> This book addresses the fundamental units of Windows and .NET >>> programming-threads. Coverage includes how .NET applications are > executed, >>> the life cycle of a thread in .NET, how the .NET Framework uses threads, >> and >>> more. >>> >>> >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 3 06:34:34 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 03 Apr 2008 13:34:34 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From accessd at shaw.ca Thu Apr 3 10:59:46 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Thu, 03 Apr 2008 08:59:46 -0700 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: References: Message-ID: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From michael at ddisolutions.com.au Thu Apr 3 18:01:55 2008 From: michael at ddisolutions.com.au (Michael Maddison) Date: Fri, 4 Apr 2008 10:01:55 +1100 Subject: [dba-VB] ASP.NET, separated programming and visual design References: <5BC301195B7047F7B59080AEB830A328@creativesystemdesigns.com> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D013BF515@ddi-01.DDI.local> I did an online seminar with MS about VS 2008 + WPF. They claim that WPF is pretty much designed to allow the UI designer to work seperately to the programmer. He could do it in NP as well as long as he knows his XML ;-) cheers Michael M -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Friday, 4 April 2008 3:00 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Visual Studio or Dreamweaver (use for FE) most like or at the very least he should be using Programmers NotePad (free download). Used notepad 10 or 15 years ago and have not used it since. Went to a seminar last night and the discussion was on how to build a proper UI.... as important as the code behind it. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 4:35 AM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Thu Apr 3 18:53:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 03:53:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <006601c895e5$fa874e80$6401a8c0@nant> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 02:14:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 09:14:35 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From shamil at users.mns.ru Fri Apr 4 04:59:56 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Fri, 4 Apr 2008 13:59:56 +0400 Subject: [dba-VB] ASP.NET, separated programming and visual design In-Reply-To: Message-ID: <002d01c8963a$a3217e90$6501a8c0@nant> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Fri Apr 4 05:17:54 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 04 Apr 2008 12:17:54 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi Shamil Great! This is exactly the information I need. /gustav >>> shamil at users.mns.ru 04-04-2008 11:59 >>> Hi Gustav, In fact I and my customer(s) sometimes do the "trick" with notepad when some "cosmetic" or minor application web page logic changes are needed: we open such a page or its accompanying code behind in notepad, do the edits and save the page - all on a live running version... Yes, if you download current version of ASP.NET application files from live site then you can open it as a web site in VS or in VS you can have solution including Web site as one of the projects.... No, ASP.NET will not be "smart enough" to not let newer version of files be replaced with the files you upload, but because ASP.NET application release is usually a mere XCOPY of files (after first version is released) then you can use XCOPY "smartness" to not replace newer version of files, or some other files' synchronization tools, which do not overwrite newer by date versions of files, or you can use setup tools with option to not overwrite newer version of files etc. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Friday, April 04, 2008 11:15 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ASP.NET, separated programming and visual design Hi Jim, Michael and Shamil First, this will be a simple application, only four tables as in most "ASP Beginner Course": Customer Product Order OrderLine I could probably just copy one of the VS samples! Then, that about Notepad was a joke. Don't know which tool he uses but he would have a look at Programmer's Notepad, that's at least one step forward. But I think I will try to persuade him for VS Express. If he is fluent in ASP, it must be a gift. Finally, as this will be the simple case as Shamil describes, if I upload the application and he does some changes to the .aspx/.ascx files, all I need to do is to download the .aspx/.ascx files and overwrite my local files to have the full application? But if I don't do that and needed to make some changes to the "mechanics", will VS be so clever when uploading the revised application that it will not touch the newer .aspx/.ascx files? That would be perfect. /gustav >>> shamil at users.mns.ru 04-04-2008 01:53 >>> Hi Gustav, "separated programming and visual design" for asp.net application in the most simple case usually means that visual design part of web pages/custom controls is kept in .aspx/.ascx files, and code to process events of these web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of code is stored in \bin subfolder; in even more advanced cases one can create (visual layer) classlib DLLs etc.... Yes, one can type entire ASP.Net application in Notepad (except graphics of course)... But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing .aspx markup in text view in Visual Studio - when used that way then one can use Intellisence, which is very handy... After release you can replace parts of ASP.NET application or the whole application just by replacing/copying over parts/all the files of the ASP.NET application... You can do such replacing/copying over even for live running ASP.NET application: when separate .aspx pages are replaced then after replacing the new version of .aspx page will be rendered on the new page request, and ASP.NET application *will not be* restarted... If you replace certain files as e.g. ASP.NET app's web.config, master page, code files/classlib .dlls in \bin folder and some others then ASP.NET application gets restarted but assuming that the usual practice is to have most of the pages of ASP.NET app stateless such restarting shouldn't harm (NB: ASP.Net application restarting clears/destroys all session level variables and application cache but preserve live user sessions, which identity is usually kept in cookies...) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Thursday, April 03, 2008 3:35 PM To: dba-vb at databaseadvisors.com Subject: [dba-VB] ASP.NET, separated programming and visual design Hi all A client asks if we can create a small database application (usual stuff for order registration) for him with only a very basic design and then he will create the visual design. That should be doable with ASP.NET and Visual Studio 2008 Standard for my part but what will he need? I would say Visual Studio Web edition but he claims to be an expert in ASP and prefers Notepad! I could leave the problem to him but I prefer to stay clear of possible disasters. Also, I must be able to pull back the application, make some changes and forward it again. Anyone having experiences with a process like this? /gustav From wdhindman at dejpolsystems.com Thu Apr 10 09:11:34 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Thu, 10 Apr 2008 10:11:34 -0400 Subject: [dba-VB] ASP.NET, separated programming and visual design References: Message-ID: <002301c89b14$c8de1d30$7e6d6c4c@jisshowsbs.local> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From Gustav at cactus.dk Thu Apr 10 09:25:30 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 10 Apr 2008 16:25:30 +0200 Subject: [dba-VB] ASP.NET, separated programming and visual design Message-ID: Hi William Fine, though it we be the other way around - I would make the mechanics first, then he would beef it up. With VSVWD8 Express as a free tool I see no reason why he should punish him self with something like Notepad. /gustav >>> wdhindman at dejpolsystems.com 10-04-2008 16:11 >>> ...late here Gustav but you can do the entire app in VSVWD8 Express ...turn theming and skins on and let him build the UI to his hearts content there, and only there, using VSVWD8 Express without ever touching the real app ...its amazing what can be done with css in VS these days via themes and skins ...and when you use CopyWeb to upload your changes to the aspx files, it lets you pick and choose which files you overwrite ...since VSVWD Express is a free dl and supports both C# and vb.net, theres no reason anyone should be playng with text editors. William ----- Original Message ----- From: "Gustav Brock" To: Sent: Friday, April 04, 2008 6:17 AM Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > Hi Shamil > > Great! This is exactly the information I need. > > /gustav > >>>> shamil at users.mns.ru 04-04-2008 11:59 >>> > Hi Gustav, > > In fact I and my customer(s) sometimes do the "trick" with notepad when > some > "cosmetic" or minor application web page logic changes are needed: we open > such a page or its accompanying code behind in notepad, do the edits and > save the page - all on a live running version... > > Yes, if you download current version of ASP.NET application files from > live > site then you can open it as a web site in VS or in VS you can have > solution > including Web site as one of the projects.... > > No, ASP.NET will not be "smart enough" to not let newer version of files > be > replaced with the files you upload, but because ASP.NET application > release > is usually a mere XCOPY of files (after first version is released) then > you > can use XCOPY "smartness" to not replace newer version of files, or some > other files' synchronization tools, which do not overwrite newer by date > versions of files, or you can use setup tools with option to not overwrite > newer version of files etc. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Friday, April 04, 2008 11:15 AM > To: dba-vb at databaseadvisors.com > Subject: Re: [dba-VB] ASP.NET, separated programming and visual design > > Hi Jim, Michael and Shamil > > First, this will be a simple application, only four tables as in most "ASP > Beginner Course": > > Customer > Product > Order > OrderLine > > I could probably just copy one of the VS samples! > > Then, that about Notepad was a joke. Don't know which tool he uses but he > would have a look at Programmer's Notepad, that's at least one step > forward. > But I think I will try to persuade him for VS Express. If he is fluent in > ASP, it must be a gift. > > Finally, as this will be the simple case as Shamil describes, if I upload > the application and he does some changes to the .aspx/.ascx files, all I > need to do is to download the .aspx/.ascx files and overwrite my local > files > to have the full application? > But if I don't do that and needed to make some changes to the "mechanics", > will VS be so clever when uploading the revised application that it will > not > touch the newer .aspx/.ascx files? > That would be perfect. > > /gustav > > >>>> shamil at users.mns.ru 04-04-2008 01:53 >>> > Hi Gustav, > > "separated programming and visual design" for asp.net application in the > most simple case usually means that visual design part of web pages/custom > controls is kept in .aspx/.ascx files, and code to process events of these > web pages/controls is kept in accompanying .aspx.cs/.ascx.cs (C#) or > .aspx.vb/.ascx.vb (VB) code behind files. In more advanced cases part of > code is stored in \bin subfolder; in even more advanced cases one can > create > (visual layer) classlib DLLs etc.... > > Yes, one can type entire ASP.Net application in Notepad (except graphics > of > course)... > > But I'd doubt such a practice - ASP.NET experts AFAIS are usually typing > .aspx markup in text view in Visual Studio - when used that way then one > can > use Intellisence, which is very handy... > > After release you can replace parts of ASP.NET application or the whole > application just by replacing/copying over parts/all the files of the > ASP.NET application... > > You can do such replacing/copying over even for live running ASP.NET > application: when separate .aspx pages are replaced then after replacing > the > new version of .aspx page will be rendered on the new page request, and > ASP.NET application *will not be* restarted... > > If you replace certain files as e.g. ASP.NET app's web.config, master > page, > code files/classlib .dlls in \bin folder and some others then ASP.NET > application gets restarted but assuming that the usual practice is to have > most of the pages of ASP.NET app stateless such restarting shouldn't harm > (NB: ASP.Net application restarting clears/destroys all session level > variables and application cache but preserve live user sessions, which > identity is usually kept in cookies...) > > > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock > Sent: Thursday, April 03, 2008 3:35 PM > To: dba-vb at databaseadvisors.com > Subject: [dba-VB] ASP.NET, separated programming and visual design > > Hi all > > A client asks if we can create a small database application (usual stuff > for > order registration) for him with only a very basic design and then he will > create the visual design. > > That should be doable with ASP.NET and Visual Studio 2008 Standard for my > part but what will he need? I would say Visual Studio Web edition but he > claims to be an expert in ASP and prefers Notepad! I could leave the > problem > to him but I prefer to stay clear of possible disasters. > > Also, I must be able to pull back the application, make some changes and > forward it again. > > Anyone having experiences with a process like this? > > /gustav From jwcolby at colbyconsulting.com Fri Apr 11 12:26:53 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:26:53 -0400 Subject: [dba-VB] Parsing strings Message-ID: <47FF9F5D.2070704@colbyconsulting.com> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Fri Apr 11 12:42:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Fri, 11 Apr 2008 19:42:18 +0200 Subject: [dba-VB] Parsing strings Message-ID: Hi John Why not use the Path class: extension = Path.GetExtension(FileName) basename = Path.GetFileNameWithoutExtension(FileName) /gustav >>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> I am trying to parse a string. I notice that the string object has many methods but it does not seem to have the left, mid and right functions that vba has. If I want to split a file name into name and extension how would you do it? I am just use the existing string functions that we all know and love but am wondering if there is a better (.Net) way. Thanks, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 11 12:49:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 11 Apr 2008 13:49:18 -0400 Subject: [dba-VB] Parsing strings In-Reply-To: References: Message-ID: <47FFA49E.2020605@colbyconsulting.com> Lovely, thanks. Gustav Brock wrote: > Hi John > > Why not use the Path class: > > extension = Path.GetExtension(FileName) > basename = Path.GetFileNameWithoutExtension(FileName) > > /gustav > >>>> jwcolby at colbyconsulting.com 11-04-2008 19:26 >>> > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, -- John W. Colby www.ColbyConsulting.com From stuart at lexacorp.com.pg Fri Apr 11 18:45:55 2008 From: stuart at lexacorp.com.pg (Stuart McLachlan) Date: Sat, 12 Apr 2008 09:45:55 +1000 Subject: [dba-VB] Parsing strings In-Reply-To: <47FF9F5D.2070704@colbyconsulting.com> References: <47FF9F5D.2070704@colbyconsulting.com> Message-ID: <480084D3.12387.2763EDB7@stuart.lexacorp.com.pg> Left,Mid,Right etc are not function of the string object, they are functions in the Strings module of the Microsoft.VisualBasic namespace. i.e., fully qualified it is Microsoft.VisualBasic.Strings.Right() On 11 Apr 2008 at 13:26, jwcolby wrote: > I am trying to parse a string. I notice that the string object has many > methods but it does not seem to have the left, mid and right functions > that vba has. > > If I want to split a file name into name and extension how would you do > it? I am just use the existing string functions that we all know and > love but am wondering if there is a better (.Net) way. > > Thanks, > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > From jwcolby at colbyconsulting.com Sat Apr 12 15:23:49 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 16:23:49 -0400 Subject: [dba-VB] Count a character in a string Message-ID: <48011A55.40402@colbyconsulting.com> I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 15:50:25 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 15:50:25 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From Gustav at cactus.dk Sat Apr 12 16:33:25 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 12 Apr 2008 23:33:25 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 12 17:48:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 12 Apr 2008 18:48:26 -0400 Subject: [dba-VB] Count a character in a string In-Reply-To: References: Message-ID: <48013C3A.9030500@colbyconsulting.com> I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com From DWUTKA at Marlow.com Sat Apr 12 18:00:38 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:00:38 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: Message-ID: True, don't use .Net much though. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Saturday, April 12, 2008 4:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] Count a character in a string Hi John and Drew The UBound method looks nice. Wouldn't it read in VB.Net: intCount = Split(strTest, "|").GetUpperBound(0) /gustav >>> DWUTKA at marlow.com 12-04-2008 22:50 >>> Hmmm, two ways I can think of. One: Dim strRecordLine() as array strRecordline=split(yourstringhere,"|") NumberOfSeperatorCharacters=UBound(strRecordline)+1 Two: Dim strTemp as String strTemp=Replace(yourstring,"|","") #ofCharacters=Len(yourstring)-len(strTemp) Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 3:24 PM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From DWUTKA at Marlow.com Sat Apr 12 18:01:05 2008 From: DWUTKA at Marlow.com (Drew Wutka) Date: Sat, 12 Apr 2008 18:01:05 -0500 Subject: [dba-VB] Count a character in a string In-Reply-To: <48013C3A.9030500@colbyconsulting.com> Message-ID: Ya, I think I posted that solution a while back on AccessD. Drew -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 12, 2008 5:48 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Count a character in a string I like #2 and in fact I seem to have read that one before somewhere. I am running through a file with millions of lines so it has to be as fast as possible. Thanks, Drew Wutka wrote: > Hmmm, two ways I can think of. > > One: > > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) > > Drew > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 12, 2008 3:24 PM > To: VBA > Subject: [dba-VB] Count a character in a string > > I do a lot of CSV stuff. One of my file suppliers gives me files that > throw an error when I try to import them and so I am writing tools to > try and troubleshoot these files. I need to be able to count the times > that the field separator character is contained in a line. The error I > am getting says the line is too long but I have determined that is not > the case. I suspect that perhaps the field separator character is in > the line too often. > > So, if I have a line that looks like: > > asdf|qwert|zxcv|mnbv > > Other than just iterating through the string character by character, is > there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com The information contained in this transmission is intended only for the person or entity to which it is addressed and may contain II-VI Proprietary and/or II-VI Business Sensitive material. If you are not the intended recipient, please contact the sender immediately and destroy the material in its entirety, whether electronic or hard copy. You are notified that any review, retransmission, copying, disclosure, dissemination, or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. From shamil at users.mns.ru Sat Apr 12 21:30:24 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 06:30:24 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <48011A55.40402@colbyconsulting.com> Message-ID: <053b01c89d0e$53c595e0$6401a8c0@nant> Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:08:59 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:08:59 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> References: <48011A55.40402@colbyconsulting.com> <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <003c01c89d46$03525ca0$8119fea9@LTVM> John, You may recall the RegExp that I posted a short while ago. You can use this to get the answer. This code returns 17: Function CountRegEx() Dim strIn As String, lngCount As Long strIn = "asdf||||||qwert|zxcv|mnbv|qwert|zxcv|mnbv||||||" lngCount = (RegExpTest("\|", strIn)) MsgBox lngCount End Function Here is the original Function Function RegExpTest(patrn, strng) As Long ' Set reference to Microsoft vbscript regular expressions 5.5 ' Cheat Sheet available at http://regexlib.com/CheatSheet.aspx Dim RegEx As New VBScript_RegExp_55.RegExp Dim Match, Matches, RetStr ' Create variable. Set RegEx = New RegExp ' Create a regular expression. RegEx.Pattern = patrn ' Set pattern. RegEx.IgnoreCase = True ' Set case insensitivity. RegEx.Global = True ' Set global applicability. Set Matches = RegEx.Execute(strng) ' Execute search. RegExpTest = 0 For Each Match In Matches ' Iterate Matches collection. RetStr = RetStr & "Match found at position " RetStr = RetStr & Match.FirstIndex & ". Match Value is '" RetStr = RetStr & Match.Value & "'." & vbCrLf RegExpTest = RegExpTest + 1 Next End Function HTH Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 3:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From max.wanadoo at gmail.com Sun Apr 13 04:14:06 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 13 Apr 2008 10:14:06 +0100 Subject: [dba-VB] Count a character in a string In-Reply-To: References: <48013C3A.9030500@colbyconsulting.com> Message-ID: <003d01c89d46$ba370a60$8119fea9@LTVM> John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max From shamil at users.mns.ru Sun Apr 13 04:26:17 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 13:26:17 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <053b01c89d0e$53c595e0$6401a8c0@nant> Message-ID: <054b01c89d48$6d543360$6401a8c0@nant> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 13 04:47:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 11:47:21 +0200 Subject: [dba-VB] Count a character in a string Message-ID: Hi Shamil Neat! I'll add that to my toolbox. /gustav >>> shamil at users.mns.ru 13-04-2008 11:26 >>> Hi All, I have also tried: string s = "asdf|qwert|zxcv|mnbv"; Regex rx = new Regex("|"); int count = rx.Matches(s).Count; and I (expectedly) found that it takes ages comparing with the first method: string s = "asdf|qwert|zxcv|mnbv"; int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; which finished 20 million iterations in 3 sec, and I cancelled the RegEx based test after several minutes waiting that it finishes for 20 million iterations... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 13, 2008 6:30 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string Hi John, My bet that this code will be the most speedy in .NET (C# sample): int count = 0; for (int index = 0; index < s.Length; index++) if ((s[index] ^ '|') == 0) count++; -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 13, 2008 12:24 AM To: VBA Subject: [dba-VB] Count a character in a string I do a lot of CSV stuff. One of my file suppliers gives me files that throw an error when I try to import them and so I am writing tools to try and troubleshoot these files. I need to be able to count the times that the field separator character is contained in a line. The error I am getting says the line is too long but I have determined that is not the case. I suspect that perhaps the field separator character is in the line too often. So, if I have a line that looks like: asdf|qwert|zxcv|mnbv Other than just iterating through the string character by character, is there a way to get a count of the | character in the string? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at users.mns.ru Sun Apr 13 05:08:53 2008 From: shamil at users.mns.ru (Shamil Salakhetdinov) Date: Sun, 13 Apr 2008 14:08:53 +0400 Subject: [dba-VB] Count a character in a string In-Reply-To: <003d01c89d46$ba370a60$8119fea9@LTVM> Message-ID: <054f01c89d4e$60f41d00$6401a8c0@nant> Hi Max, Our postings crossed - I have just posted RegEx test results... N.B.: when testing this or that method both memory consumption and execution speed should be taken into consideration: - high memory consumption giving speediest results can be neglected to solve small tasks within "tiny" applications/utilities, but - high memory consumption is the thing, which could "unexpectedly" give nasty side effects in the large application systems: One example I have got several months ago, which was so "tricky" to watch, and was not easy to find why it could ever happen in .NET: I have got a memory leakage/excessive memory consumption (in .NET!) because of "lazy loading", and that latter "lazy loading" loaded quite some data when it wasn't needed, in the cycle processing millions of records - the results were like that when I watched 'Page File Usage History' in Task Manager: /| /| /| . / | / | / | . / | / | / | . / |/ |/ |/ And in usual mode without heavy application system workload everything worked well with the same data... Recap: - Split(...) approach could result in similar to the above side effect for very large input strings; // 10 sec for 20,000,000 iterations string[] recordLine = s.Split('|'); count = recordLine.Length-1; - Replace(...) could also result in the above side effect; // 10 sec for 20,000,000 iterations string temp = s.Replace("|", Microsoft.VisualBasic.Constants.vbNullString); count = s.Length - temp.Length; - RegEx(...) seems to be the slowest - unsatisfactory slow for large input strings/heavy system workload; // ? sec (unfinished) for 20,000,000 iterations Regex rx = new Regex("|"); count = rx.Matches(s).Count; - char array iteration using char index and (XOR or char comparison) gives the fastest results and is 100% safe from memory consumption point of view.... // * using XOR: // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; // * using char comparinson... // ~3 sec for 20,000,000 iterations for (int index = 0; index < s.Length; index++) if ((s[index] == '|')) count++; Please correct me if you'll find the above results have mistakes... Any takers to find quiker code for .NET VB or C# or C++/CLI? - that would be a good weekend exercise on code optimization techniques... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Max Wanadoo Sent: Sunday, April 13, 2008 1:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Count a character in a string John, Further to my example RegExpr code that I just posted. Would you have time to compare how long it takes and how long these two take (posted by others): > One: > Dim strRecordLine() as array > strRecordline=split(yourstringhere,"|") > NumberOfSeperatorCharacters=UBound(strRecordline)+1 > > Two: > Dim strTemp as String > strTemp=Replace(yourstring,"|","") > #ofCharacters=Len(yourstring)-len(strTemp) It would be really nice to get a handle on which of these is the faster for a very large file. Thanks Max _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Johncliviger at aol.com Sun Apr 13 13:36:20 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 14:36:20 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Gustav at cactus.dk Sun Apr 13 13:50:38 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 13 Apr 2008 20:50:38 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Pick only the fields you need ... or do you wish to display 30 fields? /gustav >>> Johncliviger at aol.com 13-04-2008 20:36 >>> Hi All I'm teaching myself vb.net 2005 with the help of AppDev which is good stuff. However I've hit the knowledge buffers when trying to add a Access style subform to a mainform. I can use the DataGrid on problem. But with 30 or so fields in the child table, the screen is not wide enough and it looks naff. How does vb.net 2005 handle this one-to-many relathionship with too many fields for a datagrid? regards john cliviger From Johncliviger at aol.com Sun Apr 13 14:08:13 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Sun, 13 Apr 2008 15:08:13 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Gustav at cactus.dk Mon Apr 14 08:53:40 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 14 Apr 2008 15:53:40 +0200 Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Hi John Many ways to do this - depends a bit on your previous code and what features you need. A lot of examples can be found via Google for this - including Microsoft's pages. Key words could be master/child or parent/child. /gustav >>> Johncliviger at aol.com 13-04-2008 21:08 >>> Hi gustav OK. I've done that already. I guess that I now need to add a BindingNavigator for the child controls. Am I heading the right way? Cheers john c From Johncliviger at aol.com Mon Apr 14 09:23:22 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Mon, 14 Apr 2008 10:23:22 EDT Subject: [dba-VB] VB.net 2005 SubForm Message-ID: Gustav Many thanks I've cracked it. Quite simple when you know how. When you try it for this first time your Access thinking leads you astray. Cheers Gustav john c From Gustav at cactus.dk Tue Apr 15 07:28:29 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 15 Apr 2008 14:28:29 +0200 Subject: [dba-VB] Web colours Message-ID: Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav From accessd at shaw.ca Tue Apr 15 14:03:05 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:03:05 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From accessd at shaw.ca Tue Apr 15 14:28:06 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Tue, 15 Apr 2008 12:28:06 -0700 Subject: [dba-VB] Web colours In-Reply-To: References: Message-ID: <945F14E6828D4E92AB0736A26485C259@creativesystemdesigns.com> Hi Gustav: I should have read right through your question before I, with a sandwich in hand, responded. Use any colour you want. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Tuesday, April 15, 2008 12:03 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] Web colours Hi Gustav: Simple answer to that question is NO. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 15, 2008 5:28 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Web colours Hi all Those of you designing web pages, do you still stick to the "safe web colours" - the limited selection of 216 colours out of 256 possible colours? http://www.lynda.com/hex.asp As this Lynda writes, in 1996 this might be a concern, but today where I hardly know nobody having the old 65000 colour resolution, is this still valid in any way? Isn't is safe - at least for normal business use - to just go ahead and use the full RGB scale? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 16 08:44:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 16 Apr 2008 15:44:13 +0200 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Message-ID: Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24/microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav From accessd at shaw.ca Wed Apr 16 16:00:03 2008 From: accessd at shaw.ca (Jim Lawrence) Date: Wed, 16 Apr 2008 14:00:03 -0700 Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) In-Reply-To: References: Message-ID: <2751B45E193C418490D807B02A37C79A@creativesystemdesigns.com> Thanks for your heads up Gustav. Jim -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Wednesday, April 16, 2008 6:44 AM To: accessd at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-VB] Visual Studio: Free Developer Training (3 hours) Hi all If you like free on-line training, here are three one hour sessions on C#, VB, and Linq: http://www.innerworkings.com/promotions/3648c121-09eb-4858-b4e8-abc4c0054e24 /microsoft-learning-vs-2008-promotion I haven't seen them so I can't tell about the quality. /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From paul.hartland at googlemail.com Fri Apr 18 02:49:18 2008 From: paul.hartland at googlemail.com (Paul Hartland) Date: Fri, 18 Apr 2008 08:49:18 +0100 Subject: [dba-VB] OT - Free Twain ActiveX Control Message-ID: <38c884770804180049n76b46424y6d66cf4124792ced@mail.gmail.com> To all, I need to develop an application, which will involve scanning some documents etc. Does anyone know of or use a Twain ActiveX Control (preferably easy to use & most of all free, or very cheap) Thank you for any help on this..... -- Paul Hartland paul.hartland at googlemail.com From jwcolby at colbyconsulting.com Sat Apr 19 09:05:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 10:05:31 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <4809FC2B.6030602@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 19 10:08:14 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 19 Apr 2008 17:08:14 +0200 Subject: [dba-VB] NetGender - The Software Company Message-ID: Hi John One method would be to have a MDI parent form from which you open the other forms as child forms. Then you just to either keep track of what you open or to loop through the child forms collection to see if it contains a certain child form. In VS2008 the Add New wizard will happily create a new MDI parent form with basic functionality. /gustav >>> jwcolby at colbyconsulting.com 19-04-2008 16:05 >>> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and the code in the form shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. Modifications to the demo form to allow me to do all that stuff is just about complete. What I need to know is the syntax to get a pointer to a form being opened. I know that you just do a "Someform.Show" to open the form, but a parent form needs to get a pointer to this demo form and then use that pointer to set / retrieve properties. How do I get a pointer to it once opened? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 11:08:13 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 12:08:13 -0400 Subject: [dba-VB] Splitting a name Message-ID: <480A18ED.6020009@colbyconsulting.com> I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 19 12:27:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 19 Apr 2008 13:27:25 -0400 Subject: [dba-VB] NetGender - The Software Company Message-ID: <480A2B7D.6000202@colbyconsulting.com> I purchased a .Net library that does name splitting. http://www.softwarecompany.com/dotnet/netgender.htm It seems to work well just from using the little demo that you can download. Now I am starting to actually work with it. It consists of a single class, embedded in a dll that you reference, with properties to set to tell it all of the controlling stuff, then properties to hand it the name to split and read back the name pieces. The developer claims that it will handle between 2K and 3K names per second. We shall see. So here I am trying to work with this thing. The demo form is quite extensive and shows how to read / write all of the properties to populate the form. I have a database with (of course) about 80 million names, in a single field, in the form LastName FirstName MiddleInitial (no commas etc) which needs parsing. What I thought I would do (just for the moment) is to use the existing form and just feed in the names. Thus I need to add a couple of properties to the form such that I can feed in names and get back the parsed pieces. I will build a switch to tell the form to display the pieces or not, thus I could watch the process function if I wanted, though at the cost of slowing the process down. -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sat Apr 19 13:51:12 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sat, 19 Apr 2008 14:51:12 -0400 Subject: [dba-VB] NetGender - The Software Company References: <4809FC2B.6030602@colbyconsulting.com> Message-ID: <000301c8a24e$57223eb0$60606c4c@jisshowsbs.local> ...nice looking library ...let us know what problems you run into with it ...I may well be in the market for such myself in the near future. William ----- Original Message ----- From: "jwcolby" To: "VBA" Sent: Saturday, April 19, 2008 10:05 AM Subject: [dba-VB] NetGender - The Software Company >I purchased a .Net library that does name splitting. > > http://www.softwarecompany.com/dotnet/netgender.htm > > It seems to work well just from using the little demo that you can > download. Now I am starting to actually work with it. It consists of a > single class, embedded in a dll that you reference, with properties to > set to tell it all of the controlling stuff, then properties to hand it > the name to split and read back the name pieces. > > The developer claims that it will handle between 2K and 3K names per > second. We shall see. > > So here I am trying to work with this thing. The demo form is quite > extensive and the code in the form shows how to read / write all of the > properties to populate the form. > > I have a database with (of course) about 80 million names, in a single > field, in the form LastName FirstName MiddleInitial (no commas etc) > which needs parsing. What I thought I would do (just for the moment) is > to use the existing form and just feed in the names. Thus I need to add > a couple of properties to the form such that I can feed in names and get > back the parsed pieces. I will build a switch to tell the form to > display the pieces or not, thus I could watch the process function if I > wanted, though at the cost of slowing the process down. > > Modifications to the demo form to allow me to do all that stuff is just > about complete. What I need to know is the syntax to get a pointer to a > form being opened. I know that you just do a "Someform.Show" to open > the form, but a parent form needs to get a pointer to this demo form and > then use that pointer to set / retrieve properties. How do I get a > pointer to it once opened? > > -- > John W. Colby > www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From max.wanadoo at gmail.com Sun Apr 20 07:32:53 2008 From: max.wanadoo at gmail.com (Max Wanadoo) Date: Sun, 20 Apr 2008 13:32:53 +0100 Subject: [dba-VB] Splitting a name In-Reply-To: <480A18ED.6020009@colbyconsulting.com> References: <480A18ED.6020009@colbyconsulting.com> Message-ID: <002101c8a2e2$a84d3580$8119fea9@LTVM> John: No doubt there are many more cleverer answers then this and this may be tooooo slow for your needs. Function jc() ' will take the 1st series up to space as Xname, right series back to space as zname and bit in the middle as yname Const conQuote As String = """" Dim sql As String sql = "update tblTest set " sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & conQuote & ")-1) ," sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") - InStr(searchname, " & conQuote & " " & conQuote & "))," sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " & conQuote & ")+1)" CurrentDb.Execute (sql) End Function Max -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 19, 2008 5:08 PM To: VBA Subject: [dba-VB] Splitting a name I have a table with eight million records, with a single name field, and I have a library that splits the name into components. What I need now is a generic method of getting the name out of a table and updating a specific set of fields back in the table. It has to do so QUICKLY since I have to update 80 million names. In this case I have a single table Name NamePrefix LastName FirstName MiddleName NameSuffix Name already exists and contains a name in a consistent format LName FName MName. The other fields I have added to my table. So what I am looking for is a general strategy. I assume I will open a read recordset with the PKID and the name field, then iterate through this recordset performing the split. Once I do that should I: Build a dynamic SQL Update query and execute that? Build a stored procedure that I pass the PKID and the name parts to which then updates the data? Use an ADO recordset to do this? some other method entirely? I am assuming at this point that because of the sheer size of the data I do not want to try and do this in an ado recordset. Eight million records sitting in data at one time does not sound fun. All ideas gratefully accepted. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 20 10:29:34 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 20 Apr 2008 11:29:34 -0400 Subject: [dba-VB] Splitting a name In-Reply-To: <002101c8a2e2$a84d3580$8119fea9@LTVM> References: <480A18ED.6020009@colbyconsulting.com> <002101c8a2e2$a84d3580$8119fea9@LTVM> Message-ID: <480B615E.5040903@colbyconsulting.com> Max, The problems with names is that there are many variations and you cannot count on simple rules like that. For example Mr. Juan Miguel de la Uariate. Now you get another list with De la Uriarte Juan Miguel Sr. You get the picture I purchased a library to do the split. The library performs the split nicely, handling such variations quite well. What I need now is a strategy for reading rows or groups of rows, updating a single record at a time back to SQL Server. In the absence of experience I am going to open a chunk of records,, perhaps 10K, with a startPK and EndPK to track the chunks into an ADO recordset. I will read in 10K records (or some chunk size), then just iterate that recordset splitting each name. My first attempt will be to dynamically build up an update query to write EACH RECORD back out, one at a time. I am almost certain that this will not be fast enough but we shall see. Once I have the code I will probably move the sql into a udf and call the udf passing the name parts in to the udf which will then write one record back to the table. Also likely not fast enough but we shall see. Last will be an attempt to build an ado recordset in memory, then have ADO write the entire recordset back to SQL Server. If that is not fast enough I have run out of ideas. Max Wanadoo wrote: > John: > No doubt there are many more cleverer answers then this and this may be > tooooo slow for your needs. > > Function jc() > ' will take the 1st series up to space as Xname, right series back to > space as zname and bit in the middle as yname > Const conQuote As String = """" > Dim sql As String > sql = "update tblTest set " > sql = sql & "Xname = left(Searchname,instr(Searchname," & conQuote & " " & > conQuote & ")-1) ," > sql = sql & "Yname = Mid(searchname, InStr(searchname, " & conQuote & " " > & conQuote & ") + 1, InStrRev(searchname, " & conQuote & " " & conQuote & ") > - InStr(searchname, " & conQuote & " " & conQuote & "))," > sql = sql & "Zname = mid(Searchname,instrrev(Searchname," & conQuote & " " > & conQuote & ")+1)" > CurrentDb.Execute (sql) > End Function > > Max > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, April 19, 2008 5:08 PM > To: VBA > Subject: [dba-VB] Splitting a name > > I have a table with eight million records, with a single name field, and I > have a library that splits the name into components. What I need now is a > generic method of getting the name out of a table and updating a specific > set of fields back in the table. It has to do so QUICKLY since I have to > update 80 million names. > > In this case I have a single table > > Name > > NamePrefix > LastName > FirstName > MiddleName > NameSuffix > > Name already exists and contains a name in a consistent format LName FName > MName. > > The other fields I have added to my table. > > So what I am looking for is a general strategy. I assume I will open a read > recordset with the PKID and the name field, then iterate through this > recordset performing the split. > > Once I do that should I: > > Build a dynamic SQL Update query and execute that? > > Build a stored procedure that I pass the PKID and the name parts to which > then updates the data? > > Use an ADO recordset to do this? > > some other method entirely? > > I am assuming at this point that because of the sheer size of the data I do > not want to try and do this in an ado recordset. Eight million records > sitting in data at one time does not sound fun. > > All ideas gratefully accepted. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Johncliviger at aol.com Tue Apr 22 04:40:48 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 05:40:48 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Gustav at cactus.dk Tue Apr 22 05:57:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 12:57:51 +0200 Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi John For #3 I would use a DataTableAdapter to attach the table of the database - and then feed your data to a DataTable of the DataTableAdapter. /gustav >>> Johncliviger at aol.com 22-04-2008 11:40 >>> Hi All As part of my teach yourself vb.net 2005/08 project I'm trying to import a .txt file(with white space between columns) into an Array, clean it up and dump it into a SQL express table all nicely cut up with the appropiate datatypes. So far I've got my vb project to read the file contents into an ArrayList and using Console.Writeline() I get a pretty display of data in columns in the output window. I'm now geting an attack of self doubt that my approach is correct. Heres my thinking. 1) Build a Class to read a xxxxx.txt file into an Array (1D array) using MS.VB.FileIO.TextFieldParser(file path and name). 2) Then loop thru the Array cutting up the string using String.Split and assigning a datatype as we go thru the loop. 3) Dump each new field into a Dataset. Run and Append query to populate the SQL table. I'd be grateful for any comments as to my modus operandi. Regards johnc From Johncliviger at aol.com Tue Apr 22 06:05:04 2008 From: Johncliviger at aol.com (Johncliviger at aol.com) Date: Tue, 22 Apr 2008 07:05:04 EDT Subject: [dba-VB] Moving aText file into SQL 2005 Express Message-ID: Hi Gustav Yes. TableAdapter. I'm on the right lines then, which good news. Cheers gustav From Gustav at cactus.dk Tue Apr 22 10:51:23 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 17:51:23 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 11:10:18 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 22 Apr 2008 18:10:18 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all I located this article which will be of interest for those with an Access background: Flat-MultiColumn Combobox with Autocomplete http://www.codeproject.com/KB/architecture/MultiColumnFlatCombo.aspx /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? How do others handle this? /gustav From Gustav at cactus.dk Tue Apr 22 17:06:50 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 00:06:50 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From cfoust at infostatsystems.com Tue Apr 22 17:55:36 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Tue, 22 Apr 2008 15:55:36 -0700 Subject: [dba-VB] No flat ComboBox in .Net?? In-Reply-To: References: Message-ID: We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Wed Apr 23 01:26:27 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 08:26:27 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Wed Apr 23 16:31:09 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Wed, 23 Apr 2008 23:31:09 +0200 Subject: [dba-VB] Custom Access 2007 ribbons with Visual Studio 2008 Message-ID: Hi all Noticed this tip from FMS: Create Custom Microsoft Office 2007 Ribbons with the New Visual Designer in Visual Studio 2008 http://www.fmsinc.com/free/NewTips/NET/CreateCustomOffice2007Ribbons.asp With the new release of Visual Studio Tools for Office (VSTO) in Visual Studio 2008, you can now use a visual designer to create your own custom Ribbons in Microsoft Access, Excel, Word, Outlook, etc. These Ribbons are encapsulated as add-ins for each of the specific Office 2007 products. If you wanted to build your own custom Ribbons, but have held back by the lack of a visual designer, take a look at Visual Studio 2008. The visual designer is available under the New Project, Office, Office 2007 dialog. More resources: http://msdn2.microsoft.com/en-us/library/bb386089.aspx /gustav From mmattys at rochester.rr.com Wed Apr 23 19:58:58 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Wed, 23 Apr 2008 20:58:58 -0400 Subject: [dba-VB] No flat ComboBox in .Net?? References: Message-ID: <01c801c8a5a6$6201d2e0$0502a8c0@Laptop> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" To: Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From Gustav at cactus.dk Thu Apr 24 02:31:43 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:31:43 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi Michael That's a nice tip. Thanks. Lots of other stuff on that site ... /gustav >>> mmattys at rochester.rr.com 24-04-2008 02:58 >>> Hi Gustav, You might be able to manipulate the borderstyle using defined constants in the manner described here: http://www.bobpowell.net/addborder.htm Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "Gustav Brock" < Gustav at cactus.dk > To: < dba-vb at databaseadvisors.com > Sent: Tuesday, April 22, 2008 6:06 PM Subject: Re: [dba-VB] No flat ComboBox in .Net?? > Hi all > > No suggestions? > > I've experimented with a DataGridView of one cell as a combobox, no > headers, no scrollbars. > Looks OK but what a deroute ... I can't believe this. > > /gustav > >>>> Gustav at cactus.dk 22-04-2008 17:51 >>> > Hi All > > Where is the BorderStyle property of the ComboBox? > It can't be true that you need custom controls if you wish a flat XP style > ComboBox? From Gustav at cactus.dk Thu Apr 24 02:38:08 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 24 Apr 2008 09:38:08 +0200 Subject: [dba-VB] GDI+ graphics in .Net Message-ID: Hi all This site of Bob Powell which Michael pointed out has some very good explanation and code examples on pictures, graphics and drawing in .Net: http://www.bobpowell.net/faqmain.htm He also has a very good introduction. That info can be hard to find around the web. Remember: Graphics are fun! /gustav From jwcolby at colbyconsulting.com Fri Apr 25 11:31:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 12:31:25 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: <4812075D.6000306@colbyconsulting.com> OK, I have been badgered and goaded into coming into the 21st century. Not to mention I can't figure out how to safely and effectively share development of my .Net programs on all of my machines. So I am trying to get Subversion installed and functioning. So of course I come here seeking advice or to start a user group (goad and prod you guys) if no one is using this thing. I installed VisualSVN and it told me I had to install TortoiseSVN to get full functionality. It implied that it would install Subversion as well. The install did not ask any "install" questions such as where do you want the database to go etc. My intention is to place the Subversion database on one of my servers, share the directory, then place the various clients (TortoiseSVN and VisualSVN on each machine which I develop on. So my first question, is anyone out there using this thing? Is anyone interested in using this and sharing the pain of figuring it out? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Fri Apr 25 12:25:08 2008 From: robert at webedb.com (Robert L. Stewart) Date: Fri, 25 Apr 2008 12:25:08 -0500 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <200804251729.m3PHTFbD020137@databaseadvisors.com> Been through the pain and suffering already. I much prefer Sourcesafe. This was the tool used on one of the contracts I was on. They were always having problems with it. It is open source, so there was not what you would really call technical support. Good luck. At 12:00 PM 4/25/2008, you wrote: >Date: Fri, 25 Apr 2008 12:31:25 -0400 >From: jwcolby >Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >To: VBA >Message-ID: <4812075D.6000306 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > >OK, I have been badgered and goaded into coming into the 21st century. >Not to mention I can't figure out how to safely and effectively share >development of my .Net programs on all of my machines. > >So I am trying to get Subversion installed and functioning. So of >course I come here seeking advice or to start a user group (goad and >prod you guys) if no one is using this thing. > >I installed VisualSVN and it told me I had to install TortoiseSVN to get >full functionality. It implied that it would install Subversion as >well. The install did not ask any "install" questions such as where do >you want the database to go etc. > >My intention is to place the Subversion database on one of my servers, >share the directory, then place the various clients (TortoiseSVN and >VisualSVN on each machine which I develop on. > >So my first question, is anyone out there using this thing? Is anyone >interested in using this and sharing the pain of figuring it out? > >-- >John W. Colby >www.ColbyConsulting.com > > >------------------------------ > >_______________________________________________ >dba-VB mailing list >dba-VB at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-vb > > >End of dba-VB Digest, Vol 54, Issue 18 >************************************** From jwcolby at colbyconsulting.com Fri Apr 25 12:41:23 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 13:41:23 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <200804251729.m3PHTFbD020137@databaseadvisors.com> References: <200804251729.m3PHTFbD020137@databaseadvisors.com> Message-ID: <481217C3.7020105@colbyconsulting.com> Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 13:27:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Fri, 25 Apr 2008 22:27:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481217C3.7020105@colbyconsulting.com> Message-ID: <000901c8a701$f4afd8b0$6401a8c0@nant> Hi John, I did use Subversion for some time, then I abandoned it because customer didn't insist on using any source control tools, and because I'm still working mainly as one man shop... ...I do plan to restart using Subversion hopefully in the near future - I wanted to be 'in' in this discussion and participate or at least lurk it because of a lot of work here, and not that much time left for pleasure of AccessD discussions... ...I did also use TortoiseSVN... ...I do agree that MS support is usually useless... ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as well as I do plan to add to that "bundle" MS SoundCastle, NCover and some other useful tools... ...one or another bug/issues/features tracking/managing tools should be used - I do use Axosoft's OnTiome2008 now... ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS Build + NUnit working together and making automated builds and (unit) test runs - that's the only way to go for teamwork development these days... ...all that mentioned above stuff worked well here but I must admit it was not that easy to make the parts working in ensemble but when everything was tuned it happened to be relatively easy to handle stuff... ...I have had two large ASP.NET applications with probably several (1,2,3 ? I didn't count) thousands of source files and about 15 projects in two solutions to put into Subversion code base, automate compilation/building using CruiseCOntrol.NET and MS Build etc... ...the only issue I can remember (after everything was set) is that I didn't find how to easily manage the cases to synchronize central code db with local code versions when source files are moved from one folder into another within the same project - I could have missed something because of lack of time, and experience or that could be a real "bottleneck" of Subversion with everything else being very good I think... ...I'd participate in preparing and publishing on AccessD web site together with other interested AccessD members an instruction/manual and set of scripts and utilities on how to set up and effectively use together Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say how much time I will have for this work... Thank you. -- Shamil P.S. As far as I know Subversion "beats" MS Source Safe for the cases when the project development should be fork-ed, and then several branches merged back: the difference between Subversion and MS SourceSafe is conceptual here IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I can be wrong, please correct me then, thank you. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Thanks Robert. I find we prefer what we are familiar with. Since I am not familiar with anything I can make Subversion what I am familiar with. ;-) What I have also found is that Microsoft's "technical support" is about as useless as tits on the proverbial boar, so the fact that there is no "tech support" seems irrelevant. In the end we end up going to a forum somewhere to get support. Thanks for your input though. We shall see if anyone out there is using (or wants to use) Subversion. Robert L. Stewart wrote: > Been through the pain and suffering already. > I much prefer Sourcesafe. > > This was the tool used on one of the contracts I was on. > They were always having problems with it. > It is open source, so there was not what you would really > call technical support. > > Good luck. > > At 12:00 PM 4/25/2008, you wrote: >> Date: Fri, 25 Apr 2008 12:31:25 -0400 >> From: jwcolby >> Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN >> To: VBA >> Message-ID: <4812075D.6000306 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> OK, I have been badgered and goaded into coming into the 21st century. >> Not to mention I can't figure out how to safely and effectively share >> development of my .Net programs on all of my machines. >> >> So I am trying to get Subversion installed and functioning. So of >> course I come here seeking advice or to start a user group (goad and >> prod you guys) if no one is using this thing. >> >> I installed VisualSVN and it told me I had to install TortoiseSVN to get >> full functionality. It implied that it would install Subversion as >> well. The install did not ask any "install" questions such as where do >> you want the database to go etc. >> >> My intention is to place the Subversion database on one of my servers, >> share the directory, then place the various clients (TortoiseSVN and >> VisualSVN on each machine which I develop on. >> >> So my first question, is anyone out there using this thing? Is anyone >> interested in using this and sharing the pain of figuring it out? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> >> >> ------------------------------ >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> >> >> End of dba-VB Digest, Vol 54, Issue 18 >> ************************************** > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Fri Apr 25 14:35:03 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 15:35:03 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000901c8a701$f4afd8b0$6401a8c0@nant> References: <000901c8a701$f4afd8b0$6401a8c0@nant> Message-ID: <48123267.7030608@colbyconsulting.com> Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Apr 25 15:06:31 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 16:06:31 -0400 Subject: [dba-VB] Reading / writing SQL Server rows with ADO.Net (or something) Message-ID: <481239C7.2030506@colbyconsulting.com> I need to read blocks of data from tables in SQL Server, use values in the rows then update those rows and write back to SQL Server. I need to do this programmatically, not in a control on a form. does anyone have any example code for setting up the various ADO.Net objects, fetching rows of data from SQL Server, updating that data and writing it back to SQL Server? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:20:01 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:20:01 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000001c8a711$bdda92c0$6401a8c0@nant> John, Everything in Subversion can be controlled by using batch files - that is the best approach (IMO) for the initial setup, bulk checking and working set checkout: if I do not miss anything then I will post in the following several e-mails under this topic the set of .bat files I used to create the test environment last autumn(three projects ProjectA, ProjectB and ProjectC). Stay tuned (after posting I will be out here until Sunday). Thanks. -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Friday, April 25, 2008 11:35 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Shamil, I am also a "one man shop" however I still want a version control system. One thing that it does for me is to allow me to do development on any of my machines and not worry about keeping the sources in sync. For example, a lot of time I just want to use my laptop, but the files are in the c: drive for that. On Stonehenge they are on the C: drive there. In Azul they are on the C: drive there. Which version is the latest? I would like to be able to work wherever I want but without a VCS that is tough. I have a server that is x64 with more memory and a quad core, but I wrote it all on a different machine. Trying to get it onto that more powerful machine has been fun. If I can get it to just "check in" and "check out" then I can go to that machine and check it out and start running. Shamil Salakhetdinov wrote: > Hi John, > > I did use Subversion for some time, then I abandoned it because customer > didn't insist on using any source control tools, and because I'm still > working mainly as one man shop... > > ...I do plan to restart using Subversion hopefully in the near future - I > wanted to be 'in' in this discussion and participate or at least lurk it > because of a lot of work here, and not that much time left for pleasure of > AccessD discussions... > > ...I did also use TortoiseSVN... > > ...I do agree that MS support is usually useless... > > ...I did also use CruiseControl.NET as well as NUnit as well as MS Build as > well as I do plan to add to that "bundle" MS SoundCastle, NCover and some > other useful tools... > > ...one or another bug/issues/features tracking/managing tools should be used > - I do use Axosoft's OnTiome2008 now... > > ...I did manage (last autumn) to make CruiseCOntrol.NET + Subversion + MS > Build + NUnit working together and making automated builds and (unit) test > runs - that's the only way to go for teamwork development these days... > > ...all that mentioned above stuff worked well here but I must admit it was > not that easy to make the parts working in ensemble but when everything was > tuned it happened to be relatively easy to handle stuff... > > ...I have had two large ASP.NET applications with probably several (1,2,3 ? > I didn't count) thousands of source files and about 15 projects in two > solutions to put into Subversion code base, automate compilation/building > using CruiseCOntrol.NET and MS Build etc... > > ...the only issue I can remember (after everything was set) is that I didn't > find how to easily manage the cases to synchronize central code db with > local code versions when source files are moved from one folder into another > within the same project - I could have missed something because of lack of > time, and experience or that could be a real "bottleneck" of Subversion with > everything else being very good I think... > > ...I'd participate in preparing and publishing on AccessD web site together > with other interested AccessD members an instruction/manual and set of > scripts and utilities on how to set up and effectively use together > Subversion, CruiseCOntrol.NET, MS Build, NUnit etc. - although I can't say > how much time I will have for this work... > > Thank you. > > -- > Shamil > > P.S. As far as I know Subversion "beats" MS Source Safe for the cases when > the project development should be fork-ed, and then several branches merged > back: the difference between Subversion and MS SourceSafe is conceptual here > IOW MS SourceSafe doesn't allow to automate "project forks" in principle. I > can be wrong, please correct me then, thank you. > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Fri Apr 25 15:38:38 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sat, 26 Apr 2008 00:38:38 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48123267.7030608@colbyconsulting.com> Message-ID: <000401c8a714$5762dae0$6401a8c0@nant> Hi John, Below is the set of .batch files I have just found, which I did make last autumn to control/direct subversion server. I must say I'm not sure this is the last working version I used (AFAIKR I did write some simple custom code in .NET to control subversion server to avoid using .bat files, which have to be made for every project you'll use - and my simple custom code was parameterized to use any source project) but as you can see the set of batch files below is the minimal set to fulfill all the source control main tasks. Watch line wraps and read subversion docs for more details - svn-book.pdf - located in Sunversion server's installation folder (I'm out until Sunday). Thanks. -- Shamil P.S. .bat files to use with Subversion !svn_admin_main.bat ===================== rem Port: 3690 rem RepositoryPath: e:\Subversion\svn_repository rem X Install svnserve to run as a Windows service rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server call make_dirs.bat new_repository call check_out.bat new_repository pause make_dirs.bat ============== rmdir tmpdir /s /q mkdir tmpdir mkdir tmpdir\projectA mkdir tmpdir\projectA\trunk mkdir tmpdir\projectA\branches mkdir tmpdir\projectA\tags mkdir tmpdir\projectB mkdir tmpdir\projectB\trunk mkdir tmpdir\projectB\branches mkdir tmpdir\projectB\tags mkdir tmpdir\projectC mkdir tmpdir\projectC\trunk mkdir tmpdir\projectC\branches mkdir tmpdir\projectC\tags rem xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E rem rmdir /s /q ..\%1 call create_fs_repository.bat %1 rem svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log call verify_fs_repository.bat %1 call svnlook_fs_repository.bat %1 rmdir tmpdir /s /q rem rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA --message svnlog.log create_fs_repository.bat ======================== svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs xcopy ..\args\conf\*.* ..\%1\conf /Y /E goto :EOF verify_fs_repository.bat ======================== svnadmin verify ../%1 GOTO :EOF svnlook_fs_repository.bat ========================= svnlook info ../%1 goto :EOF check_out.bat ============= rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S mkdir E:\Temp\SV\WORKING_SET\ /Q /S svn checkout file:///E:/Subversion/%1/projectA/trunk E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log svn checkout file:///E:/Subversion/%1/projectB/trunk E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log svn checkout file:///E:/Subversion/%1/projectC/trunk E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log commit.bat ========== svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log backup_fs_repository.bat ======================== svnadmin hotcopy ../%1 ../backups/%1 GOTO :EOF From jwcolby at colbyconsulting.com Fri Apr 25 17:15:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 25 Apr 2008 18:15:27 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <000401c8a714$5762dae0$6401a8c0@nant> References: <000401c8a714$5762dae0$6401a8c0@nant> Message-ID: <481257FF.30208@colbyconsulting.com> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Sat Apr 26 06:52:35 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 13:52:35 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil I would love to follow this thread. I work both at home and at the office and is a bit tired of copying files around. I located this page: Source Control with Visual Studio .NET http://www.codeproject.com/KB/dotnet/SourceControl_VSNET.aspx?fid=471385&df=90&mpp=25&noise=3&sort=Position&view=Quick&fr=26#xx0xx which, however, seems to know nothing about VS2008. It has links to these add-ins for VS: Garry Broadsword's Add-on which is updated for VS2008: http://garrys-brain.blogspot.com/2007/07/tortoisesvn-and-visual-studio.html and VisualSVN: http://www.visualsvn.com/ This site also sports VisualSVN Server: http://www.visualsvn.com/server/ VisualSVN Server is a package that contains everything you need to install, configure and manage Subversion server for your team on Windows platform. It includes Subversion, Apache and a management console. ... And last, but not least - VisualSVN Server is completely free! This must be what John is after - and myself as neither I nor any colleague have much experience with Apache. Now, big question: Does this replace or supplement Shamil's collection of batch files? As I understand it, VisualSVN Server should take care of the setup of Subversion but I must admit I have not yet grasped all the bits and pieces of this ... A final note for others that may follow this thread: Nothing of this works with the free Visual Studio Express versions. /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil From Gustav at cactus.dk Sat Apr 26 07:04:22 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sat, 26 Apr 2008 14:04:22 +0200 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN Message-ID: Hi John and Shamil Exciting. http://www.visualsvn.com/doc/getting-started.html Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? Have never heard of Google Code. Can I use Google to store a project? http://code.google.com/ How does that work and is it only for public access project? Or how would you control access for closed source projects? /gustav >>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. From jwcolby at colbyconsulting.com Sat Apr 26 07:11:54 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 08:11:54 -0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: References: Message-ID: <48131C0A.1000302@colbyconsulting.com> the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Apr 26 14:25:56 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 15:25:56 -0400 Subject: [dba-VB] ADO.Net Message-ID: <481381C4.7000804@colbyconsulting.com> I have been setting up an application to read a block of records, perhaps 10K or something. I designed a class that will load a block of records which consist of a name to be parsed, and a set of 6 fields which will be filled in with the parsed data returned from the name parser. Thus as I load the data only the name to parse is filled in. The supervisor iterates this set of records, pulling the name to parse, passing it off to the name parser, then reading the data back from the name parser and updating the fields in the same record. Move to the next record, wash, rinse repeat. I do have a couple of questions though. I am not very familiar with how the ADO.net stuff works. I build a connection string I build a connection object which I leave not connected (for now), but feeding it the connection string. I build a sql statement that represents the fields to pull WHERE PK between StartValue and EndValue. I build a command object which I pass SQL statement and the still unconnected connection object. I build a data adapter which I feed the command object. Notice that we are still not connected to the data store. I build a DataTable object with a table name. I call DataAdapter(DataTable).Fill Does .Net then connect correctly and proceed to fill the data table? Once I have the data table filled I start to iterate the DataTable using the for each DataRow iterator I then call the name splitter passing in the name to be split. Back comes the name pieces. I fill in the row with the name pieces and move to the next row. Once every row has been filled in I need to call an update method of the DataAdapter which is SUPPOSED to write the changes back to the table in SQL Server..... The name splitter handles ~1K names / second, NOT as speedy as I had hoped. At this point I haven't a clue how fast the data adapter will write back to the table in SQL Server. What I am trying to do is write this thing such that each instance of the data class can then be handed off to a thread so that the write back to SQL Server can begin and I can instantiate another data class to load / parse the next "chunk" of data in parallel with the write to SQL Server. I am trying to discover whether I need to close connections to the database while I do the parsing or just not worry about it. Should I just leave all the objects open and close them all down as the data class closes once the write to SQL Server is done? How will I know the write is done? Does the code "hang" at the DataAdapter.Update until the entire update (all the rows) is finished? Am I going about this the right way? Should I become a used car salesman? -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <481381C4.7000804@colbyconsulting.com> Message-ID: <000501c8a7ea$a4d53080$6401a8c0@nant> Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <481257FF.30208@colbyconsulting.com> Message-ID: <000401c8a7ea$a4c949a0$6401a8c0@nant> Hi John, Yes, I'm considering to purchase it also very probably... ...I will wait how well it will work for you - OK :) ? -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 2:15 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN Wow, thanks for that. I found a widget to integrate Subversion directly into Visual Studio. It is called VisualVSN and costs $49 / developer license. I just got the eval but if it works as it says I will probably pay up. Shamil Salakhetdinov wrote: > Hi John, > > Below is the set of .batch files I have just found, which I did make last > autumn to control/direct subversion server. I must say I'm not sure this is > the last working version I used (AFAIKR I did write some simple custom code > in .NET to control subversion server to avoid using .bat files, which have > to be made for every project you'll use - and my simple custom code was > parameterized to use any source project) but as you can see the set of batch > files below is the minimal set to fulfill all the source control main tasks. > Watch line wraps and read subversion docs for more details - svn-book.pdf - > located in Sunversion server's installation folder (I'm out until Sunday). > Thanks. > > -- > Shamil > > P.S. .bat files to use with Subversion > > !svn_admin_main.bat > ===================== > rem Port: 3690 > rem RepositoryPath: e:\Subversion\svn_repository > rem X Install svnserve to run as a Windows service > rem ProgramSetupPath: D:\Program Files\CollabNet Subversion Server > > call make_dirs.bat new_repository > call check_out.bat new_repository > pause > > make_dirs.bat > ============== > rmdir tmpdir /s /q > mkdir tmpdir > mkdir tmpdir\projectA > mkdir tmpdir\projectA\trunk > mkdir tmpdir\projectA\branches > mkdir tmpdir\projectA\tags > mkdir tmpdir\projectB > mkdir tmpdir\projectB\trunk > mkdir tmpdir\projectB\branches > mkdir tmpdir\projectB\tags > mkdir tmpdir\projectC > mkdir tmpdir\projectC\trunk > mkdir tmpdir\projectC\branches > mkdir tmpdir\projectC\tags > rem > xcopy e:\temp\SV\SRC\ProjectA\*.* tmpdir\projectA\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectB\*.* tmpdir\projectB\trunk /Y /E > xcopy e:\temp\SV\SRC\ProjectC\*.* tmpdir\projectC\trunk /Y /E > rem > rmdir /s /q ..\%1 > call create_fs_repository.bat %1 > rem > svn import tmpdir file:///e:/Subversion/%1 --message svnlog.log > call verify_fs_repository.bat %1 > call svnlook_fs_repository.bat %1 > rmdir tmpdir /s /q > rem > rem svn import file:///e:/temp/sv/projecta file:///e:/Subversion/%1/projectA > --message svnlog.log > > create_fs_repository.bat > ======================== > svnadmin create ../%1 --config-dir ../args/conf --fs-type fsfs > xcopy ..\args\conf\*.* ..\%1\conf /Y /E > goto :EOF > > verify_fs_repository.bat > ======================== > svnadmin verify ../%1 > GOTO :EOF > > svnlook_fs_repository.bat > ========================= > svnlook info ../%1 > goto :EOF > > > check_out.bat > ============= > rem rmdir E:\Temp\SV\WORKING_SET\ /Q /S > mkdir E:\Temp\SV\WORKING_SET\ /Q /S > svn checkout file:///E:/Subversion/%1/projectA/trunk > E:\Temp\SV\WORKING_SET\ProjectA --message checkout.log > svn checkout file:///E:/Subversion/%1/projectB/trunk > E:\Temp\SV\WORKING_SET\ProjectB --message checkout.log > svn checkout file:///E:/Subversion/%1/projectC/trunk > E:\Temp\SV\WORKING_SET\ProjectC --message checkout.log > > commit.bat > ========== > svn commit E:\Temp\SV\WORKING_SET\ProjectA --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectB --message commit.log > svn commit E:\Temp\SV\WORKING_SET\ProjectC --message commit.log > > backup_fs_repository.bat > ======================== > svnadmin hotcopy ../%1 ../backups/%1 > GOTO :EOF > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:12:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:12:40 +0400 Subject: [dba-VB] Subversion, TortoiseSVN and VisualSVN In-Reply-To: <48131C0A.1000302@colbyconsulting.com> Message-ID: <000301c8a7ea$a4af0ae0$6401a8c0@nant> <<< running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever >>> John, The batches I posted here are just to work with Subversion's repository: create it, check-in, check-out, backup... ...to run builds and automate unit testing etc. CruiseControl.NET has to be used, or other similar tools... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 4:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Subversion, TortoiseSVN and VisualSVN the VSN server itself is completely free. However the clients cost, though the cost is reasonable at $49 for a single developer like me. And if you are working on a public domain project they will give you a license for free. As for Shamil's batch files, I suspect that his batch files does more, checking out files, running makes on each and every piece of the project, and whatever else you might want, regression testing or whatever. The integrated widget just checks things out and back in for working inside of Visual Studio. Gustav Brock wrote: > Hi John and Shamil > > Exciting. > http://www.visualsvn.com/doc/getting-started.html > > > Please note that when adding your code to an existing repository you can choose any kind of remote ones. For example, you can easily add your solution to a hosted repository provided by services like Google Code. Just copy a repository URL to the corresponding field in the dialog. > > > What is that supposed to mean? What is a remote repository? I know the words, of course, but is it just a remote Apache server hooked up with Subversion? > Have never heard of Google Code. Can I use Google to store a project? > > http://code.google.com/ > > How does that work and is it only for public access project? Or how would you control access for closed source projects? > > /gustav > >>>> jwcolby at colbyconsulting.com 26-04-2008 00:15 >>> > Wow, thanks for that. > > I found a widget to integrate Subversion directly into Visual Studio. > It is called VisualVSN and costs $49 / developer license. I just got > the eval but if it works as it says I will probably pay up. > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sat Apr 26 17:39:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 02:39:45 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <000601c8a7ee$6d9119a0$6401a8c0@nant> Hi John, Additionally to the below e-mail - you can consider using SQLDataReader to get data records batches as in this below example, then build parsed names' arrays together with ids, then Join() these arrays into strings and pass these strings as parameters to SPs, which will make actual updates by parsing strings into dynamic update sqls or something like that - T-SQL parsing functions are reasonably quick, or you can even consider managed SPs to parse the strings on server side and perform the updates... ...but that latter could be an overkill giving very little gains because .NET framework's communication with native MS SQL provider when using plain simple C#/VB.NET is very effective I have heard... ..maybe you can just process your data and generate and run update temp SPs to have update T-SQL statements batched into such temp SPs - that could give you (much) quicker updates... ...etc... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 2:13 AM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Try to use SqlDataReader and custom classes as in the sample code below (sorry C# is used - real pleasure to use C# is also coming from the fact that you can copy and paste code and do not care about line warps because even "screwed by e-mail" code should be compiled well).... ...using ADO.NET datasets and bulk update will anyway result in a series of updates on SQL server side therefore custom on-the-fly built sql updates or calls to update SPs should work as quick as ADO.NET DataSets' batch update... ...etc... ...as you can see my sample did update 25000 records in ~10 seconds - and this only for starters - if that approach will work reasonably well for you then the next step could be to introduce multi-threading etc... ...note also how MS SQL 2005's paging feature is used to get record's batches... using System; using System.Collections.Generic; using System.Data.SqlClient; namespace ORS_TDD_Console { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { get { return Name; // just for testing purposes // return real parsed name here } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader( System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { list.Add (new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID ); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { //27.04.2008 1:53:30 //Count(*) = 25045 //Counter = 25045 //27.04.2008 1:53:40 public static void Main() { int count = 0; try { //+ customize - set your connection string here string connectionString = AppConfigManager.DefaultConnectionString; //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count++; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, April 26, 2008 11:26 PM To: VBA Subject: [dba-VB] ADO.Net I have been setting up an application to read a block of records, perhaps 10K or something. <<< the rest trimmed to get message through accessd >>> _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 18:31:32 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 19:31:32 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000501c8a7ea$a4d53080$6401a8c0@nant> References: <000501c8a7ea$a4d53080$6401a8c0@nant> Message-ID: <4813BB54.4090203@colbyconsulting.com> Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sat Apr 26 18:47:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 03:47:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813BB54.4090203@colbyconsulting.com> Message-ID: <000001c8a7f7$e871c530$6401a8c0@nant> <<< I am definitely interested in doing strongly typed recordsets though. >>> John, That strongly typed recordsets are imposing big overhead - you can dig into the code generated (by VS) for these recordsets... ...as for batch updates "automagically" generated by ADO.Net - you can use SQL profiler to see what happens "under the hood"... ...I'd bet that if you generate batch updates by custom code into temp SPs and use datareader to get data to build these batches then this approach will result in a way faster data processing, and you will have many options to optimize it even more... ...just my opinion but as you can find in ASP.Net apps ADO.NET typed recordsets are not recommended to use - have a look at e.g. "ASP.NET Website Programming: Problem - Design - Solution C# Edition" by Marco Bellinaso.... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 3:32 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Thanks Shamil, Unfortunately unless you are dealing with 80 million row 200 field tables, any timing comparisons are going to be suspect. I considered doing (and may still do) a record by record update using either a stored procedure or dynamic sql right from vb.net. Then I read an article (blog) that essentially said that the batch update from an ado dataset really did work as you would hope that it would, and this in a blog from a member of Microsoft's ADO.Net dev team. He claimed that either of the other methods had network transaction overhead as each update (when done singly) required the command to do it as well as a result coming back, whereas if you used a batch it would group all of the commands out and then group all the results coming back into big packages. Understand that I do not know enough to even comment on the validity of his argument however he said to definitely try it. Set the batch size and then let ADO just send batches of updates. Given that the coding effort is smaller to do that I figured I would at least get that running. I am definitely interested in doing strongly typed recordsets though. Shamil Salakhetdinov wrote: > Hi John, > > Try to use SqlDataReader and custom classes as in the sample code below > (sorry C# is used - real pleasure to use C# is also coming from the fact > that you can copy and paste code and do not care about line warps because > even "screwed by e-mail" code should be compiled well).... > > ...using ADO.NET datasets and bulk update will anyway result in a series of > updates on SQL server side therefore custom on-the-fly built sql updates or > calls to update SPs should work as quick as ADO.NET DataSets' batch > update... > > ...etc... > > ...as you can see my sample did update 25000 records in ~10 seconds - and > this only for starters - if that approach will work reasonably well for you > then the next step could be to introduce multi-threading etc... > > ...note also how MS SQL 2005's paging feature is used to get record's > batches... -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Apr 26 19:45:47 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 26 Apr 2008 20:45:47 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a7f7$e871c530$6401a8c0@nant> References: <000001c8a7f7$e871c530$6401a8c0@nant> Message-ID: <4813CCBB.3060907@colbyconsulting.com> This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx There are many things to consider when I program in .net, not the least of which is simplicity. If I can get a simple method that gets me medium efficiency, it is better (for me) than a complex method that gets high efficiency simply because I may be able to implement the simple method whereas the complex method would never get done. I can always come back later and recode for a higher efficiency after I have a method working and chugging away. As long as the program does not take weeks to run, it is better to spend cpu time than John Colby time. As I mentioned, the name parser is not super fast. I am hoping to get the first database update batch happening in parallel with the second name parsing batch. I also hope to run this on a completely different computer which has more cores (and memory) so that I can use threads effectively. Which is not to say I am not interested in the most efficient method, simply that I will likely be unable to code it very quickly. Shamil Salakhetdinov wrote: > > <<< > I am definitely interested in doing strongly typed recordsets though. > John, > > That strongly typed recordsets are imposing big overhead - you can dig into > the code generated (by VS) for these recordsets... > > ...as for batch updates "automagically" generated by ADO.Net - you can use > SQL profiler to see what happens "under the hood"... > > ...I'd bet that if you generate batch updates by custom code into temp SPs > and use datareader to get data to build these batches then this approach > will result in a way faster data processing, and you will have many options > to optimize it even more... > > ...just my opinion but as you can find in ASP.Net apps ADO.NET typed > recordsets are not recommended to use - have a look at e.g. "ASP.NET Website > Programming: Problem - Design - Solution C# Edition" > by Marco Bellinaso.... > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 3:32 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Thanks Shamil, > > Unfortunately unless you are dealing with 80 million row 200 field > tables, any timing comparisons are going to be suspect. > > I considered doing (and may still do) a record by record update using > either a stored procedure or dynamic sql right from vb.net. Then I read > an article (blog) that essentially said that the batch update from an > ado dataset really did work as you would hope that it would, and this in > a blog from a member of Microsoft's ADO.Net dev team. > > He claimed that either of the other methods had network transaction > overhead as each update (when done singly) required the command to do it > as well as a result coming back, whereas if you used a batch it would > group all of the commands out and then group all the results coming back > into big packages. > > Understand that I do not know enough to even comment on the validity of > his argument however he said to definitely try it. Set the batch size > and then let ADO just send batches of updates. > > Given that the coding effort is smaller to do that I figured I would at > least get that running. > > I am definitely interested in doing strongly typed recordsets though. > > Shamil Salakhetdinov wrote: >> Hi John, >> >> Try to use SqlDataReader and custom classes as in the sample code below >> (sorry C# is used - real pleasure to use C# is also coming from the fact >> that you can copy and paste code and do not care about line warps because >> even "screwed by e-mail" code should be compiled well).... >> >> ...using ADO.NET datasets and bulk update will anyway result in a series > of >> updates on SQL server side therefore custom on-the-fly built sql updates > or >> calls to update SPs should work as quick as ADO.NET DataSets' batch >> update... >> >> ...etc... >> >> ...as you can see my sample did update 25000 records in ~10 seconds - and >> this only for starters - if that approach will work reasonably well for > you >> then the next step could be to introduce multi-threading etc... >> >> ...note also how MS SQL 2005's paging feature is used to get record's >> batches... > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 04:08:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:08:08 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <4813CCBB.3060907@colbyconsulting.com> Message-ID: <000001c8a846$3606c6d0$6401a8c0@nant> Hello John, OK, I will read the blog you mentioned later today... I know it's not easy to convince you to change your mind when you are "driving your way full speed" :) And I'm not trying to - I'm just musing in this coding samples I post here testing this or that approach, which I might use in my own development... <<< There are many things to consider when I program in .net, not the least of which is simplicity. >>> May I say that proposed in C# solution is simple and almost ready to use in your scenario in one thread? And also ready to be converted into a multi-threading app/utility. And you don't need to translate in manually in VB.NET - just use this free tool: http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx or this one: http://codeconverter.sharpdevelop.net/Convert.aspx and voila' the converted VB.NET is below : (watch line wraps now - this is VB.NET - it was "automagically" converted by referred above tool by copying and pasting C# code I posted yesterday + making some manual editing of several code line to make it independent of my testing/development environment) - and it seems to work even quicker than C# equivalent somehow !?. I have got additional back VB.NET -> C# conversion in the end of this message - you can use C# code copy and paste it in the tool refereed above and get VB.NET code properly formatted. Proposal: let's post C# code here for the folks who will try this code to not spend time with fixing line wraps introduced by e-mail clients): Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) ID = id Name = name End Sub Public ReadOnly Property ParsedName() As String Get ' just for testing purposes ' return real parsed name here Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry '27.04.2008 12:45:58 'Count(*) = 25045 'Counter = 25045 '27.04.2008 12:46:04 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = _ String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", _ pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list p.Update(item) count += 1 Next End While Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace Here it's converted back to C#: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB"; string dbName = "dbName"; string userName = "sa"; string password = "sapwd"; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Sunday, April 27, 2008 4:46 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net This is the blog I mentioned: http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx <<< tail skipped to get message through in AccessD >>> From Gustav at cactus.dk Sun Apr 27 04:33:02 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 11:33:02 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From shamil at smsconsulting.spb.ru Sun Apr 27 04:42:37 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 13:42:37 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: Message-ID: <000601c8a84b$0711cdc0$6401a8c0@nant> Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Sun Apr 27 05:53:36 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 14:53:36 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000601c8a84b$0711cdc0$6401a8c0@nant> Message-ID: <000301c8a854$f1c57cf0$6401a8c0@nant> ...now the VB.net version of manually batched updates (watch line wraps!) It seems to work slightly quicker than similar C# version: ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 'http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx 'http://codeconverter.sharpdevelop.net/Convert.aspx Imports System Imports System.Collections.Generic Imports System.Data.SqlClient Namespace TestSamples Public Class DataItem Public ID As Integer Public Name As String Public Sub New(ByVal id As Integer, ByVal name As String) id = id name = name End Sub Public ReadOnly Property ParsedName() As String ' just for testing purposes ' return real parsed name here Get Return Name End Get End Property End Class Public Class DataProcessor Private Const COUNT_SQL As String = "select count(*) from zsysUserSessionTrace" Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}" Private Const UPDATE_SQL As String = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}" Private Const DEFAULT_BATCH_SIZE As Integer = 1000 Private _totalCount As Integer = -1 Public ReadOnly Property TotalCount() As Integer Get Return _totalCount End Get End Property Private _currentBatchStartRowNum As Integer = 0 Private _batchSize As Integer = 0 Public ReadOnly Property BatchSize() As Integer Get Return _batchSize End Get End Property Private _connectionString As String Public Sub New(ByVal connectionString As String) initialize(connectionString, DEFAULT_BATCH_SIZE) End Sub Public Sub New(ByVal connectionString As String, ByVal batchSize As Integer) initialize(connectionString, batchSize) End Sub Private Sub initialize(ByVal connectionString As String, ByVal batchSize As Integer) _batchSize = batchSize _currentBatchStartRowNum = 1 _connectionString = connectionString Using cnn As New System.Data.SqlClient.SqlConnection(connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() cmd.CommandText = COUNT_SQL cmd.CommandType = System.Data.CommandType.Text _totalCount = CInt(cmd.ExecuteScalar()) End Using End Sub Public ReadOnly Property HasData() As Boolean Get Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < TotalCount) End Get End Property Public Function GetData() As List(Of DataItem) Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text _currentBatchStartRowNum += _batchSize + 1 Dim rdr As System.Data.SqlClient.SqlDataReader = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) Dim list As New List(Of DataItem)() While rdr.Read() 'error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), String))) End While Return list End Using End Function Public Function Update(ByVal item As DataItem) As Integer Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function Private _updateCommands As String() = Nothing Private _arrayIndex As Integer = -1 Public Sub AddUpdateCommand(ByVal item As DataItem) If _arrayIndex = -1 Then _updateCommands = DirectCast(Array.CreateInstance(GetType(String), _batchSize), String()) _arrayIndex = 0 End If Dim sql As String = String.Format(UPDATE_SQL, item.ParsedName, item.ID) 'badly translated => _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql _updateCommands(_arrayIndex) = sql _arrayIndex += 1 End Sub Public Function BatchUpdate() As Integer If _arrayIndex = -1 Then Return -1 End If If _arrayIndex < _batchSize Then Array.Resize(Of String)(_updateCommands, _arrayIndex) End If Using cnn As New System.Data.SqlClient.SqlConnection(_connectionString) cnn.Open() Dim cmd As System.Data.SqlClient.SqlCommand = cnn.CreateCommand() 'Console.WriteLine("Batch update: {0} updates", _arrayIndex) Dim sql As String = [String].Join(";", _updateCommands) _arrayIndex = -1 cmd.CommandText = sql cmd.CommandType = System.Data.CommandType.Text Return cmd.ExecuteNonQuery() End Using End Function End Class Public Class A_MainEntry ' VB.NET - individual updates '27.04.2008 13:01:44 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:49 ' C# - individual updates '27.04.2008 13:00:56 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:01:04 ' C# - manually batched updates '27.04.2008 13:37:39 'Count(*) = 25045 'Counter = 25045 '27.04.2008 13:37:42 ' VB.NET - manually batched updates '27.04.2008 14:39:11 'Count(*) = 25046 - changed because I work with this db 'Counter = 25046 '27.04.2008 14:39:13 Public Shared Sub Main() Dim count As Integer = 0 Try '+ customize - set your connection string here Dim pcName As String = "SPB" Dim dbName As String = "dbName" Dim userName As String = "sa" Dim password As String = "sapwd" Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) '- Console.WriteLine(DateTime.Now) Dim p As New DataProcessor(connectionString) While p.HasData Dim list As List(Of DataItem) = p.GetData() For Each item As DataItem In list If count > 0 AndAlso count Mod p.BatchSize = 0 Then p.BatchUpdate() End If 'p.Update(item); p.AddUpdateCommand(item) count += 1 Next End While p.BatchUpdate() Console.WriteLine("Count(*) = {0}", p.TotalCount) Console.WriteLine("Counter = {0}", count) Console.WriteLine(DateTime.Now) Catch ex As Exception Console.WriteLine(ex.Message) End Try End Sub End Class End Namespace -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 1:43 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi Gustav, I do not know what causes this difference, I will try to find it later - yes, it expected to be nearly the same - meanwhile here is another version of manually (by custom code batched updates - works quicker as expected): // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 // coverters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx using System; using System.Collections.Generic; using System.Data.SqlClient; namespace TestSamples { public class DataItem { public int ID; public string Name; public DataItem(int id, string name) { ID = id; Name = name; } public string ParsedName { // just for testing purposes // return real parsed name here get { return Name; } } } public class DataProcessor { private const string COUNT_SQL = "select count(*) from zsysUserSessionTrace"; private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and {1}"; private const string UPDATE_SQL = "update zsysUserSessionTrace set UserBrowser = '{0}' where UserSessionTraceId = {1}"; private const int DEFAULT_BATCH_SIZE = 1000; private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; }} private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(1), (string)rdr(3))); list.Add(new DataItem((int)rdr[1], (string)rdr[3])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";",_updateCommands) ; _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } } public class A_MainEntry { // VB.NET - individual updates //27.04.2008 13:01:44 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:49 // C# - individual updates //27.04.2008 13:00:56 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:01:04 // C# - manually batched updates //27.04.2008 13:37:39 //Count(*) = 25045 //Counter = 25045 //27.04.2008 13:37:42 public static void Main() { int count = 0; try { //+ customize - set your connection string here string pcName = "SPB string dbName = "dbName string userName = "sa"; string password = "sapwd string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- Console.WriteLine(DateTime.Now); DataProcessor p = new DataProcessor(connectionString); while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && count % p.BatchSize == 0) p.BatchUpdate(); //p.Update(item); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); Console.WriteLine(DateTime.Now); } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Sunday, April 27, 2008 1:33 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] ADO.Net Hi Shamil > .. VB.NET .. it seems to work even quicker than C# equivalent somehow .. Strange. Could it be caused by some caching somewhere? What happens if you run the two tests in reverse order? /gustav From jwcolby at colbyconsulting.com Sun Apr 27 07:12:19 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 08:12:19 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000001c8a846$3606c6d0$6401a8c0@nant> References: <000001c8a846$3606c6d0$6401a8c0@nant> Message-ID: <48146DA3.1010200@colbyconsulting.com> LOL, now that I can readi it... ;-) >> and voila' the converted VB.NET is below : (watch line wraps now - this is Shamil Salakhetdinov wrote: > Hello John, > > OK, I will read the blog you mentioned later today... > > I know it's not easy to convince you to change your mind when you are > "driving your way full speed" :) And I'm not trying to - I'm just musing in > this coding samples I post here testing this or that approach, which I might > use in my own development... > > <<< > There are many things to consider when I program in .net, not the least > of which is simplicity. > May I say that proposed in C# solution is simple and almost ready to use in > your scenario in one thread? And also ready to be converted into a > multi-threading app/utility. And you don't need to translate in manually in > VB.NET - just use this free tool: > > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > > or this one: > > http://codeconverter.sharpdevelop.net/Convert.aspx > > and voila' the converted VB.NET is below : (watch line wraps now - this is > VB.NET - it was "automagically" converted by referred above tool by copying > and pasting C# code I posted yesterday + making some manual editing of > several code line to make it independent of my testing/development > environment) - and it seems to work even quicker than C# equivalent somehow > !?. I have got additional back VB.NET -> C# conversion in the end of this > message - you can use C# code copy and paste it in the tool refereed above > and get VB.NET code properly formatted. Proposal: let's post C# code here > for the folks who will try this code to not spend time with fixing line > wraps introduced by e-mail clients): > > Imports System > Imports System.Collections.Generic > Imports System.Data.SqlClient > > Namespace TestSamples > > Public Class DataItem > Public ID As Integer > Public Name As String > > Public Sub New(ByVal id As Integer, ByVal name As String) > ID = id > Name = name > End Sub > > Public ReadOnly Property ParsedName() As String > Get > ' just for testing purposes > ' return real parsed name here > Return Name > End Get > End Property > End Class > > Public Class DataProcessor > Private Const COUNT_SQL As String = "select count(*) from > zsysUserSessionTrace" > Private Const PAGED_SELECT_SQL As String = "SELECT * from " + "(" + > " select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as > int) as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " > UserBrowser " + " from zsysUserSessionTrace " + ") s " + "where s.RowNum > between {0} and {1}" > Private Const UPDATE_SQL As String = "update zsysUserSessionTrace > set UserBrowser = '{0}' where UserSessionTraceId = {1}" > > Private Const DEFAULT_BATCH_SIZE As Integer = 1000 > > Private _totalCount As Integer = -1 > Public ReadOnly Property TotalCount() As Integer > Get > Return _totalCount > End Get > End Property > > Private _currentBatchStartRowNum As Integer = 0 > Private _batchSize As Integer = 0 > > Private _connectionString As String > Public Sub New(ByVal connectionString As String) > initialize(connectionString, DEFAULT_BATCH_SIZE) > End Sub > Public Sub New(ByVal connectionString As String, ByVal batchSize As > Integer) > initialize(connectionString, batchSize) > End Sub > Private Sub initialize(ByVal connectionString As String, ByVal > batchSize As Integer) > > _batchSize = batchSize > _currentBatchStartRowNum = 1 > _connectionString = connectionString > > Using cnn As New > System.Data.SqlClient.SqlConnection(connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > cmd.CommandText = COUNT_SQL > cmd.CommandType = System.Data.CommandType.Text > > _totalCount = CInt(cmd.ExecuteScalar()) > End Using > End Sub > > Public ReadOnly Property HasData() As Boolean > Get > Return (TotalCount > 0 AndAlso _currentBatchStartRowNum < > TotalCount) > End Get > End Property > > Public Function GetData() As List(Of DataItem) > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > _currentBatchStartRowNum += _batchSize + 1 > Dim rdr As System.Data.SqlClient.SqlDataReader = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess) > > Dim list As New List(Of DataItem)() > While rdr.Read() > list.Add(New DataItem(CInt(rdr(1)), DirectCast(rdr(3), > String))) > End While > Return list > End Using > End Function > > Public Function Update(ByVal item As DataItem) As Integer > Using cnn As New > System.Data.SqlClient.SqlConnection(_connectionString) > cnn.Open() > > Dim cmd As System.Data.SqlClient.SqlCommand = > cnn.CreateCommand() > Dim sql As String = String.Format(UPDATE_SQL, > item.ParsedName, item.ID) > cmd.CommandText = sql > cmd.CommandType = System.Data.CommandType.Text > Return cmd.ExecuteNonQuery() > End Using > End Function > End Class > > > Public Class A_MainEntry > '27.04.2008 12:45:58 > 'Count(*) = 25045 > 'Counter = 25045 > '27.04.2008 12:46:04 > > Public Shared Sub Main() > Dim count As Integer = 0 > Try > '+ customize - set your connection string here > Dim pcName As String = "SPB" > Dim dbName As String = "dbName" > Dim userName As String = "sa" > Dim password As String = "sapwd" > Dim connectionString As String = _ > String.Format("Data Source={0}\SQLEXPRESS;Initial > Catalog={1};User Id={2};Password={3}", _ > pcName, dbName, userName, password) > '- > > Console.WriteLine(DateTime.Now) > Dim p As New DataProcessor(connectionString) > While p.HasData > Dim list As List(Of DataItem) = p.GetData() > For Each item As DataItem In list > p.Update(item) > count += 1 > Next > End While > Console.WriteLine("Count(*) = {0}", p.TotalCount) > Console.WriteLine("Counter = {0}", count) > Console.WriteLine(DateTime.Now) > Catch ex As Exception > Console.WriteLine(ex.Message) > End Try > End Sub > > End Class > End Namespace > > Here it's converted back to C#: > > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > using System; > using System.Collections.Generic; > using System.Data.SqlClient; > > namespace TestSamples > { > > public class DataItem > { > public int ID; > public string Name; > > public DataItem(int id, string name) > { > ID = id; > Name = name; > } > > public string ParsedName > { > // just for testing purposes > // return real parsed name here > get { return Name; } > } > } > > public class DataProcessor > { > private const string COUNT_SQL = "select count(*) from > zsysUserSessionTrace"; > private const string PAGED_SELECT_SQL = "SELECT * from " + "(" + " > select " + " Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) > as RowNum, " + " UserSessionTraceId, " + " HostAddress, " + " UserBrowser " > + " from zsysUserSessionTrace " + ") s " + "where s.RowNum between {0} and > {1}"; > private const string UPDATE_SQL = "update zsysUserSessionTrace set > UserBrowser = '{0}' where UserSessionTraceId = {1}"; > > private const int DEFAULT_BATCH_SIZE = 1000; > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(1), (string)rdr(3))); > list.Add(new DataItem((int)rdr[1], (string)rdr[3])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(UPDATE_SQL, item.ParsedName, > item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > } > > > public class A_MainEntry > { > // VB.NET > //27.04.2008 13:01:44 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:49 > > // C# > //27.04.2008 13:00:56 > //Count(*) = 25045 > //Counter = 25045 > //27.04.2008 13:01:04 > > public static void Main() > { > int count = 0; > try > { > //+ customize - set your connection string here > string pcName = "SPB"; > string dbName = "dbName"; > string userName = "sa"; > string password = "sapwd"; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > Console.WriteLine(DateTime.Now); > DataProcessor p = new DataProcessor(connectionString); > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > Console.WriteLine(DateTime.Now); > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Sunday, April 27, 2008 4:46 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > This is the blog I mentioned: > > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > <<< tail skipped to get message through in AccessD >>> > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From shamil at smsconsulting.spb.ru Sun Apr 27 10:14:00 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:14:00 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48146DA3.1010200@colbyconsulting.com> Message-ID: <001401c8a879$5298bd70$6401a8c0@nant> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil From shamil at smsconsulting.spb.ru Sun Apr 27 10:24:40 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Sun, 27 Apr 2008 19:24:40 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <000001c8a87a$cfa80270$6401a8c0@nant> ...additionally to my quoted posting here is SQL script to create test table: CREATE TABLE [dbo].[zsysUserSessionTrace]( [UserSessionTraceId] [int] IDENTITY(1,1) NOT NULL, [AspNetSessionId] [nvarchar](80) NOT NULL, [HostAddress] [nvarchar](40) NULL, [HostName] [nvarchar](40) NULL, [PageViewCount] [int] NULL, [UserName] [nvarchar](40) NOT NULL, [UserBrowser] [nvarchar](255) NOT NULL, [UpdatedBy] [nvarchar](20) NOT NULL, [UpdatedDate] [datetime] NOT NULL, [UpdatedLocation] [nvarchar](20) NOT NULL, [ParsedUserBrowserName] [nvarchar](255) NULL, CONSTRAINT [PK_zsysUserSessionTrace] PRIMARY KEY CLUSTERED ( [UserSessionTraceId] ASC )WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] ...and here is insert startemet to populate one row - you can run it many times: insert into zsysUserSessionTrace (AspNetSessionId, UserName, UserBrowser, UpdatedBy, UpdatedDate, UpdatedLocation) values (1,'TEST','IE7','SA','2008-04-27','SPB') ... and here are select statements to run e.g. in MS SQL Management console to see that test updates really happened: select count(*) from zsysUserSessionTrace go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId]) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go SELECT * from ( select Cast(ROW_NUMBER() over (ORDER BY [UserSessionTraceId] desc) as int) as RowNum, UserSessionTraceId, UserBrowser, ParsedUserBrowserName from zsysUserSessionTrace ) s where s.RowNum between 1 and 100 go -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Shamil Salakhetdinov Sent: Sunday, April 27, 2008 7:14 PM To: 'Discussion concerning Visual Basic and related programming issues.' Subject: Re: [dba-VB] ADO.Net Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? I wasn't aware of MS SQL TDS feature you mentioned but as you can see from stats it gives just slightly better results than the manual batching. Of course playing with batch size might give better gains of this method (my tests didn't show that gains depending on batch size) but I must say for me it looks more complicated and less flexible that using SQLDatareader and custom classes - that could be just my perception.... The C# code is below in the end of this message - its VB.NET equivalent can be obtained using automatic conversion - http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one code line have to be manually patched 'conversion issue replace the following code line => ==================================================== _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra yIndex), _arrayIndex - 1)) = sql with this one ============= _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 : Here are the stats: ==================== C# ==================== Strategy 1: ----------- Start time: 27.04.2008 19:02:58 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:15 Elapsed time: 00:00:16.7968750 Strategy 2: ----------- Start time: 27.04.2008 19:03:15 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:23 Elapsed time: 00:00:08.4218750 Strategy 3: ----------- Start time: 27.04.2008 19:03:23 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:03:31 Elapsed time: 00:00:07.9062500 ==================== VB.NET ==================== Strategy 1: ----------- Start time: 27.04.2008 19:05:20 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:36 Elapsed time: 00:00:16.7500000 Strategy 2: ----------- Start time: 27.04.2008 19:05:36 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:45 Elapsed time: 00:00:08.2812500 Strategy 3: ----------- Start time: 27.04.2008 19:05:45 Count(*) = 25046 Counter = 25046 End time: 27.04.2008 19:05:52 Elapsed time: 00:00:07.5000000 ==================== // Converters: //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx //http://codeconverter.sharpdevelop.net/Convert.aspx // Sql TDS batched update //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx using System; using System.Collections.Generic; using System.Data; using System.Data.SqlClient; namespace TestSamples { public sealed class Constants { //+ ///////////// C U S T O M I Z E ///////////////////// public const string PcName = "NANT"; public const string DbName = "ORS2005"; public const string UserName = "sa"; public const string Password = "shms"; public const string TABLE_NAME = "zsysUserSessionTrace"; public const string ID_COL_NAME = "UserSessionTraceId"; public const string NAME_COL_NAME = "UserBrowser"; public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; public const int DEFAULT_BATCH_SIZE = 1000; public static string GetParsedName(string name) { // just for testing purposes - customzie // to return your parsed name here if (name == null) name = ""; name = DateTime.Now.ToString() + ": " + name; if (name.Length > 255) name = name.Substring(0, 255); return name; } //- ///////////// C U S T O M I Z E ///////////////////// public const int ID_COL_INDEX = 1; public const int NAME_COL_INDEX = 2; public const int PARSED_NAME_COL_INDEX = 3; public static string COUNT_SQL { get { return String.Format("select count(*) from {0}", TABLE_NAME); } } public static string PAGED_SELECT_SQL { get { return string.Format( "SELECT * from " + "(" + " select " + " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as RowNum, " + " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, PARSED_NAME_COL_NAME, TABLE_NAME) + "where s.RowNum between {0} and {1}"; }} public static string UPDATE_SQL { get { return string.Format("update {0} ",TABLE_NAME) + string.Format(" set {0} = '{1}' where {2} = {3}", PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); }} public static string UPDATE_SQL_PARAMETERIZED { get { return string.Format("update {0} set {1} = @{2} where {3} = @{4}", TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, ID_COL_NAME, ID_COL_NAME); }} } public class DataItem { private int _id; public int ID { get { return _id; } } private string _name; public string Name { get { return _name; } } public DataItem(int id, string name) { _id = id; _name = name; } public string ParsedName { get { return Constants.GetParsedName(_name); } } } public class DataProcessor { private int _totalCount = -1; public int TotalCount { get { return _totalCount; } } private int _currentBatchStartRowNum = 0; private int _batchSize = 0; public int BatchSize { get { return _batchSize; } } private string _connectionString; public DataProcessor(string connectionString) { initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); } public DataProcessor(string connectionString, int batchSize) { initialize(connectionString, batchSize); } private void initialize(string connectionString, int batchSize) { _batchSize = batchSize; _currentBatchStartRowNum = 1; _connectionString = connectionString; using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); cmd.CommandText = Constants.COUNT_SQL; cmd.CommandType = System.Data.CommandType.Text; _totalCount = (int)cmd.ExecuteScalar(); } } public bool HasData { get { return (TotalCount > 0 && _currentBatchStartRowNum < TotalCount); } } public List GetData() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; _currentBatchStartRowNum += _batchSize + 1; System.Data.SqlClient.SqlDataReader rdr = cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); List list = new List(); while (rdr.Read()) { //error in automated conversion => list.Add(new DataItem((int)rdr(Constants.ID_COL_INDEX), (string)rdr(Constants.NAME_COL_INDEX))); list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], (string)rdr[Constants.NAME_COL_INDEX])); } return list; } } public int Update(DataItem item) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private string[] _updateCommands = null; private int _arrayIndex = -1; public void AddUpdateCommand(DataItem item) { if (_arrayIndex == -1) { _updateCommands = (string[])Array.CreateInstance(typeof(string), _batchSize); _arrayIndex = 0; } string sql = string.Format(Constants.UPDATE_SQL, item.ParsedName, item.ID); _updateCommands[_arrayIndex++] = sql; } public int BatchUpdate() { if (_arrayIndex == -1) return -1; if (_arrayIndex < _batchSize) Array.Resize(ref _updateCommands, _arrayIndex); using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); //Console.WriteLine("Batch update: {0} updates", _arrayIndex); string sql = String.Join(";", _updateCommands); _arrayIndex = -1; cmd.CommandText = sql; cmd.CommandType = System.Data.CommandType.Text; return cmd.ExecuteNonQuery(); } } private SqlDataAdapter _da = null; private DataSet _ds = null; public DataTable SQLTDSGetDataTablePaged() { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); int batchSize = _batchSize; if (_currentBatchStartRowNum + _batchSize >= _totalCount) batchSize = _totalCount - _currentBatchStartRowNum; string sql = string.Format(Constants.PAGED_SELECT_SQL, _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); _currentBatchStartRowNum += _batchSize + 1; _da = new SqlDataAdapter(sql, cnn); _ds = new DataSet(); _da.Fill(_ds); _da.UpdateCommand = new SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); _da.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, Constants.PARSED_NAME_COL_NAME); _da.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; _da.UpdateBatchSize = batchSize; _da.SelectCommand.Connection = null; _da.UpdateCommand.Connection = null; return _ds.Tables[0]; } } public void SQLTDSUpdate() { SQLTDSUpdate(false); } public void SQLTDSUpdate(bool forceUpdate) { using (System.Data.SqlClient.SqlConnection cnn = new System.Data.SqlClient.SqlConnection(_connectionString)) { cnn.Open(); _da.SelectCommand.Connection = cnn; _da.UpdateCommand.Connection = cnn; if (forceUpdate) _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; _da.Update(_ds.Tables[0]); //Console.WriteLine("*** Forced test exit ***"); //System.Environment.Exit(100); } } } public class A_MainEntry { /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate and execute individual update SQL statements /// public static void Strategy1(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { p.Update(item); count += 1; } //Console.WriteLine("{0} - rows processed...", p.BatchSize); } } /// /// Strategy 1 - use SqlDataReader to get data pages, and /// generate batched update SQL statements, and /// execute update batch when it has all update /// statements added /// public static void Strategy2(DataProcessor p, ref int count) { while (p.HasData) { List list = p.GetData(); foreach (DataItem item in list) { if (count > 0 && (count % p.BatchSize) == 0) p.BatchUpdate(); p.AddUpdateCommand(item); count += 1; } } p.BatchUpdate(); } /// /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, and /// use SQL TDS bacthed updates facility /// public static void Strategy3(DataProcessor p, ref int count) { while (p.HasData) { DataTable table = p.SQLTDSGetDataTablePaged(); foreach (DataRow row in table.Rows) { row.SetModified(); row[Constants.PARSED_NAME_COL_INDEX] = Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); count += 1; } p.SQLTDSUpdate(!p.HasData); } } public static void Main() { try { //+ customize - set your connection string here string pcName = Constants.PcName; string dbName = Constants.DbName; string userName = Constants.UserName; string password = Constants.Password; string connectionString = string.Format("Data Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password); //- for (int i = 1; i <= 3; i++) { int count = 0; DateTime startTime, endTime; startTime = DateTime.Now; Console.WriteLine(""); Console.WriteLine("Strategy {0}:", i); Console.WriteLine("-----------"); Console.WriteLine("Start time: {0}", startTime); DataProcessor p = new DataProcessor(connectionString); switch (i) { case 1: Strategy1(p, ref count); break; case 2: Strategy2(p, ref count); break; case 3: Strategy3(p, ref count); break; default: Console.WriteLine("{0} - undefined strategy number", i); break; } Console.WriteLine("Count(*) = {0}", p.TotalCount); Console.WriteLine("Counter = {0}", count); endTime = DateTime.Now; Console.WriteLine("End time: {0}", endTime); TimeSpan ts = TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); Console.WriteLine("Elapsed time: {0}", ts); } } catch (Exception ex) { Console.WriteLine(ex.Message); } } } } Enjoy! -- Shamil _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From Gustav at cactus.dk Sun Apr 27 11:42:13 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Sun, 27 Apr 2008 18:42:13 +0200 Subject: [dba-VB] ADO.Net Message-ID: Hi Shamil I'm following this thread with great interest because it touches some tasks I may soon encounter. It's a pleasure to study your work, and I really appreciate that you take your time to post it here. /gustav >>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> Hi John, Below is fully parameterized sample C# code, which uses three update strategies including the one you mentioned http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx The C# and VB.Net is nearly the same now - the difference of runtime of previous version was because of one bug. (There could be still others - if you find them please inform me - thank you). I'd think this thread's subject could be interested to many developers - proposal (to moderators): is there anybody to take on this current sample code and stats to test and to add some more tests and write some text to publish this code and accompanying article on accessD web site? From jwcolby at colbyconsulting.com Sun Apr 27 11:56:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 12:56:40 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <001401c8a879$5298bd70$6401a8c0@nant> References: <001401c8a879$5298bd70$6401a8c0@nant> Message-ID: <4814B048.6070704@colbyconsulting.com> What I come hear is that using the built-in batch is as fast as any other method, so if that is what I find easier to use there is no issue using that. I will try to understand your other methods (in VB, since the language makes very little apparent speed difference) and implement that in parallel so that I can switch between them on demand. If I can do that I will do some timings on my big database, which is what I am doing the updates on. Shamil Salakhetdinov wrote: > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > I wasn't aware of MS SQL TDS feature you mentioned but as you can see from > stats it gives just slightly better results than the manual batching. Of > course playing with batch size might give better gains of this method (my > tests didn't show that gains depending on batch size) but I must say for me > it looks more complicated and less flexible that using SQLDatareader and > custom classes - that could be just my perception.... > > The C# code is below in the end of this message - its VB.NET equivalent can > be obtained using automatic conversion - > http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx - and then one > code line have to be manually patched > > 'conversion issue replace the following code line => > ==================================================== > _updateCommands(System.Math.Max(System.Threading.Interlocked.Increment(_arra > yIndex), _arrayIndex - 1)) = sql > > with this one > ============= > _updateCommands(_arrayIndex) = sql : _arrayIndex += 1 > : > > Here are the stats: > > ==================== > C# > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:02:58 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:15 > Elapsed time: 00:00:16.7968750 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:03:15 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:23 > Elapsed time: 00:00:08.4218750 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:03:23 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:03:31 > Elapsed time: 00:00:07.9062500 > > ==================== > VB.NET > ==================== > > Strategy 1: > ----------- > Start time: 27.04.2008 19:05:20 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:36 > Elapsed time: 00:00:16.7500000 > > Strategy 2: > ----------- > Start time: 27.04.2008 19:05:36 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:45 > Elapsed time: 00:00:08.2812500 > > Strategy 3: > ----------- > Start time: 27.04.2008 19:05:45 > Count(*) = 25046 > Counter = 25046 > End time: 27.04.2008 19:05:52 > Elapsed time: 00:00:07.5000000 > ==================== > > // Converters: > //http://labs.developerfusion.co.uk/convert/csharp-to-vb.aspx > //http://codeconverter.sharpdevelop.net/Convert.aspx > > // Sql TDS batched update > //http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > using System; > using System.Collections.Generic; > using System.Data; > using System.Data.SqlClient; > > namespace TestSamples > { > public sealed class Constants > { > //+ ///////////// C U S T O M I Z E ///////////////////// > public const string PcName = "NANT"; > public const string DbName = "ORS2005"; > public const string UserName = "sa"; > public const string Password = "shms"; > > public const string TABLE_NAME = "zsysUserSessionTrace"; > public const string ID_COL_NAME = "UserSessionTraceId"; > public const string NAME_COL_NAME = "UserBrowser"; > public const string PARSED_NAME_COL_NAME = "ParsedUserBrowserName"; > > public const int DEFAULT_BATCH_SIZE = 1000; > > public static string GetParsedName(string name) > { > // just for testing purposes - customzie > // to return your parsed name here > if (name == null) name = ""; > name = DateTime.Now.ToString() + ": " + name; > if (name.Length > 255) name = name.Substring(0, 255); > > return name; > } > //- ///////////// C U S T O M I Z E ///////////////////// > > public const int ID_COL_INDEX = 1; > public const int NAME_COL_INDEX = 2; > public const int PARSED_NAME_COL_INDEX = 3; > > public static string COUNT_SQL { get { > return String.Format("select count(*) from {0}", TABLE_NAME); } > } > > public static string PAGED_SELECT_SQL { get { > return string.Format( > "SELECT * from " + "(" + " select " + > " Cast(ROW_NUMBER() over (ORDER BY [{0}]) as int) as > RowNum, " + > " [{1}], [{2}],[{3}] " + " from [{4}] " + ") s ", > ID_COL_NAME, ID_COL_NAME, NAME_COL_NAME, > PARSED_NAME_COL_NAME, TABLE_NAME) + > "where s.RowNum between {0} and {1}"; }} > > public static string UPDATE_SQL { get { > return string.Format("update {0} ",TABLE_NAME) + > string.Format(" set {0} = '{1}' where {2} = {3}", > PARSED_NAME_COL_NAME,"{0}",ID_COL_NAME,"{1}"); > }} > > public static string UPDATE_SQL_PARAMETERIZED { get { > return string.Format("update {0} set {1} = @{2} where {3} = > @{4}", > TABLE_NAME, PARSED_NAME_COL_NAME, PARSED_NAME_COL_NAME, > ID_COL_NAME, ID_COL_NAME); > }} > } > > public class DataItem > { > private int _id; > public int ID > { > get { return _id; } > } > > private string _name; > public string Name > { > get { return _name; } > } > > public DataItem(int id, string name) > { > _id = id; > _name = name; > } > > public string ParsedName > { > get > { > return Constants.GetParsedName(_name); > } > } > > > } > > public class DataProcessor > { > > private int _totalCount = -1; > public int TotalCount > { > get { return _totalCount; } > } > > private int _currentBatchStartRowNum = 0; > private int _batchSize = 0; > public int BatchSize { get { return _batchSize; } } > > private string _connectionString; > public DataProcessor(string connectionString) > { > initialize(connectionString, Constants.DEFAULT_BATCH_SIZE); > } > public DataProcessor(string connectionString, int batchSize) > { > initialize(connectionString, batchSize); > } > private void initialize(string connectionString, int batchSize) > { > > _batchSize = batchSize; > _currentBatchStartRowNum = 1; > _connectionString = connectionString; > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > cmd.CommandText = Constants.COUNT_SQL; > cmd.CommandType = System.Data.CommandType.Text; > > _totalCount = (int)cmd.ExecuteScalar(); > } > } > > public bool HasData > { > get { return (TotalCount > 0 && _currentBatchStartRowNum < > TotalCount); } > } > > public List GetData() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > _currentBatchStartRowNum += _batchSize + 1; > System.Data.SqlClient.SqlDataReader rdr = > cmd.ExecuteReader(System.Data.CommandBehavior.SequentialAccess); > > List list = new List(); > while (rdr.Read()) > { > //error in automated conversion => list.Add(new > DataItem((int)rdr(Constants.ID_COL_INDEX), > (string)rdr(Constants.NAME_COL_INDEX))); > list.Add(new DataItem((int)rdr[Constants.ID_COL_INDEX], > (string)rdr[Constants.NAME_COL_INDEX])); > } > return list; > } > } > > public int Update(DataItem item) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private string[] _updateCommands = null; > private int _arrayIndex = -1; > public void AddUpdateCommand(DataItem item) > { > if (_arrayIndex == -1) > { > _updateCommands = > (string[])Array.CreateInstance(typeof(string), _batchSize); > _arrayIndex = 0; > } > string sql = string.Format(Constants.UPDATE_SQL, > item.ParsedName, item.ID); > _updateCommands[_arrayIndex++] = sql; > } > > public int BatchUpdate() > { > if (_arrayIndex == -1) return -1; > if (_arrayIndex < _batchSize) > Array.Resize(ref _updateCommands, _arrayIndex); > > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > System.Data.SqlClient.SqlCommand cmd = cnn.CreateCommand(); > > //Console.WriteLine("Batch update: {0} updates", > _arrayIndex); > > string sql = String.Join(";", _updateCommands); > _arrayIndex = -1; > > cmd.CommandText = sql; > cmd.CommandType = System.Data.CommandType.Text; > return cmd.ExecuteNonQuery(); > } > } > > private SqlDataAdapter _da = null; > private DataSet _ds = null; > public DataTable SQLTDSGetDataTablePaged() > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > > int batchSize = _batchSize; > if (_currentBatchStartRowNum + _batchSize >= _totalCount) > batchSize = _totalCount - _currentBatchStartRowNum; > string sql = string.Format(Constants.PAGED_SELECT_SQL, > _currentBatchStartRowNum, _currentBatchStartRowNum + _batchSize); > > _currentBatchStartRowNum += _batchSize + 1; > > _da = new SqlDataAdapter(sql, cnn); > _ds = new DataSet(); > _da.Fill(_ds); > > _da.UpdateCommand = new > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn); > _da.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, 255, > Constants.PARSED_NAME_COL_NAME); > _da.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME); > _da.UpdateCommand.UpdatedRowSource = UpdateRowSource.None; > _da.UpdateBatchSize = batchSize; > > _da.SelectCommand.Connection = null; > _da.UpdateCommand.Connection = null; > > return _ds.Tables[0]; > } > } > > public void SQLTDSUpdate() > { > SQLTDSUpdate(false); > } > public void SQLTDSUpdate(bool forceUpdate) > { > using (System.Data.SqlClient.SqlConnection cnn = new > System.Data.SqlClient.SqlConnection(_connectionString)) > { > cnn.Open(); > _da.SelectCommand.Connection = cnn; > _da.UpdateCommand.Connection = cnn; > if (forceUpdate) > _da.UpdateBatchSize = _ds.Tables[0].Rows.Count; > _da.Update(_ds.Tables[0]); > > //Console.WriteLine("*** Forced test exit ***"); > //System.Environment.Exit(100); > } > } > } > > public class A_MainEntry > { > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate and execute individual update SQL > statements > /// > public static void Strategy1(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > p.Update(item); > count += 1; > } > //Console.WriteLine("{0} - rows processed...", p.BatchSize); > > } > } > > /// > /// Strategy 1 - use SqlDataReader to get data pages, and > /// generate batched update SQL statements, and > /// execute update batch when it has all update > /// statements added > /// > public static void Strategy2(DataProcessor p, ref int count) > { > while (p.HasData) > { > List list = p.GetData(); > foreach (DataItem item in list) > { > if (count > 0 && (count % p.BatchSize) == 0) > p.BatchUpdate(); > p.AddUpdateCommand(item); > count += 1; > } > } > p.BatchUpdate(); > } > > /// > /// Strategy 3 - use SqlDataAdapter and DataSet to get data pages, > and > /// use SQL TDS bacthed updates facility > /// > public static void Strategy3(DataProcessor p, ref int count) > { > while (p.HasData) > { > DataTable table = p.SQLTDSGetDataTablePaged(); > > foreach (DataRow row in table.Rows) > { > row.SetModified(); > row[Constants.PARSED_NAME_COL_INDEX] = > Constants.GetParsedName((string)row[Constants.NAME_COL_INDEX]); > count += 1; > } > p.SQLTDSUpdate(!p.HasData); > } > } > > public static void Main() > { > try > { > //+ customize - set your connection string here > string pcName = Constants.PcName; > string dbName = Constants.DbName; > string userName = Constants.UserName; > string password = Constants.Password; > string connectionString = string.Format("Data > Source={0}\\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password); > //- > > for (int i = 1; i <= 3; i++) > { > int count = 0; > DateTime startTime, endTime; > startTime = DateTime.Now; > Console.WriteLine(""); > Console.WriteLine("Strategy {0}:", i); > Console.WriteLine("-----------"); > Console.WriteLine("Start time: {0}", startTime); > DataProcessor p = new DataProcessor(connectionString); > > switch (i) > { > case 1: > Strategy1(p, ref count); > break; > case 2: > Strategy2(p, ref count); > break; > case 3: > Strategy3(p, ref count); > break; > default: > Console.WriteLine("{0} - undefined strategy > number", i); > break; > } > Console.WriteLine("Count(*) = {0}", p.TotalCount); > Console.WriteLine("Counter = {0}", count); > endTime = DateTime.Now; > Console.WriteLine("End time: {0}", endTime); > TimeSpan ts = > TimeSpan.FromTicks((endTime.Subtract(startTime)).Ticks); > Console.WriteLine("Elapsed time: {0}", ts); > } > } > catch (Exception ex) > { > Console.WriteLine(ex.Message); > } > } > > } > } > > Enjoy! > > -- > Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > -- John W. Colby www.ColbyConsulting.com From wdhindman at dejpolsystems.com Sun Apr 27 14:13:04 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Sun, 27 Apr 2008 15:13:04 -0400 Subject: [dba-VB] ADO.Net References: Message-ID: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Sun Apr 27 16:02:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 01:02:11 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <9A52D3B152D8433FA75924D302966DD0@jislaptopdev> Message-ID: <000401c8a8a9$f695a200$6401a8c0@nant> Here are the sources I promised to publish in my previous post: http://smsconsulting.spb.ru/samples/Stats.txt http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt Your turn, guys, to fix and improve them... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of William Hindman Sent: Sunday, April 27, 2008 11:13 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net ...ditto gustav ...I'm not contributing ...just lurking and learning ...much as I did with Access :) William -------------------------------------------------- From: "Gustav Brock" Sent: Sunday, April 27, 2008 12:42 PM To: Subject: Re: [dba-VB] ADO.Net > Hi Shamil > > I'm following this thread with great interest because it touches some > tasks I may soon encounter. > It's a pleasure to study your work, and I really appreciate that you take > your time to post it here. > > /gustav > >>>> shamil at smsconsulting.spb.ru 27-04-2008 17:14 >>> > Hi John, > > Below is fully parameterized sample C# code, which uses three update > strategies including the one you mentioned > http://blogs.msdn.com/dataaccess/archive/2005/05/19/420065.aspx > > The C# and VB.Net is nearly the same now - the difference of runtime of > previous version was because of one bug. (There could be still others - if > you find them please inform me - thank you). > > I'd think this thread's subject could be interested to many developers - > proposal (to moderators): is there anybody to take on this current sample > code and stats to test and to add some more tests and write some text to > publish this code and accompanying article on accessD web site? > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sun Apr 27 22:01:58 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 27 Apr 2008 23:01:58 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <000401c8a8a9$f695a200$6401a8c0@nant> References: <000401c8a8a9$f695a200$6401a8c0@nant> Message-ID: <48153E26.3040803@colbyconsulting.com> Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Mon Apr 28 01:52:21 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 08:52:21 +0200 Subject: [dba-VB] No flat ComboBox in .Net?? Message-ID: Hi all What happened? How have I been fooling around with covered eyes? If I select Flat for property FlatStyle the ComboBox changes to ... a flat design. No visible borders. Only issue is that the invisible borders of the ComboBox appears to be as wide as if the ComboBox has 3D-style, thus leaving a larger total area than an equivalent TextBox with identical font settings. If I force the size of the ComboBox to be lower, it won't display the content (not very useful). /gustav >>> Gustav at cactus.dk 23-04-2008 08:26 >>> Hi Charlotte OK, thanks. Looks like I have to accept the facts. If no other ideas for Visual Studio 2011, this is one. /gustav >>> cfoust at infostatsystems.com 23-04-2008 00:55 >>> We use 3rd party controls where one of the properties of a combobox is "UseFlatMode". Since these are usercontrols built on the winforms base, there has to be a way to do it, but I've never felt compelled to reinvent that wheel. If we want to get fancy with a cell in a grid, we assign a combobox as the control editor for that column and then set the properties of the combobox which passes them on to the cell. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Gustav Brock Sent: Tuesday, April 22, 2008 3:07 PM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] No flat ComboBox in .Net?? Hi all No suggestions? I've experimented with a DataGridView of one cell as a combobox, no headers, no scrollbars. Looks OK but what a deroute ... I can't believe this. /gustav >>> Gustav at cactus.dk 22-04-2008 17:51 >>> Hi All Where is the BorderStyle property of the ComboBox? It can't be true that you need custom controls if you wish a flat XP style ComboBox? From Gustav at cactus.dk Mon Apr 28 10:53:55 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Mon, 28 Apr 2008 17:53:55 +0200 Subject: [dba-VB] C# event handler syntax Message-ID: Hi all Just browsed this page: http://blueonionsoftware.com/Blog.aspx?p=8afe238f-87c2-400b-9440-8231280f7576 which has a brief explanation on the syntax from the beginning to the ultra compact of version 3.5 using Lambda expression: clock.Alarm += (o, e) => Console.Write("Alarm!"); This is _not_ what IntelliSense suggests, so you are a little on your own. /gustav From shamil at smsconsulting.spb.ru Mon Apr 28 11:40:25 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 20:40:25 +0400 Subject: [dba-VB] ADO.Net In-Reply-To: <48153E26.3040803@colbyconsulting.com> Message-ID: <00c301c8a94e$8f660430$6401a8c0@nant> Hi John, Thank you for your proposal to run the code against your huge db... OK, let's try this remote/offline Q&A/code musing/consulting/education in several installments published here: I must say I have quite some work to do for customers and I can't spend a lot of time for postings here/explaining what and why is done in the sample code - I will try just give refs on "concepts" I use for you to read more in many open sources... I made first adjustment to the yesterday's code - new code is here: http://smsconsulting.spb.ru/samples/Program.cs.txt http://smsconsulting.spb.ru/samples/Module.vb.txt http://smsconsulting.spb.ru/samples/Stats.txt To run it (let's use VB.NET): - in VS2005 create VB.NET Console application, replace the code of the module, which VS2005 creates by default (Module1.vb) with the code from http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines Public Const PcName As String = "LOCALHOST" Public Const DbName As String = "NamesParser" Public Const UserName As String = "TESTER" Public Const Password As String = "TEST.1959" to point to your db and its credentials, push CTRL+F5, and you should see the sample app running in console window and producing something like this report: Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:13 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:19 Elapsed time: 00:00:05.5156250 Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:19 Count(*) = 1000 Counter = 1000 End time: 28.04.2008 20:18:22 Elapsed time: 00:00:03.5468750 Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms Start time: 28.04.2008 20:18:22 Count(*) = 1000 Counter = 1009 End time: 28.04.2008 20:18:26 Elapsed time: 00:00:03.3906250 Try. If you tell me the name of your database and the name of your table I will change them here, as well as you can create username and password as above to not do any changes later for the new versions of the code. N.B: Make sure you will run the code against sample not the live db. The code assumes that the length of the target field FName is 50 chars - it uses just one this column currently to put "parsed" value - just first 50 chars of OWNERNAME field. Please have a look through the code, and ask your questions in the order you wanted them to be answered... Please everybody who wanted to participate in this discussion - you're very welcome! On debugging multi-threaded apps: that's a real PITA - yes you can debug/trace with breakpoints etc. but you have to keep in mind that as many threads as you have will cause debugger to stop on breakpoints etc. IOW better debug/trace single threaded code and them run it in multi-threaded mode, and use some indirect artefacts/techniques (as unit testing), which will show you that your app is running OK: as you can find in the new version of the code it can run both single and multi-threaded depending on the value of this constant: Public Const MULTI_THREADED_MODE As Boolean = True -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 7:02 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Shamil, I would like to thank you for taking the time to do this. Now... I will thank you in advance for taking the time to explain it to me. As happened many years ago when you taught me WithEvents in Access, I don't know enough to understand what the heck you are doing. In the absence of any instructions on how to use this thing, I cut and pasted the entire (VB) shebang into a class in a Windows Form project. It compiled flawlessly and the blank form opened. Now what? I hope that you can appreciate my lack of knowledge here. I have built several projects, with (to my pitiful level of expertise) quite extensive stuff going on but... there are a lot of concepts in your code that I have not used. Where to start. I guess where to start is a good spot. What do I do to cause this to run? Coming from Access I have always started with a form, which I promptly place a button on so I can call some function which "starts" things. That form will have class variables if necessary to instantiate a supervisor class, the supervisor class will then load recordsets, load data into data classes as required etc. So what I do not understand is "what do I call to make this run"? I see the MainEntry at the bottom but do I just call MainEntry from a button? Does the code belong in a class or a plain module? I assume that if I put it in a class then I would have to dim a variable in my form. Second (I am totally excited about this whole thing!!!) what are the replaceable parameters? Dim connectionString As String = String.Format("Data Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", pcName, dbName, userName, password) I assume here that pcName gets fed in to {0}? THAT is cool, and demonstrates where I start from in understanding your code. Third (also exciting!) I have never used threads yet so here we go. How do I debug that? Can I step through a thread in the same way I step through any other code? Set break points etc. And finally (for now) it just occurred to me to ask... can you help me (online of offline) to modify this example to actually use my own table that I need to parse the name for. I think I can figure out how to modify the stuff for the server, database, table etc. I have done that already in my own code for doing this. I will also worry about the parsing code itself, though just for a demo we can just hard code some values. My table Data: PKID (int32) OWNERNAME (to be parsed) FName (parsed) MName (parsed) LName (parsed) NamePrefix (parsed) NameSuffix (parsed) Gender (parsed) If I can modify they example to actually read / write to my table then I kill two birds with one stone, learn a bunch of cool new stuff including getting it all happening with threads, and get my real work done. For my real life purposes I am going to have to have a PKStart, PKEnd pair which tracks a block (remember I am doing 80 million rows eventually), and increment those start/end by the chunk size to repeat over and over (eventually). What I have found convenient in the past (and have code already written to do) is write status log files out to disk for each block processed. I write them as XML and include things like the start / end PKID, status memo fields, time start / stop and the like. As I said, I have already written that code (I'm not completely helpless, though it sometimes appears that way) and sitting out in a library ready to use for the logging. All I would need to do is build the specific data class to store the log data for this project. I will handle that. I will promise to absorb your example if you will promise to take the time to help me when I get stuck. Once I have done so I will then be able to apply it to my specific problem and give you back some real life timings from my largish database. I can vary the chunk size from 1k to any upward limit we find useful to test. My experience has been that anything above 10K can get problematic, though that was using the bulk import widget in past projects. I was getting timeouts if I went with too large a chunk size. VB.Net is only a small part of my work life, though I would like to make it the main part. I still spend most of my work life stuck in Access, maintaining applications I wrote (or inherited) long ago. While it pays the bills I would LOVE to get to the point where I could bang out code like this in the short time that it probably took you. Again thanks for your time and effort and patience. Shamil Salakhetdinov wrote: > Here are the sources I promised to publish in my previous post: > > http://smsconsulting.spb.ru/samples/Stats.txt > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > Your turn, guys, to fix and improve them... > > -- > Shamil > -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 11:55:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 12:55:08 -0400 Subject: [dba-VB] ADO.Net In-Reply-To: <00c301c8a94e$8f660430$6401a8c0@nant> References: <00c301c8a94e$8f660430$6401a8c0@nant> Message-ID: <4816016C.4010709@colbyconsulting.com> Thanks for the response Shamil, I know that you have lots of stuff to get done as do I. I am already back to the code I know in order to get this thing processed, however I will get your sample code running in the coming days. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > Thank you for your proposal to run the code against your huge db... > > OK, let's try this remote/offline Q&A/code musing/consulting/education in > several installments published here: I must say I have quite some work to do > for customers and I can't spend a lot of time for postings here/explaining > what and why is done in the sample code - I will try just give refs on > "concepts" I use for you to read more in many open sources... > > I made first adjustment to the yesterday's code - new code is here: > > http://smsconsulting.spb.ru/samples/Program.cs.txt > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > http://smsconsulting.spb.ru/samples/Stats.txt > > To run it (let's use VB.NET): > > - in VS2005 create VB.NET Console application, replace the code of the > module, which VS2005 creates by default (Module1.vb) with the code from > http://smsconsulting.spb.ru/samples/Module.vb.txt , correct these code lines > > Public Const PcName As String = "LOCALHOST" > Public Const DbName As String = "NamesParser" > Public Const UserName As String = "TESTER" > Public Const Password As String = "TEST.1959" > > to point to your db and its credentials, push CTRL+F5, and you should see > the sample app running in console window and producing something like this > report: > > Strategy = 1, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:13 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:19 > Elapsed time: 00:00:05.5156250 > > Strategy = 2, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:19 > Count(*) = 1000 > Counter = 1000 > End time: 28.04.2008 20:18:22 > Elapsed time: 00:00:03.5468750 > > Strategy = 3, PageSize = 100, ThreadsQty = 10, NameParsingDuration = 1 ms > > Start time: 28.04.2008 20:18:22 > Count(*) = 1000 > Counter = 1009 > End time: 28.04.2008 20:18:26 > Elapsed time: 00:00:03.3906250 > > Try. > > If you tell me the name of your database and the name of your table I will > change them here, as well as you can create username and password as above > to not do any changes later for the new versions of the code. > > N.B: Make sure you will run the code against sample not the live db. The > code assumes that the length of the target field FName is 50 chars - it uses > just one this column currently to put "parsed" value - just first 50 chars > of OWNERNAME field. > > Please have a look through the code, and ask your questions in the order you > wanted them to be answered... > > Please everybody who wanted to participate in this discussion - you're very > welcome! > > On debugging multi-threaded apps: that's a real PITA - yes you can > debug/trace with breakpoints etc. but you have to keep in mind that as many > threads as you have will cause debugger to stop on breakpoints etc. IOW > better debug/trace single threaded code and them run it in multi-threaded > mode, and use some indirect artefacts/techniques (as unit testing), which > will show you that your app is running OK: as you can find in the new > version of the code it can run both single and multi-threaded depending on > the value of this constant: > > Public Const MULTI_THREADED_MODE As Boolean = True > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 7:02 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net > > Shamil, > > I would like to thank you for taking the time to do this. Now... I will > thank you in advance for taking the time to explain it to me. As > happened many years ago when you taught me WithEvents in Access, I don't > know enough to understand what the heck you are doing. > > In the absence of any instructions on how to use this thing, I cut and > pasted the entire (VB) shebang into a class in a Windows Form project. > It compiled flawlessly and the blank form opened. > > Now what? > > I hope that you can appreciate my lack of knowledge here. I have built > several projects, with (to my pitiful level of expertise) quite > extensive stuff going on but... there are a lot of concepts in your code > that I have not used. Where to start. > > I guess where to start is a good spot. What do I do to cause this to > run? Coming from Access I have always started with a form, which I > promptly place a button on so I can call some function which "starts" > things. That form will have class variables if necessary to instantiate > a supervisor class, the supervisor class will then load recordsets, load > data into data classes as required etc. So what I do not understand is > "what do I call to make this run"? I see the MainEntry at the bottom > but do I just call MainEntry from a button? Does the code belong in a > class or a plain module? I assume that if I put it in a class then I > would have to dim a variable in my form. > > Second (I am totally excited about this whole thing!!!) what are the > replaceable parameters? > > Dim connectionString As String = String.Format("Data > Source={0}\SQLEXPRESS;Initial Catalog={1};User Id={2};Password={3}", > pcName, dbName, userName, password) > > I assume here that pcName gets fed in to {0}? THAT is cool, and > demonstrates where I start from in understanding your code. > > Third (also exciting!) I have never used threads yet so here we go. How > do I debug that? Can I step through a thread in the same way I step > through any other code? Set break points etc. > > And finally (for now) it just occurred to me to ask... can you help me > (online of offline) to modify this example to actually use my own table > that I need to parse the name for. I think I can figure out how to > modify the stuff for the server, database, table etc. I have done that > already in my own code for doing this. I will also worry about the > parsing code itself, though just for a demo we can just hard code some > values. > > > My table Data: > > PKID (int32) > OWNERNAME (to be parsed) > FName (parsed) > MName (parsed) > LName (parsed) > NamePrefix (parsed) > NameSuffix (parsed) > Gender (parsed) > > If I can modify they example to actually read / write to my table then I > kill two birds with one stone, learn a bunch of cool new stuff including > getting it all happening with threads, and get my real work done. > > For my real life purposes I am going to have to have a PKStart, PKEnd > pair which tracks a block (remember I am doing 80 million rows > eventually), and increment those start/end by the chunk size to repeat > over and over (eventually). What I have found convenient in the past > (and have code already written to do) is write status log files out to > disk for each block processed. I write them as XML and include things > like the start / end PKID, status memo fields, time start / stop and the > like. As I said, I have already written that code (I'm not completely > helpless, though it sometimes appears that way) and sitting out in a > library ready to use for the logging. All I would need to do is build > the specific data class to store the log data for this project. I will > handle that. > > I will promise to absorb your example if you will promise to take the > time to help me when I get stuck. Once I have done so I will then be > able to apply it to my specific problem and give you back some real life > timings from my largish database. I can vary the chunk size from 1k to > any upward limit we find useful to test. My experience has been that > anything above 10K can get problematic, though that was using the bulk > import widget in past projects. I was getting timeouts if I went with > too large a chunk size. > > VB.Net is only a small part of my work life, though I would like to make > it the main part. I still spend most of my work life stuck in Access, > maintaining applications I wrote (or inherited) long ago. While it pays > the bills I would LOVE to get to the point where I could bang out code > like this in the short time that it probably took you. > > Again thanks for your time and effort and patience. > > Shamil Salakhetdinov wrote: >> Here are the sources I promised to publish in my previous post: >> >> http://smsconsulting.spb.ru/samples/Stats.txt >> >> http://smsconsulting.spb.ru/samples/Program.cs.txt >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> Your turn, guys, to fix and improve them... >> >> -- >> Shamil >> > From jwcolby at colbyconsulting.com Mon Apr 28 13:10:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:10:08 -0400 Subject: [dba-VB] ADO.Net data table update Message-ID: <48161300.8050807@colbyconsulting.com> I have a datatable that I opened. It contains a set of 10K records which I have updated pieces of. I now need to write it back to SQL Server. If I just call the da.update it errors saying it needs an update command. Do I have to build an update SQL statement or will the DA just figure out where the data fields came from and update them back out to SQL Server? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 13:49:38 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 14:49:38 -0400 Subject: [dba-VB] ADO.Net Update SQL Server Message-ID: <48161C42.1030500@colbyconsulting.com> Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Mon Apr 28 14:04:08 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Mon, 28 Apr 2008 12:04:08 -0700 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> References: <48161C42.1030500@colbyconsulting.com> Message-ID: UpdateBatch should work. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 11:50 AM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Mon Apr 28 14:09:11 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Mon, 28 Apr 2008 23:09:11 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <48161C42.1030500@colbyconsulting.com> Message-ID: <010001c8a963$57cd7c00$6401a8c0@nant> Hi John, The answer is in the code I published today on my site related to 'Re: [dba-VB] ADO.Net': http://smsconsulting.spb.ru/samples/Module.vb.txt - Strategy 3 - trace in single threaded mode and you'll find what to do. Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As DataSet) As DataTable ... And Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet As DataSet) .... HTH, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 10:50 PM To: VBA Subject: [dba-VB] ADO.Net Update SQL Server Does ANYBODY know how to write a table back out to SQL Server? All of my books show cute little examples of building up a data table manually then BINDING it to a control or whatever. I am not finding anything with code example that shows how to WRITE changed data back to SQL Server. I have managed to load a data table, write changes to the rows and now... I have to write it back out to SQL Server. The table has the PK, NameToSplit, LName, FName etc etc. What am I missing? I desperately need help (yea, yea, not MENTAL help) and I need it quickly. I have been trying to do this last piece for DAYS and I am not succeeding. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 14:57:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 15:57:15 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: References: <48161C42.1030500@colbyconsulting.com> Message-ID: <48162C1B.8060207@colbyconsulting.com> I hate to sound ignorant but what is the UpdateBatch a method of? I have a connection, command, dataset and datatable and AFAICT none of those things have an UpdateBatch method. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > UpdateBatch should work. > > Charlotte Foust > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 11:50 AM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 15:14:01 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 16:14:01 -0400 Subject: [dba-VB] ADO.Net Update SQL Server References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> Message-ID: <01d001c8a96c$676b5a70$0502a8c0@Laptop> It is a method of the ADO.Recordset. I was ignorant of this, also, until I looked it up in a book. Ignorant<>something you can sound like Either you know or you don't know :) Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 3:57 PM Subject: Re: [dba-VB] ADO.Net Update SQL Server >I hate to sound ignorant but what is the UpdateBatch a method of? > > I have a connection, command, dataset and datatable and AFAICT none of > those things have an UpdateBatch method. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> UpdateBatch should work. >> >> Charlotte Foust >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 11:50 AM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> >> -- >> John W. Colby >> www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 15:17:48 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:17:48 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <010001c8a963$57cd7c00$6401a8c0@nant> References: <010001c8a963$57cd7c00$6401a8c0@nant> Message-ID: <481630EC.5060600@colbyconsulting.com> When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > From jwcolby at colbyconsulting.com Mon Apr 28 15:29:52 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 16:29:52 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <01d001c8a96c$676b5a70$0502a8c0@Laptop> References: <48161C42.1030500@colbyconsulting.com> <48162C1B.8060207@colbyconsulting.com> <01d001c8a96c$676b5a70$0502a8c0@Laptop> Message-ID: <481633C0.9060808@colbyconsulting.com> ROTFL, Oh I think I succeed very well at sounding ignorant. 8-( And yup, I don't know. I have beaten my head against this all day with absolutely zero results. I find it difficult to believe that I am doing anything unique but I'll be damned if I can find an article or code that just says "here is how you write back the updated table to the sql server... (data store)". John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > It is a method of the ADO.Recordset. > I was ignorant of this, also, until I looked it up in a book. > Ignorant<>something you can sound like > Either you know or you don't know :) > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Monday, April 28, 2008 3:57 PM > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > >> I hate to sound ignorant but what is the UpdateBatch a method of? >> >> I have a connection, command, dataset and datatable and AFAICT none of >> those things have an UpdateBatch method. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Charlotte Foust wrote: >>> UpdateBatch should work. >>> >>> Charlotte Foust >>> >>> -----Original Message----- >>> From: dba-vb-bounces at databaseadvisors.com >>> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >>> Sent: Monday, April 28, 2008 11:50 AM >>> To: VBA >>> Subject: [dba-VB] ADO.Net Update SQL Server >>> >>> Does ANYBODY know how to write a table back out to SQL Server? >>> >>> All of my books show cute little examples of building up a data table >>> manually then BINDING it to a control or whatever. I am not finding >>> anything with code example that shows how to WRITE changed data back to >>> SQL Server. >>> >>> I have managed to load a data table, write changes to the rows and >>> now... I have to write it back out to SQL Server. The table has the PK, >>> NameToSplit, LName, FName etc etc. What am I missing? >>> >>> I desperately need help (yea, yea, not MENTAL help) and I need it >>> quickly. I have been trying to do this last piece for DAYS and I am not >>> succeeding. >>> >>> -- >>> John W. Colby >>> www.ColbyConsulting.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From James at fcidms.com Mon Apr 28 16:02:31 2008 From: James at fcidms.com (James Barash) Date: Mon, 28 Apr 2008 17:02:31 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> John: If you already have a data adapter that you used to fill your data table, you need to add an UpdateCommand to that data adapter and then use it to update the datastore. Below is a very quick and dirty version from an existing program I've used. It should at least give you a place to start. It assumes you already have a data adapter named da, a data table named dt and a connection named conn. You will have to change the table and field names to match yours. da.UpdateCommand = New SqlClient.SqlCommand da.UpdateCommand.Connection = conn da.UpdateCommand.CommandType = CommandType.Text da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldDisplayValue", System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.UpdateCommand.Parameters.Add(New System.Data.SqlClient.SqlParameter("@FieldNumber", System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", "", "")) da.Update(dt) James Barash -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 4:18 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 17:11:07 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:11:07 -0400 Subject: [dba-VB] vb.net - Updates Message-ID: <48164B7B.2070808@colbyconsulting.com> I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Apr 28 17:12:37 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 18:12:37 -0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> References: <00c901c8a973$2d0a44c0$6d0101c0@fci.local> Message-ID: <48164BD5.60906@colbyconsulting.com> Thanks James. I also found a piece of code that shows how to use a commandbuilder object to generate those update commands automatically. At my skill level, automatic is a good thing. John W. Colby www.ColbyConsulting.com James Barash wrote: > John: > If you already have a data adapter that you used to fill your data table, > you need to add an UpdateCommand to that data adapter and then use it to > update the datastore. Below is a very quick and dirty version from an > existing program I've used. It should at least give you a place to start. It > assumes you already have a data adapter named da, a data table named dt and > a connection named conn. You will have to change the table and field names > to match yours. > > > da.UpdateCommand = New SqlClient.SqlCommand > da.UpdateCommand.Connection = conn > da.UpdateCommand.CommandType = CommandType.Text > da.UpdateCommand.CommandText = "Update tblValues set FieldValue=@FieldValue, > FieldDisplayValue = @FieldDisplayValue where FieldNumber=@FieldNumber" > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldValue", System.Data.DataRowVersion.Current, False, Nothing, "", "", > "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldDisplayValue", > System.Data.SqlDbType.VarChar, 0, System.Data.ParameterDirection.Input, 0, > 0, "FieldDisplayValue", System.Data.DataRowVersion.Current, False, > Nothing, "", "", "")) > da.UpdateCommand.Parameters.Add(New > System.Data.SqlClient.SqlParameter("@FieldNumber", > System.Data.SqlDbType.Int, 0, System.Data.ParameterDirection.Input, 0, 0, > "FieldNumber", System.Data.DataRowVersion.Current, False, Nothing, "", > "", "")) > da.Update(dt) > > James Barash > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 4:18 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] ADO.Net Update SQL Server > > When I try to call the update method of the data set object I get an error: > > "Update requires a valid update command when passed datarow collection > with modified rows." > > I probably sound ungrateful but in your code you appear to set up a sql > statement for an update statement: > > adapter.UpdateCommand = New > SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, > Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) > adapter.UpdateCommand.Parameters.Add("@" + > Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) > adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None > adapter.UpdateBatchSize = pageSize > > > My problem is that is just Chinese to me. I have no background > knowledge in plain English of what an update sql statement is supposed > to look like (in the ADO context) and absolutely no understanding of > even what I am trying to accomplish (expressed in plain English). I am > uneducated, and I can find no way to get educated. > > I must own 6 books on ADO.Net and VB and every single one of them > expects me to drag and drop objects out onto a form. OR they very > helpfully set everything up and show me how to do stuff RIGHT UP TO > USING IT, but never go that last step of sending updates back to the > database. Open, update, display in a grid, cleanup and close. Hmm.... > and I paid good money for these books. > > They do not discuss anything at all about what is going on at the code > level. So when I try to walk through your code it is simply > indecipherable to me. If I don't know what you are trying to accomplish > I can't understand how you code will accomplish that thing. > > Please do not take these statements as criticisms of your code, simply > that I am a freshman student still taking the basket weaving classes > looking at masters code. Thus it isn't useful to tell me to trace > through the code. > > MY code doesn't do any of that stuff. My code is pretty simple: > > Set up a connection, a data adapter, a table. Iterate the table row by > row, updating fields of the rows. > > Now... how to update the table back to SQL Server... > > Given how simple everything has been up to this point I have to believe > that I am just missing a concept and code to implement that concept. I > have called the update method and I have googled the error messages and > nothing... > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> The answer is in the code I published today on my site related to 'Re: >> [dba-VB] ADO.Net': >> >> http://smsconsulting.spb.ru/samples/Module.vb.txt >> >> - Strategy 3 - trace in single threaded mode and you'll find what to do. >> >> Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, > ByVal >> pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As >> DataSet) As DataTable >> ... >> >> And >> >> Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet >> As DataSet) >> .... >> >> >> HTH, >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Monday, April 28, 2008 10:50 PM >> To: VBA >> Subject: [dba-VB] ADO.Net Update SQL Server >> >> Does ANYBODY know how to write a table back out to SQL Server? >> >> All of my books show cute little examples of building up a data table >> manually then BINDING it to a control or whatever. I am not finding >> anything with code example that shows how to WRITE changed data back to >> SQL Server. >> >> I have managed to load a data table, write changes to the rows and >> now... I have to write it back out to SQL Server. The table has the PK, >> NameToSplit, LName, FName etc etc. What am I missing? >> >> I desperately need help (yea, yea, not MENTAL help) and I need it >> quickly. I have been trying to do this last piece for DAYS and I am not >> succeeding. >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Mon Apr 28 18:31:14 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 19:31:14 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> References: <48164B7B.2070808@colbyconsulting.com> Message-ID: <48165E42.20404@colbyconsulting.com> We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. From ebarro at verizon.net Mon Apr 28 19:44:24 2008 From: ebarro at verizon.net (Eric Barro) Date: Mon, 28 Apr 2008 17:44:24 -0700 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48164B7B.2070808@colbyconsulting.com> Message-ID: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Yeah it is a terrible thing from a DBA point of view because it opens it up to SQL injection attacks since it builds your SQL statement on the fly. I've tried it before and since it is generating it on the fly it makes it all the more harder to debug and offers the least amount of control from a DBA point of view. So...I let SQL server do all the work on the back end when it comes to batch updates by collecting the parameters and sending it to SQL server to process for the records it needs to process. Based on what you are trying to do you are building an image of the data structure in memory, manipulating the data in memory and then committing the changes to the database in memory basically sending the update command for each record to the database back end right? Or am I mistaken in thinking that all of this magically happens by sending a single update command containing several individual update commands strung together? -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, April 28, 2008 3:11 PM To: VBA Subject: [dba-VB] vb.net - Updates I THINK I have found my solution, called the SQLCommandBuilder. I found reference to the command build saying how terrible it is, then a piece of code that, in simple plain language, shows how to use it all, using the simple objects I am already using. We shall see. -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Apr 28 20:08:20 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 28 Apr 2008 21:08:20 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> Message-ID: <48167504.6070909@colbyconsulting.com> It all happens automagically. At this point I modify about 10K records at a clip, then issue a single update and it goes. I set the batch size to 100 for this experiment but I will probably play with that to see what can be done. I tried a thousand but it timed out. I don't see any timeout period property though there might well be one somewhere. I understand the SQL Injection attack thing but you need to remember that this is a database running on my system that only I ever access. Even more importantly though, I need this kind of thing to just work, as quickly as I can make it happen. I am not a high powered DBA (nor any other powered DBA) nor a high powered .Net programmer. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Yeah it is a terrible thing from a DBA point of view because it opens it up > to SQL injection attacks since it builds your SQL statement on the fly. > > I've tried it before and since it is generating it on the fly it makes it > all the more harder to debug and offers the least amount of control from a > DBA point of view. So...I let SQL server do all the work on the back end > when it comes to batch updates by collecting the parameters and sending it > to SQL server to process for the records it needs to process. > > Based on what you are trying to do you are building an image of the data > structure in memory, manipulating the data in memory and then committing the > changes to the database in memory basically sending the update command for > each record to the database back end right? Or am I mistaken in thinking > that all of this magically happens by sending a single update command > containing several individual update commands strung together? > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 3:11 PM > To: VBA > Subject: [dba-VB] vb.net - Updates > > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece of > code that, in simple plain language, shows how to use it all, using the > simple objects I am already using. > > We shall see. > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Mon Apr 28 21:26:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Mon, 28 Apr 2008 22:26:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net> <48167504.6070909@colbyconsulting.com> Message-ID: <034b01c8a9a0$7b784440$0502a8c0@Laptop> These fit the bill ... How To: Protect From SQL Injection in ASP.NET http://msdn2.microsoft.com/en-us/library/ms998271.aspx Using Stored Procedures with SQLDataAdapter? http://forums.devx.com/showthread.php?t=148707 SQLDataAdapter without using SQLCommandBuilder http://www.codeproject.com/KB/database/sqldawithoutsqlcb.aspx Using Stored Procedures in Conjunction with the SqlDataAdapter http://www.code-magazine.com/Article.aspx?quickid=0303141 Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Monday, April 28, 2008 9:08 PM Subject: Re: [dba-VB] vb.net - Updates > I understand the SQL Injection attack thing but you need to remember > that this is a database running on my system that only I ever access. From jwcolby at colbyconsulting.com Tue Apr 29 00:15:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 01:15:45 -0400 Subject: [dba-VB] Name parsing system timing Message-ID: <4816AF01.8090500@colbyconsulting.com> OK, it is running in a loop now, processing 10K records every 19 seconds. It will take 50 hours at the current rate to parse 96.4 million records, or about 2 million records / hour. the SQL Server update portion is taking about 5 seconds of each 19 second cycle, 14 seconds to parse, 5 seconds to write back to the database, virtually nothing to load. At least it is running now, making progress. Thanks for all the help from everyone getting this thing to happen. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 08:53:26 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 09:53:26 -0400 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: <48172856.9080502@colbyconsulting.com> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Tue Apr 29 09:39:45 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 16:39:45 +0200 Subject: [dba-VB] Update gets string or binary truncation error Message-ID: Hi John That is what the DataTableAdapter is for. If you use the wizard for creating these to attach your tables, it will do all the plumbing and create a (giant) class which wraps all this for you taking care of relations, not null, max lengths, strong types, error handling etc. etc. not to say provide the basis for creating all the queries you need including ready-made basic queries for CRUD as well as giving you the choice to use stored procedures, pass-through or normal queries. All of this is controlled from a graphic layout like a relations view. If that isn't enough it also lets you carry on with Data Access Layers and business rules. Nothing is hidden or secret. You can browse the class should you have the patience and wish so. This is a snippet: private void InitClass() { this.columnKundeNr = new global::System.Data.DataColumn("KundeNr", typeof(int), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnKundeNr); this.columnNavn = new global::System.Data.DataColumn("Navn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnNavn); this.columnAdr1 = new global::System.Data.DataColumn("Adr1", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr1); this.columnAdr2 = new global::System.Data.DataColumn("Adr2", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnAdr2); this.columnPostNr = new global::System.Data.DataColumn("PostNr", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnPostNr); this.columnBynavn = new global::System.Data.DataColumn("Bynavn", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnBynavn); this.columnTlf = new global::System.Data.DataColumn("Tlf", typeof(string), null, global::System.Data.MappingType.Element); base.Columns.Add(this.columnTlf); this.Constraints.Add(new global::System.Data.UniqueConstraint("Constraint1", new global::System.Data.DataColumn[] { this.columnKundeNr}, true)); this.columnKundeNr.AllowDBNull = false; this.columnKundeNr.Unique = true; this.columnNavn.AllowDBNull = false; this.columnNavn.MaxLength = 50; this.columnAdr1.MaxLength = 50; this.columnAdr2.MaxLength = 50; this.columnPostNr.MaxLength = 50; this.columnBynavn.MaxLength = 50; this.columnTlf.MaxLength = 50; } If you redesign the table structure from within Visual Studio, it will track the changes and adjust the class accordingly. It is __so clever__ that I wonder why most are still moving on with SQL builders, hand built command strings and the like - pure waste of time in my opinion - except for programmers with special needs and knowledge like Shamil who knows what he is doing. William, I think, will amen this. I must admit that because of VS2008 I'm ready to forgive MS much of their bad doings. /gustav >>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> I am doing the update thing, updating data back to SQL Server. I have been updating an ADO recordset in memory and when I try to update it back to SQL Server it throws an error that the update failed because it would cause a string value to be truncated. How is this possible. I thought that the table structure in memory mirrored the structure in the data store (SQL Server in this instance). Doesn't the read of the data out of SQL Service discover the field size and type and build a similar structure in memory? Apparently not, but if not how do I handle a case like this? -- John W. Colby www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 09:46:24 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 09:46:24 -0500 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: References: Message-ID: <200804291451.m3TEpqu6010208@databaseadvisors.com> Yes, that is correct. He is ending up loading the records into memory. Then passing individual updates back for each row of data that he changed. This method is not designed for doing large datasets like he is doing. It should be done on the SQL Server side. John, Just what are you updating...the UPDATE statement you would use to do what you are doing. Robert At 09:40 AM 4/29/2008, you wrote: >Date: Mon, 28 Apr 2008 17:44:24 -0700 >From: "Eric Barro" >Subject: Re: [dba-VB] vb.net - Updates >To: "'Discussion concerning Visual Basic and related programming > issues.'" >Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >Content-Type: text/plain; charset=us-ascii > >Yeah it is a terrible thing from a DBA point of view because it opens it up >to SQL injection attacks since it builds your SQL statement on the fly. > >I've tried it before and since it is generating it on the fly it makes it >all the more harder to debug and offers the least amount of control from a >DBA point of view. So...I let SQL server do all the work on the back end >when it comes to batch updates by collecting the parameters and sending it >to SQL server to process for the records it needs to process. > >Based on what you are trying to do you are building an image of the data >structure in memory, manipulating the data in memory and then committing the >changes to the database in memory basically sending the update command for >each record to the database back end right? Or am I mistaken in thinking >that all of this magically happens by sending a single update command >containing several individual update commands strung together? From wdhindman at dejpolsystems.com Tue Apr 29 10:26:31 2008 From: wdhindman at dejpolsystems.com (William Hindman) Date: Tue, 29 Apr 2008 11:26:31 -0400 Subject: [dba-VB] Update gets string or binary truncation error References: Message-ID: ...sigh ...I look at some of the stuff Shamil posts and haven't a clue what he's doing, much less why ...waaaaaaaay over my head ...so much so I get a bad headache every time I try to read some of it ...but you definitely get an AMEN! ...and then some ...on VS2K8 ...when MS finally gets something right, its RIGHT!!! :) William -------------------------------------------------- From: "Gustav Brock" Sent: Tuesday, April 29, 2008 10:39 AM To: Subject: Re: [dba-VB] Update gets string or binary truncation error > Hi John > > That is what the DataTableAdapter is for. If you use the wizard for > creating these to attach your tables, it will do all the plumbing and > create a (giant) class which wraps all this for you taking care of > relations, not null, max lengths, strong types, error handling etc. etc. > not to say provide the basis for creating all the queries you need > including ready-made basic queries for CRUD as well as giving you the > choice to use stored procedures, pass-through or normal queries. > > All of this is controlled from a graphic layout like a relations view. If > that isn't enough it also lets you carry on with Data Access Layers and > business rules. > > Nothing is hidden or secret. You can browse the class should you have the > patience and wish so. This is a snippet: > > private void InitClass() { > this.columnKundeNr = new > global::System.Data.DataColumn("KundeNr", typeof(int), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnKundeNr); > this.columnNavn = new > global::System.Data.DataColumn("Navn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnNavn); > this.columnAdr1 = new > global::System.Data.DataColumn("Adr1", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr1); > this.columnAdr2 = new > global::System.Data.DataColumn("Adr2", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnAdr2); > this.columnPostNr = new > global::System.Data.DataColumn("PostNr", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnPostNr); > this.columnBynavn = new > global::System.Data.DataColumn("Bynavn", typeof(string), null, > global::System.Data.MappingType.Element); > base.Columns.Add(this.columnBynavn); > this.columnTlf = new global::System.Data.DataColumn("Tlf", > typeof(string), null, global::System.Data.MappingType.Element); > base.Columns.Add(this.columnTlf); > this.Constraints.Add(new > global::System.Data.UniqueConstraint("Constraint1", new > global::System.Data.DataColumn[] { > this.columnKundeNr}, true)); > this.columnKundeNr.AllowDBNull = false; > this.columnKundeNr.Unique = true; > this.columnNavn.AllowDBNull = false; > this.columnNavn.MaxLength = 50; > this.columnAdr1.MaxLength = 50; > this.columnAdr2.MaxLength = 50; > this.columnPostNr.MaxLength = 50; > this.columnBynavn.MaxLength = 50; > this.columnTlf.MaxLength = 50; > } > > If you redesign the table structure from within Visual Studio, it will > track the changes and adjust the class accordingly. > > It is __so clever__ that I wonder why most are still moving on with SQL > builders, hand built command strings and the like - pure waste of time in > my opinion - except for programmers with special needs and knowledge like > Shamil who knows what he is doing. > > William, I think, will amen this. > > I must admit that because of VS2008 I'm ready to forgive MS much of their > bad doings. > > /gustav > >>>> jwcolby at colbyconsulting.com 29-04-2008 15:53 >>> > I am doing the update thing, updating data back to SQL Server. I have > been updating an ADO recordset in memory and when I try to update it > back to SQL Server it throws an error that the update failed because it > would cause a string value to be truncated. > > How is this possible. I thought that the table structure in memory > mirrored the structure in the data store (SQL Server in this instance). > Doesn't the read of the data out of SQL Service discover the field > size and type and build a similar structure in memory? > > Apparently not, but if not how do I handle a case like this? > -- > John W. Colby > www.ColbyConsulting.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 10:43:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 11:43:45 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <200804291451.m3TEpqu6010208@databaseadvisors.com> References: <200804291451.m3TEpqu6010208@databaseadvisors.com> Message-ID: <48174231.4070501@colbyconsulting.com> > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 12:13:02 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 12:13:02 -0500 (CDT) Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 Message-ID: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Trying to understand the scenario... 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Questions: 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? From: jwcolby Date: 2008/04/29 Tue AM 10:43:45 CDT > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. Yes, and most records have updates. > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. ROTFL. That is a definite NSS. Unfortunately I am not a DBA and with the responses I get from the SQL group I am not even going there. I can do this purely in VB.Net (it is cranking as we speak) so I will. > Just what are you updating...the UPDATE statement you would use to > do what you are doing. No idea. I thought I could find out from the DA after the command builder was finished building the update statement but I am getting an error if I try to access the update property. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Yes, that is correct. He is ending up loading the records into memory. > Then passing individual updates back for each row of data that he > changed. > > This method is not designed for doing large datasets like he is doing. > > It should be done on the SQL Server side. > > > John, > > Just what are you updating...the UPDATE statement you would use to > do what you are doing. > > Robert > > At 09:40 AM 4/29/2008, you wrote: >> Date: Mon, 28 Apr 2008 17:44:24 -0700 >> From: "Eric Barro" >> Subject: Re: [dba-VB] vb.net - Updates >> To: "'Discussion concerning Visual Basic and related programming >> issues.'" >> Message-ID: <0K02006A29SFSYJ0 at vms173005.mailsrvcs.net> >> Content-Type: text/plain; charset=us-ascii >> >> Yeah it is a terrible thing from a DBA point of view because it opens it up >> to SQL injection attacks since it builds your SQL statement on the fly. >> >> I've tried it before and since it is generating it on the fly it makes it >> all the more harder to debug and offers the least amount of control from a >> DBA point of view. So...I let SQL server do all the work on the back end >> when it comes to batch updates by collecting the parameters and sending it >> to SQL server to process for the records it needs to process. >> >> Based on what you are trying to do you are building an image of the data >> structure in memory, manipulating the data in memory and then committing the >> changes to the database in memory basically sending the update command for >> each record to the database back end right? Or am I mistaken in thinking >> that all of this magically happens by sending a single update command >> containing several individual update commands strung together? From jwcolby at colbyconsulting.com Tue Apr 29 12:48:28 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 13:48:28 -0400 Subject: [dba-VB] dba-VB Digest, Vol 54, Issue 28 In-Reply-To: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> References: <4949775.1970271209489183051.JavaMail.root@vms126.mailsrvcs.net> Message-ID: <48175F6C.5050804@colbyconsulting.com> > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? Correct. I have a 96 million record table, 155 fields. Of that I pull the PKID, NameToParse (not called that obviously) and then 6 "parsed name pieces" fields, lname, fname etc (which I added to the table to hold the pieces). > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. Not quite correct. The vb.Net results are due to the fact that I am using a 3rd party library which does the parsing of the name. Name parsing is non-trivial to say the least. That library is a .Net object (DLL actually) which you set up parameters in and then pass in a name to parse and back comes 5 to 8 "pieces". In fact fluid results are the least of my worries in SQL Server, ANY results are more like it. > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? In fact I do but that is not necessary. The name parser came with a demo form and I just modified that to display every 1000th record (which I pass to the form every 1000th record in the table), just so I could "watch it". The form is not a necessity and I may or may not get rid of it in a later iteration (probably will). I have a form (I created) which has a ServerName, DatabaseName, TableName (being parsed), PKFldName, ParseFldName, StartID, EndID, and button to trigger the parse. I fill in the pieces and push the button. Since I have control of the tables, I "hard code" the "parsed parts" field names and they will (at least for now) always be those names. I have to add those fields to the source table anyway. > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? Correct. I grab X records into an ADO.Net DataTable, iterate that table row by row, passing the Unparsed name field off to the parser. I then write the resulting "Pieces" that the parser returns (properties of the parser object) into the matching fields in the ADO.Net DataTable. The name parser handles roughly 800 names / second and is the real bottle neck. I have a StartPK / EndPK which I then manipulate in a while loop, as I finish a "chunk" I write it to SQL Server, update the StartPK / EndPK and grab another chunk to process. I have to do this for 96 million names, however I also receive weekly updates to this table which I will also have to process in a similar manner before I append them to the table. The results of this will be used to send through Accuzip for Address validation / NCOA processing. Address validation can be done just on an address, but NCOA is change of address which is about a person at an address. Once that is done... there is more but you get the picture. And in fact there is another different table from the same data source that I will have to perform this same process on. It is highly likely that I will get other lists in the future with single name fields. The world is full of idiots. ;-) John W. Colby www.ColbyConsulting.com Eric Barro wrote: > Trying to understand the scenario... > > 1. So...you have a recordset in SQL server that you pull into a VB.NET app to process the records whose main purpose is to parse field values and then updates the same records to make those values into meaningful records that you can hand over to another process, right? > > 2. The VB.Net approach is due to the fact that you can't achieve the same fluid results using just SQL server. > > Questions: > > 1. Do you actually display the data you grab for your datatable like in a grid? Or is it just a form with a button that triggers the underlying processes? > > 2. Do you grab x number of records, process them in VB.Net and then write the changes back to SQL server and then retrieve the next x number of records and so on? > > > From: jwcolby > Date: 2008/04/29 Tue AM 10:43:45 CDT > > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > > Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > > ROTFL. That is a definite NSS. > > Unfortunately I am not a DBA and with the responses I get from the SQL > group I am not even going there. I can do this purely in VB.Net (it is > cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > > No idea. I thought I could find out from the DA after the command > builder was finished building the update statement but I am getting an > error if I try to access the update property. > > John W. Colby > www.ColbyConsulting.com From robert at webedb.com Tue Apr 29 13:13:49 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 13:13:49 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804291817.m3TIH41W019987@databaseadvisors.com> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert At 12:00 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 11:43:45 -0400 >From: jwcolby >Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >To: "Discussion concerning Visual Basic and related programming > issues." >Message-ID: <48174231.4070501 at colbyconsulting.com> >Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > > Yes, that is correct. He is ending up loading the records into memory. > > Then passing individual updates back for each row of data that he > > changed. > >Yes, and most records have updates. > > > This method is not designed for doing large datasets like he is doing. > > > > It should be done on the SQL Server side. > >ROTFL. That is a definite NSS. > >Unfortunately I am not a DBA and with the responses I get from the SQL >group I am not even going there. I can do this purely in VB.Net (it is >cranking as we speak) so I will. > > > Just what are you updating...the UPDATE statement you would use to > > do what you are doing. > >No idea. I thought I could find out from the DA after the command >builder was finished building the update statement but I am getting an >error if I try to access the update property. > >John W. Colby From Gustav at cactus.dk Tue Apr 29 13:24:51 2008 From: Gustav at cactus.dk (Gustav Brock) Date: Tue, 29 Apr 2008 20:24:51 +0200 Subject: [dba-VB] SQL VS VB Message-ID: Hi Robert If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? /gustav >>> robert at webedb.com 29-04-2008 20:13 >>> Which is why it is going to take 50 hours instead of 2 or 3. If you will give me an idea of what you are updating, I can send you the SQL for the update. "I know how to do it with a hammer, therefor, I will make everything a nail." Robert From jwcolby at colbyconsulting.com Tue Apr 29 13:33:22 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:33:22 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804291817.m3TIH41W019987@databaseadvisors.com> References: <200804291817.m3TIH41W019987@databaseadvisors.com> Message-ID: <481769F2.2090303@colbyconsulting.com> It is going to take 50 hours because an external program which does the name parsing will take at least 40 hours just to parse the data. Writing to the database is only about 25% of the total time. If you managed to completely eliminate my database update time it would still take about 40 hours to complete. Thanks for the offer though. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > At 12:00 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 11:43:45 -0400 >> From: jwcolby >> Subject: Re: [dba-VB] dba-VB Digest, Vol 54, Issue 28 >> To: "Discussion concerning Visual Basic and related programming >> issues." >> Message-ID: <48174231.4070501 at colbyconsulting.com> >> Content-Type: text/plain; charset=ISO-8859-1; format=flowed >> >> > Yes, that is correct. He is ending up loading the records into memory. >> > Then passing individual updates back for each row of data that he >> > changed. >> >> Yes, and most records have updates. >> >> > This method is not designed for doing large datasets like he is doing. >> > >> > It should be done on the SQL Server side. >> >> ROTFL. That is a definite NSS. >> >> Unfortunately I am not a DBA and with the responses I get from the SQL >> group I am not even going there. I can do this purely in VB.Net (it is >> cranking as we speak) so I will. >> >> > Just what are you updating...the UPDATE statement you would use to >> > do what you are doing. >> >> No idea. I thought I could find out from the DA after the command >> builder was finished building the update statement but I am getting an >> error if I try to access the update property. >> >> John W. Colby > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 13:44:40 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 14:44:40 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <48176C98.8030201@colbyconsulting.com> > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? Precisely Gustav. What I WOULD like to do is spin off a thread to do the SQL Server update so the parser could go back to work on the next chunk. IOW put the update in parallel with the parser instead of in series with it. In fact if I had this running on the 4 core system I could probably call the parser DLL in several threads as well. In fact if I ran the VB.Net application on a separate machine, then the SQL Server machine could be just running updates and another machine running the application. What I REALLY need is a thousand blade mini-super that I could just run the entire thing in parallel. It would only take a few minutes then. ;-) John W. Colby www.ColbyConsulting.com Gustav Brock wrote: > Hi Robert > > If the parser dll runs at about 800 records/s, 93 mio. records will need 33 hours as a minimum. Or? > > /gustav > >>>> robert at webedb.com 29-04-2008 20:13 >>> > Which is why it is going to take 50 hours instead of 2 or 3. > > > If you will give me an idea of what you are updating, I can send > you the SQL for the update. > > "I know how to do it with a hammer, therefor, I will make everything > a nail." > > Robert > > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] ADO.Net Update SQL Server In-Reply-To: <481630EC.5060600@colbyconsulting.com> Message-ID: <022401c8aa2b$786352f0$6401a8c0@nant> <<< Please do not take these statements as criticisms of your code >>> Hi John, No problem with criticism - in fact it's very welcome! :) -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 12:18 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] ADO.Net Update SQL Server When I try to call the update method of the data set object I get an error: "Update requires a valid update command when passed datarow collection with modified rows." I probably sound ungrateful but in your code you appear to set up a sql statement for an update statement: adapter.UpdateCommand = New SqlCommand(Constants.UPDATE_SQL_PARAMETERIZED, cnn) adapter.UpdateCommand.Parameters.Add("@" + Constants.PARSED_NAME_COL_NAME, SqlDbType.NVarChar, Constants.NAME_COL_LENGTH, Constants.PARSED_NAME_COL_NAME) adapter.UpdateCommand.Parameters.Add("@" + Constants.ID_COL_NAME, SqlDbType.Int, 0, Constants.ID_COL_NAME) adapter.UpdateCommand.UpdatedRowSource = UpdateRowSource.None adapter.UpdateBatchSize = pageSize My problem is that is just Chinese to me. I have no background knowledge in plain English of what an update sql statement is supposed to look like (in the ADO context) and absolutely no understanding of even what I am trying to accomplish (expressed in plain English). I am uneducated, and I can find no way to get educated. I must own 6 books on ADO.Net and VB and every single one of them expects me to drag and drop objects out onto a form. OR they very helpfully set everything up and show me how to do stuff RIGHT UP TO USING IT, but never go that last step of sending updates back to the database. Open, update, display in a grid, cleanup and close. Hmm.... and I paid good money for these books. They do not discuss anything at all about what is going on at the code level. So when I try to walk through your code it is simply indecipherable to me. If I don't know what you are trying to accomplish I can't understand how you code will accomplish that thing. Please do not take these statements as criticisms of your code, simply that I am a freshman student still taking the basket weaving classes looking at masters code. Thus it isn't useful to tell me to trace through the code. MY code doesn't do any of that stuff. My code is pretty simple: Set up a connection, a data adapter, a table. Iterate the table row by row, updating fields of the rows. Now... how to update the table back to SQL Server... Given how simple everything has been up to this point I have to believe that I am just missing a concept and code to implement that concept. I have called the update method and I have googled the error messages and nothing... John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > The answer is in the code I published today on my site related to 'Re: > [dba-VB] ADO.Net': > > http://smsconsulting.spb.ru/samples/Module.vb.txt > > - Strategy 3 - trace in single threaded mode and you'll find what to do. > > Public Function SQLTDSGetDataTablePaged(ByVal startRow As Integer, ByVal > pageSize As Integer, ByRef adapter As SqlDataAdapter, ByRef dataSet As > DataSet) As DataTable > ... > > And > > Public Sub SQLTDSUpdate(ByVal adapter As SqlDataAdapter, ByVal dataSet > As DataSet) > .... > > > HTH, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, April 28, 2008 10:50 PM > To: VBA > Subject: [dba-VB] ADO.Net Update SQL Server > > Does ANYBODY know how to write a table back out to SQL Server? > > All of my books show cute little examples of building up a data table > manually then BINDING it to a control or whatever. I am not finding > anything with code example that shows how to WRITE changed data back to > SQL Server. > > I have managed to load a data table, write changes to the rows and > now... I have to write it back out to SQL Server. The table has the PK, > NameToSplit, LName, FName etc etc. What am I missing? > > I desperately need help (yea, yea, not MENTAL help) and I need it > quickly. I have been trying to do this last piece for DAYS and I am not > succeeding. > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:01:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:01:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48165E42.20404@colbyconsulting.com> Message-ID: <022801c8aa2b$787b20b0$6401a8c0@nant> <<< ...it is definitely going to be desirable to feed the update process off to a thread... >>> Hi John, I was lucky with multi-threaded batch updates' samples I posted here the day before yesterday - and the new tests show that batch updates in multi-threading environments result in MS SQL deadlocks: I should have thought about those issues before... Recap: forget batch updates - one of the scenarios of your task could be: - many readers; - many parsers; - but just one updater making batch updates, or many updaters but updating just one row in one cycle... HTH to save your time, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 3:31 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates We have ignition folks. I just updated the first 10K records in the database. I had to set the batch size to 100, as I was getting a timeout with a batch size as small as 1000 records. I haven't done any timings but it is definitely going to be desirable to feed the update process off to a thread if I can figure out how to do that. John W. Colby www.ColbyConsulting.com jwcolby wrote: > I THINK I have found my solution, called the SQLCommandBuilder. I found > reference to the command build saying how terrible it is, then a piece > of code that, in simple plain language, shows how to use it all, using > the simple objects I am already using. > > We shall see. _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 14:11:36 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 15:11:36 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022801c8aa2b$787b20b0$6401a8c0@nant> References: <022801c8aa2b$787b20b0$6401a8c0@nant> Message-ID: <481772E8.3050505@colbyconsulting.com> Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Tue Apr 29 14:43:53 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:43:53 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <30862132.1249691209498233814.JavaMail.root@vms124.mailsrvcs.net> I believe that Shamil was referring to multi-threaded updates. If you used the multi-threaded approach you could potentially end up slowing the process more than you could ever imagine because of the way SQL server locks pages of records when you run an update. From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Tue Apr 29 14:51:42 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Tue, 29 Apr 2008 23:51:42 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <481772E8.3050505@colbyconsulting.com> Message-ID: <022c01c8aa32$7269c440$6401a8c0@nant> John, I'm not sure yet what happens< I will try to find out more details in the coming days - here is the message I'm currently getting with new Strategy 4 , which I'm making similar to your way of coding this task but running in multi-threading environment: +++ System.Data.SqlClient.SqlException: Transaction (Process ID 57) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction. --- Just wanted to warn you - it might happen you'll never see this message but if you'll see it then you may find there is no workaround except abandoning the idea of batch updates running in several threads - and I mean parallel batch updates: of course, as I noted in my previous e-mail parallel reading and parsing could be done but batch updates very probably can't be run in parallel threads because they might result in deadlocks... But it looks like (from your recent posting here) that you don't need such parallel batch updates at all? Thanks. Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Tuesday, April 29, 2008 11:12 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 14:59:51 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 14:59:51 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Tue Apr 29 15:02:10 2008 From: ebarro at verizon.net (Eric Barro) Date: Tue, 29 Apr 2008 15:02:10 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <29420937.173771209499330766.JavaMail.root@vms246.mailsrvcs.net> John, This article might be of use in your scenario... http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " --Eric From: jwcolby Date: 2008/04/29 Tue PM 02:11:36 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates Interesting. Do you think that you would get deadlocks for different blocks of records? In my case there is not "a million users" scenario. This process (and its threads) are the only things modifying data. Each "chunk" is a defined block of records. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > <<< > ...it is definitely going to be desirable to feed the update > process off to a thread... > Hi John, > > I was lucky with multi-threaded batch updates' samples I posted here the day > before yesterday - and the new tests show that batch updates in > multi-threading environments result in MS SQL deadlocks: I should have > thought about those issues before... > > Recap: forget batch updates - one of the scenarios of your task could be: > > - many readers; > - many parsers; > - but just one updater making batch updates, or many updaters but updating > just one row in one cycle... > > HTH to save your time, > Shamil > > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Tuesday, April 29, 2008 3:31 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > We have ignition folks. I just updated the first 10K records in the > database. I had to set the batch size to 100, as I was getting a > timeout with a batch size as small as 1000 records. I haven't done any > timings but it is definitely going to be desirable to feed the update > process off to a thread if I can figure out how to do that. > > John W. Colby > www.ColbyConsulting.com > > > jwcolby wrote: >> I THINK I have found my solution, called the SQLCommandBuilder. I found >> reference to the command build saying how terrible it is, then a piece >> of code that, in simple plain language, shows how to use it all, using >> the simple objects I am already using. >> >> We shall see. > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 15:13:29 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:13:29 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <022c01c8aa32$7269c440$6401a8c0@nant> References: <022c01c8aa32$7269c440$6401a8c0@nant> Message-ID: <48178169.70002@colbyconsulting.com> Shamil, I have just moved the whole project to my quad core x64 server and am bringing it up now. If I can get that code running over on another machine (especially IF it is a faster machine) then the update part will be less of an issue. In any event the parse WAS taking about 15 seconds for 10K records on the dual core x32 machine which was also running SQL Server. I have designed this thing as a set of classes. There is a "data class" which is opened by a class that parses the names. The NameParser just writes the data directly into the ADO recordset in the data class. I MIGHT be able to just shuffle the class holding the open ado recordsets off into a collection. Another thread could pull those class instances out of the collection and call the update method which would start the update. As it finished the update, it would destroy that data class instance and pull the next one out of the collection. This would effectively separate the parsing and the updating, and each could proceed at it's own speed. I have not investigated yet but I might be able to have multiple threads doing the name parsing, and storing the resulting data class in that collection. It depends on whether the dll is reentrant I suppose and I really don't have a clue yet.. There would only be a single thread processing the dataset classes "update". I would need to watch that I didn't get too many open data classes at once time I suppose, though this machine has Windows x64 and 8 gigs of RAM so it has a little more breathing room than the Windows x32 machine I was running it on. Of course all of this is just "play". At least it is running now. I was also thinking about running two threads in "opposite directions", one processing data PK high to low, and the other processing data PK Low to High. Until they met in the middle that MIGHT avoid the deadlock thing. Again though, that is just play. NOT TO MENTION that I have never run a thread anyway so i have to figure THAT out. ;-) John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > I'm not sure yet what happens< I will try to find out more details in the > coming days - here is the message I'm currently getting with new Strategy 4 > , which I'm making similar to your way of coding this task but running in > multi-threading environment: > > +++ > System.Data.SqlClient.SqlException: Transaction (Process ID > 57) was deadlocked on lock resources with another process and has been > chosen as the deadlock victim. Rerun the transaction. > --- > > Just wanted to warn you - it might happen you'll never see this message but > if you'll see it then you may find there is no workaround except abandoning > the idea of batch updates running in several threads - and I mean parallel > batch updates: of course, as I noted in my previous e-mail parallel reading > and parsing could be done but batch updates very probably can't be run in > parallel threads because they might result in deadlocks... > > But it looks like (from your recent posting here) that you don't need such > parallel batch updates at all? > > Thanks. > > Shamil From jwcolby at colbyconsulting.com Tue Apr 29 15:18:50 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 16:18:50 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> References: <19901810.172101209499194554.JavaMail.root@vms246.mailsrvcs.net> Message-ID: <481782AA.6030204@colbyconsulting.com> Eric, I will read it. Unfortunately that puts me back in the SQL forum and you know how I ... uhh... well, let's just say I would rather muddle along in VB.Net. That is something that I can get better at and will spend enough time at to pay back my efforts. Besides... I come from the "just throw more hardware at it" school of Engineering. ;-) Seriously though, of course I'd LOVE to just pass the ADO recordset over and let SQL Server do it. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > John, > > This article might be of use in your scenario... > > http://www.codeproject.com/KB/database/PassingArraysIntoSPs.aspx?msg=937012 > > It talks about passing an array or datatable to a stored procedure for CRUD operations. The introductory sentence says it all... > > "Have you ever wanted to pass a DataTable, or array, containing tens or tens of thousands (or even millions) of records, into a SQL Server2000 stored procedure in just one database call? " > > --Eric > > From: jwcolby > Date: 2008/04/29 Tue PM 02:11:36 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > Interesting. Do you think that you would get deadlocks for different > blocks of records? In my case there is not "a million users" scenario. > This process (and its threads) are the only things modifying data. > Each "chunk" is a defined block of records. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> <<< >> ...it is definitely going to be desirable to feed the update >> process off to a thread... >> Hi John, >> >> I was lucky with multi-threaded batch updates' samples I posted here the day >> before yesterday - and the new tests show that batch updates in >> multi-threading environments result in MS SQL deadlocks: I should have >> thought about those issues before... >> >> Recap: forget batch updates - one of the scenarios of your task could be: >> >> - many readers; >> - many parsers; >> - but just one updater making batch updates, or many updaters but updating >> just one row in one cycle... >> >> HTH to save your time, >> Shamil >> >> >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Tuesday, April 29, 2008 3:31 AM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> We have ignition folks. I just updated the first 10K records in the >> database. I had to set the batch size to 100, as I was getting a >> timeout with a batch size as small as 1000 records. I haven't done any >> timings but it is definitely going to be desirable to feed the update >> process off to a thread if I can figure out how to do that. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> jwcolby wrote: >>> I THINK I have found my solution, called the SQLCommandBuilder. I found >>> reference to the command build saying how terrible it is, then a piece >>> of code that, in simple plain language, shows how to use it all, using >>> the simple objects I am already using. >>> >>> We shall see. >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From robert at webedb.com Tue Apr 29 16:21:31 2008 From: robert at webedb.com (Robert L. Stewart) Date: Tue, 29 Apr 2008 16:21:31 -0500 Subject: [dba-VB] SQL VS VB In-Reply-To: References: Message-ID: <200804292127.m3TLRIWT012647@databaseadvisors.com> His original estimate was 50 hours. (not mine) But, he would have to do a bit of fancy coding in SQL Server to use the same DLL he is using. That bit of information was not in the first post. At 02:02 PM 4/29/2008, you wrote: >Date: Tue, 29 Apr 2008 20:24:51 +0200 >From: "Gustav Brock" >Subject: Re: [dba-VB] SQL VS VB >To: >Message-ID: >Content-Type: text/plain; charset=US-ASCII > >Hi Robert > >If the parser dll runs at about 800 records/s, 93 mio. records will >need 33 hours as a minimum. Or? > >/gustav From jwcolby at colbyconsulting.com Tue Apr 29 16:28:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:28:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <48178169.70002@colbyconsulting.com> References: <022c01c8aa32$7269c440$6401a8c0@nant> <48178169.70002@colbyconsulting.com> Message-ID: <4817930D.3080400@colbyconsulting.com> Well I shoulda done that sooner. With the process running on Azul (the quad core x64) machine and Stonehenge (dual core x32) just processing the SQL Server thing, the parse time dropped dramatically! I am now processing about 2800 - 2900 records / second. Interestingly the update time is about the same, ~5 seconds, but that is for ~80K records. I am wondering if I am timing what i think I am timing on the update thing. 8~/ PERHAPS there is some kind of "connection" time that is the majority of the time At any rate I will finish MUCH sooner, and that with no thread stuff yet. John W. Colby www.ColbyConsulting.com jwcolby wrote: > Shamil, > > I have just moved the whole project to my quad core x64 server and am > bringing it up now. If I can get that code running over on another > machine (especially IF it is a faster machine) then the update part will > be less of an issue. In any event the parse WAS taking about 15 seconds > for 10K records on the dual core x32 machine which was also running SQL > Server. > > I have designed this thing as a set of classes. There is a "data class" > which is opened by a class that parses the names. The NameParser just > writes the data directly into the ADO recordset in the data class. I > MIGHT be able to just shuffle the class holding the open ado recordsets > off into a collection. Another thread could pull those class instances > out of the collection and call the update method which would start the > update. As it finished the update, it would destroy that data class > instance and pull the next one out of the collection. > > This would effectively separate the parsing and the updating, and each > could proceed at it's own speed. > > I have not investigated yet but I might be able to have multiple threads > doing the name parsing, and storing the resulting data class in that > collection. It depends on whether the dll is reentrant I suppose and I > really don't have a clue yet.. > > There would only be a single thread processing the dataset classes "update". > > I would need to watch that I didn't get too many open data classes at > once time I suppose, though this machine has Windows x64 and 8 gigs of > RAM so it has a little more breathing room than the Windows x32 machine > I was running it on. > > Of course all of this is just "play". At least it is running now. > > I was also thinking about running two threads in "opposite directions", > one processing data PK high to low, and the other processing data PK Low > to High. Until they met in the middle that MIGHT avoid the deadlock > thing. Again though, that is just play. > > NOT TO MENTION that I have never run a thread anyway so i have to figure > THAT out. ;-) > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> John, >> >> I'm not sure yet what happens< I will try to find out more details in the >> coming days - here is the message I'm currently getting with new Strategy 4 >> , which I'm making similar to your way of coding this task but running in >> multi-threading environment: >> >> +++ >> System.Data.SqlClient.SqlException: Transaction (Process ID >> 57) was deadlocked on lock resources with another process and has been >> chosen as the deadlock victim. Rerun the transaction. >> --- >> >> Just wanted to warn you - it might happen you'll never see this message but >> if you'll see it then you may find there is no workaround except abandoning >> the idea of batch updates running in several threads - and I mean parallel >> batch updates: of course, as I noted in my previous e-mail parallel reading >> and parsing could be done but batch updates very probably can't be run in >> parallel threads because they might result in deadlocks... >> >> But it looks like (from your recent posting here) that you don't need such >> parallel batch updates at all? >> >> Thanks. >> >> Shamil > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 16:45:10 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 17:45:10 -0400 Subject: [dba-VB] SQL VS VB In-Reply-To: <200804292127.m3TLRIWT012647@databaseadvisors.com> References: <200804292127.m3TLRIWT012647@databaseadvisors.com> Message-ID: <481796E6.7060300@colbyconsulting.com> That was indeed my original estimate. The PARSER was running at ~800 names / second. The UPDATE occurs after the parser finishes its thing., and is thus "in series with" (time wise) the parser. I just moved the VB.Net off to a more powerful machine and it is now running the parser at just over 3K records / second. I am in fact not parsing every record, I am actually processing ~80% of the records. Now I am off to see if I can figure out how to use threads. John W. Colby www.ColbyConsulting.com Robert L. Stewart wrote: > His original estimate was 50 hours. (not mine) > > But, he would have to do a bit of fancy coding in > SQL Server to use the same DLL he is using. That > bit of information was not in the first post. > > > > At 02:02 PM 4/29/2008, you wrote: >> Date: Tue, 29 Apr 2008 20:24:51 +0200 >> From: "Gustav Brock" >> Subject: Re: [dba-VB] SQL VS VB >> To: >> Message-ID: >> Content-Type: text/plain; charset=US-ASCII >> >> Hi Robert >> >> If the parser dll runs at about 800 records/s, 93 mio. records will >> need 33 hours as a minimum. Or? >> >> /gustav > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 19:03:46 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 20:03:46 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> Message-ID: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> I've fallen behind quite a bit, I see. Well, I'll catch up. Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com From jwcolby at colbyconsulting.com Tue Apr 29 20:13:25 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 21:13:25 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030f01c8aa55$aa6384d0$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop> <030f01c8aa55$aa6384d0$0502a8c0@Laptop> Message-ID: <4817C7B5.6020005@colbyconsulting.com> > I've fallen behind quite a bit, I see. eh? John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > I've fallen behind quite a bit, I see. > Well, I'll catch up. > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From mmattys at rochester.rr.com Tue Apr 29 20:23:54 2008 From: mmattys at rochester.rr.com (Michael R Mattys) Date: Tue, 29 Apr 2008 21:23:54 -0400 Subject: [dba-VB] vb.net - Updates References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> Message-ID: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Hi John, I was looking over Gustav's posts of Feb 20 with Charlotte and posts of this and nearby threads. I've been doing stuff with maps that didn't require use of large databases. The difference between an sqladapter and a tableadapter were unknown to me until today. C#, SQL Server 200X, ASP.Net All very compelling! Michael R. Mattys MapPoint & Access Dev www.mattysconsulting.com ----- Original Message ----- From: "jwcolby" To: "Discussion concerning Visual Basic and related programming issues." Sent: Tuesday, April 29, 2008 9:13 PM Subject: Re: [dba-VB] vb.net - Updates > > I've fallen behind quite a bit, I see. > > eh? > > John W. Colby > www.ColbyConsulting.com > > > Michael R Mattys wrote: >> I've fallen behind quite a bit, I see. >> Well, I'll catch up. >> >> Michael R. Mattys >> MapPoint & Access Dev >> www.mattysconsulting.com >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Tue Apr 29 21:14:27 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 29 Apr 2008 22:14:27 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <03d401c8aa60$dc0cae20$0502a8c0@Laptop> References: <0K02006A29SFSYJ0@vms173005.mailsrvcs.net><48167504.6070909@colbyconsulting.com> <034b01c8a9a0$7b784440$0502a8c0@Laptop><030f01c8aa55$aa6384d0$0502a8c0@Laptop> <4817C7B5.6020005@colbyconsulting.com> <03d401c8aa60$dc0cae20$0502a8c0@Laptop> Message-ID: <4817D603.9050308@colbyconsulting.com> Michael, > C#, SQL Server 200X, ASP.Net > All very compelling! Indeed they are. So much to know, so little time. I am slowly learning some of this, painfully slowly. John W. Colby www.ColbyConsulting.com Michael R Mattys wrote: > Hi John, > > I was looking over Gustav's posts of Feb 20 > with Charlotte and posts of this and nearby threads. > > I've been doing stuff with maps that didn't require use > of large databases. The difference between an sqladapter > and a tableadapter were unknown to me until today. > > C#, SQL Server 200X, ASP.Net > All very compelling! > > Michael R. Mattys > MapPoint & Access Dev > www.mattysconsulting.com > > ----- Original Message ----- > From: "jwcolby" > To: "Discussion concerning Visual Basic and related programming issues." > > Sent: Tuesday, April 29, 2008 9:13 PM > Subject: Re: [dba-VB] vb.net - Updates > > >>> I've fallen behind quite a bit, I see. >> eh? >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Michael R Mattys wrote: >>> I've fallen behind quite a bit, I see. >>> Well, I'll catch up. >>> >>> Michael R. Mattys >>> MapPoint & Access Dev >>> www.mattysconsulting.com >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Tue Apr 29 23:54:18 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 00:54:18 -0400 Subject: [dba-VB] Worker threads and class global variables Message-ID: <4817FB7A.4000002@colbyconsulting.com> Can a backgroundworker access a class global variable in the class that dimensions the background worker. I have a process that will be creating data classes. these classes will fill ADO data sets, modify them and then place them in a collection global to the class. This same class would then dim a backgroundworker object. The way I envision this working is that the backgroundworker is initialized and started. It simply sits in a loop checking the class global collection for data class instances deposited into the collection by the main thread. If a class instance is found in the thread it pulls it out of the collection and processes it. This backgroundworker thread's job is to run the SQL Server update code in the data class (call a method of that class). Somehow that backgroundworker has to be told that the main thread is done parsing names, ran through the entire table and is done. So I thought the main thread could set a class global boolean done flag. Or something. Can I do this at all the way I am thinking? My problem is that there is no way to predict which process will be faster, name parse or SQL Server update. My idea was simply to use a collection to act as an interface between the two. The name parser instantiates a data class which loads the data to be parsed into an ADO.Net datatable. The name parser updates the data in the the data class. When it is done updating the data it drops the class into the collection. the backgroundworker watches the collection and when it finds a data class in the collection, it pulls it out and calls the update method of that class to process the data. when it is done it cleans up the ADO objects and stores the class in another "history" collection up in the parse class. The data class will have start / stop times, start / end PKIDs, record count etc information which will be of interest to me. I have never used Background worker objects so I don't know whether it can interface in the manner described, using collections to pass objects. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Wed Apr 30 08:09:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 09:09:45 -0400 Subject: [dba-VB] Background worker threads Message-ID: <48186F99.6090401@colbyconsulting.com> I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 10:19:03 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 08:19:03 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <48186F99.6090401@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> Message-ID: You can create your own events with whatever parameters you want and raise those. Just Dim Public Event Whatever(parameter) Then raise it as needed. Charlotte Foust -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 6:10 AM To: VBA Subject: [dba-VB] Background worker threads I am new to this worker thread thing. I know that the worker can report status but it does so by raising an event and passing a single value (or object) to the event handler. I currently have a user form that in the past I simply raised an event from my class and the form sank the event and updated a text box. Which event I raised determined which text box got updated. Now my thread can't raise that same event(s) AFAICT because the event sink writes to the text box on the form. However the progress changed event that it can raise is limited to a single parameter, but I need to update several different status boxes on the main form. My thought is to create a status class which has a couple of fields, the name of the control to update and the data to put in the control. The worker thread would then instantiate one of these status classes, fill it in, and raise its ProgressChanged event passing out the status class instance. The progressChanged sink receives that status class instance and processes it. It can use the control name to raise the correct event, passing the value to be stuffed into the control. Does that make sense or is there a simpler way? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 11:05:17 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 12:05:17 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> Message-ID: <481898BD.2050301@colbyconsulting.com> Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust From shamil at smsconsulting.spb.ru Wed Apr 30 11:07:47 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:07:47 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4817D603.9050308@colbyconsulting.com> Message-ID: <02cc01c8aadc$55aeb860$6401a8c0@nant> Hi John, May I ask you to run this query (e.g. in MS SQL Management Studio) against your huge db table? --- cut here --- SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID, OWNERNAME, FName, MName, LName, NamePrefix, NameSuffix, Gender from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] --- cut here --- M.B. [Names] have to be substituted with the name of your table - what is it BTW? This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... I'm curious to know how much time approx. this query will take to execute... Thank you. -- Shamil P.S. BTW, here is a useful for your case article on multi-threading (practical solution) with good introduction - http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik Sorry, I can't react promptly on your messages in this discussion thread: a lot of custom work here, long weekends and our time zones difference but I do continue to work in this direction and if I get some useful results I will post them here... P.P.S. This book could be also useful: C# Threading Handbook by Tobin Titus et al. ISBN:1861008295 APress, LLC C 2004 (288 pages) This book addresses the fundamental units of Windows and .NET programming-threads. Coverage includes how .NET applications are executed, the life cycle of a thread in .NET, how the .NET Framework uses threads, and more. From shamil at smsconsulting.spb.ru Wed Apr 30 11:25:15 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 20:25:15 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> Message-ID: <02d901c8aade$c66c3120$6401a8c0@nant> John, Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that the fact that "The basic code WAS working before I carved it out into a worker." doesn't mean that the same code (copied & pasted) into multi-threading environment will work OK - very often the opposite will be the truth - it could produce (very) faulty solution in multi-threaded environment, which will not be easy to sort out into properly running multi-threaded code... I'd dare to advise you to prepare and test your code in VS console application (or .net console app as test-bed app and .net classlib(s) keeping all your non-UI code), and when that will work OK then add some more code to run your classlibs with your .NET WinForms application: you have a lot to do, to learn and to test - multi-threading and .NET programming can't be "captured by attack" - it's a painful and lengthy natural process, which usually takes a lot of time - I know that from my own experience and experience of others - and better not "read all that books" - find just one but good and then experiment a lot on your own - then you'll get it working properly and you'll teach all of us here - I'm quite sure about that latter! Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 8:05 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:17:04 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:17:04 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: <02d901c8aade$c66c3120$6401a8c0@nant> References: <02d901c8aade$c66c3120$6401a8c0@nant> Message-ID: <4818A990.2020306@colbyconsulting.com> Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com From cfoust at infostatsystems.com Wed Apr 30 12:28:02 2008 From: cfoust at infostatsystems.com (Charlotte Foust) Date: Wed, 30 Apr 2008 10:28:02 -0700 Subject: [dba-VB] Background worker threads In-Reply-To: <481898BD.2050301@colbyconsulting.com> References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: I'm sorry, John, I missed the contextt of your question. We don't use background workers that way, so I don't have an answer for you. Charlotte -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:05 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Charlotte, Could you be a little more verbose? I have spent about 6 hours now Googling and reading EXACTLY the same (though written by different authors) code "the button starts the thread, the thread increments the number, the thread raises its event and passes a "percent done" to the form where it is used to update a progress bar". Give me a break! To be honest I am astonished that FORTY different people thought that it was necessary to publish identical content articles. None of which answers either of my questions. I have a class. The class has TWO BackgroundWorker objects defined Withevents. bgwParseData is running a function that creates a class to load data from a SQL server, and once loaded, parse that data. The basic code WAS working before I carved it out into a worker. bgwUpdateData is running a method of a class that updates data back to SQL Server. The basic code WAS working before I carved it out into a worker. So... a single data class, two methods, one reads / parses the data in an ADO DataTable, the second writes the datatable back to SQL Server. Two threads, bgwParseData generates the data class and calls the read/Parse method, then stores the class in a collection. bgwUpdateData watches the collection, pulls the class out and calls the update method. Or that is the theory. I have asked whether it is legal for one background worker to write to a collection while a second background worker reads that collection. No answer so far. The second issue is how can bgwParseData update THREE different text boxes on the main form. All of the examples show a cute "progress bar". I don't want a progress bar, I want THREE text boxes to display data from the process running in the background worker. I know that I can create events and raise them, and I DID THAT before I started converting to backgroundworkers. That was easy. HOWEVER the way I read it I can't raise those same events from the BackgroundWorker thread because the event sink updates a physical control on the form, and that is a no-no from a thread. Unfortunately, as always happens with the books, they don't discuss anything more complicated than incrementing a number and passing that number back to a progress bar on the form. I have seen that (almost) identical code in probably FORTY different articles on using the backgroundworker. All saying EXACTLY the same, useless (to me) thing. Sigh. John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > You can create your own events with whatever parameters you want and > raise those. Just Dim Public Event Whatever(parameter) Then raise it > as needed. > > Charlotte Foust _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 12:32:45 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:32:45 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AD3D.6030003@colbyconsulting.com> It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:36:02 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:36:02 -0400 Subject: [dba-VB] Background worker threads In-Reply-To: References: <48186F99.6090401@colbyconsulting.com> <481898BD.2050301@colbyconsulting.com> Message-ID: <4818AE02.2080601@colbyconsulting.com> Bueno, thanks! John W. Colby www.ColbyConsulting.com Charlotte Foust wrote: > I'm sorry, John, I missed the contextt of your question. We don't use > background workers that way, so I don't have an answer for you. > > Charlotte > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:05 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com > > > Charlotte Foust wrote: >> You can create your own events with whatever parameters you want and >> raise those. Just Dim Public Event Whatever(parameter) Then raise it > >> as needed. >> >> Charlotte Foust > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 12:41:16 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 13:41:16 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02cc01c8aadc$55aeb860$6401a8c0@nant> References: <02cc01c8aadc$55aeb860$6401a8c0@nant> Message-ID: <4818AF3C.3040601@colbyconsulting.com> Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 13:06:29 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:06:29 +0400 Subject: [dba-VB] Background worker threads In-Reply-To: <4818A990.2020306@colbyconsulting.com> Message-ID: <02f601c8aaec$ea01f620$6401a8c0@nant> John, Please read the article I referred already: "Simple and Efficient Multi-threading with Blocking Queues" http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d cik I think it is "what doctor ordered" for your task, and it has code, C# but you can convert it if needed using free converters.... As for collections and variables - they are not thread safe by default. Read these two articles (they have sample code): "Static & Thread Safety: Part I and II" http://www.odetocode.com/Articles/313.aspx http://www.odetocode.com/Articles/314.aspx I believe the only book you need now is the one I have also mentioned already: "C# Threading Handbook by Tobin Titus et al.", ISBN:1861008295 APress, 2004 - it has a lot of code samples (C# sorry) and very good .NET multi-threading features coverage And this is an excerpt from MSDN on collections (MSDN is our best friend!): +++ Collection Object (Visual Basic) Caution Iterating through a Visual Basic Collection is not a thread-safe procedure. Even if the collection is synchronized, other threads can still modify the collection, causing the enumerator to throw an exception. To guarantee thread safety during enumeration, either lock the collection or catch the exceptions resulting from changes made by other threads. For more information on locking a programming element, see SyncLock Statement. ---- The referred above as "what doctor ordered for your task" article uses .NET's Queue class - here is the info on it from MSDN on thread safety: +++ Queue Class Thread Safety Public static (Shared in Visual Basic) members of this type are thread safe. Any instance members are not guaranteed to be thread safe. To guarantee the thread safety of the Queue, all operations must be done through the wrapper returned by the Synchronized method. Enumerating through a collection is intrinsically not a thread safe procedure. Even when a collection is synchronized, other threads can still modify the collection, which causes the enumerator to throw an exception. To guarantee thread safety during enumeration, you can either lock the collection during the entire enumeration or catch the exceptions resulting from changes made by other threads. ---- The "Simple and Efficient Multi-threading with Blocking Queues" article has sample code, which shows what simple coding is needed to implement thread-safe collections and queues... HTH, Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:17 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] Background worker threads Shamil, I have no dispute with Charlotte, other than perhaps her response didn't really tell me anything. But that brings up an good question - "one but good". All of my books (each one of them) has exactly one to 1.5 pages of working with BackGroundWorker. This is precisely my problem, that they all want to do the same "increment a number and call a progress event" example. How about 4 different threads, all asynchronous processes, each of them handing off work to another thread (or multiple threads). THAT is real life. I have a process that reads and parses data. There is no way to predict how long that will take. However long it takes it needs to feed another process that performs the next step, in this case writing it back to the database. There is no way to know how long that step will take. One thread needs to write to a queue, the next thread needs to read from the queue. Potentially that step needs to write to a new queue which is read by a third thread. So what book handles this subject in depth? AFAICT your code uses threads directly, correct? Not a backgroundWorker object? I do NOT want to jump off of that bridge just yet. Just a quick question though, no one is answering whether one thread can WRITE to a variable and another thread READ from that variable. Are variables thread safe? Are collections thread safe? If so then I can at least pass off the data class between the two threads by a collection in the middle. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Sorry, I'm jumping in your dispute with Charlotte - just wanted to note that > the fact that "The basic code WAS working before I carved it out into a > worker." doesn't mean that the same code (copied & pasted) into > multi-threading environment will work OK - very often the opposite will be > the truth - it could produce (very) faulty solution in multi-threaded > environment, which will not be easy to sort out into properly running > multi-threaded code... > > I'd dare to advise you to prepare and test your code in VS console > application (or .net console app as test-bed app and .net classlib(s) > keeping all your non-UI code), and when that will work OK then add some more > code to run your classlibs with your .NET WinForms application: you have a > lot to do, to learn and to test - multi-threading and .NET programming can't > be "captured by attack" - it's a painful and lengthy natural process, which > usually takes a lot of time - I know that from my own experience and > experience of others - and better not "read all that books" - find just one > but good and then experiment a lot on your own - then you'll get it working > properly and you'll teach all of us here - I'm quite sure about that latter! > > > Thank you. > > -- > Shamil > > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 8:05 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] Background worker threads > > Charlotte, > > Could you be a little more verbose? > > I have spent about 6 hours now Googling and reading EXACTLY the same > (though written by different authors) code "the button starts the > thread, the thread increments the number, the thread raises its event > and passes a "percent done" to the form where it is used to update a > progress bar". > > Give me a break! > > To be honest I am astonished that FORTY different people thought that it > was necessary to publish identical content articles. None of which > answers either of my questions. > > I have a class. > > The class has TWO BackgroundWorker objects defined Withevents. > > bgwParseData is running a function that creates a class to load data > from a SQL server, and once loaded, parse that data. The basic code WAS > working before I carved it out into a worker. > > bgwUpdateData is running a method of a class that updates data back to > SQL Server. The basic code WAS working before I carved it out into a > worker. > > So... a single data class, two methods, one reads / parses the data in > an ADO DataTable, the second writes the datatable back to SQL Server. > > Two threads, bgwParseData generates the data class and calls the > read/Parse method, then stores the class in a collection. bgwUpdateData > watches the collection, pulls the class out and calls the update method. > > Or that is the theory. I have asked whether it is legal for one > background worker to write to a collection while a second background > worker reads that collection. No answer so far. > > The second issue is how can bgwParseData update THREE different text > boxes on the main form. All of the examples show a cute "progress bar". > I don't want a progress bar, I want THREE text boxes to display data > from the process running in the background worker. > > I know that I can create events and raise them, and I DID THAT before I > started converting to backgroundworkers. That was easy. HOWEVER the > way I read it I can't raise those same events from the BackgroundWorker > thread because the event sink updates a physical control on the form, > and that is a no-no from a thread. > > Unfortunately, as always happens with the books, they don't discuss > anything more complicated than incrementing a number and passing that > number back to a progress bar on the form. I have seen that (almost) > identical code in probably FORTY different articles on using the > backgroundworker. All saying EXACTLY the same, useless (to me) thing. > > Sigh. > > John W. Colby > www.ColbyConsulting.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:22:08 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:22:08 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AD3D.6030003@colbyconsulting.com> Message-ID: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 13:28:58 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 22:28:58 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818AF3C.3040601@colbyconsulting.com> Message-ID: <02fb01c8aaf0$0e512110$6401a8c0@nant> Thank you, John, You have a very good PC, lucky man! :) As I wrote already I (I'm sorry) did post incorrect query - I wanted to run this one: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] It will also return 96K rows for your huge table, but it's expected (here) it will get first results instantly, and it will finish within just several (3?) seconds... Please prove am I wrong with that bright above expectations on the execution time of the fixed version of the query?... Thank you. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:41 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server x32) 13:51 to pull 96,417 rows for the result set. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From ebarro at verizon.net Wed Apr 30 13:50:38 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 13:50:38 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Wednesday, April 30, 2008 9:33 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates It started returning results in about 12-13 seconds. You are asking for 96 thousand result rows you realize. This database is 96 million rows, so every thousandth row is 96K rows. It is still processing at 9 minutes. I'll email the final execution time whenever the query finishes. This is fascinating because I pull just the fields into an ADO.Net table for 100K rows in just a second or two. Of course these are consecutive rows, it is not scanning the entire table. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Hi John, > > May I ask you to run this query (e.g. in MS SQL Management Studio) against > your huge db table? > > --- cut here --- > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, > PKID, OWNERNAME, FName, MName, LName, NamePrefix, > NameSuffix, Gender from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > --- cut here --- > > M.B. [Names] have to be substituted with the name of your table - what is it > BTW? > > This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... > > I'm curious to know how much time approx. this query will take to execute... > > Thank you. > > -- > Shamil > > P.S. BTW, here is a useful for your case article on multi-threading > (practical solution) with good introduction - > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d > cik > > Sorry, I can't react promptly on your messages in this discussion thread: a > lot of custom work here, long weekends and our time zones difference but I > do continue to work in this direction and if I get some useful results I > will post them here... > > P.P.S. This book could be also useful: > > C# Threading Handbook > by Tobin Titus et al. > > ISBN:1861008295 > > APress, LLC C 2004 (288 pages) > > This book addresses the fundamental units of Windows and .NET > programming-threads. Coverage includes how .NET applications are executed, > the life cycle of a thread in .NET, how the .NET Framework uses threads, and > more. > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 14:31:28 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Wed, 30 Apr 2008 23:31:28 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <822189.2679181209581438859.JavaMail.root@vms075.mailsrvcs.net> Message-ID: <030a01c8aaf8$c971d130$6401a8c0@nant> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead of ~3 seconds.... SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc] (nolock)) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] (nolock) BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of useful advices including all kinds of locking - the first time I see so detailed advices and samples on that subject (but I must say I didn't read that much computer books in the past)... Thanks. -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Wednesday, April 30, 2008 10:51 PM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates try to add (nolock) after the table names in your select queries... From: Shamil Salakhetdinov Date: 2008/04/30 Wed PM 01:22:08 CDT To: "'Discussion concerning Visual Basic and related programming issues.'" Subject: Re: [dba-VB] vb.net - Updates John, Yes, I do realize it returns 96K records... I'm sorry - I should have not used anything except PK in the query fields' list - i.e. I wanted to run this query: SELECT RowNum, PKID from ( select Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID from [Names]) s where (s.RowNum % 1000) = 1 union all Select NULL as RowNum, Max(PKID) as PKID from [Names] I expected it will use only the index and therefore should run rather quickly producing first results instantly and finishing within 3(? three ?) seconds? Here I have tested similar query against a table with only 1,374,163 rows but queried every 10th row, i.e. it returned 137K rows, and this query finished in 3 seconds: SELECT RowNum, CompetitionresultId from ( select Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as RowNum, CompetitionresultId from [CompetitionresultArc]) s where (s.RowNum % 10) = 1 union all Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from [CompetitionresultArc] Thanks. -- Shamil From jwcolby at colbyconsulting.com Wed Apr 30 14:55:15 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:55:15 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> References: <02fa01c8aaef$19a3ccd0$6401a8c0@nant> Message-ID: <4818CEA3.5000704@colbyconsulting.com> Well, I do not know what you want me to do with the result. To just open a query window in SSMS and execute it in there I get pretty much the same result. First results in 14 seconds and ten minutes and counting. I don't have "just the PK" indexed, at least not intentionally and visibly. I am now using cover indexes that index the PKID and one or more fields that I am interested in. Whether there is a PKID index behind the scenes I don't know. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:33 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > It started returning results in about 12-13 seconds. You are asking for > 96 thousand result rows you realize. This database is 96 million rows, > so every thousandth row is 96K rows. > > It is still processing at 9 minutes. I'll email the final execution > time whenever the query finishes. > > This is fascinating because I pull just the fields into an ADO.Net table > for 100K rows in just a second or two. Of course these are consecutive > rows, it is not scanning the entire table. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 14:59:44 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 15:59:44 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <030a01c8aaf8$c971d130$6401a8c0@nant> References: <030a01c8aaf8$c971d130$6401a8c0@nant> Message-ID: <4818CFB0.4020402@colbyconsulting.com> I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Wed Apr 30 15:19:00 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:19:00 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <02fb01c8aaf0$0e512110$6401a8c0@nant> References: <02fb01c8aaf0$0e512110$6401a8c0@nant> Message-ID: <4818D434.4090104@colbyconsulting.com> Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From ebarro at verizon.net Wed Apr 30 15:24:32 2008 From: ebarro at verizon.net (Eric Barro) Date: Wed, 30 Apr 2008 15:24:32 -0500 (CDT) Subject: [dba-VB] vb.net - Updates Message-ID: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 15:32:43 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 16:32:43 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> References: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <4818D76B.7000602@colbyconsulting.com> Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4270063.1747621209587072803.JavaMail.root@vms172.mailsrvcs.net> Message-ID: <031501c8ab05$422d2460$6401a8c0@nant> <<< I see the UNION ALL and ORDER BY as bottlenecks. >>> Eric, But union all is executed with two row sets: - first having every 1000th record, and - second having ONLY ONE record... When query execution plan is analyzed then I see that first operation for both parts of UNION ALL query is index scan IOW UNION ALL and ORDER BY do not look as a bottleneck because UNION ALL merges (without any attempts to suppress full duplicate) a set of records with just one record, and because ORDER by of clustered index is 0% cost operation? Thank you -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro Sent: Thursday, May 01, 2008 12:25 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates How about this approach? 1. Create a temp table with the output fields you want in the final result. 2. INSERT the results of the first SELECT statement into the temp table. 3. INSERT the results of the second SELECT statement into the same temp table. 4. Run the SELECT statement on the temp table. I see the UNION ALL and ORDER BY as bottlenecks. From: jwcolby Date: 2008/04/30 Wed PM 02:59:44 CDT To: "Discussion concerning Visual Basic and related programming issues." Subject: Re: [dba-VB] vb.net - Updates I added the nolock in both places and it returned first results in 8 seconds instead of 14. However the whole result set is still taking awhile. I will post the result when it is in. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead > of ~3 seconds.... > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc] (nolock)) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] (nolock) > > BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers > for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of > useful advices including all kinds of locking - the first time I see so > detailed advices and samples on that subject (but I must say I didn't read > that much computer books in the past)... > > Thanks. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro > Sent: Wednesday, April 30, 2008 10:51 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > try to add (nolock) after the table names in your select queries... > > From: Shamil Salakhetdinov > Date: 2008/04/30 Wed PM 01:22:08 CDT > To: > "'Discussion concerning Visual Basic and related programming > issues.'" > Subject: Re: [dba-VB] vb.net - Updates > > John, > > Yes, I do realize it returns 96K records... > > I'm sorry - I should have not used anything except PK in the query fields' > list - i.e. I wanted to run this query: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > I expected it will use only the index and therefore should run rather > quickly producing first results instantly and finishing within 3(? three ?) > seconds? > > Here I have tested similar query against a table with only 1,374,163 rows > but queried every 10th row, i.e. it returned 137K rows, and this query > finished in 3 seconds: > > SELECT RowNum, CompetitionresultId from ( select > Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as > RowNum, CompetitionresultId > from [CompetitionresultArc]) s > where (s.RowNum % 10) = 1 > union all > Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from > [CompetitionresultArc] > > Thanks. > > -- > Shamil > > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:00:45 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:00:45 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D434.4090104@colbyconsulting.com> Message-ID: <031601c8ab05$42476320$6401a8c0@nant> <<< Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? >>> Yes, it was, but I expected you'll get results much quicker... Could that be that you do not have (clustered)index on your PK field? Can you: Right-Click on table name -> Script Table as -> CREATE to -> New Query Window => I'm getting the following for your simulated table. CREATE TABLE [dbo].[Names]( [PKID] [int] IDENTITY(1,1) NOT NULL, [OWNERNAME] [nvarchar](255) NOT NULL, [FName] [nvarchar](50) NULL, [MName] [nvarchar](50) NULL, [LName] [nvarchar](50) NULL, [NamePrefix] [nvarchar](50) NULL, [NameSuffix] [nvarchar](50) NULL, [Gender] [nvarchar](50) NULL, [ParsedName] [nvarchar](255) NULL, CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED ( [PKID] ASC ) WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] ) ON [PRIMARY] Also if you select query's text in SSMS query window, and then Ctrl+L then you'll get query execution plan - here as the last step with 99% cost I'm getting: "Clustered Index Scan" IOW the query doesn't scan table's data at all - it just scans the index... Thanks, -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:19 AM To: Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil, > You have a very good PC, lucky man! :) As I mentioned elsewhere, I come from the "throw more hardware at it" school of Engineering. ;-) I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster in many cases. I do not have this large database over on that machine, and it would take me hours to get it over there so i can't test this query on that, at least today. that database has not been compacted recently and is about 300 Gigs with 136 gigs free space. I will compact it tonight and then copy the smaller file to Azul. As for your new query, with the NoLocks in place it returned first results in only 8 seconds as opposed to 14 before, but it still took 13:59 to pull the entire result set. Again, that is in SSMS, directly running a query and "displaying" it in a query pane. Was that your intention? John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > Thank you, John, > > You have a very good PC, lucky man! :) > > As I wrote already I (I'm sorry) did post incorrect query - I wanted to run > this one: > > SELECT RowNum, PKID from ( select > Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID > from [Names]) s > where (s.RowNum % 1000) = 1 > union all > Select NULL as RowNum, Max(PKID) as PKID from [Names] > > It will also return 96K rows for your huge table, but it's expected (here) > it will get first results instantly, and it will finish within just several > (3?) seconds... > > Please prove am I wrong with that bright above expectations on the execution > time of the fixed version of the query?... > > Thank you. > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Wednesday, April 30, 2008 9:41 PM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server > x32) 13:51 to pull 96,417 rows for the result set. > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Hi John, >> >> May I ask you to run this query (e.g. in MS SQL Management Studio) against >> your huge db table? >> >> --- cut here --- >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >> NameSuffix, Gender from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> --- cut here --- >> >> M.B. [Names] have to be substituted with the name of your table - what is > it >> BTW? >> >> This query returns PKID of every 1000th row sorted by PKID + MAX(PKID).... >> >> I'm curious to know how much time approx. this query will take to > execute... >> Thank you. >> >> -- >> Shamil >> >> P.S. BTW, here is a useful for your case article on multi-threading >> (practical solution) with good introduction - >> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >> cik >> >> Sorry, I can't react promptly on your messages in this discussion thread: > a >> lot of custom work here, long weekends and our time zones difference but I >> do continue to work in this direction and if I get some useful results I >> will post them here... >> >> P.P.S. This book could be also useful: >> >> C# Threading Handbook >> by Tobin Titus et al. >> >> ISBN:1861008295 >> >> APress, LLC C 2004 (288 pages) >> >> This book addresses the fundamental units of Windows and .NET >> programming-threads. Coverage includes how .NET applications are executed, >> the life cycle of a thread in .NET, how the .NET Framework uses threads, > and >> more. >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From shamil at smsconsulting.spb.ru Wed Apr 30 16:07:55 2008 From: shamil at smsconsulting.spb.ru (Shamil Salakhetdinov) Date: Thu, 1 May 2008 01:07:55 +0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <4818D76B.7000602@colbyconsulting.com> Message-ID: <032201c8ab06$42a01500$6401a8c0@nant> John, This ORDER BY used here to get ROW_NUMBER is/was expected as very speedy "virtual" zero cost operation because I thought/expected your PK (PKID) was a (clustered) index... -- Shamil -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, May 01, 2008 12:33 AM To: ebarro at verizon.net; Discussion concerning Visual Basic and related programming issues. Subject: Re: [dba-VB] vb.net - Updates Shamil will have to do that. I am just running the tests. 8-) In fact I am not sure why the OrderBy anyway. The PKID is a PK and is in order anyway, though you couldn't be sure of that in the generic sense. John W. Colby www.ColbyConsulting.com Eric Barro wrote: > How about this approach? > > 1. Create a temp table with the output fields you want in the final result. > 2. INSERT the results of the first SELECT statement into the temp table. > 3. INSERT the results of the second SELECT statement into the same temp table. > 4. Run the SELECT statement on the temp table. > > I see the UNION ALL and ORDER BY as bottlenecks. > > From: jwcolby > Date: 2008/04/30 Wed PM 02:59:44 CDT > To: > "Discussion concerning Visual Basic and related programming issues." > Subject: Re: [dba-VB] vb.net - Updates > > I added the nolock in both places and it returned first results in 8 > seconds instead of 14. However the whole result set is still taking > awhile. I will post the result when it is in. > > > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Yes, it helped, Eric, thanks - the query finished now in ~2 seconds instead >> of ~3 seconds.... >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc] (nolock)) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] (nolock) >> >> BTW, Re: JC question on books - I have here "SQL Server 2000, Fast Answers >> for DBAs and Developers" by Joseph Sack - ISBN : 1-59059-592-0 - full of >> useful advices including all kinds of locking - the first time I see so >> detailed advices and samples on that subject (but I must say I didn't read >> that much computer books in the past)... >> >> Thanks. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Eric Barro >> Sent: Wednesday, April 30, 2008 10:51 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> try to add (nolock) after the table names in your select queries... >> >> From: Shamil Salakhetdinov >> Date: 2008/04/30 Wed PM 01:22:08 CDT >> To: >> "'Discussion concerning Visual Basic and related programming >> issues.'" >> Subject: Re: [dba-VB] vb.net - Updates >> >> John, >> >> Yes, I do realize it returns 96K records... >> >> I'm sorry - I should have not used anything except PK in the query fields' >> list - i.e. I wanted to run this query: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> I expected it will use only the index and therefore should run rather >> quickly producing first results instantly and finishing within 3(? three ?) >> seconds? >> >> Here I have tested similar query against a table with only 1,374,163 rows >> but queried every 10th row, i.e. it returned 137K rows, and this query >> finished in 3 seconds: >> >> SELECT RowNum, CompetitionresultId from ( select >> Cast(ROW_NUMBER() over (ORDER BY [CompetitionresultId]) as int) as >> RowNum, CompetitionresultId >> from [CompetitionresultArc]) s >> where (s.RowNum % 10) = 1 >> union all >> Select NULL as RowNum, Max(CompetitionresultId) as CompetitionresultId from >> [CompetitionresultArc] >> >> Thanks. >> >> -- >> Shamil >> >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Wed Apr 30 16:16:08 2008 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 30 Apr 2008 17:16:08 -0400 Subject: [dba-VB] vb.net - Updates In-Reply-To: <031601c8ab05$42476320$6401a8c0@nant> References: <031601c8ab05$42476320$6401a8c0@nant> Message-ID: <4818E198.30501@colbyconsulting.com> I do have the clustered stuff exactly as you show: > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] Likewise I am getting the same 99% cost clustered index scan. I have to guess it is something else. There is a lot of empty space in the database file which I will compact out tonight. It takes hours because of the size and when it is doing that I cannot use the db. Perhaps I have severe fragmentation or something. The database size is 300 GIGS. And finally just remember that when all is said and done it is still a hundred times larger than your database. No matter how you slice it, it has to process more physical disk sectors etc. John W. Colby www.ColbyConsulting.com Shamil Salakhetdinov wrote: > > <<< > Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. > Was that your intention? > Yes, it was, but I expected you'll get results much quicker... > > Could that be that you do not have (clustered)index on your PK field? > > Can you: Right-Click on table name -> Script Table as -> CREATE to -> New > Query Window => I'm getting the following for your simulated table. > > CREATE TABLE [dbo].[Names]( > [PKID] [int] IDENTITY(1,1) NOT NULL, > [OWNERNAME] [nvarchar](255) NOT NULL, > [FName] [nvarchar](50) NULL, > [MName] [nvarchar](50) NULL, > [LName] [nvarchar](50) NULL, > [NamePrefix] [nvarchar](50) NULL, > [NameSuffix] [nvarchar](50) NULL, > [Gender] [nvarchar](50) NULL, > [ParsedName] [nvarchar](255) NULL, > CONSTRAINT [PK_Names] PRIMARY KEY CLUSTERED > ( > [PKID] ASC > ) > WITH (PAD_INDEX = OFF, > STATISTICS_NORECOMPUTE = OFF, > IGNORE_DUP_KEY = OFF, > ALLOW_ROW_LOCKS = ON, > ALLOW_PAGE_LOCKS = ON) ON [PRIMARY] > ) ON [PRIMARY] > > Also if you select query's text in SSMS query window, and then Ctrl+L then > you'll get query execution plan - here as the last step with 99% cost I'm > getting: "Clustered Index Scan" IOW the query doesn't scan table's data at > all - it just scans the index... > > Thanks, > > -- > Shamil > > -----Original Message----- > From: dba-vb-bounces at databaseadvisors.com > [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Thursday, May 01, 2008 12:19 AM > To: Discussion concerning Visual Basic and related programming issues. > Subject: Re: [dba-VB] vb.net - Updates > > Shamil, > > > You have a very good PC, lucky man! :) > > As I mentioned elsewhere, I come from the "throw more hardware at it" > school of Engineering. ;-) > > I have another machine (Azul) with a quad core (AMDPhenom) 2.2ghz, 8 > gigs ram, Win2k3 x64 SQL Server 2005 x64. That machine is MUCH faster > in many cases. > > I do not have this large database over on that machine, and it would > take me hours to get it over there so i can't test this query on that, > at least today. that database has not been compacted recently and is > about 300 Gigs with 136 gigs free space. I will compact it tonight and > then copy the smaller file to Azul. > > As for your new query, with the NoLocks in place it returned first > results in only 8 seconds as opposed to 14 before, but it still took > 13:59 to pull the entire result set. Again, that is in SSMS, directly > running a query and "displaying" it in a query pane. Was that your > intention? > > John W. Colby > www.ColbyConsulting.com > > > Shamil Salakhetdinov wrote: >> Thank you, John, >> >> You have a very good PC, lucky man! :) >> >> As I wrote already I (I'm sorry) did post incorrect query - I wanted to > run >> this one: >> >> SELECT RowNum, PKID from ( select >> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, PKID >> from [Names]) s >> where (s.RowNum % 1000) = 1 >> union all >> Select NULL as RowNum, Max(PKID) as PKID from [Names] >> >> It will also return 96K rows for your huge table, but it's expected (here) >> it will get first results instantly, and it will finish within just > several >> (3?) seconds... >> >> Please prove am I wrong with that bright above expectations on the > execution >> time of the fixed version of the query?... >> >> Thank you. >> >> -- >> Shamil >> >> -----Original Message----- >> From: dba-vb-bounces at databaseadvisors.com >> [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Wednesday, April 30, 2008 9:41 PM >> To: Discussion concerning Visual Basic and related programming issues. >> Subject: Re: [dba-VB] vb.net - Updates >> >> Shamil, >> >> It took Stonehenge (2.8ghz AMD x2, 4 gigs Ram, Windows x32, SQL Server >> x32) 13:51 to pull 96,417 rows for the result set. >> >> John W. Colby >> www.ColbyConsulting.com >> >> >> Shamil Salakhetdinov wrote: >>> Hi John, >>> >>> May I ask you to run this query (e.g. in MS SQL Management Studio) > against >>> your huge db table? >>> >>> --- cut here --- >>> >>> SELECT RowNum, PKID from ( select >>> Cast(ROW_NUMBER() over (ORDER BY [PKID]) as int) as RowNum, >>> PKID, OWNERNAME, FName, MName, LName, NamePrefix, >>> NameSuffix, Gender from [Names]) s >>> where (s.RowNum % 1000) = 1 >>> union all >>> Select NULL as RowNum, Max(PKID) as PKID from [Names] >>> >>> --- cut here --- >>> >>> M.B. [Names] have to be substituted with the name of your table - what is >> it >>> BTW? >>> >>> This query returns PKID of every 1000th row sorted by PKID + > MAX(PKID).... >>> I'm curious to know how much time approx. this query will take to >> execute... >>> Thank you. >>> >>> -- >>> Shamil >>> >>> P.S. BTW, here is a useful for your case article on multi-threading >>> (practical solution) with good introduction - >>> > http://www.dotnetjunkies.com/Tutorial/D7E688B8-0BDD-4D44-9A0F-4CD26FB35F51.d >>> cik >>> >>> Sorry, I can't react promptly on your messages in this discussion thread: >> a >>> lot of custom work here, long weekends and our time zones difference but > I >>> do continue to work in this direction and if I get some useful results I >>> will post them here... >>> >>> P.P.S. This book could be also useful: >>> >>> C# Threading Handbook >>> by Tobin Titus et al. >>> >>> ISBN:1861008295 >>> >>> APress, LLC C 2004 (288 pages) >>> >>> This book addresses the fundamental units of Windows and .NET >>> programming-threads. Coverage includes how .NET applications are > executed, >>> the life cycle of a thread in .NET, how the .NET Framework uses threads, >> and >>> more. >>> >>> >>> _______________________________________________ >>> dba-VB mailing list >>> dba-VB at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-vb >>> http://www.databaseadvisors.com >>> >>> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-VB mailing list >> dba-VB at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-vb >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > >