From jwcolby at colbyconsulting.com Mon Nov 2 07:58:56 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 02 Nov 2009 08:58:56 -0500 Subject: [dba-SQLServer] SQL Server 2005: Running stored procedures from C# / VBA Message-ID: <4AEEE5A0.4040901@colbyconsulting.com> I am embarking on the task of running the "DB from hell" from C#. I have several dozen stored procedures that perform various tasks and I currently run many of them from Access using a single procedure provided by Charlotte. I need a similar function written in C# (preferably) or VB.Net. ATM I have an Access database which has two specific forms, bound to two tables. These two forms each execute a set of SPs which build temp tables of data to be exported to an external process, build indexes to make the processes faster, BCP the data out to CSV files, and then the reverse on the way back in. I have another Access database that creates an order for the same client. Same kind of thing, temp tables created, indexes, fields added and so forth. My intention is to port each of these processes to C#. The reason is simply that C# has a couple of huge advantages over VBA, such as child threads and built-in capabilities in the framework that I have to really scrounge to make work in VBA. The first thing I need though is code that can execute a stored procedure, pass parameters to it, retrieve parameters back etc. With demo code preferably. Also I would like to open a conversation about how using .Net from inside of SQL Server (2005) works. Does anyone on the list use C# or VB.Net from inside of SQL Server? Where is the code stored? How to do you call it? What do you use it for? TIA, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Tue Nov 3 21:12:23 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 03 Nov 2009 22:12:23 -0500 Subject: [dba-SQLServer] How do you run SQL Server processes Message-ID: <4AF0F117.20209@colbyconsulting.com> I have processes that I run which use sets of stored procedures. For example two of the processes export / import records from large tables for address validation. The addresses need to be sorted in Zip5/Zip4/Address order in order to make the external validation process as fast as possible so I create a table that pulls millions of records, often tens of millions of records, into a table (created on-the-fly) in sorted order, then pulls out 2 million records at a time and BCPs them out for processing in an external program. The inverse occurs when the records finish processing, 2 million record chunks are imported back into a temp table, then each 2 million record chunk is appended to a main table. Once back in SQL Server I have processes that build clustered and non-clustered indexes, run sha-1 hash functions to generate hash keys, update address valid codes etc. I will run this entire project at least once a month (to keep the addresses valid as people move), and I do so for many tables - a table of about 65 million addresses, another of 21 million, another of 11 million etc. I store all of my stored procedures in a dedicated database that pretty much just contains the SPs and a few UDFs as well as a couple of logging type tables. At the moment I use an Access database to run the individual stored procedures in specific order. I am just embarking on a project to move that functioning but not very efficient Access database to C#. I want to be able to have better monitoring of SP completion, records processed (counts), time to complete each SP etc. I found a pretty nice set of C# classes that allow creation of "stored procedure" objects, with parameter objects, then stored into a collection as the SPs (class instances) are assembled and finally the whole collection of SPs executed. I created a test database of a million names / addresses and today I started testing my existing SPs on this small data set. In the next few days I expect to build out a preliminary "port" of the existing Access VBA code. I am hoping to eventually make the entire process "table driven" where I can store (in a table) the name of the sp, the parameters in a child table etc. and then have a supervisor pull out the records and execute the SPs based on what is in the table. This would allow me to create the SPs and then use a C# application (data entry forms) to organize the SPs, add / delete SPs from the process and so forth. That is down the road of course, for now I will have hard coded sequences - as I already do in supervisor SPs as well as in hard coded Access VBA modules. As a programmer, doing this supervisor level in C# feels more natural to me (than doing it inside of SQL Server), and gives me the flexibility to do what I want in code external to the SQL Server database being manipulated. I want to be able to do things external to the database processes, things like monitor the directories that the BCP processes export to / from, move files popping out of a bcp process off to a VM on a different server, monitor those VMs for files coming out of their processes, moving the files back over to the SQL Server directories, and trigger BCP processes that import the data back in to SQL server. I need to be able to FTP order and data files to clients, and get (FTP) files from process bureaus. So it FEELS to me like using a language such as C# to build a system outside of SQL Server is the way to go. BTW I stumbled across something called SMO (SQL Server Management Objects) which makes it really easy to get at the SQL Server object model from C#. I can apparently see the entire tree structure that is a server/database/tables/etc. and do I know not what (yet) with those objects. Looks pretty cool though. The database is a large part of what I do but it is not the whole picture and I have been handcuffed by my lack of ability to effectively do these external parts. But I am a programmer at heart. I want to know how you do this kind of stuff. I know that running such "systems" of stored procedures tied to external processes must be quite common and I am wondering what you guys do for supervising your processes. If anyone doing anything remotely similar would take the time to explain how you do it, I would be most appreciative. Thanks, -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Wed Nov 4 10:55:01 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Wed, 04 Nov 2009 11:55:01 -0500 Subject: [dba-SQLServer] Lock Pages in Memory Message-ID: <4AF1B1E5.4010205@colbyconsulting.com> I ran across this today. http://www.sqlservercentral.com/blogs/glennberry/archive/2009/10/29/suggested-max-memory-settings-for-sql-server-2005_2F00_2008.aspx It mentions in passing the "lock pages in memory" setting. http://support.microsoft.com/kb/918483 Apparently SQL Server can swap some fairly critical SQL Server pages to disk at inopportune times and this setting prevents that. It was only available on the Enterprise edition but apparently MS relented and made it available to the masses (standard version as well). I truly have no clue (as you guys know) ;) but we shall see whether that helps me at all. -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Thu Nov 5 07:52:53 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 05 Nov 2009 08:52:53 -0500 Subject: [dba-SQLServer] [dba-VB] Fwd: How do you run SQL Server processes In-Reply-To: References: <4AF207CD.2060307@colbyconsulting.com> Message-ID: <4AF2D8B5.1030507@colbyconsulting.com> Mark, > However, I think that Robert is correct in suggestion to try, as much as possible to not cross the technology stack with volumes of data when you do not have to. I have never had a problem with letting each tool do what it does best. In my case (or for this specific client) the database is decidedly not "normal". I have many different databases, each with a few tables, and ZERO (yes ZERO) parent / child tables. I keep tables of names / addresses. That's it. So I am not doing transactions, I am not doing reports, I am not doing data entry, I am not doing any of the "normal" things that people do with databases. What I have to do is get the entire table of name / address out to an external process and then back in to the same database, and I have to do this at least monthly. We are talking 20, 50, 80 million names exported into 2 million record files, processed through this external program, and then imported back in again. This is a "huge" process, not in terms of sheer SQL complexity but rather in terms of the steps involved, the monitoring that has to occur, and the time it takes. And I have to do this on a half dozen large databases, monthly. The external program processes about 2 million records an hour start to finish. Do the math and 60 million records turns into 30 hours of external processing. I use three virtual machines to get three times the records / hour but you are still talking 10 hours and that doesn't count the time to get the records out and back in to SQL Server again. My thought was to start a discussion about that "periphery processing" so that I could get ideas from people who already do it. I am getting almost no response however with actual "I do that using..." responses, and I am wondering why. Perhaps because I deal with small clients, the cost of manual labor to do those kinds of things by hand is too high so they hire me to automate it. Perhaps in a big organization they simply throw people at the problem? Believe me, when you get down to a "one person company" (me!) you can't do this stuff manually for very long, you MUST automate it or you will spend you entire day doing stuff that should be done by the computer. My original question was aimed at discovering how other people handle this kind of processing. I have another client (in Access in this case) that routinely exports data using very specialized export functions (that I wrote) dictated by mainframe interface specs. The data has to be pulled, exported, formatted, sometimes encrypted, sometimes zipped, sometimes FTPd, sometimes emailed, (the "sometimes" part depending on the recipient's preferences), and emails sent announcing the data is on the way etc. I do all of this in Access currently but it is ugly in Access because VBA does not have the built-in tools to do ftp/zip/encrypt etc. C# has some of that built-in and you can usually find C# routines to do the rest. BTW, as I have mentioned, I am taking a C# class at the local community college. I met a Russian gentleman, 60ish, who has been in programming / IT all of his life, recently unemployed. He and I are the only two in the class that have actually worked in the industry (other than the teacher). We struck up a conversation and he asked me if I would like to bid with him on a couple of small C# jobs on a jobs board "just to get some real world experience". I counter-offered that he join me in automating some of my real-world tasks. He came over yesterday and we are going to be working together on the kinds of tasks I described above. He has a LOT of C# experience but almost no database experience beyond simple select queries. We should each learn a lot from the other, it should be fun, and hopefully I can get some of this automation done! John W. Colby www.ColbyConsulting.com Mark Breen wrote: > Hello John, > > FWIW, I agree with you that you need a rich environment to do the work that > you plan to do. > > However, I think that Robert is correct in suggestion to try, as much as > possible to not cross the technology stack with volumes of data when you do > not have to. > > IOW, do all the heavy lifting in the database, and by all means return as > many parameters or log results back to the front end you are building in C#, > but try to avoid returning your data when ever possible. > > I guess I was suggesting the same with with SSIS. Where you have to export > data based on decisions, SSIS is an extremely fast option. > > Have you already used EXECUTE command to call an sproc within an sproc? > This is a useful means of creating an sproc, that running a set of sprocs, > and can insert / update a log table as it progresses. For data only > actvity, this might be effective. > > For all the front end work however, I guess you will make C# sing, like only > you can, so work away, and let TSQL do what it is good at, all the rest > should not involve enormous volumes of data, so the technology bridge will > be irrelevant. > > Good luck, you know you are lucky. Most Database people never get to read > about a db as big as what you have :) > > On last thing, in SQL Server, Cursors are a dirty word, I never see them > used. However, when you have low volumes of iteration (thousands or a few > tens of thousands), cursors are a very powerful way of enumerating tasks. > > Eg, lets imagine you had a table of states, and you needed to perform ten > seperate sprocs for each state. You could use a cursor to enumerate the > states table and pass in the state name to the ten different sprocs. > > You could log between each iteration, and you could have the whole thing > running in 30 minutes. In this sort of context, I agree with Robert, that > TSQL can give you Raw power quickly and easily. > > C# could be watching your log table and your 'Status Update' table and > simply returning 1 or 2 records from the db to C# per minute, whereas within > your sproc that is enumerating it might be handling millions of records each > two minutes. Those millions would never cross the stack divide of DB Server > to .Net. Again this sharing of processing power is probably what Robert was > referring to. > > BTW I recently started using FileZilla and find it super. > > Good Luck, > > Mark > > > > Mark > _______________________________________________ > dba-VB mailing list > dba-VB at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-vb > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Thu Nov 5 20:16:51 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 05 Nov 2009 21:16:51 -0500 Subject: [dba-SQLServer] Joins don't discriminate Message-ID: <4AF38713.7030008@colbyconsulting.com> For ages I have been using a SHA1 hash of my name / address fields to create what I call hashfields. I feed Addr + zip5 + zip4 into SHA1 and store the result into HashAddr - varbool(200). Add last name to that string and feed to SHA1 to create HashFamily. And finally add first name to that string and feed to Sha1 to create HashPerson. I have been pulling my hair out trying to discover why a simple join on those fields would yield a different number of records than a join on my hash fields. IOW join Addr, Zip5, Zip4 and you pull X records but join HashAddr and you get Y records. It turns out that a join on the fields themselves ignore text case. SHA1 does not. Thus to a multifield join Twin Pines Drive and twin pines drive are identical but SHA1 returns a different hash string. Sigh! At least the mystery is solved. -- John W. Colby www.ColbyConsulting.com From dwaters at usinternet.com Mon Nov 9 10:10:03 2009 From: dwaters at usinternet.com (Dan Waters) Date: Mon, 9 Nov 2009 10:10:03 -0600 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express Message-ID: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> When I open SQL Express on my PC, it can take 20 or 30 minutes to finally open to where I can work. I have TCP/IP and Shared Memory enabled. What could be causing this? From fhtapia at gmail.com Mon Nov 9 10:27:22 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Mon, 9 Nov 2009 08:27:22 -0800 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express In-Reply-To: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> References: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> Message-ID: what is your hardware configuration and OS you are working on? -- Francisco On Nov 9, 2009, at 8:10 AM, Dan Waters wrote: > When I open SQL Express on my PC, it can take 20 or 30 minutes to > finally > open to where I can work. > > I have TCP/IP and Shared Memory enabled. > > What could be causing this? > > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > From dwaters at usinternet.com Mon Nov 9 13:01:48 2009 From: dwaters at usinternet.com (Dan Waters) Date: Mon, 9 Nov 2009 13:01:48 -0600 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express In-Reply-To: References: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> Message-ID: I have a fast PC I built about two years ago. XP Pro, Quad core, 2 Gb fast ram, etc. It has opened very quickly in the past. Also, even opening a small table to look at data is quite slow in addition to a slow opening. Management Studio did open this morning in about 7 or 8 minutes, not 20 or 30. Thanks! Dan -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Francisco Tapia Sent: Monday, November 09, 2009 10:27 AM To: Discussion concerning MS SQL Server Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express what is your hardware configuration and OS you are working on? -- Francisco On Nov 9, 2009, at 8:10 AM, Dan Waters wrote: > When I open SQL Express on my PC, it can take 20 or 30 minutes to > finally > open to where I can work. > > I have TCP/IP and Shared Memory enabled. > > What could be causing this? > > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fhtapia at gmail.com Mon Nov 9 21:31:41 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Mon, 9 Nov 2009 19:31:41 -0800 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express In-Reply-To: References: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> Message-ID: check the size of the transaction logs, they maybe overgrown, and causes slow performance, in general they ought to be 10% of the data files, to shrink them do a backup log databasename with truncate_only -- this empties the log but the space is still reserved followed by dbcc shrinkfile ('transaction log logical name', 1) -- replace the 1 with whatever 10% of your datafile is. -Francisco http://sqlthis.blogspot.com | Tsql and More... On Mon, Nov 9, 2009 at 11:01 AM, Dan Waters wrote: > I have a fast PC I built about two years ago. XP Pro, Quad core, 2 Gb fast > ram, etc. > > It has opened very quickly in the past. Also, even opening a small table > to > look at data is quite slow in addition to a slow opening. > > Management Studio did open this morning in about 7 or 8 minutes, not 20 or > 30. > > Thanks! > Dan > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Francisco > Tapia > Sent: Monday, November 09, 2009 10:27 AM > To: Discussion concerning MS SQL Server > Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express > > what is your hardware configuration and OS you are working on? > > -- > Francisco > On Nov 9, 2009, at 8:10 AM, Dan Waters wrote: > > > When I open SQL Express on my PC, it can take 20 or 30 minutes to > > finally > > open to where I can work. > > > > I have TCP/IP and Shared Memory enabled. > > > > What could be causing this? > > > > > > > > > > _______________________________________________ > > dba-SQLServer mailing list > > dba-SQLServer at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > > http://www.databaseadvisors.com > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From dwaters at usinternet.com Wed Nov 11 14:58:36 2009 From: dwaters at usinternet.com (Dan Waters) Date: Wed, 11 Nov 2009 14:58:36 -0600 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express In-Reply-To: References: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> Message-ID: <134B564B45534127993F7A696FF57927@danwaters> Hi Francisco, I finally got a chance to try this. Both my mdf and ldf files for the only database I have are about 30 Mb in size. I suspect that this is not an issue for SQL Server if everything else is going well. Also, when I opened the Back Up Database window, under Options, the two options under Transaction Log choices, Truncate the transaction log and Back up the tail of the log, are both disabled. By the way, what is DBCC? At my customer's site I've been getting errors trying to do certain things because I don't have 'DBCC Permissions', although I am the owner of that database in SQL Server. The DBA there says that she doesn't understand this, and is currently to busy too investigate. Any thoughts on this problem? Thanks! Dan -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Francisco Tapia Sent: Monday, November 09, 2009 9:32 PM To: Discussion concerning MS SQL Server Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express check the size of the transaction logs, they maybe overgrown, and causes slow performance, in general they ought to be 10% of the data files, to shrink them do a backup log databasename with truncate_only -- this empties the log but the space is still reserved followed by dbcc shrinkfile ('transaction log logical name', 1) -- replace the 1 with whatever 10% of your datafile is. -Francisco http://sqlthis.blogspot.com | Tsql and More... On Mon, Nov 9, 2009 at 11:01 AM, Dan Waters wrote: > I have a fast PC I built about two years ago. XP Pro, Quad core, 2 Gb fast > ram, etc. > > It has opened very quickly in the past. Also, even opening a small table > to > look at data is quite slow in addition to a slow opening. > > Management Studio did open this morning in about 7 or 8 minutes, not 20 or > 30. > > Thanks! > Dan > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Francisco > Tapia > Sent: Monday, November 09, 2009 10:27 AM > To: Discussion concerning MS SQL Server > Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express > > what is your hardware configuration and OS you are working on? > > -- > Francisco > On Nov 9, 2009, at 8:10 AM, Dan Waters wrote: > > > When I open SQL Express on my PC, it can take 20 or 30 minutes to > > finally > > open to where I can work. > > > > I have TCP/IP and Shared Memory enabled. > > > > What could be causing this? > > > > > > > > > > _______________________________________________ > > dba-SQLServer mailing list > > dba-SQLServer at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > > http://www.databaseadvisors.com > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fhtapia at gmail.com Wed Nov 11 17:23:38 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Wed, 11 Nov 2009 15:23:38 -0800 Subject: [dba-SQLServer] Very Long Opening Time for SQL Express In-Reply-To: <134B564B45534127993F7A696FF57927@danwaters> References: <4BD2E48CBCE541DF914DFD688980E8F7@danwaters> <134B564B45534127993F7A696FF57927@danwaters> Message-ID: DBCC : Database console command On the security issue That's because you are only a database owner and not a system admin Sent from my mobile On Nov 11, 2009, at 12:58 PM, "Dan Waters" wrote: > Hi Francisco, > > I finally got a chance to try this. Both my mdf and ldf files for > the only > database I have are about 30 Mb in size. I suspect that this is not > an > issue for SQL Server if everything else is going well. > > Also, when I opened the Back Up Database window, under Options, the > two > options under Transaction Log choices, Truncate the transaction log > and Back > up the tail of the log, are both disabled. > > By the way, what is DBCC? At my customer's site I've been getting > errors > trying to do certain things because I don't have 'DBCC Permissions', > although I am the owner of that database in SQL Server. The DBA > there says > that she doesn't understand this, and is currently to busy too > investigate. > Any thoughts on this problem? > > Thanks! > Dan > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of > Francisco > Tapia > Sent: Monday, November 09, 2009 9:32 PM > To: Discussion concerning MS SQL Server > Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express > > check the size of the transaction logs, they maybe overgrown, and > causes > slow performance, in general they ought to be 10% of the data files, > to > shrink them do a > > backup log databasename with truncate_only -- this empties the log > but the > space is still reserved > followed by > dbcc shrinkfile ('transaction log logical name', 1) -- replace the 1 > with > whatever 10% of your datafile is. > > > -Francisco > http://sqlthis.blogspot.com | Tsql and More... > > > On Mon, Nov 9, 2009 at 11:01 AM, Dan Waters > wrote: > >> I have a fast PC I built about two years ago. XP Pro, Quad core, 2 >> Gb fast >> ram, etc. >> >> It has opened very quickly in the past. Also, even opening a small >> table >> to >> look at data is quite slow in addition to a slow opening. >> >> Management Studio did open this morning in about 7 or 8 minutes, >> not 20 or >> 30. >> >> Thanks! >> Dan >> >> -----Original Message----- >> From: dba-sqlserver-bounces at databaseadvisors.com >> [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of >> Francisco >> Tapia >> Sent: Monday, November 09, 2009 10:27 AM >> To: Discussion concerning MS SQL Server >> Subject: Re: [dba-SQLServer] Very Long Opening Time for SQL Express >> >> what is your hardware configuration and OS you are working on? >> >> -- >> Francisco >> On Nov 9, 2009, at 8:10 AM, Dan Waters wrote: >> >>> When I open SQL Express on my PC, it can take 20 or 30 minutes to >>> finally >>> open to where I can work. >>> >>> I have TCP/IP and Shared Memory enabled. >>> >>> What could be causing this? >>> >>> >>> >>> >>> _______________________________________________ >>> dba-SQLServer mailing list >>> dba-SQLServer at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >>> http://www.databaseadvisors.com >>> >> >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > From jwcolby at colbyconsulting.com Thu Nov 12 07:42:35 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 12 Nov 2009 08:42:35 -0500 Subject: [dba-SQLServer] An interesting tale Message-ID: <4AFC10CB.2070007@colbyconsulting.com> I need to export 40 million records, from a table of 50 million records where these records 40 million records are not in another list. So I build up a "not in" kind of query to grab the PKs, and a count to verify I got the right things. The count runs in about 6 minutes. I join that NotIn to the table from which I am pulling the data to export and start the export running (to a csv file). This was yesterday afternoon. By midnight, still no file. By this morning still no file BUT... by this morning the export dialog said I had managed to export 20 thousand records so far. WHAT??? So I go create a table to hold just the PK of the records NotIn the other table. Populate that with the PKs (took about 5 minutes), index it (took about 30 seconds) removed the NotIn query and replaced with the NotIn table and start the export running. Two minutes later I have 40 million records in a csv file. Obviously I am not real hip on the query analyzer, but would it or some other tool warn me that something is going to take the rest of eternity to complete? -- John W. Colby www.ColbyConsulting.com From Gustav at cactus.dk Thu Nov 12 07:50:58 2009 From: Gustav at cactus.dk (Gustav Brock) Date: Thu, 12 Nov 2009 14:50:58 +0100 Subject: [dba-SQLServer] An interesting tale Message-ID: Hi John I have the same experience though in a much smaller scale - going from close-to-impossible to a snap; temp tables are perfect in some cases. /gustav >>> jwcolby at colbyconsulting.com 12-11-2009 14:42 >>> I need to export 40 million records, from a table of 50 million records where these records 40 million records are not in another list. So I build up a "not in" kind of query to grab the PKs, and a count to verify I got the right things. The count runs in about 6 minutes. I join that NotIn to the table from which I am pulling the data to export and start the export running (to a csv file). This was yesterday afternoon. By midnight, still no file. By this morning still no file BUT... by this morning the export dialog said I had managed to export 20 thousand records so far. WHAT??? So I go create a table to hold just the PK of the records NotIn the other table. Populate that with the PKs (took about 5 minutes), index it (took about 30 seconds) removed the NotIn query and replaced with the NotIn table and start the export running. Two minutes later I have 40 million records in a csv file. Obviously I am not real hip on the query analyzer, but would it or some other tool warn me that something is going to take the rest of eternity to complete? -- John W. Colby www.ColbyConsulting.com From jwcolby at colbyconsulting.com Thu Nov 12 08:57:13 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 12 Nov 2009 09:57:13 -0500 Subject: [dba-SQLServer] An interesting tale In-Reply-To: References: Message-ID: <4AFC2249.5030107@colbyconsulting.com> > I have the same experience though in a much smaller scale - going from close-to-impossible to a snap; temp tables are perfect in some cases. Yep, in agreement there. There must be some strange interplay with the export engine and the data engine though. If I can built a count query on top of the query I am about to use for export, and that count returns in about 6 minutes, there is no way on God's green earth it should take overnight to export 20 thousand of my 40 million records using that same export query. John W. Colby www.ColbyConsulting.com Gustav Brock wrote: > Hi John > > I have the same experience though in a much smaller scale - going from close-to-impossible to a snap; temp tables are perfect in some cases. > > /gustav > >>>> jwcolby at colbyconsulting.com 12-11-2009 14:42 >>> > I need to export 40 million records, from a table of 50 million records where these records 40 > million records are not in another list. So I build up a "not in" kind of query to grab the PKs, > and a count to verify I got the right things. The count runs in about 6 minutes. I join that NotIn > to the table from which I am pulling the data to export and start the export running (to a csv file). > > This was yesterday afternoon. By midnight, still no file. By this morning still no file BUT... by > this morning the export dialog said I had managed to export 20 thousand records so far. > > WHAT??? > > So I go create a table to hold just the PK of the records NotIn the other table. Populate that with > the PKs (took about 5 minutes), index it (took about 30 seconds) removed the NotIn query and > replaced with the NotIn table and start the export running. Two minutes later I have 40 million > records in a csv file. > > Obviously I am not real hip on the query analyzer, but would it or some other tool warn me that > something is going to take the rest of eternity to complete? > From fhtapia at gmail.com Thu Nov 12 09:36:03 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Thu, 12 Nov 2009 07:36:03 -0800 Subject: [dba-SQLServer] An interesting tale In-Reply-To: <4AFC10CB.2070007@colbyconsulting.com> References: <4AFC10CB.2070007@colbyconsulting.com> Message-ID: <6CBC9AF5-50C1-4E42-BA8C-1F521FA5F198@gmail.com> So how long does the query take and what tool are you using when you say you are exporting? With query analyzer you just have to change the export location be it grid text or file. Query Analyzer will display an estimated execution plan so you can review what theengine wants to do such as doing a full table scan or an index scan. It is with these tools that you can add query optimizer hints if these things need to be analyzed. Sent from my mobile On Nov 12, 2009, at 5:42 AM, jwcolby wrote: > I need to export 40 million records, from a table of 50 million > records where these records 40 > million records are not in another list. So I build up a "not in" > kind of query to grab the PKs, > and a count to verify I got the right things. The count runs in > about 6 minutes. I join that NotIn > to the table from which I am pulling the data to export and start > the export running (to a csv file). > > This was yesterday afternoon. By midnight, still no file. By this > morning still no file BUT... by > this morning the export dialog said I had managed to export 20 > thousand records so far. > > WHAT??? > > So I go create a table to hold just the PK of the records NotIn the > other table. Populate that with > the PKs (took about 5 minutes), index it (took about 30 seconds) > removed the NotIn query and > replaced with the NotIn table and start the export running. Two > minutes later I have 40 million > records in a csv file. > > Obviously I am not real hip on the query analyzer, but would it or > some other tool warn me that > something is going to take the rest of eternity to complete? > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > From accessd at shaw.ca Thu Nov 12 11:46:15 2009 From: accessd at shaw.ca (Jim Lawrence) Date: Thu, 12 Nov 2009 09:46:15 -0800 Subject: [dba-SQLServer] An interesting tale In-Reply-To: <4AFC10CB.2070007@colbyconsulting.com> References: <4AFC10CB.2070007@colbyconsulting.com> Message-ID: Ahhhh, the old imfamous 'NOT IN' clause. Only good if you are going for a coffee but in your case going camping. The other point you noted is that even a 'SELECT * ...' would take a weekend unless indexed. ;-) Jim -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, November 12, 2009 5:43 AM To: Dba-Sqlserver Subject: [dba-SQLServer] An interesting tale I need to export 40 million records, from a table of 50 million records where these records 40 million records are not in another list. So I build up a "not in" kind of query to grab the PKs, and a count to verify I got the right things. The count runs in about 6 minutes. I join that NotIn to the table from which I am pulling the data to export and start the export running (to a csv file). This was yesterday afternoon. By midnight, still no file. By this morning still no file BUT... by this morning the export dialog said I had managed to export 20 thousand records so far. WHAT??? So I go create a table to hold just the PK of the records NotIn the other table. Populate that with the PKs (took about 5 minutes), index it (took about 30 seconds) removed the NotIn query and replaced with the NotIn table and start the export running. Two minutes later I have 40 million records in a csv file. Obviously I am not real hip on the query analyzer, but would it or some other tool warn me that something is going to take the rest of eternity to complete? -- John W. Colby www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fhtapia at gmail.com Thu Nov 12 12:35:46 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Thu, 12 Nov 2009 10:35:46 -0800 Subject: [dba-SQLServer] An interesting tale In-Reply-To: References: <4AFC10CB.2070007@colbyconsulting.com> Message-ID: Instead of using not in, use an outter join and it will return quicker. On 11/12/09, Jim Lawrence wrote: > Ahhhh, the old imfamous 'NOT IN' clause. Only good if you are going for a > coffee but in your case going camping. The other point you noted is that > even a 'SELECT * ...' would take a weekend unless indexed. ;-) > > Jim > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Thursday, November 12, 2009 5:43 AM > To: Dba-Sqlserver > Subject: [dba-SQLServer] An interesting tale > > I need to export 40 million records, from a table of 50 million records > where these records 40 > million records are not in another list. So I build up a "not in" kind of > query to grab the PKs, > and a count to verify I got the right things. The count runs in about 6 > minutes. I join that NotIn > to the table from which I am pulling the data to export and start the export > running (to a csv file). > > This was yesterday afternoon. By midnight, still no file. By this morning > still no file BUT... by > this morning the export dialog said I had managed to export 20 thousand > records so far. > > WHAT??? > > So I go create a table to hold just the PK of the records NotIn the other > table. Populate that with > the PKs (took about 5 minutes), index it (took about 30 seconds) removed the > NotIn query and > replaced with the NotIn table and start the export running. Two minutes > later I have 40 million > records in a csv file. > > Obviously I am not real hip on the query analyzer, but would it or some > other tool warn me that > something is going to take the rest of eternity to complete? > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- Sent from Gmail for mobile | mobile.google.com -Francisco http://sqlthis.blogspot.com | Tsql and More... From jwcolby at colbyconsulting.com Thu Nov 12 14:35:49 2009 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 12 Nov 2009 15:35:49 -0500 Subject: [dba-SQLServer] An interesting tale In-Reply-To: References: <4AFC10CB.2070007@colbyconsulting.com> Message-ID: <4AFC71A5.6030602@colbyconsulting.com> well... that is what I did. Outer join where FieldOverThere is null "Not in" John W. Colby www.ColbyConsulting.com Francisco Tapia wrote: > Instead of using not in, use an outter join and it will return quicker. > > > > > On 11/12/09, Jim Lawrence wrote: >> Ahhhh, the old imfamous 'NOT IN' clause. Only good if you are going for a >> coffee but in your case going camping. The other point you noted is that >> even a 'SELECT * ...' would take a weekend unless indexed. ;-) >> >> Jim >> >> -----Original Message----- >> From: dba-sqlserver-bounces at databaseadvisors.com >> [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby >> Sent: Thursday, November 12, 2009 5:43 AM >> To: Dba-Sqlserver >> Subject: [dba-SQLServer] An interesting tale >> >> I need to export 40 million records, from a table of 50 million records >> where these records 40 >> million records are not in another list. So I build up a "not in" kind of >> query to grab the PKs, >> and a count to verify I got the right things. The count runs in about 6 >> minutes. I join that NotIn >> to the table from which I am pulling the data to export and start the export >> running (to a csv file). >> >> This was yesterday afternoon. By midnight, still no file. By this morning >> still no file BUT... by >> this morning the export dialog said I had managed to export 20 thousand >> records so far. >> >> WHAT??? >> >> So I go create a table to hold just the PK of the records NotIn the other >> table. Populate that with >> the PKs (took about 5 minutes), index it (took about 30 seconds) removed the >> NotIn query and >> replaced with the NotIn table and start the export running. Two minutes >> later I have 40 million >> records in a csv file. >> >> Obviously I am not real hip on the query analyzer, but would it or some >> other tool warn me that >> something is going to take the rest of eternity to complete? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> > From fhtapia at gmail.com Thu Nov 12 15:45:47 2009 From: fhtapia at gmail.com (Francisco Tapia) Date: Thu, 12 Nov 2009 13:45:47 -0800 Subject: [dba-SQLServer] An interesting tale In-Reply-To: <4AFC71A5.6030602@colbyconsulting.com> References: <4AFC10CB.2070007@colbyconsulting.com> <4AFC71A5.6030602@colbyconsulting.com> Message-ID: <540126B9-1BEB-4579-ACBD-C9954AFDC7A4@gmail.com> so are you running this in query analyzer and it's taking for ever? or are you exporting using some other tool and it's taking forever? -- Francisco On Nov 12, 2009, at 12:35 PM, jwcolby wrote: > well... that is what I did. > > Outer join where FieldOverThere is null > > "Not in" > > John W. Colby > www.ColbyConsulting.com > > > Francisco Tapia wrote: >> Instead of using not in, use an outter join and it will return quicker. >> >> >> >> >> On 11/12/09, Jim Lawrence wrote: >>> Ahhhh, the old imfamous 'NOT IN' clause. Only good if you are going for a >>> coffee but in your case going camping. The other point you noted is that >>> even a 'SELECT * ...' would take a weekend unless indexed. ;-) >>> >>> Jim >>> >>> -----Original Message----- >>> From: dba-sqlserver-bounces at databaseadvisors.com >>> [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby >>> Sent: Thursday, November 12, 2009 5:43 AM >>> To: Dba-Sqlserver >>> Subject: [dba-SQLServer] An interesting tale >>> >>> I need to export 40 million records, from a table of 50 million records >>> where these records 40 >>> million records are not in another list. So I build up a "not in" kind of >>> query to grab the PKs, >>> and a count to verify I got the right things. The count runs in about 6 >>> minutes. I join that NotIn >>> to the table from which I am pulling the data to export and start the export >>> running (to a csv file). >>> >>> This was yesterday afternoon. By midnight, still no file. By this morning >>> still no file BUT... by >>> this morning the export dialog said I had managed to export 20 thousand >>> records so far. >>> >>> WHAT??? >>> >>> So I go create a table to hold just the PK of the records NotIn the other >>> table. Populate that with >>> the PKs (took about 5 minutes), index it (took about 30 seconds) removed the >>> NotIn query and >>> replaced with the NotIn table and start the export running. Two minutes >>> later I have 40 million >>> records in a csv file. >>> >>> Obviously I am not real hip on the query analyzer, but would it or some >>> other tool warn me that >>> something is going to take the rest of eternity to complete? >>> >>> -- >>> John W. Colby >>> www.ColbyConsulting.com >>> _______________________________________________ >>> dba-SQLServer mailing list >>> dba-SQLServer at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >>> http://www.databaseadvisors.com >>> >>> _______________________________________________ >>> dba-SQLServer mailing list >>> dba-SQLServer at databaseadvisors.com >>> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >>> http://www.databaseadvisors.com >>> >>> >> > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > From marklbreen at gmail.com Fri Nov 13 00:55:04 2009 From: marklbreen at gmail.com (Mark Breen) Date: Fri, 13 Nov 2009 14:55:04 +0800 Subject: [dba-SQLServer] An interesting tale In-Reply-To: <4AFC10CB.2070007@colbyconsulting.com> References: <4AFC10CB.2070007@colbyconsulting.com> Message-ID: Hello John, Another thing to add to your toolset of handy tips is to avail of the SELECT TOP command in SQL. I use that then I am about to launch a set of routines that I think may be slow. It enables me, in advance to know whether the SQL is efficient enough to run or whether I need to consider breaking it into two ore more steps. In your case, I would first write all the code with Select top 10, then expand that to top 100,000 or even 1M, and only if all looks Ok would I kick it off. Just a hint as to how I do larger transactions, HTH, Mark 2009/11/12 jwcolby > I need to export 40 million records, from a table of 50 million records > where these records 40 > million records are not in another list. So I build up a "not in" kind of > query to grab the PKs, > and a count to verify I got the right things. The count runs in about 6 > minutes. I join that NotIn > to the table from which I am pulling the data to export and start the > export running (to a csv file). > > This was yesterday afternoon. By midnight, still no file. By this morning > still no file BUT... by > this morning the export dialog said I had managed to export 20 thousand > records so far. > > WHAT??? > > So I go create a table to hold just the PK of the records NotIn the other > table. Populate that with > the PKs (took about 5 minutes), index it (took about 30 seconds) removed > the NotIn query and > replaced with the NotIn table and start the export running. Two minutes > later I have 40 million > records in a csv file. > > Obviously I am not real hip on the query analyzer, but would it or some > other tool warn me that > something is going to take the rest of eternity to complete? > > -- > John W. Colby > www.ColbyConsulting.com > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From ab-mi at post3.tele.dk Fri Nov 13 17:30:08 2009 From: ab-mi at post3.tele.dk (Asger Blond) Date: Sat, 14 Nov 2009 00:30:08 +0100 Subject: [dba-SQLServer] BULK INSERT versus OPENROWSET(BULK...) Message-ID: Importing flat textfiles to an SQL Server table which is the fastest: BULK INSERT myTable FROM '' Or INSERT myTable SELECT a.* FROM OPENROWSET(BULK '') AS a Eventually using format files (same question): BULK INSERT myTable FROM '' WITH(FORMATFILE=') Or INSERT myTable SELECT a.* FROM OPENROWSET(BULK '', FORMATFILE='') AS a Do you have any experience or documentation hint? TIA Asger