From jwcolby at colbyconsulting.com Sat Dec 1 11:04:04 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 1 Dec 2007 12:04:04 -0500 Subject: [dba-SQLServer] Converting the value in a field Message-ID: <005501c8343c$2deeb580$647aa8c0@M90> I have lines of code like this: mintNumOfRecsToExport = !EDS_FileSize The compile complains because !EDS_FileSize is an object (a field) and mintNumOfRecsToExport is an integer variable. The error is: Error 2 Option Strict On disallows implicit conversions from 'Object' to 'Integer'. There is no .Value property for the field object !EDS_FileSize. Will simply wrapping the object in CInt() fix the problem, or will I get an integer pointer to the object? John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Dec 1 11:30:31 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 1 Dec 2007 12:30:31 -0500 Subject: [dba-SQLServer] Ionforge Message-ID: <005601c8343f$e5533a40$647aa8c0@M90> Is anyone using Ionforge VCS? Like it? Easy to use, hard to use? Gotchas for .Net? Misc comments? John W. Colby Colby Consulting www.ColbyConsulting.com From dwaters at usinternet.com Sun Dec 2 15:35:13 2007 From: dwaters at usinternet.com (Dan Waters) Date: Sun, 2 Dec 2007 15:35:13 -0600 Subject: [dba-SQLServer] Finding Fields Using Reserved Words Message-ID: <000001c8352b$39836f40$0300a8c0@danwaters> I have a medium sized BE that I will be converting to SQL Server soon. Before that, I'd like to fix the reserved words that are there. I know of a couple, but I want to be confident that I get them all. Does anyone know of a utility to compare table field names to the reserved words for Access, Jet, and SQL Server? I can make one, but if it's already out there . . . Thanks! Dan From michael at ddisolutions.com.au Sun Dec 2 16:35:37 2007 From: michael at ddisolutions.com.au (Michael Maddison) Date: Mon, 3 Dec 2007 09:35:37 +1100 Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size References: <20071201153715.DIX24665@dommail.onthenet.com.au> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D0128A152@ddi-01.DDI.local> Hi Borge, You can create a job to shrink then db and/or the log file. You must backup the files before you can shrink the db file and truncate the log file. Data Files. SQL will grow the data file when it needs to based on the parameters you set when the db is created, by default the value is 10%. Growing the db is 'expensive' and will lock the db while it occurs. A common practice is to set the start size at the size required for 1 years activity and grow it if required during downtime. Log Files. Log transactions in the db and can grow very fast. Backup and truncate as required. The above is a very simplistic explanation of how SQL manages its files. In general DBA's don't worry too much about db size, they just keep adding more disks ;-) Logs size is managed because its part of the backup routine. HTH Michael M Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size Hi all, SQL2005 : If I daily create a table, populate it with a bunch of data, use it to perform some updates and then drop it - will this just keep growing the database or will SQL Server manage the disk space.... I am asking the question as my experience with Access Dbs is that ever so often you had to compact the Db if you did something like this.... Does SQL Server handle this better, i.e. without user intervention? Regards Borge _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From pcs at azizaz.com Sun Dec 2 20:37:43 2007 From: pcs at azizaz.com (pcs at azizaz.com) Date: Mon, 3 Dec 2007 12:37:43 +1000 (EST) Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size Message-ID: <20071203123743.DIZ37863@dommail.onthenet.com.au> Michael, Thanks for reply. I user drop and insert into for the temp table and truncate for the 'live' table. SP via ADODB connection and with BeginsTrans - CommitTrans and RollBackTrans .... the temp table takes up about 30Mb - I ran procedure to create and drop the temp table 10 times .... The .mdf and .ldf did not increase in size.... So far so good.... I have set the Db to auto shrink. I understand that on Simple Full Recovery backup the Log File will be truncated automatically following a backup .... So once the Db is placed in production you recommend to increase the size to allow for say a year's growth.... Can I change the size using SQL Server Management Studio or do I need to run an SP? In regards to the Insert Into SP that creates the temp table.... It's a mirrored table of the 'live ' one but with no indexes..... Can I just tag a create PK index to the end of the Insert Into SP or do I need to run it separate? The PK is a clustered index on two columns. When filtering on just one of the columns is the PK clustered index good for that, or do I need to create an index on each of the two columns that make up the PK to get best performance?? Regards Borge ---- Original message ---- >Date: Mon, 3 Dec 2007 09:35:37 +1100 >From: "Michael Maddison" >Subject: Re: [dba-SQLServer] Temporary Table - Impact on SQL Db size >To: > >Hi Borge, > >You can create a job to shrink then db and/or the log file. >You must backup the files before you can shrink the db file and truncate >the log file. > >Data Files. >SQL will grow the data file when it needs to based on the parameters you >set when the db is created, by >default the value is 10%. Growing the db is 'expensive' and will lock >the db while it occurs. A common practice >is to set the start size at the size required for 1 years activity and >grow it if required during downtime. > >Log Files. >Log transactions in the db and can grow very fast. Backup and truncate >as required. > >The above is a very simplistic explanation of how SQL manages its files. >In general DBA's don't worry too much >about db size, they just keep adding more disks ;-) Logs size is >managed because its part of the backup routine. > >HTH > > >Michael M > >Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size > >Hi all, > >SQL2005 : >If I daily create a table, populate it with a bunch of data, use it to >perform some updates and then drop it - will this just keep growing the >database or will SQL Server manage the disk space.... I am asking the >question as my experience with Access Dbs is that ever so often you had >to compact the Db if you did something like this.... Does SQL Server >handle this better, i.e. without user intervention? > >Regards >Borge > >_______________________________________________ >dba-SQLServer mailing list >dba-SQLServer at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >http://www.databaseadvisors.com > > >_______________________________________________ >dba-SQLServer mailing list >dba-SQLServer at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >http://www.databaseadvisors.com > From michael at ddisolutions.com.au Sun Dec 2 21:53:37 2007 From: michael at ddisolutions.com.au (Michael Maddison) Date: Mon, 3 Dec 2007 14:53:37 +1100 Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size References: <20071203123743.DIZ37863@dommail.onthenet.com.au> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D0128A162@ddi-01.DDI.local> See inline === Michael, Thanks for reply. I user drop and insert into for the temp table and truncate for the 'live' table. SP via ADODB connection and with BeginsTrans - CommitTrans and RollBackTrans .... the temp table takes up about 30Mb - I ran procedure to create and drop the temp table 10 times .... The .mdf and .ldf did not increase in size.... So far so good.... === If you are using a # temp table then the table is being created and dropped from tempdb. (if not you should be...) ie select * from #t I have set the Db to auto shrink. === Is it a multiuser app? If so I wouldn't. I understand that on Simple Full Recovery backup the Log File will be truncated automatically following a backup .... So once the Db is placed in production you recommend to increase the size to allow for say a year's growth.... === Can't hurt ;-) Can I change the size using SQL Server Management Studio === Yes or do I need to run an SP? === No, but possible. In regards to the Insert Into SP that creates the temp table.... It's a mirrored table of the 'live ' one but with no indexes..... Can I just tag a create PK index to the end of the Insert Into SP or do I need to run it separate? === I would do it like this... --EXEC dbo.x CREATE procedure dbo.x as CREATE TABLE [#ControlCentre] ( [ControlCentreID] [int] IDENTITY (1, 1) NOT NULL , [Location] [nvarchar] (50) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL , [Phone] [nvarchar] (16) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [Fax] [nvarchar] (16) COLLATE SQL_Latin1_General_CP1_CI_AS NULL , [Display] [bit] NULL CONSTRAINT [DF_ControlCentre_Display] DEFAULT (1), [AgencyID] [int] NULL , CONSTRAINT [PK_CONTROLCENTRE] PRIMARY KEY CLUSTERED ( [ControlCentreID] ) ON [PRIMARY] ) ON [PRIMARY] INSERT INTO #ControlCentre (Location, Phone, AgencyID, Display) VALUES (N'aaa', N'123456', 1, 0) select * from #ControlCentre drop table #ControlCentre The PK is a clustered index on two columns. When filtering on just one of the columns is the PK clustered index good for that, or do I need to create an index on each of the two columns that make up the PK to get best performance?? === Looking at the execution plan will tell you if it uses the idx or not. HTH Michael M Regards Borge ---- Original message ---- >Date: Mon, 3 Dec 2007 09:35:37 +1100 >From: "Michael Maddison" >Subject: Re: [dba-SQLServer] Temporary Table - Impact on SQL Db size >To: > >Hi Borge, > >You can create a job to shrink then db and/or the log file. >You must backup the files before you can shrink the db file and truncate >the log file. > >Data Files. >SQL will grow the data file when it needs to based on the parameters you >set when the db is created, by >default the value is 10%. Growing the db is 'expensive' and will lock >the db while it occurs. A common practice is to set the start size at >the size required for 1 years activity and >grow it if required during downtime. > >Log Files. >Log transactions in the db and can grow very fast. Backup and truncate >as required. > >The above is a very simplistic explanation of how SQL manages its files. >In general DBA's don't worry too much >about db size, they just keep adding more disks ;-) Logs size is >managed because its part of the backup routine. > >HTH > > >Michael M > >Subject: [dba-SQLServer] Temporary Table - Impact on SQL Db size > >Hi all, > >SQL2005 : >If I daily create a table, populate it with a bunch of data, use it to >perform some updates and then drop it - will this just keep growing the >database or will SQL Server manage the disk space.... I am asking the >question as my experience with Access Dbs is that ever so often you had >to compact the Db if you did something like this.... Does SQL Server >handle this better, i.e. without user intervention? > >Regards >Borge > >_______________________________________________ >dba-SQLServer mailing list >dba-SQLServer at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >http://www.databaseadvisors.com > > >_______________________________________________ >dba-SQLServer mailing list >dba-SQLServer at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >http://www.databaseadvisors.com > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From pcs at azizaz.com Mon Dec 3 00:57:59 2007 From: pcs at azizaz.com (pcs at azizaz.com) Date: Mon, 3 Dec 2007 16:57:59 +1000 (EST) Subject: [dba-SQLServer] Update Query Access vs SQL Server Message-ID: <20071203165759.DIZ56716@dommail.onthenet.com.au> I have the following update query running in Access: UPDATE tblRS_ReliefStaff AS T1 INNER JOIN tmpRS_ReliefStaffByAreaID AS Q1 ON T1.EmployeeID = Q1.EmployeeID SET T1.ServicingAreas = IIf(Len([ServicingAreas])>0,[T1].[ServicingAreas] & ";" & [Q1].[AreaID],[Q1].[AreaID]); It will create values in the column ServicingAreas like this: 236;123;567;432;260;230 Running this query in SQL Server will only add the first encountered instance of AreaID into column ServicingAreas, like 236 UPDATE tblRS_ReliefStaff SET ServicingAreas = CASE WHEN Len([ServicingAreas])>0 THEN [T1].[ServicingAreas] + ';' + [Q1].[AreaID] ELSE [Q1].[AreaID] END FROM tblRS_ReliefStaff AS T1 INNER JOIN tmpRS_ReliefStaffByAreaID AS Q1 ON T1.EmployeeID = Q1.EmployeeID END How do I get the SQL Server query to add all encountered values of AreaID of the inner join into the column 'ServicingAreas' ???? Regards Borge From mikedorism at verizon.net Mon Dec 3 07:11:36 2007 From: mikedorism at verizon.net (Doris Manning) Date: Mon, 03 Dec 2007 08:11:36 -0500 Subject: [dba-SQLServer] Converting the value in a field In-Reply-To: <005501c8343c$2deeb580$647aa8c0@M90> References: <005501c8343c$2deeb580$647aa8c0@M90> Message-ID: <000b01c835ae$090f7a10$2f01a8c0@Kermit> Wrapping the object in CInt() should fix the problem. Doris Manning Database Administrator Hargrove Inc. -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 01, 2007 12:04 PM To: dba-vb at databaseadvisors.com; dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Converting the value in a field I have lines of code like this: mintNumOfRecsToExport = !EDS_FileSize The compile complains because !EDS_FileSize is an object (a field) and mintNumOfRecsToExport is an integer variable. The error is: Error 2 Option Strict On disallows implicit conversions from 'Object' to 'Integer'. There is no .Value property for the field object !EDS_FileSize. Will simply wrapping the object in CInt() fix the problem, or will I get an integer pointer to the object? John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From newsgrps at dalyn.co.nz Mon Dec 3 10:49:42 2007 From: newsgrps at dalyn.co.nz (David Emerson) Date: Tue, 04 Dec 2007 05:49:42 +1300 Subject: [dba-SQLServer] Slow Sprocs Message-ID: <20071203164646.TBNV17371.fep04.xtra.co.nz@Dalyn.dalyn.co.nz> I have an SQL2000 database with a number of tables. I have a second database which has a number of identically structured tables (but with different data) as well as tables with data that is used by both databases. Both databases are on the same server. In both databases I have a number of sprocs that join data from their unique data tables and the common tables. The sprocs work fine in the database that the houses the shared tables. However, the sprocs in the database which doesn't house the shared tables run very slow (for example in the second database a sproc may take 15 seconds to run, while on the first it might take over 5 minutes). What are the things I can do to speed up the sprocs that are pulling data from the other database? Combining the databases into one file is not an option as for company reasons they require the data separate. Repeating the shared tables in each database is not an option as this would double up the maintenance required to update the data. Regards David Emerson Dalyn Software Ltd Wellington, New Zealand From jwcolby at colbyconsulting.com Mon Dec 3 12:22:01 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 3 Dec 2007 13:22:01 -0500 Subject: [dba-SQLServer] Listing distinct values in field Message-ID: <00db01c835d9$66d83580$647aa8c0@M90> In my client's business he needs a "data dictionary" kind of thing which essentially tells him the valid values in any given field (for WHERE clause purposes). I receive CSV and fixed width files which I import into SQL Server. I need to give him a list of the fields and the valid values in each field (actual different data items). I will be writing such a widget if I can't find one already out there. Does anyone have such a widget or know where to find one? John W. Colby Colby Consulting www.ColbyConsulting.com From James at fcidms.com Mon Dec 3 13:47:51 2007 From: James at fcidms.com (James Barash) Date: Mon, 3 Dec 2007 14:47:51 -0500 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: <004701c833d7$99526c00$647aa8c0@M90> Message-ID: <010d01c835e5$6424b7d0$800101df@fci.local> John: Below is a stored procedure I downloaded from somewhere that does exactly what you want, forces a truncate of the log file. I use it in a system that also imports large text files, often 10-15 million records. Use the code below to create the stored procedure in the target database and then execute it after each import. As I recall, you are using .Net to do your import so that should be easy to code. I've used it in SQL Server 2000 and 2005 without any problems. James Barash /* Start Code */ CREATE proc sp_force_shrink_log @target_percent tinyint = 0, @target_size_MB int = 10, @max_iterations int = 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount on declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration int, @file_max int, @file int, @fileid varchar(5) select @db = db_name(), @iteration = 0 create table #loginfo ( id int identity, FileId int, FileSize numeric(22,0), StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, CreateTime nvarchar(255) ) create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, StartOffset ) create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles where status & 0x40 = 0x40 select @file_max = @@rowcount if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table table_to_force_shrink_log ( x nchar(3000) not null )' ) insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select @unused1 = @unused -- save for later select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > @target_size_MB and @iteration < @max_iterations begin select @iteration = @iteration + 1 -- this is just a precaution exec( 'insert table_to_force_shrink_log select name from sysobjects delete table_to_force_shrink_log') select @file = 0 while @file < @file_max begin select @file = @file + 1 select @fileid = fileid from #logfiles where id = @file exec( 'dbcc shrinkfile( ' + @fileid + ' )' ) end exec( 'backup log [' + @db + '] ' + @backup_log_opt ) truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) end if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' iterations the unused portion of the log has grown from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the remaining unused portion is larger than 10 MB,' where @unused > 10 union all select 'you may try running this procedure again with a higher number of iterations.' where @unused > 10 union all select 'Sometimes the log would not shrink to a size smaller than several Megabytes.' where @unused <= 10 else select 'It took ' + convert( varchar, @iteration ) + ' iterations to shrink the unused portion of the log from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB' exec( 'drop table table_to_force_shrink_log' ) GO /* End code */ -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 01, 2007 12:04 AM To: dba-sqlserver at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-SQLServer] Truncate a log file from code I am doing an entire directory of files, using SQLBulkCopy to import the text files into a temp table, and from there into a permanent table. The log file is about the same size as the data file. Is it possible to truncate the log file after every file import in order to minimize the disk impact? If so can you point me to example code? John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Dec 3 14:08:45 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 3 Dec 2007 15:08:45 -0500 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: <010d01c835e5$6424b7d0$800101df@fci.local> References: <004701c833d7$99526c00$647aa8c0@M90> <010d01c835e5$6424b7d0$800101df@fci.local> Message-ID: <00e001c835e8$4fc49ff0$647aa8c0@M90> Wonderful, thanks! John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James Barash Sent: Monday, December 03, 2007 2:48 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Truncate a log file from code John: Below is a stored procedure I downloaded from somewhere that does exactly what you want, forces a truncate of the log file. I use it in a system that also imports large text files, often 10-15 million records. Use the code below to create the stored procedure in the target database and then execute it after each import. As I recall, you are using .Net to do your import so that should be easy to code. I've used it in SQL Server 2000 and 2005 without any problems. James Barash /* Start Code */ CREATE proc sp_force_shrink_log @target_percent tinyint = 0, @target_size_MB int = 10, @max_iterations int = 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount on declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration int, @file_max int, @file int, @fileid varchar(5) select @db = db_name(), @iteration = 0 create table #loginfo ( id int identity, FileId int, FileSize numeric(22,0), StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, CreateTime nvarchar(255) ) create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, StartOffset ) create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles where status & 0x40 = 0x40 select @file_max = @@rowcount if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table table_to_force_shrink_log ( x nchar(3000) not null )' ) insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select @unused1 = @unused -- save for later select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > @target_size_MB and @iteration < @max_iterations begin select @iteration = @iteration + 1 -- this is just a precaution exec( 'insert table_to_force_shrink_log select name from sysobjects delete table_to_force_shrink_log') select @file = 0 while @file < @file_max begin select @file = @file + 1 select @fileid = fileid from #logfiles where id = @file exec( 'dbcc shrinkfile( ' + @fileid + ' )' ) end exec( 'backup log [' + @db + '] ' + @backup_log_opt ) truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) end if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' iterations the unused portion of the log has grown from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the remaining unused portion is larger than 10 MB,' where @unused > 10 union all select 'you may try running this procedure again with a higher number of iterations.' where @unused > 10 union all select 'Sometimes the log would not shrink to a size smaller than several Megabytes.' where @unused <= 10 else select 'It took ' + convert( varchar, @iteration ) + ' iterations to shrink the unused portion of the log from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB' exec( 'drop table table_to_force_shrink_log' ) GO /* End code */ -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 01, 2007 12:04 AM To: dba-sqlserver at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-SQLServer] Truncate a log file from code I am doing an entire directory of files, using SQLBulkCopy to import the text files into a temp table, and from there into a permanent table. The log file is about the same size as the data file. Is it possible to truncate the log file after every file import in order to minimize the disk impact? If so can you point me to example code? John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Mon Dec 3 14:19:38 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 3 Dec 2007 15:19:38 -0500 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: <010d01c835e5$6424b7d0$800101df@fci.local> References: <004701c833d7$99526c00$647aa8c0@M90> <010d01c835e5$6424b7d0$800101df@fci.local> Message-ID: <00e101c835e9$d49ab560$647aa8c0@M90> I assume you manually copy this stuff into a SP and then edit it each time? John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James Barash Sent: Monday, December 03, 2007 2:48 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Truncate a log file from code John: Below is a stored procedure I downloaded from somewhere that does exactly what you want, forces a truncate of the log file. I use it in a system that also imports large text files, often 10-15 million records. Use the code below to create the stored procedure in the target database and then execute it after each import. As I recall, you are using .Net to do your import so that should be easy to code. I've used it in SQL Server 2000 and 2005 without any problems. James Barash /* Start Code */ CREATE proc sp_force_shrink_log @target_percent tinyint = 0, @target_size_MB int = 10, @max_iterations int = 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount on declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration int, @file_max int, @file int, @fileid varchar(5) select @db = db_name(), @iteration = 0 create table #loginfo ( id int identity, FileId int, FileSize numeric(22,0), StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, CreateTime nvarchar(255) ) create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, StartOffset ) create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles where status & 0x40 = 0x40 select @file_max = @@rowcount if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table table_to_force_shrink_log ( x nchar(3000) not null )' ) insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select @unused1 = @unused -- save for later select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > @target_size_MB and @iteration < @max_iterations begin select @iteration = @iteration + 1 -- this is just a precaution exec( 'insert table_to_force_shrink_log select name from sysobjects delete table_to_force_shrink_log') select @file = 0 while @file < @file_max begin select @file = @file + 1 select @fileid = fileid from #logfiles where id = @file exec( 'dbcc shrinkfile( ' + @fileid + ' )' ) end exec( 'backup log [' + @db + '] ' + @backup_log_opt ) truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) end if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' iterations the unused portion of the log has grown from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the remaining unused portion is larger than 10 MB,' where @unused > 10 union all select 'you may try running this procedure again with a higher number of iterations.' where @unused > 10 union all select 'Sometimes the log would not shrink to a size smaller than several Megabytes.' where @unused <= 10 else select 'It took ' + convert( varchar, @iteration ) + ' iterations to shrink the unused portion of the log from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB' exec( 'drop table table_to_force_shrink_log' ) GO /* End code */ -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 01, 2007 12:04 AM To: dba-sqlserver at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-SQLServer] Truncate a log file from code I am doing an entire directory of files, using SQLBulkCopy to import the text files into a temp table, and from there into a permanent table. The log file is about the same size as the data file. Is it possible to truncate the log file after every file import in order to minimize the disk impact? If so can you point me to example code? John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From James at fcidms.com Mon Dec 3 14:49:33 2007 From: James at fcidms.com (James Barash) Date: Mon, 3 Dec 2007 15:49:33 -0500 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: <00e101c835e9$d49ab560$647aa8c0@M90> Message-ID: <011a01c835ee$02cf0770$800101df@fci.local> No, this is the script that creates the stored procedure. Copy it into Query Analyzer for 2000 or a New Query in Management Studio for 2005 select the appropriate database and execute. That will create the stored procedure. You only need to create it once per database. Once you've created it, you just need to execute it. If you are using VB.Net: Dim Cmd as Data.SqlClient.SqlCommand Cmd.Commandtext = "sp_force_shrink_log" Cmd.CommandType = CommandType.StoredProcedure Cmd.Connection = conn Cmd.ExecuteNonQuery() This assumes you already have a connection (conn) to the database. James Barash -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Monday, December 03, 2007 3:20 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Truncate a log file from code I assume you manually copy this stuff into a SP and then edit it each time? John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James Barash Sent: Monday, December 03, 2007 2:48 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Truncate a log file from code John: Below is a stored procedure I downloaded from somewhere that does exactly what you want, forces a truncate of the log file. I use it in a system that also imports large text files, often 10-15 million records. Use the code below to create the stored procedure in the target database and then execute it after each import. As I recall, you are using .Net to do your import so that should be easy to code. I've used it in SQL Server 2000 and 2005 without any problems. James Barash /* Start Code */ CREATE proc sp_force_shrink_log @target_percent tinyint = 0, @target_size_MB int = 10, @max_iterations int = 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount on declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration int, @file_max int, @file int, @fileid varchar(5) select @db = db_name(), @iteration = 0 create table #loginfo ( id int identity, FileId int, FileSize numeric(22,0), StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, CreateTime nvarchar(255) ) create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, StartOffset ) create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles where status & 0x40 = 0x40 select @file_max = @@rowcount if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table table_to_force_shrink_log ( x nchar(3000) not null )' ) insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select @unused1 = @unused -- save for later select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > @target_size_MB and @iteration < @max_iterations begin select @iteration = @iteration + 1 -- this is just a precaution exec( 'insert table_to_force_shrink_log select name from sysobjects delete table_to_force_shrink_log') select @file = 0 while @file < @file_max begin select @file = @file + 1 select @fileid = fileid from #logfiles where id = @file exec( 'dbcc shrinkfile( ' + @fileid + ' )' ) end exec( 'backup log [' + @db + '] ' + @backup_log_opt ) truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / 1048576.00 from #loginfo select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( decimal(6,2), @shrinkable * 100 / @log_size ) end if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' iterations the unused portion of the log has grown from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the remaining unused portion is larger than 10 MB,' where @unused > 10 union all select 'you may try running this procedure again with a higher number of iterations.' where @unused > 10 union all select 'Sometimes the log would not shrink to a size smaller than several Megabytes.' where @unused <= 10 else select 'It took ' + convert( varchar, @iteration ) + ' iterations to shrink the unused portion of the log from ' + convert( varchar, @unused1 ) + ' MB to ' + convert( varchar, @unused ) + ' MB' exec( 'drop table table_to_force_shrink_log' ) GO /* End code */ -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 01, 2007 12:04 AM To: dba-sqlserver at databaseadvisors.com; dba-vb at databaseadvisors.com Subject: [dba-SQLServer] Truncate a log file from code I am doing an entire directory of files, using SQLBulkCopy to import the text files into a temp table, and from there into a permanent table. The log file is about the same size as the data file. Is it possible to truncate the log file after every file import in order to minimize the disk impact? If so can you point me to example code? John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fuller.artful at gmail.com Tue Dec 4 10:57:09 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Tue, 4 Dec 2007 11:57:09 -0500 Subject: [dba-SQLServer] Listing distinct values in field In-Reply-To: <00db01c835d9$66d83580$647aa8c0@M90> References: <00db01c835d9$66d83580$647aa8c0@M90> Message-ID: <29f585dd0712040857s3f800ea7gb649bd19de08cbba@mail.gmail.com> I've got one. I think you can find it at TechRepublic.com by searching for "Data Values Dictionary". If you don't find it, I'll dig it up and send it. Let me know. (I'm sick today so I'm moving slowly.) Arthur On 12/3/07, jwcolby wrote: > > In my client's business he needs a "data dictionary" kind of thing which > essentially tells him the valid values in any given field (for WHERE > clause > purposes). I receive CSV and fixed width files which I import into SQL > Server. I need to give him a list of the fields and the valid values in > each field (actual different data items). I will be writing such a widget > if I can't find one already out there. > > Does anyone have such a widget or know where to find one? > From fuller.artful at gmail.com Tue Dec 4 11:02:44 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Tue, 4 Dec 2007 12:02:44 -0500 Subject: [dba-SQLServer] Listing distinct values in field In-Reply-To: <29f585dd0712040857s3f800ea7gb649bd19de08cbba@mail.gmail.com> References: <00db01c835d9$66d83580$647aa8c0@M90> <29f585dd0712040857s3f800ea7gb649bd19de08cbba@mail.gmail.com> Message-ID: <29f585dd0712040902we16fbe6heafad319c3271139@mail.gmail.com> JC, Here's the link. I found it. As I wrote, I'm moving slowly today. Here you go: http://articles.techrepublic.com.com/5100-10878_11-5683261.html hth, Arthur On 12/4/07, Arthur Fuller wrote: > > I've got one. I think you can find it at TechRepublic.com by searching for > "Data Values Dictionary". If you don't find it, I'll dig it up and send it. > Let me know. (I'm sick today so I'm moving slowly.) > > Arthur > > On 12/3/07, jwcolby wrote: > > > > In my client's business he needs a "data dictionary" kind of thing which > > essentially tells him the valid values in any given field (for WHERE > > clause > > purposes). I receive CSV and fixed width files which I import into SQL > > Server. I need to give him a list of the fields and the valid values in > > each field (actual different data items). I will be writing such a > > widget > > if I can't find one already out there. > > > > Does anyone have such a widget or know where to find one? > > > > From jwcolby at colbyconsulting.com Tue Dec 4 11:35:20 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Tue, 4 Dec 2007 12:35:20 -0500 Subject: [dba-SQLServer] Listing distinct values in field In-Reply-To: <29f585dd0712040902we16fbe6heafad319c3271139@mail.gmail.com> References: <00db01c835d9$66d83580$647aa8c0@M90><29f585dd0712040857s3f800ea7gb649bd19de08cbba@mail.gmail.com> <29f585dd0712040902we16fbe6heafad319c3271139@mail.gmail.com> Message-ID: <001701c8369c$0b9373b0$977aa8c0@M90> I figured you had to have this Arthur. Thanks, John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Tuesday, December 04, 2007 12:03 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Listing distinct values in field JC, Here's the link. I found it. As I wrote, I'm moving slowly today. Here you go: http://articles.techrepublic.com.com/5100-10878_11-5683261.html hth, Arthur On 12/4/07, Arthur Fuller wrote: > > I've got one. I think you can find it at TechRepublic.com by searching > for "Data Values Dictionary". If you don't find it, I'll dig it up and send it. > Let me know. (I'm sick today so I'm moving slowly.) > > Arthur > > On 12/3/07, jwcolby wrote: > > > > In my client's business he needs a "data dictionary" kind of thing > > which essentially tells him the valid values in any given field (for > > WHERE clause purposes). I receive CSV and fixed width files which I > > import into SQL Server. I need to give him a list of the fields and > > the valid values in each field (actual different data items). I > > will be writing such a widget if I can't find one already out there. > > > > Does anyone have such a widget or know where to find one? > > > > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fhtapia at gmail.com Wed Dec 5 00:59:30 2007 From: fhtapia at gmail.com (Francisco Tapia) Date: Tue, 4 Dec 2007 22:59:30 -0800 Subject: [dba-SQLServer] [dba-VB] Truncate a log file from code In-Reply-To: <011a01c835ee$02cf0770$800101df@fci.local> References: <00e101c835e9$d49ab560$647aa8c0@M90> <011a01c835ee$02cf0770$800101df@fci.local> Message-ID: One thing that I find very useful with ss is that you can schedule these kinds of tasks with alerts. I am not on my pc right now but tomorrow I can post some examples of what I typically do. Ie, I set the tlog to auto backup everytime it reaches 60% of its size. I also have a second alert set up so when the tlog grows beyond 10gb I have it auto backup/shrink Hth On 12/3/07, James Barash wrote: > No, this is the script that creates the stored procedure. Copy it into Query > Analyzer for 2000 or a New Query in Management Studio for 2005 select the > appropriate database and execute. That will create the stored procedure. You > only need to create it once per database. Once you've created it, you just > need to execute it. If you are using VB.Net: > > Dim Cmd as Data.SqlClient.SqlCommand > Cmd.Commandtext = "sp_force_shrink_log" > Cmd.CommandType = CommandType.StoredProcedure > Cmd.Connection = conn > Cmd.ExecuteNonQuery() > > This assumes you already have a connection (conn) to the database. > > James Barash > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Monday, December 03, 2007 3:20 PM > To: dba-sqlserver at databaseadvisors.com > Subject: Re: [dba-SQLServer] Truncate a log file from code > > I assume you manually copy this stuff into a SP and then edit it each time? > > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James > Barash > Sent: Monday, December 03, 2007 2:48 PM > To: dba-sqlserver at databaseadvisors.com > Subject: Re: [dba-SQLServer] Truncate a log file from code > > John: > > Below is a stored procedure I downloaded from somewhere that does exactly > what you want, forces a truncate of the log file. I use it in a system that > also imports large text files, often 10-15 million records. Use the code > below to create the stored procedure in the target database and then execute > it after each import. As I recall, you are using .Net to do your import so > that should be easy to code. I've used it in SQL Server 2000 and 2005 > without any problems. > > James Barash > > > /* Start Code */ > CREATE proc sp_force_shrink_log > > > @target_percent tinyint = 0, @target_size_MB int = 10, @max_iterations int = > 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount > on > > declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 > decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration > int, @file_max int, @file int, @fileid > varchar(5) > > select @db = db_name(), @iteration = 0 > > create table #loginfo ( id int identity, FileId int, FileSize numeric(22,0), > StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, > CreateTime > nvarchar(255) ) > > create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, > StartOffset ) > > create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) > insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles > where status & 0x40 = 0x40 select @file_max = @@rowcount > > if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table > table_to_force_shrink_log ( x nchar(3000) not null )' ) > > insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, > CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > 1048576.00 from #loginfo > > select @unused1 = @unused -- save for later > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( > decimal(6,2), @shrinkable > * 100 / @log_size ) > > while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > > @target_size_MB and @iteration < @max_iterations begin select @iteration = > @iteration + 1 -- this is just a precaution > > exec( 'insert table_to_force_shrink_log select name from sysobjects delete > table_to_force_shrink_log') > > select @file = 0 while @file < @file_max begin select @file = @file + 1 > select @fileid = fileid from #logfiles where id = @file exec( 'dbcc > shrinkfile( ' + @fileid + ' )' ) end > > exec( 'backup log [' + @db + '] ' + @backup_log_opt ) > > truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, > FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select > @last_row = @@rowcount > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > 1048576.00 from #loginfo > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' = convert( > decimal(6,2), @shrinkable > * 100 / @log_size ) end > > if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' > iterations the unused > portion of the log has grown from ' + convert( varchar, @unused1 ) + ' MB to > ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the > remaining unused portion is larger than 10 MB,' where @unused > 10 union all > select 'you may try running this procedure again with a higher number of > iterations.' where @unused > 10 union all select 'Sometimes the log would > not shrink to a size smaller than several Megabytes.' where @unused <= 10 > > else select 'It took ' + convert( varchar, @iteration ) + ' iterations to > shrink the unused portion of the log from ' + convert( varchar, @unused1 ) + > ' MB to ' + convert( varchar, @unused ) + ' MB' > > exec( 'drop table table_to_force_shrink_log' ) > > GO > /* End code */ > > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, December 01, 2007 12:04 AM > To: dba-sqlserver at databaseadvisors.com; dba-vb at databaseadvisors.com > Subject: [dba-SQLServer] Truncate a log file from code > > I am doing an entire directory of files, using SQLBulkCopy to import the > text files into a temp table, and from there into a permanent table. The > log file is about the same size as the data file. Is it possible to > truncate the log file after every file import in order to minimize the disk > impact? If so can you point me to example code? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- -Francisco http://sqlthis.blogspot.com | Tsql and More... From fhtapia at gmail.com Wed Dec 5 01:03:51 2007 From: fhtapia at gmail.com (Francisco Tapia) Date: Tue, 4 Dec 2007 23:03:51 -0800 Subject: [dba-SQLServer] Slow Sprocs In-Reply-To: <20071203164646.TBNV17371.fep04.xtra.co.nz@Dalyn.dalyn.co.nz> References: <20071203164646.TBNV17371.fep04.xtra.co.nz@Dalyn.dalyn.co.nz> Message-ID: What do the query plan execution times look like? How do you ref the data when you call it from the second db (the 5 min running one?) ? Have you tried minimizing the returning data and utilizing the openquery method? On 12/3/07, David Emerson wrote: > I have an SQL2000 database with a number of tables. I have a second > database which has a number of identically structured tables (but > with different data) as well as tables with data that is used by both > databases. Both databases are on the same server. > > In both databases I have a number of sprocs that join data from their > unique data tables and the common tables. The sprocs work fine in > the database that the houses the shared tables. However, the sprocs > in the database which doesn't house the shared tables run very slow > (for example in the second database a sproc may take 15 seconds to > run, while on the first it might take over 5 minutes). > > What are the things I can do to speed up the sprocs that are pulling > data from the other database? > > Combining the databases into one file is not an option as for company > reasons they require the data separate. > Repeating the shared tables in each database is not an option as this > would double up the maintenance required to update the data. > > Regards > > David Emerson > Dalyn Software Ltd > Wellington, New Zealand > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- -Francisco http://sqlthis.blogspot.com | Tsql and More... From newsgrps at dalyn.co.nz Wed Dec 5 09:40:02 2007 From: newsgrps at dalyn.co.nz (David Emerson) Date: Thu, 06 Dec 2007 04:40:02 +1300 Subject: [dba-SQLServer] Slow Sprocs In-Reply-To: References: <20071203164646.TBNV17371.fep04.xtra.co.nz@Dalyn.dalyn.co.nz> Message-ID: <20071205153717.RWUZ17371.fep04.xtra.co.nz@Dalyn.dalyn.co.nz> Thanks for the response. In getting info to answer your questions I found that the second db is not actually referring to the first database for this query at all. This makes it even stranger as the database structures should be identical and the second database is about 3 times smaller than the first one. I have tried to reduce the data returned by running sprocs that put data into tables first (not true SQL temporary tables, but ones already created which have their records deleted prior to being filled with new records for the report), then use these tables for the main sproc. What I might try is replacing the views and sprocs from the second database with scripts from the first database. See if this helps. David At 5/12/2007, you wrote: >What do the query plan execution times look like? How do you ref the >data when you call it from the second db (the 5 min running one?) ? > >Have you tried minimizing the returning data and utilizing the >openquery method? > >On 12/3/07, David Emerson wrote: > > I have an SQL2000 database with a number of tables. I have a second > > database which has a number of identically structured tables (but > > with different data) as well as tables with data that is used by both > > databases. Both databases are on the same server. > > > > In both databases I have a number of sprocs that join data from their > > unique data tables and the common tables. The sprocs work fine in > > the database that the houses the shared tables. However, the sprocs > > in the database which doesn't house the shared tables run very slow > > (for example in the second database a sproc may take 15 seconds to > > run, while on the first it might take over 5 minutes). > > > > What are the things I can do to speed up the sprocs that are pulling > > data from the other database? > > > > Combining the databases into one file is not an option as for company > > reasons they require the data separate. > > Repeating the shared tables in each database is not an option as this > > would double up the maintenance required to update the data. > > > > Regards > > > > David Emerson > > Dalyn Software Ltd > > Wellington, New Zealand > > > > _______________________________________________ > > dba-SQLServer mailing list > > dba-SQLServer at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > > http://www.databaseadvisors.com > > > > > > >-- >-Francisco >http://sqlthis.blogspot.com | Tsql and More... >_______________________________________________ >dba-SQLServer mailing list >dba-SQLServer at databaseadvisors.com >http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Thu Dec 6 13:54:17 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 6 Dec 2007 14:54:17 -0500 Subject: [dba-SQLServer] Where do I find Message-ID: <00b301c83841$c9939060$977aa8c0@M90> What would the SQL look like to get a list of all the field names in a table? I assume that there are system tables with this stuff but it is kind of slow poking around trying to find that. John W. Colby Colby Consulting www.ColbyConsulting.com From ssharkins at gmail.com Thu Dec 6 14:02:27 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Thu, 6 Dec 2007 15:02:27 -0500 Subject: [dba-SQLServer] Where do I find References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: <00ad01c83842$f02dceb0$4b3a8343@SusanOne> Try a schema recordset object. Susan H. > What would the SQL look like to get a list of all the field names in a > table? I assume that there are system tables with this stuff but it is > kind > of slow poking around trying to find that. From kens.programming at verizon.net Thu Dec 6 14:07:40 2007 From: kens.programming at verizon.net (kens.programming) Date: Thu, 06 Dec 2007 12:07:40 -0800 Subject: [dba-SQLServer] Where do I find In-Reply-To: <00b301c83841$c9939060$977aa8c0@M90> References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: <006501c83843$a8c2c480$6b01a8c0@Stoker.com> There are two ways to find this, John. The two system tables you would find this in are sysobjects (tables) and syscolumns (columns). However, there are also a set of specialized views called INFORMATION_SCHEMA, in this case you would use INFORMATION_SCHEMA.COLUMNS. This way you don't have to interpret the data in the sys... tables. It will give you the tablename, columname, datatype, fieldlength, etc., etc. SELECT * FROM INFORMATION_SCHEMA.COLUMNS Ken -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, December 06, 2007 11:54 AM To: dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Where do I find What would the SQL look like to get a list of all the field names in a table? I assume that there are system tables with this stuff but it is kind of slow poking around trying to find that. John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.16.15/1174 - Release Date: 12/6/2007 10:11 AM No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.503 / Virus Database: 269.16.15/1174 - Release Date: 12/6/2007 10:11 AM From garykjos at gmail.com Thu Dec 6 14:05:43 2007 From: garykjos at gmail.com (Gary Kjos) Date: Thu, 6 Dec 2007 14:05:43 -0600 Subject: [dba-SQLServer] Where do I find In-Reply-To: <00b301c83841$c9939060$977aa8c0@M90> References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: In Oracle it would be DESCRIBE tablename. Be worth a shot in SQL Server On 12/6/07, jwcolby wrote: > What would the SQL look like to get a list of all the field names in a > table? I assume that there are system tables with this stuff but it is kind > of slow poking around trying to find that. > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- Gary Kjos garykjos at gmail.com From garykjos at gmail.com Thu Dec 6 14:11:14 2007 From: garykjos at gmail.com (Gary Kjos) Date: Thu, 6 Dec 2007 14:11:14 -0600 Subject: [dba-SQLServer] Where do I find In-Reply-To: References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: oops, no Describe in SQL Server. sp_help is a near equivalent though.. SP_Help tablename GK On 12/6/07, Gary Kjos wrote: > In Oracle it would be DESCRIBE tablename. Be worth a shot in SQL Server > > On 12/6/07, jwcolby wrote: > > What would the SQL look like to get a list of all the field names in a > > table? I assume that there are system tables with this stuff but it is kind > > of slow poking around trying to find that. > > > > John W. Colby > > Colby Consulting > > www.ColbyConsulting.com > > > > _______________________________________________ > > dba-SQLServer mailing list > > dba-SQLServer at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > > http://www.databaseadvisors.com > > > > > > > -- > Gary Kjos > garykjos at gmail.com > -- Gary Kjos garykjos at gmail.com From garykjos at gmail.com Thu Dec 6 14:13:47 2007 From: garykjos at gmail.com (Gary Kjos) Date: Thu, 6 Dec 2007 14:13:47 -0600 Subject: [dba-SQLServer] Where do I find In-Reply-To: References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: sp_columns tablename also works. Slightly different output. GK On 12/6/07, Gary Kjos wrote: > oops, no Describe in SQL Server. sp_help is a near equivalent though.. > > SP_Help tablename > > GK > > On 12/6/07, Gary Kjos wrote: > > In Oracle it would be DESCRIBE tablename. Be worth a shot in SQL Server > > > > On 12/6/07, jwcolby wrote: > > > What would the SQL look like to get a list of all the field names in a > > > table? I assume that there are system tables with this stuff but it is kind > > > of slow poking around trying to find that. > > > > > > John W. Colby > > > Colby Consulting > > > www.ColbyConsulting.com > > > > > > _______________________________________________ > > > dba-SQLServer mailing list > > > dba-SQLServer at databaseadvisors.com > > > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > > > http://www.databaseadvisors.com > > > > > > > > > > > > -- > > Gary Kjos > > garykjos at gmail.com > > > > > -- > Gary Kjos > garykjos at gmail.com > -- Gary Kjos garykjos at gmail.com From jwcolby at colbyconsulting.com Thu Dec 6 14:37:51 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 6 Dec 2007 15:37:51 -0500 Subject: [dba-SQLServer] A step backwards Message-ID: <00bb01c83847$dff2e490$977aa8c0@M90> I am taking a stab at what Arthur has been harping on about, which is turning my database from hell into a hellish normalized table with something like PKID, FieldName, Value. This essentially stacks every piece of data into a single table of three fields (four if you include a table name column). I would normally be skeptical of such a schema however in this specific case I have a table of 51 million rows with 600 fields where in come cases there is data in fieldXYZ for 30 million records, but for field ABC only 5 million rows. As you can see, by doing it this way I pull out exactly the number of values for a given field as there actual data values in the source table / that field, and then place the value and field name into a new table. Index it and when I ask for fieldABC I only get back 5 million records. Furthermore my table is only 3 fields wide instead of 600 fields wide. However... the field names are things like Forwarding_Address_Or_Nixie_Run_Codes, Number_of_Persons and Second_Name_of_Opposite_Gender_Match_Indicator. My issue now is that the data storage for the FIELD NAMES becomes gigantic. 37 million records where the DATA is a single character 0-9 or A-T (good) but the NAME is Narrow_Income_Band (bad). Thus I REAAAALY need to turn the field names into a lookup table so that the nameID (integer) is used instead of a FieldName (varchar(50)). Doing it that way would then reduce my table to PKID (int), FldNameID (int) and FldVal (VarChar(50)). Throwing a cover index on that would allow me to ask any question with something like SELECT PKID from tblHSIDVertical WHERE FldNameID = 13 and FldVal = 'A' Once constructed the table would have exactly as many records as there were actual data values in ALL of the fields, and there would be ~600 "sets" of answers. All queries would look like the above query. Let's pick a number out of the air and say that there are ON AVERAGE 10 million records with values in any given field, that means that the resulting database would have 10 million * 600 records, BUT the records would only be three fields and there would be exactly ONE index (a cover index). Further I could do queries with ANDS and ORs much more simply (I think). Of course I would also have to do lookups to get the FieldNameID but it seems like it should work. Arthur has been hammering me that this is the "right way" to do this, and I kind of agree. I am currently doing table scans in many cases (across 50 million records) because there are only two values in many of the columns, 'Y' and '' (true or no value). Your thoughts on this? John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Thu Dec 6 15:40:43 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Thu, 6 Dec 2007 16:40:43 -0500 Subject: [dba-SQLServer] Stored Procedure Assisted Editor Message-ID: <00bf01c83850$a7fe7730$977aa8c0@M90> I am looking in SQL Server 2005 by gunderloy & co, chapter 14, Stored Procedures. he has me open the programmability tab and right click "new stored procedure. In the book a "wizard" pops up with the created stored procedure in the bottom. I don't see the wizard, only the text "model" of a new SP. Is this wizard no more? I hate it when the books are just completely different than what I see. John W. Colby Colby Consulting www.ColbyConsulting.com From fahooper at trapo.com Thu Dec 6 15:54:36 2007 From: fahooper at trapo.com (Fred Hooper) Date: Thu, 6 Dec 2007 16:54:36 -0500 Subject: [dba-SQLServer] Where do I find In-Reply-To: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: <002c01c83852$97f070d0$8151dd48@fredxp> Hi John, I use the code below to get Describe-type information from SQL Server. It gets it for the whole database but you can limit it, of course. There's separate code for 2000 and earlier versus 2005; both are below. Fred For SQL Server 2000 and earlier I use: SELECT distinct M.owner, M.table_name, M.column_name, M.colorder, M.field_type + case when M.field_type in ('int','bigint','tinyint','datetime','money','bit','smallint','smallmoney') then '' Else '(' + convert(varchar,case when M.field_type in ('nvarchar','varchar','char') then case when M.length > 255 then 255 else M.length end else M.length end) + ')' end as data_type, M.isnullable as nullable, IsNull(K.key_fld,0) as primary_key, IsNull(I.keyno, 0) As Indexed, case when M.xtype = 'U' then 'T' else 'V' end as TableOrView FROM /* Basic information on the tables and fields */ (select user_name(O.uid) as owner, O.name as table_name, C.name as column_name, C.colorder, T.name as field_type, C.length, C.isnullable, O.xtype from sysobjects O inner join syscolumns C on O.id = C.id inner join systypes T on C.xtype = T.xtype where (O.xtype = 'U' or O.xtype = 'V') and T.name <> 'sysname' and user_name(O.uid) = 'dbo') M /* Returns all of the key fields */ left outer join (select SO.name as table_name, SC.name as column_name, SIK.keyno AS key_fld from sysobjects SO, sysobjects SPK, sysindexes SI, sysindexkeys SIK, syscolumns SC where SO.xtype = 'U' and SO.id = SPK.parent_obj and SPK.xtype = 'PK' and SO.id = SI.id and SPK.name = SI.name and SO.id = SIK.id and SI.indid = SIK.indid and SO.id = SC.id and SIK.colid = SC.colid) K on M.table_name = K.table_name and M.column_name = K.column_name /* Returns all of the indexed fields */ left outer join (select object_name(si.id) as table_name, col_name(sic.id, sic.colid) as column_name, min(sic.keyno) as keyno from sysindexes si, sysindexkeys sic where si.id = sic.id and si.indid = sic.indid and left(si.name,8) <> '_WA_Sys_' /* Omit statistics indexes */ group by object_name(si.id), col_name(sic.id, sic.colid)) I on M.table_name = I.table_name and M.column_name = I.column_name For SQL Server 2005 I use: SELECT distinct M.owner, M.table_name, M.column_name, M.colorder, M.field_type + case when M.field_type in ('int','bigint','tinyint','datetime','smalldatetime', 'money','bit','smallint','smallmoney') then '' Else '(' + convert(varchar,case when M.field_type in ('nvarchar','varchar','char') then case when M.length > 255 then 255 else M.length end else M.length end) + ')' end as data_type, M.isnullable as nullable, IsNull(K.key_fld,0) as primary_key, IsNull(I.keyno, 0) As Indexed, case when M.xtype = 'U' then 'T' else 'V' end as TableOrView FROM /* Basic information on the tables and fields */ (select s.name as owner, o.name as table_name, c.name as column_name, column_id as colorder, t.name as field_type, c.max_length as length, c.is_nullable as isnullable, case when type_desc = 'USER_TABLE' then 'U' else 'V' end as xtype from sys.objects o inner join sys.schemas s on o.schema_id = s.schema_id inner join sys.columns c on o.object_id = c.object_id inner join sys.types t on c.system_type_id = t.system_type_id where type_desc in ('USER_TABLE','VIEW') and t.name <> 'sysname' and s.name = 'dbo') M /* Returns all of the key fields */ left outer join (select distinct o.name as table_name, c.name as column_name, i.index_column_id as key_fld from sys.objects o inner join sys.columns c on o.object_id = c.object_id inner join sys.index_columns i on c.object_id = i.object_id and c.column_id = i.column_id inner join sys.key_constraints k on i.object_id = k.parent_object_id and i.index_id = k.unique_index_id where k.type = 'PK') K on M.table_name = K.table_name and M.column_name = K.column_name /* Returns all of the indexed fields */ left outer join (select o.name as table_name, c.name as column_name, min(index_column_id) as keyno from sys.objects o inner join sys.columns c on o.object_id = c.object_id inner join sys.index_columns i on c.object_id = i.object_id and c.column_id = i.column_id group by o.name, c.name) I on M.table_name = I.table_name and M.column_name = I.column_name -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, December 06, 2007 2:54 PM To: dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Where do I find What would the SQL look like to get a list of all the field names in a table? I assume that there are system tables with this stuff but it is kind of slow poking around trying to find that. John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From ssharkins at gmail.com Thu Dec 6 17:33:46 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Thu, 6 Dec 2007 18:33:46 -0500 Subject: [dba-SQLServer] Stored Procedure Assisted Editor References: <00bf01c83850$a7fe7730$977aa8c0@M90> Message-ID: <001b01c83860$7f8752d0$4b3a8343@SusanOne> >I am looking in SQL Server 2005 by gunderloy & co, chapter 14, Stored > Procedures. he has me open the programmability tab and right click "new > stored procedure. In the book a "wizard" pops up with the created stored > procedure in the bottom. I don't see the wizard, only the text "model" of > a > new SP. Is this wizard no more? > > I hate it when the books are just completely different than what I see. ======You're seeing a template -- I find it totally useless. Just enter the sp as you normally would and delete all that crap. Susan H. From bheid at sc.rr.com Thu Dec 6 21:05:33 2007 From: bheid at sc.rr.com (Bobby Heid) Date: Thu, 6 Dec 2007 22:05:33 -0500 Subject: [dba-SQLServer] Where do I find In-Reply-To: <00b301c83841$c9939060$977aa8c0@M90> References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: <00c401c8387e$083895e0$18a9c1a0$@rr.com> I found this little snippet that works in SQL 2005. You should be able to modify it for your purposes. Note that the table I was searching on is videos in the line: AND OBJECT_NAME(c.object_id) = 'videos' SELECT [Table Name] = OBJECT_NAME(c.object_id), [Column Name] = c.name, [Description] = ex.value FROM sys.columns c LEFT OUTER JOIN sys.extended_properties ex ON ex.major_id = c.object_id AND ex.minor_id = c.column_id AND ex.name = 'MS_Description' WHERE OBJECTPROPERTY(c.object_id, 'IsMsShipped')=0 AND OBJECT_NAME(c.object_id) = 'videos' ORDER BY OBJECT_NAME(c.object_id), c.column_id Bobby -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, December 06, 2007 2:54 PM To: dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Where do I find What would the SQL look like to get a list of all the field names in a table? I assume that there are system tables with this stuff but it is kind of slow poking around trying to find that. John W. Colby Colby Consulting www.ColbyConsulting.com From bheid at sc.rr.com Thu Dec 6 21:06:44 2007 From: bheid at sc.rr.com (Bobby Heid) Date: Thu, 6 Dec 2007 22:06:44 -0500 Subject: [dba-SQLServer] Where do I find References: <00b301c83841$c9939060$977aa8c0@M90> Message-ID: <00c501c8387e$32e28580$98a79080$@rr.com> I found that here: http://databases.aspfaq.com/schema-tutorials/schema-how-do-i-show-the-descri ption-property-of-a-column.html Bobby -----Original Message----- From: Bobby Heid [mailto:bheid at sc.rr.com] Sent: Thursday, December 06, 2007 10:06 PM To: 'dba-sqlserver at databaseadvisors.com' Subject: RE: [dba-SQLServer] Where do I find I found this little snippet that works in SQL 2005. You should be able to modify it for your purposes. Note that the table I was searching on is videos in the line: AND OBJECT_NAME(c.object_id) = 'videos' SELECT [Table Name] = OBJECT_NAME(c.object_id), [Column Name] = c.name, [Description] = ex.value FROM sys.columns c LEFT OUTER JOIN sys.extended_properties ex ON ex.major_id = c.object_id AND ex.minor_id = c.column_id AND ex.name = 'MS_Description' WHERE OBJECTPROPERTY(c.object_id, 'IsMsShipped')=0 AND OBJECT_NAME(c.object_id) = 'videos' ORDER BY OBJECT_NAME(c.object_id), c.column_id Bobby -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Thursday, December 06, 2007 2:54 PM To: dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Where do I find What would the SQL look like to get a list of all the field names in a table? I assume that there are system tables with this stuff but it is kind of slow poking around trying to find that. John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Dec 7 00:09:15 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 7 Dec 2007 01:09:15 -0500 Subject: [dba-SQLServer] How do I Message-ID: <00de01c83897$b2190360$977aa8c0@M90> I have created an SProc that accepts a field name and a Field ID, and looks up the data in that field name (in a fixed source table for now). It then writes the data and the Field ID into my "vertical table". The next "higher" piece needs to look up that Field Name and Field ID in a table of field names (I have that already) and the call the SProc mentioned above that I just created, passing off the values from the Field name and Field ID columns. I can get the table values (build the SQL to do that). What I do not know how to do is to execute a loop moving through the recordset and calling the SProc with the values pulled all inside of a "supervisor" SProc. Can someone point me to or write air code to do this? Basically I have a table with 600 fields. I have a field name table where I pulled the field names and poured them into a field name table with an autoincrement ID so I now have a Fld_ID, Fld_Name. I need to read the ~600 records in this table and one by one pass off the FLD_ID and FLD_Name to my existing sproc. That existing sproc will pull the data from the "horizontal table" and store it into my "vertical table". Once I finish this exercise I will have the system that Arthur has been harping on me to build. Then I can build a cover index and start some testing to see how fast it ends up being vs. running a similar query on the original "horizontal table". Any help would be appreciated. John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Fri Dec 7 07:23:02 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 7 Dec 2007 08:23:02 -0500 Subject: [dba-SQLServer] Where does the data go Message-ID: <00e801c838d4$4bd60ac0$977aa8c0@M90> As I mentioned last night, I need to iterate a recordset in a stored proc in SQL, sending the fields in the recordset off to another stored procedure. Is this possible? I have my books out and am Googling like mad but I am not seeing anything like recordset objects in tsql. I could switch to VB.Net and do it there but given that I am already in stored procedure mode (which I do not do a lot of, but need to get conversant in) I thought I would try and do it there. So is this possible at all in TSQL? John W. Colby Colby Consulting www.ColbyConsulting.com From ssharkins at gmail.com Fri Dec 7 10:05:52 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Fri, 7 Dec 2007 11:05:52 -0500 Subject: [dba-SQLServer] Where does the data go References: <00e801c838d4$4bd60ac0$977aa8c0@M90> Message-ID: <000801c838eb$10868320$4b3a8343@SusanOne> Have you taken a look at Catalog views? Not sure they'll help you. Susan H. > As I mentioned last night, I need to iterate a recordset in a stored proc > in > SQL, sending the fields in the recordset off to another stored procedure. > > Is this possible? I have my books out and am Googling like mad but I am > not > seeing anything like recordset objects in tsql. I could switch to VB.Net > and do it there but given that I am already in stored procedure mode > (which > I do not do a lot of, but need to get conversant in) I thought I would try > and do it there. > > So is this possible at all in TSQL? From fuller.artful at gmail.com Fri Dec 7 10:07:11 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Fri, 7 Dec 2007 11:07:11 -0500 Subject: [dba-SQLServer] Where does the data go In-Reply-To: <00e801c838d4$4bd60ac0$977aa8c0@M90> References: <00e801c838d4$4bd60ac0$977aa8c0@M90> Message-ID: <29f585dd0712070807w5fef5211v88901ee4c6692c41@mail.gmail.com> There are no recordsets in SQL, John. There are cursors but in general they are despised by most experienced SQL programmers. It is also not clear why you would want to pass the fields off to another sproc. I would guess that you intend to call the second sproc once per field in the recordset, but in that case why the second sproc? Why not just fold that logic into the first sproc? An explanation of what you are trying to accomplish would help me help you. Oh, was the table-values dictionary stuff any help? A. On 12/7/07, jwcolby wrote: > > As I mentioned last night, I need to iterate a recordset in a stored proc > in > SQL, sending the fields in the recordset off to another stored procedure. > > Is this possible? I have my books out and am Googling like mad but I am > not > seeing anything like recordset objects in tsql. I could switch to VB.Net > and do it there but given that I am already in stored procedure mode > (which > I do not do a lot of, but need to get conversant in) I thought I would try > and do it there. > > So is this possible at all in TSQL? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From robert at webedb.com Fri Dec 7 12:45:24 2007 From: robert at webedb.com (Robert L. Stewart) Date: Fri, 07 Dec 2007 12:45:24 -0600 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: References: Message-ID: <200712071849.lB7In0kX031948@databaseadvisors.com> Here is a much simpler way on SQL Server to truncate the log. Replace PT_Data_Org with you database name and run it in each database. And, no, you do not have to do a backup first. CREATE PROCEDURE [dbo].[__TruncateLog] AS BEGIN ALTER DATABASE PT_Data_Org SET RECOVERY SIMPLE END BEGIN EXEC sp_dboption 'PT_Data_Org','trunc. log on chkpt.','TRUE' END BEGIN CHECKPOINT END BEGIN DBCC SHRINKFILE (PT_Data_Org_Log,1) END BEGIN EXEC sp_dboption 'PT_Data_Org','trunc. log on chkpt.','FALSE' END BEGIN ALTER DATABASE PT_Data_Org SET RECOVERY FULL END Robert At 12:00 PM 12/5/2007, you wrote: >Date: Tue, 4 Dec 2007 22:59:30 -0800 >From: "Francisco Tapia" >Subject: Re: [dba-SQLServer] [dba-VB] Truncate a log file from code >To: dba-sqlserver at databaseadvisors.com >Message-ID: > >Content-Type: text/plain; charset=ISO-8859-1 > >One thing that I find very useful with ss is that you can schedule >these kinds of tasks with alerts. I am not on my pc right now but >tomorrow I can post some examples of what I typically do. > >Ie, I set the tlog to auto backup everytime it reaches 60% of its >size. I also have a second alert set up so when the tlog grows beyond >10gb I have it auto backup/shrink > >Hth > > > > >On 12/3/07, James Barash wrote: > > No, this is the script that creates the stored procedure. Copy it > into Query > > Analyzer for 2000 or a New Query in Management Studio for 2005 select the > > appropriate database and execute. That will create the stored > procedure. You > > only need to create it once per database. Once you've created it, you just > > need to execute it. If you are using VB.Net: > > > > Dim Cmd as Data.SqlClient.SqlCommand > > Cmd.Commandtext = "sp_force_shrink_log" > > Cmd.CommandType = CommandType.StoredProcedure > > Cmd.Connection = conn > > Cmd.ExecuteNonQuery() > > > > This assumes you already have a connection (conn) to the database. > > > > James Barash > > > > -----Original Message----- > > From: dba-sqlserver-bounces at databaseadvisors.com > > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby > > Sent: Monday, December 03, 2007 3:20 PM > > To: dba-sqlserver at databaseadvisors.com > > Subject: Re: [dba-SQLServer] Truncate a log file from code > > > > I assume you manually copy this stuff into a SP and then edit it each time? > > > > > > John W. Colby > > Colby Consulting > > www.ColbyConsulting.com > > -----Original Message----- > > From: dba-sqlserver-bounces at databaseadvisors.com > > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James > > Barash > > Sent: Monday, December 03, 2007 2:48 PM > > To: dba-sqlserver at databaseadvisors.com > > Subject: Re: [dba-SQLServer] Truncate a log file from code > > > > John: > > > > Below is a stored procedure I downloaded from somewhere that does exactly > > what you want, forces a truncate of the log file. I use it in a system that > > also imports large text files, often 10-15 million records. Use the code > > below to create the stored procedure in the target database and > then execute > > it after each import. As I recall, you are using .Net to do your import so > > that should be easy to code. I've used it in SQL Server 2000 and 2005 > > without any problems. > > > > James Barash > > > > > > /* Start Code */ > > CREATE proc sp_force_shrink_log > > > > > > @target_percent tinyint = 0, @target_size_MB int = 10, > @max_iterations int = > > 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set nocount > > on > > > > declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 > > decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), @iteration > > int, @file_max int, @file int, @fileid > > varchar(5) > > > > select @db = db_name(), @iteration = 0 > > > > create table #loginfo ( id int identity, FileId int, FileSize > numeric(22,0), > > StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, > > CreateTime > > nvarchar(255) ) > > > > create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, > > StartOffset ) > > > > create table #logfiles ( id int identity(1,1), fileid varchar(5) not null ) > > insert #logfiles ( fileid ) select convert( varchar, fileid ) from sysfiles > > where status & 0x40 = 0x40 select @file_max = @@rowcount > > > > if object_id( 'table_to_force_shrink_log' ) is null exec( 'create table > > table_to_force_shrink_log ( x nchar(3000) not null )' ) > > > > insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, Parity, > > CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount > > > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when > > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case > > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > > 1048576.00 from #loginfo > > > > select @unused1 = @unused -- save for later > > > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, > > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' > = convert( > > decimal(6,2), @shrinkable > > * 100 / @log_size ) > > > > while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > > > @target_size_MB and @iteration < @max_iterations begin select @iteration = > > @iteration + 1 -- this is just a precaution > > > > exec( 'insert table_to_force_shrink_log select name from sysobjects delete > > table_to_force_shrink_log') > > > > select @file = 0 while @file < @file_max begin select @file = @file + 1 > > select @fileid = fileid from #logfiles where id = @file exec( 'dbcc > > shrinkfile( ' + @fileid + ' )' ) end > > > > exec( 'backup log [' + @db + '] ' + @backup_log_opt ) > > > > truncate table #loginfo insert #loginfo ( FileId, FileSize, StartOffset, > > FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select > > @last_row = @@rowcount > > > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case when > > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( case > > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > > 1048576.00 from #loginfo > > > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused log, > > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' > = convert( > > decimal(6,2), @shrinkable > > * 100 / @log_size ) end > > > > if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) + ' > > iterations the unused > > portion of the log has grown from ' + convert( varchar, @unused1 > ) + ' MB to > > ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the > > remaining unused portion is larger than 10 MB,' where @unused > > 10 union all > > select 'you may try running this procedure again with a higher number of > > iterations.' where @unused > 10 union all select 'Sometimes the log would > > not shrink to a size smaller than several Megabytes.' where @unused <= 10 > > > > else select 'It took ' + convert( varchar, @iteration ) + ' iterations to > > shrink the unused portion of the log from ' + convert( varchar, > @unused1 ) + > > ' MB to ' + convert( varchar, @unused ) + ' MB' > > > > exec( 'drop table table_to_force_shrink_log' ) > > > > GO > > /* End code */ > > From fhtapia at gmail.com Fri Dec 7 13:31:38 2007 From: fhtapia at gmail.com (Francisco Tapia) Date: Fri, 7 Dec 2007 11:31:38 -0800 Subject: [dba-SQLServer] Truncate a log file from code In-Reply-To: <200712071849.lB7In0kX031948@databaseadvisors.com> References: <200712071849.lB7In0kX031948@databaseadvisors.com> Message-ID: I currently deal with mission critical DBs so I need to continuously have ready backups in case of a failure. I've had hardware fail on me and have been able to restore the system up to the minute before failure. Log backups help achieve this. On Dec 7, 2007 10:45 AM, Robert L. Stewart wrote: > Here is a much simpler way on SQL Server to truncate the log. > > Replace PT_Data_Org with you database name and run it in each database. > And, no, you do not have to do a backup first. > > CREATE PROCEDURE [dbo].[__TruncateLog] > AS > BEGIN > ALTER DATABASE PT_Data_Org > SET RECOVERY SIMPLE > END > BEGIN > EXEC sp_dboption 'PT_Data_Org','trunc. log on chkpt.','TRUE' > END > BEGIN > CHECKPOINT > END > BEGIN > DBCC SHRINKFILE (PT_Data_Org_Log,1) > END > BEGIN > EXEC sp_dboption 'PT_Data_Org','trunc. log on chkpt.','FALSE' > END > BEGIN > ALTER DATABASE PT_Data_Org > SET RECOVERY FULL > END > > > Robert > > At 12:00 PM 12/5/2007, you wrote: > >Date: Tue, 4 Dec 2007 22:59:30 -0800 > >From: "Francisco Tapia" > >Subject: Re: [dba-SQLServer] [dba-VB] Truncate a log file from code > >To: dba-sqlserver at databaseadvisors.com > >Message-ID: > > > >Content-Type: text/plain; charset=ISO-8859-1 > > > >One thing that I find very useful with ss is that you can schedule > >these kinds of tasks with alerts. I am not on my pc right now but > >tomorrow I can post some examples of what I typically do. > > > >Ie, I set the tlog to auto backup everytime it reaches 60% of its > >size. I also have a second alert set up so when the tlog grows beyond > >10gb I have it auto backup/shrink > > > >Hth > > > > > > > > > >On 12/3/07, James Barash wrote: > > > No, this is the script that creates the stored procedure. Copy it > > into Query > > > Analyzer for 2000 or a New Query in Management Studio for 2005 select > the > > > appropriate database and execute. That will create the stored > > procedure. You > > > only need to create it once per database. Once you've created it, you > just > > > need to execute it. If you are using VB.Net: > > > > > > Dim Cmd as Data.SqlClient.SqlCommand > > > Cmd.Commandtext = "sp_force_shrink_log" > > > Cmd.CommandType = CommandType.StoredProcedure > > > Cmd.Connection = conn > > > Cmd.ExecuteNonQuery() > > > > > > This assumes you already have a connection (conn) to the database. > > > > > > James Barash > > > > > > -----Original Message----- > > > From: dba-sqlserver-bounces at databaseadvisors.com > > > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of > jwcolby > > > Sent: Monday, December 03, 2007 3:20 PM > > > To: dba-sqlserver at databaseadvisors.com > > > Subject: Re: [dba-SQLServer] Truncate a log file from code > > > > > > I assume you manually copy this stuff into a SP and then edit it each > time? > > > > > > > > > John W. Colby > > > Colby Consulting > > > www.ColbyConsulting.com > > > -----Original Message----- > > > From: dba-sqlserver-bounces at databaseadvisors.com > > > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of James > > > Barash > > > Sent: Monday, December 03, 2007 2:48 PM > > > To: dba-sqlserver at databaseadvisors.com > > > Subject: Re: [dba-SQLServer] Truncate a log file from code > > > > > > John: > > > > > > Below is a stored procedure I downloaded from somewhere that does > exactly > > > what you want, forces a truncate of the log file. I use it in a system > that > > > also imports large text files, often 10-15 million records. Use the > code > > > below to create the stored procedure in the target database and > > then execute > > > it after each import. As I recall, you are using .Net to do your > import so > > > that should be easy to code. I've used it in SQL Server 2000 and 2005 > > > without any problems. > > > > > > James Barash > > > > > > > > > /* Start Code */ > > > CREATE proc sp_force_shrink_log > > > > > > > > > @target_percent tinyint = 0, @target_size_MB int = 10, > > @max_iterations int = > > > 1000, @backup_log_opt nvarchar(1000) = 'with truncate_only' as set > nocount > > > on > > > > > > declare @db sysname, @last_row int, @log_size decimal(15,2), @unused1 > > > decimal(15,2), @unused decimal(15,2), @shrinkable decimal(15,2), > @iteration > > > int, @file_max int, @file int, @fileid > > > varchar(5) > > > > > > select @db = db_name(), @iteration = 0 > > > > > > create table #loginfo ( id int identity, FileId int, FileSize > > numeric(22,0), > > > StartOffset numeric(22,0), FSeqNo int, Status int, Parity smallint, > > > CreateTime > > > nvarchar(255) ) > > > > > > create unique clustered index loginfo_FSeqNo on #loginfo ( FSeqNo, > > > StartOffset ) > > > > > > create table #logfiles ( id int identity(1,1), fileid varchar(5) not > null ) > > > insert #logfiles ( fileid ) select convert( varchar, fileid ) from > sysfiles > > > where status & 0x40 = 0x40 select @file_max = @@rowcount > > > > > > if object_id( 'table_to_force_shrink_log' ) is null exec( 'create > table > > > table_to_force_shrink_log ( x nchar(3000) not null )' ) > > > > > > insert #loginfo ( FileId, FileSize, StartOffset, FSeqNo, Status, > Parity, > > > CreateTime ) exec ( 'dbcc loginfo' ) select @last_row = @@rowcount > > > > > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case > when > > > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( > case > > > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > > > 1048576.00 from #loginfo > > > > > > select @unused1 = @unused -- save for later > > > > > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused > log, > > > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' > > = convert( > > > decimal(6,2), @shrinkable > > > * 100 / @log_size ) > > > > > > while @shrinkable * 100 / @log_size > @target_percent and @shrinkable > > > > > @target_size_MB and @iteration < @max_iterations begin select > @iteration = > > > @iteration + 1 -- this is just a precaution > > > > > > exec( 'insert table_to_force_shrink_log select name from sysobjects > delete > > > table_to_force_shrink_log') > > > > > > select @file = 0 while @file < @file_max begin select @file = @file + > 1 > > > select @fileid = fileid from #logfiles where id = @file exec( 'dbcc > > > shrinkfile( ' + @fileid + ' )' ) end > > > > > > exec( 'backup log [' + @db + '] ' + @backup_log_opt ) > > > > > > truncate table #loginfo insert #loginfo ( FileId, FileSize, > StartOffset, > > > FSeqNo, Status, Parity, CreateTime ) exec ( 'dbcc loginfo' ) select > > > @last_row = @@rowcount > > > > > > select @log_size = sum( FileSize ) / 1048576.00, @unused = sum( case > when > > > Status = 0 then FileSize else 0 end ) / 1048576.00, @shrinkable = sum( > case > > > when id < @last_row - 1 and Status = 0 then FileSize else 0 end ) / > > > 1048576.00 from #loginfo > > > > > > select 'iteration' = @iteration, 'log size, MB' = @log_size, 'unused > log, > > > MB' = @unused, 'shrinkable log, MB' = @shrinkable, 'shrinkable %' > > = convert( > > > decimal(6,2), @shrinkable > > > * 100 / @log_size ) end > > > > > > if @unused1 < @unused select 'After ' + convert( varchar, @iteration ) > + ' > > > iterations the unused > > > portion of the log has grown from ' + convert( varchar, @unused1 > > ) + ' MB to > > > ' + convert( varchar, @unused ) + ' MB.' union all select 'Since the > > > remaining unused portion is larger than 10 MB,' where @unused > > > 10 union all > > > select 'you may try running this procedure again with a higher number > of > > > iterations.' where @unused > 10 union all select 'Sometimes the log > would > > > not shrink to a size smaller than several Megabytes.' where @unused <= > 10 > > > > > > else select 'It took ' + convert( varchar, @iteration ) + ' iterations > to > > > shrink the unused portion of the log from ' + convert( varchar, > > @unused1 ) + > > > ' MB to ' + convert( varchar, @unused ) + ' MB' > > > > > > exec( 'drop table table_to_force_shrink_log' ) > > > > > > GO > > > /* End code */ > > > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- -Francisco http://sqlthis.blogspot.com | Tsql and More... From jwcolby at colbyconsulting.com Fri Dec 7 14:27:30 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Fri, 7 Dec 2007 15:27:30 -0500 Subject: [dba-SQLServer] Connection closed Message-ID: <012701c8390f$978270e0$977aa8c0@M90> I am trying to run the following code: Private Const cstrCnn2 As String = "Data Source = Azul;Initial Catalog = HSID;Integrated Security=SSPI" Private Sub btnMigrateData_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles btnMigrateData.Click Dim lrow As DataRow Dim tbl As DataTable tbl = clsDataAccess.mtblGet("tblHSIDFldName") Dim lcnn As New SqlConnection(cstrCnn2) Dim cmdFldToVertical As New SqlCommand("USP_FldToVertical", lcnn) cmdFldToVertical.CommandType = CommandType.StoredProcedure For Each lrow In tbl.Rows If CInt(lrow("FLD_ID")) > 2 Then Debug.Print(lrow("Fld_ID").ToString & ", " & lrow("Fld_Name").ToString) cmdFldToVertical.Parameters.Add("@SrcTblName", SqlDbType.NVarChar, 100).Value = "" cmdFldToVertical.Parameters.Add("@FldNameID", SqlDbType.Int).Value = CInt(lrow("Fld_ID")) cmdFldToVertical.Parameters.Add("@SrcFldName", SqlDbType.NVarChar, 100).Value = lrow("Fld_Name").ToString Try cmdFldToVertical.ExecuteNonQuery() Catch ex As Exception MsgBox(ex.Message) End Try End If Next End Sub I am getting an error on the try that the connection is closed: "ExecuteNonQuery requires an open and available Connection. The connection's current state is closed." I did not get an error when I dimmed the cmd object which passes in the connection object. How do I check that the connection is open, or troubleshoot why it is not open? John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Dec 8 09:45:57 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 10:45:57 -0500 Subject: [dba-SQLServer] [dba-VB] Connection closed In-Reply-To: <003201c839a6$359f0ef0$4b3a8343@SusanOne> References: <012701c8390f$978270e0$977aa8c0@M90><014a01c8395b$a91ebbf0$977aa8c0@M90> <003201c839a6$359f0ef0$4b3a8343@SusanOne> Message-ID: <016401c839b1$6d4f7e60$977aa8c0@M90> Susan, Thanks for the reply. I don't have a clue and no time to screw around with it more. I was trying to use a stored procedure to migrate data from a "horizontal" to a "vertical" schema. In the end I just resorted to VB.Net. I have a class with a bunch of prewritten functions that allows me to just instantiate and call my class to do things including executing action queries directly out on SQL Server. It took me about 1/2 hour to just redo the whole process, replace the stored procedure with a new function in a form in VB.Net and voila, I am crunching away doing the conversion. At first it was whizzing through the fields of the horizontal table, throwing their data and field ID into the vertical table. It has slowed down now as the number of records in the vertical table has grown. I was reading just the other night about all the work SQL Server has to do to maintain a balanced tree and I think that is what I am seeing now. I have processed 18 fields from the horizontal table and pulled 142 million pieces of actual data out of those 18 fields. There are 584 fields to process in the horizontal table so this could take till next week to process. And then once the process is finished I have to build an index on the table. All in order to discover whether it will be faster to execute a given query using this vertical schema. It should be interesting. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-vb-bounces at databaseadvisors.com [mailto:dba-vb-bounces at databaseadvisors.com] On Behalf Of Susan Harkins Sent: Saturday, December 08, 2007 9:24 AM To: dba-vb at databaseadvisors.com Subject: Re: [dba-VB] Connection closed John, this is just a wilda*s guess because I've seen this happen to me, but it may be the syntax you're using and you're just getting one of the less the helpful error messages. The interesting word is "available" -- my guess is that one of your parameters isn't the right type of something obscure like that, but I really am only guessing. Susan H. > Any ideas out there? The ExecuteNonQuery complains that the connection > is closed. If I break at that point (the catch) and do a lcnn.open it > says the connection is open. Further it tells me the name of the > server, the table name etc. Doesn't sound closed. _______________________________________________ dba-VB mailing list dba-VB at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-vb http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Dec 8 10:20:13 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 11:20:13 -0500 Subject: [dba-SQLServer] Log file reuse Message-ID: <016601c839b6$36c83260$977aa8c0@M90> If a SQL Server action query completes, does SQL server reuse the now empty log file? I have a raid array that is huge and I use to hold these large databases. I have another single drive sitting on a SATA channel. I am experimenting by placing the log file onto that single SATA drive and the data file on the raid. I am running about 660 more action queries to move data from fields in a 50 million record 680 field wide table to a 3 field wide table. The source table and the destination table both sit on the raid, with the log file out on the non-raid. I am hoping that because of IO issues, placing the log file on a different physical drive will speed things up. I have no way to really know however. I am hoping that once the log file is expanded (it is about 13 g now) that unless the data in the subsequent fields requires more log file room, the "cost" of expanding the log file will go away and the log file empty space will be reused for each subsequent action query. John W. Colby Colby Consulting www.ColbyConsulting.com From jwcolby at colbyconsulting.com Sat Dec 8 12:04:20 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 13:04:20 -0500 Subject: [dba-SQLServer] Log file reuse In-Reply-To: <016601c839b6$36c83260$977aa8c0@M90> References: <016601c839b6$36c83260$977aa8c0@M90> Message-ID: <016a01c839c4$c24193f0$977aa8c0@M90> Well, the log file on another drive failed with errors. No time to troubleshoot, I am just moving the log back to the same drive. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 08, 2007 11:20 AM To: dba-sqlserver at databaseadvisors.com Subject: [dba-SQLServer] Log file reuse If a SQL Server action query completes, does SQL server reuse the now empty log file? I have a raid array that is huge and I use to hold these large databases. I have another single drive sitting on a SATA channel. I am experimenting by placing the log file onto that single SATA drive and the data file on the raid. I am running about 660 more action queries to move data from fields in a 50 million record 680 field wide table to a 3 field wide table. The source table and the destination table both sit on the raid, with the log file out on the non-raid. I am hoping that because of IO issues, placing the log file on a different physical drive will speed things up. I have no way to really know however. I am hoping that once the log file is expanded (it is about 13 g now) that unless the data in the subsequent fields requires more log file room, the "cost" of expanding the log file will go away and the log file empty space will be reused for each subsequent action query. John W. Colby Colby Consulting www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From ssharkins at gmail.com Sat Dec 8 13:20:02 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Sat, 8 Dec 2007 14:20:02 -0500 Subject: [dba-SQLServer] about password_hash Message-ID: <004f01c839cf$5735b450$4b3a8343@SusanOne> Can someone explain the password_hash column in the sys.server_principals catalog view. My guess is this hasn't a thing to do with the actual password you assign to individual users and that it has something to do with synchronizing or security, but I don't really understand it. Where can you find a list of "assigned" passwords or is that information available? Susan H. From jwcolby at colbyconsulting.com Sat Dec 8 14:20:51 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 15:20:51 -0500 Subject: [dba-SQLServer] Just an FYI Message-ID: <017701c839d7$d41e5a50$977aa8c0@M90> In converting the database from hell to a "vertical" table, I have processed 36 fields out of 584 fields from a table with 51.157 million records.. It has found 573,397,612 data elements in those 36 fields. 15.93 million data points per field average. It has consumed 12.246 GBytes of data space (no indexes yet). 340 megs / field average. It is taking roughly 300 seconds per field to process at this point in time, though that bounces around a lot depending on how many data elements it finds in a field. Assuming that the statistics so far hold true over the rest of the data (not a real good assumption) and assuming that the insert time / element remains the same (a horrid assumption) it will take (684-36) * 5 minutes to finish the conversion - 3240 minutes, 54 hours. It will also take 340 * 684 megs of data space - 232 gigs. The table when finished would hold 684 * 15.93 million data points - 10896.12 million data points - 10.896 Gig data points. The point of this exercise is to build a table with three fields. The FieldID from a field lookup table, the PKID from the record that the data point is found in, and the data itself. I will have converted a table containing 50 million records but 684 fields, into a table with 11 gig records, but 3 fields. Obviously a table scan is being used on the wide table most of the time, I cannot realistically index 684 fields, and even if I did, many (most) of the fields are a single character consisting of 'Y'. Once the table is built I will build a single index on the table, a clustered index with the fields arranged FieldID, Data and PKID on the theory that most operations will be "where" on a field (thus the field ID first) where the data is a certain value (thus the data next) and returning only the PKID. If I understand things (a HUGE assumption) this clustered index would be optimized for that kind of search. So in the case of the "vertical" table, an "index scan" will be performed BUT all the records for a given field will be contiguous, BUT it will have to scan (worst case) 11 gig records, BUT it will only have to read three fields consisting of two integers and a varchar(100). What would be interesting but probably not feasible would be to analyze which fields are used the most by my client and place them first in the table. That would guarantee that the most used fields occur earliest in the search. THAT would make an enormous difference I think. Then if there are only 5 million data points in a field and they are contiguous, and they are at the front of the table... My good friend Arthur assures me that the new table will on average win the race. We shall see. Does anyone have a got feeling for the results? John W. Colby Colby Consulting www.ColbyConsulting.com From fuller.artful at gmail.com Sat Dec 8 14:37:50 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sat, 8 Dec 2007 15:37:50 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <017701c839d7$d41e5a50$977aa8c0@M90> References: <017701c839d7$d41e5a50$977aa8c0@M90> Message-ID: <29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> Thanks for promoting me to your Good Friends list! Perhaps I forgot to suggest that the only rows/columns of interest are those with values that are NOT NULL or equivalent to "". The entire point of this exercise is that we record only meaningful values. In the case of your YesNo columns, if the column contains neither Yes nor No then it ought not be written to the child table. All we care about are the Yes and No answers. The empty values are of no interest, since they contain no information. A. P.S. We have had our wars, which is one of several reasons why I deem you a Good Friend too, John. On 12/8/07, jwcolby wrote: > > In converting the database from hell to a "vertical" table, I have > processed > 36 fields out of 584 fields from a table with 51.157 million records.. > > It has found 573,397,612 data elements in those 36 fields. 15.93 million > data points per field average. > > It has consumed 12.246 GBytes of data space (no indexes yet). 340 megs / > field average. > > It is taking roughly 300 seconds per field to process at this point in > time, > though that bounces around a lot depending on how many data elements it > finds in a field. > > Assuming that the statistics so far hold true over the rest of the data > (not > a real good assumption) and assuming that the insert time / element > remains > the same (a horrid assumption) it will take (684-36) * 5 minutes to finish > the conversion - 3240 minutes, 54 hours. It will also take 340 * 684 megs > of data space - 232 gigs. The table when finished would hold 684 * 15.93 > million data points - 10896.12 million data points - 10.896 Gig data > points. > > The point of this exercise is to build a table with three fields. The > FieldID from a field lookup table, the PKID from the record that the data > point is found in, and the data itself. I will have converted a table > containing 50 million records but 684 fields, into a table with 11 gig > records, but 3 fields. Obviously a table scan is being used on the wide > table most of the time, I cannot realistically index 684 fields, and even > if > I did, many (most) of the fields are a single character consisting of 'Y'. > > Once the table is built I will build a single index on the table, a > clustered index with the fields arranged FieldID, Data and PKID on the > theory that most operations will be "where" on a field (thus the field ID > first) where the data is a certain value (thus the data next) and > returning > only the PKID. > > If I understand things (a HUGE assumption) this clustered index would be > optimized for that kind of search. > > So in the case of the "vertical" table, an "index scan" will be performed > BUT all the records for a given field will be contiguous, BUT it will have > to scan (worst case) 11 gig records, BUT it will only have to read three > fields consisting of two integers and a varchar(100). > > What would be interesting but probably not feasible would be to analyze > which fields are used the most by my client and place them first in the > table. That would guarantee that the most used fields occur earliest in > the > search. THAT would make an enormous difference I think. Then if there > are > only 5 million data points in a field and they are contiguous, and they > are > at the front of the table... > > My good friend Arthur assures me that the new table will on average win > the > race. We shall see. > > Does anyone have a got feeling for the results? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From fhtapia at gmail.com Sat Dec 8 14:58:23 2007 From: fhtapia at gmail.com (Francisco Tapia) Date: Sat, 8 Dec 2007 12:58:23 -0800 Subject: [dba-SQLServer] Log file reuse In-Reply-To: <016a01c839c4$c24193f0$977aa8c0@M90> References: <016601c839b6$36c83260$977aa8c0@M90> <016a01c839c4$c24193f0$977aa8c0@M90> Message-ID: It will reuse space if you are clearing the data at the check points. One of the many reasons I have auto backups when the log hits > 60%. You can choose juse to runt the truncate log backup command that just simply flushes out committed transactions, in my environment I cannot do that but should not be too bad for you if you so need this functionality. Btw out of curiosity what was the error you received? On 12/8/07, jwcolby wrote: > Well, the log file on another drive failed with errors. No time to > troubleshoot, I am just moving the log back to the same drive. > > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby > Sent: Saturday, December 08, 2007 11:20 AM > To: dba-sqlserver at databaseadvisors.com > Subject: [dba-SQLServer] Log file reuse > > If a SQL Server action query completes, does SQL server reuse the now empty > log file? > > I have a raid array that is huge and I use to hold these large databases. I > have another single drive sitting on a SATA channel. I am experimenting by > placing the log file onto that single SATA drive and the data file on the > raid. I am running about 660 more action queries to move data from fields > in a 50 million record 680 field wide table to a 3 field wide table. The > source table and the destination table both sit on the raid, with the log > file out on the non-raid. I am hoping that because of IO issues, placing > the log file on a different physical drive will speed things up. I have no > way to really know however. > > I am hoping that once the log file is expanded (it is about 13 g now) that > unless the data in the subsequent fields requires more log file room, the > "cost" of expanding the log file will go away and the log file empty space > will be reused for each subsequent action query. > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- -Francisco http://sqlthis.blogspot.com | Tsql and More... From jwcolby at colbyconsulting.com Sat Dec 8 15:08:50 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 16:08:50 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> References: <017701c839d7$d41e5a50$977aa8c0@M90> <29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> Message-ID: <017801c839de$8871a010$977aa8c0@M90> >Thanks for promoting me to your Good Friends list! We have our differences but "I love ya Arthur" (in that sobbing tone). >Perhaps I forgot to suggest that the only rows/columns of interest Yes, that is what I am doing. Any fields with no data is a "" and the transform only pulls WHERE <> ''. So I only have valid data points. You can see that because the average data points per field is 15.93 million, not 51 million (the number of records). John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Saturday, December 08, 2007 3:38 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI Thanks for promoting me to your Good Friends list! Perhaps I forgot to suggest that the only rows/columns of interest are those with values that are NOT NULL or equivalent to "". The entire point of this exercise is that we record only meaningful values. In the case of your YesNo columns, if the column contains neither Yes nor No then it ought not be written to the child table. All we care about are the Yes and No answers. The empty values are of no interest, since they contain no information. A. P.S. We have had our wars, which is one of several reasons why I deem you a Good Friend too, John. On 12/8/07, jwcolby wrote: > > In converting the database from hell to a "vertical" table, I have > processed > 36 fields out of 584 fields from a table with 51.157 million records.. > > It has found 573,397,612 data elements in those 36 fields. 15.93 > million data points per field average. > > It has consumed 12.246 GBytes of data space (no indexes yet). 340 megs > / field average. > > It is taking roughly 300 seconds per field to process at this point in > time, though that bounces around a lot depending on how many data > elements it finds in a field. > > Assuming that the statistics so far hold true over the rest of the > data (not a real good assumption) and assuming that the insert time / > element remains the same (a horrid assumption) it will take (684-36) * > 5 minutes to finish the conversion - 3240 minutes, 54 hours. It will > also take 340 * 684 megs of data space - 232 gigs. The table when > finished would hold 684 * 15.93 million data points - 10896.12 million > data points - 10.896 Gig data points. > > The point of this exercise is to build a table with three fields. The > FieldID from a field lookup table, the PKID from the record that the > data point is found in, and the data itself. I will have converted a > table containing 50 million records but 684 fields, into a table with > 11 gig records, but 3 fields. Obviously a table scan is being used on > the wide table most of the time, I cannot realistically index 684 > fields, and even if I did, many (most) of the fields are a single > character consisting of 'Y'. > > Once the table is built I will build a single index on the table, a > clustered index with the fields arranged FieldID, Data and PKID on the > theory that most operations will be "where" on a field (thus the field > ID > first) where the data is a certain value (thus the data next) and > returning only the PKID. > > If I understand things (a HUGE assumption) this clustered index would > be optimized for that kind of search. > > So in the case of the "vertical" table, an "index scan" will be > performed BUT all the records for a given field will be contiguous, > BUT it will have to scan (worst case) 11 gig records, BUT it will only > have to read three fields consisting of two integers and a varchar(100). > > What would be interesting but probably not feasible would be to > analyze which fields are used the most by my client and place them > first in the table. That would guarantee that the most used fields > occur earliest in the search. THAT would make an enormous difference > I think. Then if there are only 5 million data points in a field and > they are contiguous, and they are at the front of the table... > > My good friend Arthur assures me that the new table will on average > win the race. We shall see. > > Does anyone have a got feeling for the results? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Dec 8 15:10:45 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 16:10:45 -0500 Subject: [dba-SQLServer] Log file reuse In-Reply-To: References: <016601c839b6$36c83260$977aa8c0@M90><016a01c839c4$c24193f0$977aa8c0@M90> Message-ID: <017901c839de$ccbcd730$977aa8c0@M90> One of those generic "second chance error" thingies. No meaningful interpretation provided. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Francisco Tapia Sent: Saturday, December 08, 2007 3:58 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Log file reuse It will reuse space if you are clearing the data at the check points. One of the many reasons I have auto backups when the log hits > 60%. You can choose juse to runt the truncate log backup command that just simply flushes out committed transactions, in my environment I cannot do that but should not be too bad for you if you so need this functionality. Btw out of curiosity what was the error you received? On 12/8/07, jwcolby wrote: > Well, the log file on another drive failed with errors. No time to > troubleshoot, I am just moving the log back to the same drive. > > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > -----Original Message----- > From: dba-sqlserver-bounces at databaseadvisors.com > [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of > jwcolby > Sent: Saturday, December 08, 2007 11:20 AM > To: dba-sqlserver at databaseadvisors.com > Subject: [dba-SQLServer] Log file reuse > > If a SQL Server action query completes, does SQL server reuse the now > empty log file? > > I have a raid array that is huge and I use to hold these large > databases. I have another single drive sitting on a SATA channel. I > am experimenting by placing the log file onto that single SATA drive > and the data file on the raid. I am running about 660 more action > queries to move data from fields in a 50 million record 680 field wide > table to a 3 field wide table. The source table and the destination > table both sit on the raid, with the log file out on the non-raid. I > am hoping that because of IO issues, placing the log file on a > different physical drive will speed things up. I have no way to really know however. > > I am hoping that once the log file is expanded (it is about 13 g now) > that unless the data in the subsequent fields requires more log file > room, the "cost" of expanding the log file will go away and the log > file empty space will be reused for each subsequent action query. > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > -- -Francisco http://sqlthis.blogspot.com | Tsql and More... _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Dec 8 15:25:17 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 16:25:17 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <017801c839de$8871a010$977aa8c0@M90> References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> <017801c839de$8871a010$977aa8c0@M90> Message-ID: <017a01c839e0$d4719450$977aa8c0@M90> I am up to field 63 and now have 954 million data points, in 20 gig of data space. BTW I broke this destination (vertical) table out into its own db but on the same drive (raid) as the source (wide) table. I doubt that is the reason, however I am only averaging about 200 seconds per field at this point and the insert time isn't climbing as I had expected. I have to wonder what the effect would be of running Windows 2003 x64 with SQL Server x64 and 8 gigs of ram (the max I can put in the machine) instead of 4 gigs (the max that Windows 2003 x32 can handle). I tried BTW to do the install, but it failed and I didn't have time to pursue it. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of jwcolby Sent: Saturday, December 08, 2007 4:09 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI >Thanks for promoting me to your Good Friends list! We have our differences but "I love ya Arthur" (in that sobbing tone). >Perhaps I forgot to suggest that the only rows/columns of interest Yes, that is what I am doing. Any fields with no data is a "" and the transform only pulls WHERE <> ''. So I only have valid data points. You can see that because the average data points per field is 15.93 million, not 51 million (the number of records). John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Saturday, December 08, 2007 3:38 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI Thanks for promoting me to your Good Friends list! Perhaps I forgot to suggest that the only rows/columns of interest are those with values that are NOT NULL or equivalent to "". The entire point of this exercise is that we record only meaningful values. In the case of your YesNo columns, if the column contains neither Yes nor No then it ought not be written to the child table. All we care about are the Yes and No answers. The empty values are of no interest, since they contain no information. A. P.S. We have had our wars, which is one of several reasons why I deem you a Good Friend too, John. On 12/8/07, jwcolby wrote: > > In converting the database from hell to a "vertical" table, I have > processed > 36 fields out of 584 fields from a table with 51.157 million records.. > > It has found 573,397,612 data elements in those 36 fields. 15.93 > million data points per field average. > > It has consumed 12.246 GBytes of data space (no indexes yet). 340 megs > / field average. > > It is taking roughly 300 seconds per field to process at this point in > time, though that bounces around a lot depending on how many data > elements it finds in a field. > > Assuming that the statistics so far hold true over the rest of the > data (not a real good assumption) and assuming that the insert time / > element remains the same (a horrid assumption) it will take (684-36) * > 5 minutes to finish the conversion - 3240 minutes, 54 hours. It will > also take 340 * 684 megs of data space - 232 gigs. The table when > finished would hold 684 * 15.93 million data points - 10896.12 million > data points - 10.896 Gig data points. > > The point of this exercise is to build a table with three fields. The > FieldID from a field lookup table, the PKID from the record that the > data point is found in, and the data itself. I will have converted a > table containing 50 million records but 684 fields, into a table with > 11 gig records, but 3 fields. Obviously a table scan is being used on > the wide table most of the time, I cannot realistically index 684 > fields, and even if I did, many (most) of the fields are a single > character consisting of 'Y'. > > Once the table is built I will build a single index on the table, a > clustered index with the fields arranged FieldID, Data and PKID on the > theory that most operations will be "where" on a field (thus the field > ID > first) where the data is a certain value (thus the data next) and > returning only the PKID. > > If I understand things (a HUGE assumption) this clustered index would > be optimized for that kind of search. > > So in the case of the "vertical" table, an "index scan" will be > performed BUT all the records for a given field will be contiguous, > BUT it will have to scan (worst case) 11 gig records, BUT it will only > have to read three fields consisting of two integers and a varchar(100). > > What would be interesting but probably not feasible would be to > analyze which fields are used the most by my client and place them > first in the table. That would guarantee that the most used fields > occur earliest in the search. THAT would make an enormous difference > I think. Then if there are only 5 million data points in a field and > they are contiguous, and they are at the front of the table... > > My good friend Arthur assures me that the new table will on average > win the race. We shall see. > > Does anyone have a got feeling for the results? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From ssharkins at gmail.com Sat Dec 8 15:30:09 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Sat, 8 Dec 2007 16:30:09 -0500 Subject: [dba-SQLServer] Just an FYI References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com><017801c839de$8871a010$977aa8c0@M90> <017a01c839e0$d4719450$977aa8c0@M90> Message-ID: <00af01c839e1$83f9f340$4b3a8343@SusanOne> You are giving me a headache. ;) Susan H. >I am up to field 63 and now have 954 million data points, in 20 gig of data > space. BTW I broke this destination (vertical) table out into its own db > but on the same drive (raid) as the source (wide) table. I doubt that is > the reason, however I am only averaging about 200 seconds per field at > this > point and the insert time isn't climbing as I had expected. > > I have to wonder what the effect would be of running Windows 2003 x64 with > SQL Server x64 and 8 gigs of ram (the max I can put in the machine) > instead > of 4 gigs (the max that Windows 2003 x32 can handle). I tried BTW to do > the > install, but it failed and I didn't have time to pursue it. From jwcolby at colbyconsulting.com Sat Dec 8 15:41:12 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 16:41:12 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <00af01c839e1$83f9f340$4b3a8343@SusanOne> References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com><017801c839de$8871a010$977aa8c0@M90><017a01c839e0$d4719450$977aa8c0@M90> <00af01c839e1$83f9f340$4b3a8343@SusanOne> Message-ID: <017b01c839e3$0e0cabd0$977aa8c0@M90> 8-( This database is largish and I have to do a LOT of counts of people, almost always using many different fields. These things can take 30 minutes or more EACH and I currently have a request for individual counts each count using a different field, about 20 of these counts. I simply have to find a better way. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Susan Harkins Sent: Saturday, December 08, 2007 4:30 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI You are giving me a headache. ;) Susan H. >I am up to field 63 and now have 954 million data points, in 20 gig of >data space. BTW I broke this destination (vertical) table out into >its own db but on the same drive (raid) as the source (wide) table. I >doubt that is the reason, however I am only averaging about 200 >seconds per field at this point and the insert time isn't climbing as >I had expected. > > I have to wonder what the effect would be of running Windows 2003 x64 > with SQL Server x64 and 8 gigs of ram (the max I can put in the > machine) instead of 4 gigs (the max that Windows 2003 x32 can handle). > I tried BTW to do the install, but it failed and I didn't have time to > pursue it. _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From ssharkins at gmail.com Sat Dec 8 16:01:37 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Sat, 8 Dec 2007 17:01:37 -0500 Subject: [dba-SQLServer] Just an FYI References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com><017801c839de$8871a010$977aa8c0@M90><017a01c839e0$d4719450$977aa8c0@M90><00af01c839e1$83f9f340$4b3a8343@SusanOne> <017b01c839e3$0e0cabd0$977aa8c0@M90> Message-ID: <00b401c839e5$e8e95940$4b3a8343@SusanOne> John, I wish I could help, but I just don't have the background for what you're doing -- I barely understand your problem. No way could I come up with a solution. Susan H. > 8-( > > This database is largish and I have to do a LOT of counts of people, > almost > always using many different fields. These things can take 30 minutes or > more EACH and I currently have a request for individual counts each count > using a different field, about 20 of these counts. I simply have to find > a > better way. From jwcolby at colbyconsulting.com Sat Dec 8 16:33:23 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 17:33:23 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <00b401c839e5$e8e95940$4b3a8343@SusanOne> References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com><017801c839de$8871a010$977aa8c0@M90><017a01c839e0$d4719450$977aa8c0@M90><00af01c839e1$83f9f340$4b3a8343@SusanOne><017b01c839e3$0e0cabd0$977aa8c0@M90> <00b401c839e5$e8e95940$4b3a8343@SusanOne> Message-ID: <017c01c839ea$58562430$977aa8c0@M90> And I am not expecting one, from you or anyone else on this list. I think that Arthur has presented a possible solution. I think I am going to break this down into 6 or so different result tables, each with data from a consecutive range of fields across the big (wide) table. I need to build a little "query generator" anyway to write queries for me, and if I add a field to my field lookup table to hold which vertical table the field is held in, then the query generator can write the query to pull data from the correct table. That has some advantages and few disadvantages. The storage tables hold fewer data elements so the scan for any given field is automatically faster. It turns out that the data is grouped such that my client asks for fields "close to each other", i.e. they tend to come from the same survey. Thinking about it in that manner I think I can get this thing pretty fast. Essentially the wide table gets riced and diced and "partitioned" vertically, each vertical partition being N fields wide, but with the advantage that the resulting "vertical" tables hold only the records that actually contain data elements. I am running a program I wrote that builds up this vertical table. I just told it to stop at field 83 which is the end of a logical group. That will be the end of one table. The next table will start at field 84 and go to... not sure yet but the same idea. I think this will work... John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Susan Harkins Sent: Saturday, December 08, 2007 5:02 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI John, I wish I could help, but I just don't have the background for what you're doing -- I barely understand your problem. No way could I come up with a solution. Susan H. > 8-( > > This database is largish and I have to do a LOT of counts of people, > almost always using many different fields. These things can take 30 > minutes or more EACH and I currently have a request for individual > counts each count using a different field, about 20 of these counts. > I simply have to find a better way. _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From fuller.artful at gmail.com Sat Dec 8 18:41:07 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sat, 8 Dec 2007 19:41:07 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <017c01c839ea$58562430$977aa8c0@M90> References: <017701c839d7$d41e5a50$977aa8c0@M90> <29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> <017801c839de$8871a010$977aa8c0@M90> <017a01c839e0$d4719450$977aa8c0@M90> <00af01c839e1$83f9f340$4b3a8343@SusanOne> <017b01c839e3$0e0cabd0$977aa8c0@M90> <00b401c839e5$e8e95940$4b3a8343@SusanOne> <017c01c839ea$58562430$977aa8c0@M90> Message-ID: <29f585dd0712081641n66d243f4s5ad948f3659e9db0@mail.gmail.com> I think it will work too. On 12/8/07, jwcolby wrote: > > And I am not expecting one, from you or anyone else on this list. I think > that Arthur has presented a possible solution. I think I am going to > break > this down into 6 or so different result tables, each with data from a > consecutive range of fields across the big (wide) table. I need to build > a > little "query generator" anyway to write queries for me, and if I add a > field to my field lookup table to hold which vertical table the field is > held in, then the query generator can write the query to pull data from > the > correct table. That has some advantages and few disadvantages. The > storage > tables hold fewer data elements so the scan for any given field is > automatically faster. It turns out that the data is grouped such that my > client asks for fields "close to each other", i.e. they tend to come from > the same survey. > > Thinking about it in that manner I think I can get this thing pretty fast. > Essentially the wide table gets riced and diced and "partitioned" > vertically, each vertical partition being N fields wide, but with the > advantage that the resulting "vertical" tables hold only the records that > actually contain data elements. > > I am running a program I wrote that builds up this vertical table. I just > told it to stop at field 83 which is the end of a logical group. That > will > be the end of one table. The next table will start at field 84 and go > to... > not sure yet but the same idea. I think this will work... > From jwcolby at colbyconsulting.com Sat Dec 8 19:58:30 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 20:58:30 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <29f585dd0712081641n66d243f4s5ad948f3659e9db0@mail.gmail.com> References: <017701c839d7$d41e5a50$977aa8c0@M90><29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com><017801c839de$8871a010$977aa8c0@M90><017a01c839e0$d4719450$977aa8c0@M90><00af01c839e1$83f9f340$4b3a8343@SusanOne><017b01c839e3$0e0cabd0$977aa8c0@M90><00b401c839e5$e8e95940$4b3a8343@SusanOne><017c01c839ea$58562430$977aa8c0@M90> <29f585dd0712081641n66d243f4s5ad948f3659e9db0@mail.gmail.com> Message-ID: <017e01c83a06$ffe85490$977aa8c0@M90> LOL, you BETTER think it will work, you were the one harping at me for the last year to do this! ;-) The first table contains fields 001-083. It contains 1.07 Gig data points and uses 23.624 gig data space. The next table will contain fields 084-142. It is almost finished now. The next table will contain fields 144-233. It will run overnight. I will also start indexing tbl_084_143 overnight as I need to do counts on that table tomorrow. All of these tables are in a single database file. Do I need to be concerned about fragmentation if I start the indexing while the next table is building? John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Saturday, December 08, 2007 7:41 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Just an FYI I think it will work too. On 12/8/07, jwcolby wrote: > > And I am not expecting one, from you or anyone else on this list. I > think that Arthur has presented a possible solution. I think I am > going to break this down into 6 or so different result tables, each > with data from a consecutive range of fields across the big (wide) > table. I need to build a little "query generator" anyway to write > queries for me, and if I add a field to my field lookup table to hold > which vertical table the field is held in, then the query generator > can write the query to pull data from the correct table. That has > some advantages and few disadvantages. The storage tables hold fewer > data elements so the scan for any given field is automatically faster. > It turns out that the data is grouped such that my client asks for > fields "close to each other", i.e. they tend to come from the same > survey. > > Thinking about it in that manner I think I can get this thing pretty fast. > Essentially the wide table gets riced and diced and "partitioned" > vertically, each vertical partition being N fields wide, but with the > advantage that the resulting "vertical" tables hold only the records > that actually contain data elements. > > I am running a program I wrote that builds up this vertical table. I > just told it to stop at field 83 which is the end of a logical group. > That will be the end of one table. The next table will start at field > 84 and go to... > not sure yet but the same idea. I think this will work... > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From jwcolby at colbyconsulting.com Sat Dec 8 20:55:10 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sat, 8 Dec 2007 21:55:10 -0500 Subject: [dba-SQLServer] Clustered index Message-ID: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> I have a table with physical fields PKID, FldNameID, Data. I want to create a clustered index on that based on the field order FldNameID, Data, PKID. Is the index based on the physical order of the fields or the order of the fields specified when you create the index? IOW I went into the widget that creates indexes, specified the fields in the order I desire them in the index, and checked the Clustered check box. It certainly appears that the index will create by the order I specified them in the index widget (which is what I want). But is that true? John W. Colby Colby Consulting www.ColbyConsulting.com From fuller.artful at gmail.com Sat Dec 8 23:09:49 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sun, 9 Dec 2007 00:09:49 -0500 Subject: [dba-SQLServer] Just an FYI In-Reply-To: <017e01c83a06$ffe85490$977aa8c0@M90> References: <017701c839d7$d41e5a50$977aa8c0@M90> <29f585dd0712081237q5e6bcc27p4b063f4878bc6aeb@mail.gmail.com> <017801c839de$8871a010$977aa8c0@M90> <017a01c839e0$d4719450$977aa8c0@M90> <00af01c839e1$83f9f340$4b3a8343@SusanOne> <017b01c839e3$0e0cabd0$977aa8c0@M90> <00b401c839e5$e8e95940$4b3a8343@SusanOne> <017c01c839ea$58562430$977aa8c0@M90> <29f585dd0712081641n66d243f4s5ad948f3659e9db0@mail.gmail.com> <017e01c83a06$ffe85490$977aa8c0@M90> Message-ID: <29f585dd0712082109t600ffe59m8ac44634380c2e22@mail.gmail.com> No. On 12/8/07, jwcolby wrote: > > Do I need to be concerned about fragmentation if I start the indexing > while the next table > is building? > From fuller.artful at gmail.com Sat Dec 8 23:16:57 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sun, 9 Dec 2007 00:16:57 -0500 Subject: [dba-SQLServer] Clustered index In-Reply-To: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> Message-ID: <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> A clustered index physically re-orders the table into the specified sequence. Given the size of your table, this may take some time. The point of a clustered index (AFAIK) is to group objects in contiguous proximity for the most rapid retrieval of proximate objects. For example, given Customers and Orders, a covering index might be clustered so as to guarantee that Customer X's Orders are available within a single page or perhaps two, thus causing minimal disk hits. If this sort of thing is not important, then clustered indexes are beside the point, and create needless index-creation time. A. On 12/8/07, jwcolby wrote: > > I have a table with physical fields PKID, FldNameID, Data. I want to > create > a clustered index on that based on the field order FldNameID, Data, PKID. > Is the index based on the physical order of the fields or the order of the > fields specified when you create the index? > > IOW I went into the widget that creates indexes, specified the fields in > the > order I desire them in the index, and checked the Clustered check box. It > certainly appears that the index will create by the order I specified them > in the index widget (which is what I want). > > But is that true? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From jwcolby at colbyconsulting.com Sun Dec 9 03:50:28 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 9 Dec 2007 04:50:28 -0500 Subject: [dba-SQLServer] Clustered index In-Reply-To: <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> Message-ID: <018301c83a48$eece8f70$977aa8c0@M90> Arthur, >If this sort of thing is not important The reason I am doing this is that: 1) Once created this particular table will never change, EVER. We don't get updates to this one. 2) I am endlessly doing counts and the like based on WHERE clauses. WHERE INCOME IN ('D','E','F') etc. >From my reading, the point of a clustered index is that: a) Once SQL Server finds the first of a series (in this case the FieldNameID) it now knows that all the rest of the records for that field are contiguous b) The data is right there in the index c) That when it gets to the END of a given part of the index it can quit looking - there will not be any FieldNameIDs on down the road because they are all contiguous. Thus to find Field Income it can scan down the index until it finds the first integer value 111 in that field. It has now found the values (actual data for) the first Income field and all the rest of the Income fields are coming next. It starts looking for DATA values of 'D'. Once it finds the first DATA value of D, all the rest are coming up next. It can start gathering the PKIDs for FieldNameID 111 and Data values 'D'. Once it hits the LAST data value 'D' it knows that it can stop searching for 111 / 'D' because all the values are contiguous and it just found the last one. It can now start looking for 111 / 'E'. It is still in 111 (and they are contiguous) so it is already in the right area etc. >From my reading, the "contiguous" part is the key here. It allows SQL Server to STOP the search once the last value is found for a particular field because it knows that there will be no more scattered around in the remaining records. Furthermore, the fact that the data is right there is the next key to the efficiency. Once 111 / 'E' is found, it can just grab the PKID and stuff it in the result set since it doesn't have to do any more leaf searching to find the actual data. So it SOUNDS like a clustered index should make this thing as fast as I can get for situations where there no updates to spoil the party, and there are none for this specific table. The data is entirely static. Given that the data is now only three fields total - FieldNameID, VALUE and PKID, a clustered covering index is doable and sounds ideal. I get to index every single "field" in the old "wide" table with a single covering index. It also turns out that since I have been doing business with Stan he has always asked for information from fields 084-233 of the table. These fields are physically grouped into two "vertical" tables - 084-143 and 144-233, with about 500 million data points in the first (084-143). The sec (144-233) is still processing and I cant get the properties for it yet but I suspect that it will be around twice that number of points. Thus out of a total of 11 Gig data points I will only have to search through 1.5 Gig points to answer all the questions so far encountered. If this works, I will owe you a bottle of your favorite whatever. John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Sunday, December 09, 2007 12:17 AM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Clustered index A clustered index physically re-orders the table into the specified sequence. Given the size of your table, this may take some time. The point of a clustered index (AFAIK) is to group objects in contiguous proximity for the most rapid retrieval of proximate objects. For example, given Customers and Orders, a covering index might be clustered so as to guarantee that Customer X's Orders are available within a single page or perhaps two, thus causing minimal disk hits. If this sort of thing is not important, then clustered indexes are beside the point, and create needless index-creation time. A. On 12/8/07, jwcolby wrote: > > I have a table with physical fields PKID, FldNameID, Data. I want to > create a clustered index on that based on the field order FldNameID, > Data, PKID. > Is the index based on the physical order of the fields or the order of > the fields specified when you create the index? > > IOW I went into the widget that creates indexes, specified the fields > in the order I desire them in the index, and checked the Clustered > check box. It certainly appears that the index will create by the > order I specified them in the index widget (which is what I want). > > But is that true? > > John W. Colby > Colby Consulting > www.ColbyConsulting.com From fuller.artful at gmail.com Sun Dec 9 12:11:31 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sun, 9 Dec 2007 13:11:31 -0500 Subject: [dba-SQLServer] Clustered index In-Reply-To: <018301c83a48$eece8f70$977aa8c0@M90> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> <018301c83a48$eece8f70$977aa8c0@M90> Message-ID: <29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> You're right about the nature and function of clustered indexes. I think it will work, John, and that your search times will diminish dramatically. A. On 12/9/07, jwcolby wrote: > > Arthur, > > >If this sort of thing is not important > > The reason I am doing this is that: > > 1) Once created this particular table will never change, EVER. We don't > get > updates to this one. > 2) I am endlessly doing counts and the like based on WHERE clauses. WHERE > INCOME IN ('D','E','F') etc. > > >From my reading, the point of a clustered index is that: > > a) Once SQL Server finds the first of a series (in this case the > FieldNameID) it now knows that all the rest of the records for that field > are contiguous > b) The data is right there in the index > c) That when it gets to the END of a given part of the index it can quit > looking - there will not be any FieldNameIDs on down the road because they > are all contiguous. > > Thus to find Field Income it can scan down the index until it finds the > first integer value 111 in that field. It has now found the values > (actual > data for) the first Income field and all the rest of the Income fields are > coming next. It starts looking for DATA values of 'D'. Once it finds the > first DATA value of D, all the rest are coming up next. It can start > gathering the PKIDs for FieldNameID 111 and Data values 'D'. Once it hits > the LAST data value 'D' it knows that it can stop searching for 111 / 'D' > because all the values are contiguous and it just found the last one. It > can now start looking for 111 / 'E'. It is still in 111 (and they are > contiguous) so it is already in the right area etc. > > >From my reading, the "contiguous" part is the key here. It allows SQL > Server to STOP the search once the last value is found for a particular > field because it knows that there will be no more scattered around in the > remaining records. Furthermore, the fact that the data is right there is > the next key to the efficiency. Once 111 / 'E' is found, it can just grab > the PKID and stuff it in the result set since it doesn't have to do any > more > leaf searching to find the actual data. > > So it SOUNDS like a clustered index should make this thing as fast as I > can > get for situations where there no updates to spoil the party, and there > are > none for this specific table. The data is entirely static. > > Given that the data is now only three fields total - FieldNameID, VALUE > and > PKID, a clustered covering index is doable and sounds ideal. I get to > index > every single "field" in the old "wide" table with a single covering index. > > It also turns out that since I have been doing business with Stan he has > always asked for information from fields 084-233 of the table. These > fields > are physically grouped into two "vertical" tables - 084-143 and 144-233, > with about 500 million data points in the first (084-143). The sec > (144-233) is still processing and I cant get the properties for it yet but > I > suspect that it will be around twice that number of points. Thus out of a > total of 11 Gig data points I will only have to search through 1.5 Gig > points to answer all the questions so far encountered. > > If this works, I will owe you a bottle of your favorite whatever. > > John W. Colby > Colby Consulting > www.ColbyConsulting.com > From jwcolby at colbyconsulting.com Sun Dec 9 14:33:58 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 9 Dec 2007 15:33:58 -0500 Subject: [dba-SQLServer] My learning experience - was RE: Clustered index In-Reply-To: <29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90><29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com><018301c83a48$eece8f70$977aa8c0@M90> <29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> Message-ID: <018e01c83aa2$d3cc1760$977aa8c0@M90> This email is going to present results of a "side by side" (but not quite apples to apples) comparison of the work I have done over the last few years on a huge database which I lovingly call "the database from hell". But first some background. I have three different databases. One called HSIDRaw (first generation), the next called HSID (second generation), the next called HSIDVert. tblHSIDCean: HSIDRaw has a table called tblHSIDClean which is all of the original data, 65 million records, with close to 700 fields. This data comes from an old "fixed width" text file which I imported years ago. It was this database / table that I have mostly been using for the last couple of years. Each record in tblHSIDClean has an autonumber PKID field which is the PK of each record in tblHSIDClean, created at the time I did the data import. This PKID carries through to everything else I do. The data in this table consists of responses to polls, so although there are 65 million records in the database, not all people answered every poll. There is data about boat ownership (about 7 million responders), income (about 40 million responders), travel (unknown number of responders ATM) etc. So every field does not contain data for every record, in fact in some fields the data is sparsely populated. I took the name / address info from this table and had it "address validated", the results of which sit in another table of ~51 million valid deliverable name / address records (tblAZHSID). That data carries the PKID from tblHSIDClean as well so that I can relate a clean address back to these tblHSIDClean records. That "address validated" table contains ONLY address validation info, and NONE of the data in tblHSIDClean. tblAZHSID is of no interest to this discussion other than so that you can conceptualize how the data might be used to get "names / addresses of people WHERE...", i.e. mailing list data, which is the ultimate requirement of this data. tblAZHSID has address info, tblHSIDClean contains WHERE info, and they are related on PKID. INDEXES: In tblHSIDClean I used indexes on the fields that the client would ask about. I would build the index on demand (as I needed it) and they slowly accumulated. The indexes were always just a single field, no "cover index" concept at this point. These indexes provided fast COUNTS but did not work well for returning PKID sets for further processing in complex queries with WHERE clauses covering many different unrelated fields - WHERE Income IN() AND BoatLength IN() AND HasDogs = 'Y'. tblHSID: In the last month I extracted tblHSID From HSIDClean, out into a new database called HSID. tblHSID has the leading ~100 fields stripped off. These 100 fields are OLD address validation stuff and is never used at all for any reason. Thus tblHSID has 584 fields. Additionally, tblHSID only has 51 million records. When I did address validation years ago It turned out that about 14 million records "fell out" of the 65 million records because of bad address info. The 51 million records represent all the records that were deliverable. This table carries the PKID field from tblHSIDClean so that I know which record in tblHSID (new) relates to which record in tblHSIDClean (old). To summarize, tblHSID has ~100 fewer fields and 14 million fewer records than tblHSIDClean. INDEXES: In this database I started building "cover indexes" where each index was GENERALLY still for a single field but would also include the PKID as well. In a couple of cases I build a cover index where a logical group of fields were all contained in the index, as well as the PKID. In these cover indexes I place the data field first in the index, and then the PKID, on the theory that I would be doing WHERE clauses using this index to look for specific values in the data field. In some cases the results are simple counts, how many records with HullType = 'A' (boaters with aluminum hulls) or NarrowIncomeBand = 'T' (people with incomes > 150K). So covering indexes SHOULD make the process of getting counts of specific data elements faster, and in particular should make the process of getting actual PKID sets for these data elements faster. One of the uses of the database is to get actual addresses for all of the people WHERE..., for which I need PKID sets in order to facilitate comparisons and joins, and also to get at the name / address data over in tblAZHSID. tblHSIDVert: In the last few days I created HSIDVert database with tblHSIDVert_001_083, tblHSIDVert_084_143, tblHSIDVert_144_233 etc. Each table contains data points from specific sets of fields. You can think of each table as a vertical partition of tblHSID, although of course it is more than that. In these tables I completely restructured the data in the manner I have discussed in the last few days. I got a list of all of the remaining data fields (all of the fields in tblHSID), 584 fields total, and placed the field names into tblHSIDFldName with an autoincrement PKID which becomes the FieldNameID. I then created tables named as above, with just three columns, FldNameID, FldValue, and PKID. PKID of course is the PKID from records in tblHSID and tblHSIDClean and simply tells me which RECORD back in tblHSID / tblHSIDClean each "data point record" is coming from. FldNameID comes from the above mentioned tblHSIDFldName and simply tells me which FIELD each data point comes from. Data is the data point or element from the field specified. I then created a process that pulls ONLY the data points out of tblHSID for each field in tblHSID, and place those data points into tblHSIDVert. IOW I grabbed "just the facts ma'am" and "stacked them" into a single column of data, with a pair of other fields to keep it all straight - the FieldNameID and the PKID. INDEXES: In HSIDVert I created a single clustered index covering all three fields, in the order of FldNameID, Data, PKID. If you think about the concept, the "second generation" (tblHSID) is of course better than the "first generation" (tblHSIDClean) for three reasons, fewer records, fewer fields, and covering indexes. HOWEVER I still have to create these covering indexes as the client asks for results using a specific field. I.e. each field has its own distinct covering index, and there are still 584 possible fields needing a cover index. That's a lot of indexes. The "third generation" has some advantages in that ONLY the actual data is contained in the table, (no empty fields) and EVERY field gets an index automatically due to the fact that this table uses a cover index (the clustered index). So which "schema" will win the race? Results in part two of this email. John W. Colby Colby Consulting www.ColbyConsulting.com From fuller.artful at gmail.com Sun Dec 9 15:36:47 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sun, 9 Dec 2007 16:36:47 -0500 Subject: [dba-SQLServer] My learning experience - was RE: Clustered index In-Reply-To: <018e01c83aa2$d3cc1760$977aa8c0@M90> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> <018301c83a48$eece8f70$977aa8c0@M90> <29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> <018e01c83aa2$d3cc1760$977aa8c0@M90> Message-ID: <29f585dd0712091336t275226d5qf8bf8493294f9f4d@mail.gmail.com> I'm dying to learn the results, John. From jwcolby at colbyconsulting.com Sun Dec 9 20:32:31 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Sun, 9 Dec 2007 21:32:31 -0500 Subject: [dba-SQLServer] My learning experience In-Reply-To: <018e01c83aa2$d3cc1760$977aa8c0@M90> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90><29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com><018301c83a48$eece8f70$977aa8c0@M90><29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> <018e01c83aa2$d3cc1760$977aa8c0@M90> Message-ID: <01b701c83ad4$eb0748a0$977aa8c0@M90> OK, so here I am going to run some timings on various things, using all three databases. My client gave me an actual order to provide counts of the HSID data against the smoker data (a different table). From the HSID data I need to do selections: NarrowIncomeBand codes 1-5 codes 6-8 codes 9, A-D codes E-I codes J-S codes T Gender - Male Code 1 Gender - Female Code 2 YearModelOfBoat Code <> '' Pool Code 'Y' Etc Etc. The client is trying to understand the data in two different completely unrelated tables, the HSID file and the TFF smokers, as well as the HSID file and the SMK smokers. The order actually has me join PKID result sets to another pair of tables the TFF smoker file and the SMK smoker file. IOW count: TFF smokers WHERE narrow income band 1-5 SMK smokers WHERE narrow income band 1-5 TFF Smokers WHERE Gender 1 SMK Smokers WHERE Gender 1 TFF Smokers WHERE Gender 2 SMK Smokers WHERE Gender 2 etc etc. The problem is that TFF and SMK smokers are unrelated between themselves, and also unrelated to HSID. In order to get counts, I need to somehow get something in common, which will be the name / address data. That part of the equation is really too complex to go into here ("beyond the scope of this discussion") and so I am going to punt and ignore the join between HSID and any other file (smokers in this case) and instead simply discuss how I "prep" the HSID file to allow me to compare it later on, and then compare timings of those "prepped" queries. Create order structures: I typically create a directory structure - PSM (the client) \ Orders \ OrderNum. In that directory on the disk I place the spreadsheet containing the order as well as any queries that I create that I want to save out as actual query files, any documents I get or create, and any CSVs or fixed with output files in the case of orders for names or counts. Counts generally go into spreadsheets directly. I also create a new database container to hold the views and temp tables required to fill that specific order, with the name of the order in the database name. Once I have a database container I... Create the Indexes: The next thing I have to do in the case of tblHSIDClean and tblHSID is to go in and make sure that I have indexes on the fields that I am going to do WHERE comparisons on. Sometimes they already exist from a previous order, sometimes I have to create the indexes because there isn't one already. In the case of tblHSIDClean I will create indexes of just each field individually. In tblHSID I will create "cover indexes" of either the individual field AND the PKID, or if there are several RELATED fields in the WHERE clause I will create a cover index including all of the related fields AND the PKID field. Creating these indexes typically takes a few minutes per index, with the exact time varying by how many actual data points are in a specific field, as well as how many fields are in the cover index (in the case of tblHSID). In the case of tblHSIDVert, I already have a single clustered index and that is all I need. Create views: Next I create views to pull each field with the PKID for each WHERE column. Strictly speaking individual queries are not required, i.e. I could create one mongo query (and I used to do so), but in the past anyway these mongo queries would take anywhere from 20 minutes to an hour to run. The problem was simply that if anything was not right I would have to make changes and rerun, leading to LOOOOONG times to get the thing done, though not necessarily lots of billable time. So I learned to create "field specific" views that would pull just the PKIDs required for one piece of the final result, and later join the pieces into a whole once the pieces pulled the right things. With a query for each piece of the bigger WHERE clause, I would then start the process of combining the queries to AND and OR them into the larger "whole WHERE clause". Joins between the PKIDs form the ANDS. So if I need (for example) WHERE Income in('D', 'E', 'F') AND CasionGambler = 'Y', I would create two queries, one to get all the PKIDs matching the income criteria, the other with all the PKIDs that match the CasinoGambler, and then join the PKIDs in a higher level query to AND the result set. Often times I need to get COUNTS so I would build a query on top of the queries that and / or the smaller queries to COUNT() the PKIDs that make it through the subqueries. If you have gotten this far then you understand how I go about constructing the stuff required to fill the order. The order might just be "counts" of how many people in an income range gamble, or how many people have dogs AND have cats, or how many people have children, are in a specific income range, and have cats. My client then knows how many "addresses" he can deliver for those criteria, and can go to his client with a proposal for supplying addresses. He might get the order, in which case he comes back to me and says, "now deliver 100K name / Address records in zip codes X, Y and Z". And of course this assumes he asked for counts in those zip codes, and there were at least 100K records made it through the filters. That's the idea and now I will present some timing results for some of the sub queries used in an actual order, this order is just for counts as discussed at the top of this email. I will construct the sub queries for all three different databases and time them. I will NOT take it to the point where I tie it into the smoker databases because that "is beyond the scope of this discussion". I am also going to bring AccessD into this because there might be folks on that list interested in this subject. OTOH maybe NOBODY except Arthur is interested! ;-) Next email, the results. It may be a day or so. I have to actually get this out TONIGHT so I will be doing whatever I have to do to get results. I will do timings tomorrow sometime. John W. Colby Colby Consulting www.ColbyConsulting.com From dwaters at usinternet.com Sun Dec 9 22:06:00 2007 From: dwaters at usinternet.com (Dan Waters) Date: Sun, 9 Dec 2007 22:06:00 -0600 Subject: [dba-SQLServer] Table Prefix Message-ID: <000001c83ae1$f9bf5e20$0300a8c0@danwaters> I just upsized a BE to SQL Server from A2003. The tables, which were all prefixed with tbl, and now prefixed with dbo_tbl. Is there any reason to keep the dbo_ prefix? Thanks, Dan From fuller.artful at gmail.com Sun Dec 9 22:27:56 2007 From: fuller.artful at gmail.com (Arthur Fuller) Date: Sun, 9 Dec 2007 23:27:56 -0500 Subject: [dba-SQLServer] Table Prefix In-Reply-To: <000001c83ae1$f9bf5e20$0300a8c0@danwaters> References: <000001c83ae1$f9bf5e20$0300a8c0@danwaters> Message-ID: <29f585dd0712092027j2788aea0p5cf2d31affcc8538@mail.gmail.com> No. This is a sad accident of the upgrading process. A. On 12/9/07, Dan Waters wrote: > > I just upsized a BE to SQL Server from A2003. The tables, which were all > prefixed with tbl, and now prefixed with dbo_tbl. > > Is there any reason to keep the dbo_ prefix? > > Thanks, > Dan > > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > From dwaters at usinternet.com Mon Dec 10 07:12:15 2007 From: dwaters at usinternet.com (Dan Waters) Date: Mon, 10 Dec 2007 07:12:15 -0600 Subject: [dba-SQLServer] Table Prefix In-Reply-To: <29f585dd0712092027j2788aea0p5cf2d31affcc8538@mail.gmail.com> References: <000001c83ae1$f9bf5e20$0300a8c0@danwaters> <29f585dd0712092027j2788aea0p5cf2d31affcc8538@mail.gmail.com> Message-ID: <000601c83b2e$48c4f1a0$0300a8c0@danwaters> Good! I've only got about 30 tables so it'll be quick. Thanks! Dan -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Arthur Fuller Sent: Sunday, December 09, 2007 10:28 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] Table Prefix No. This is a sad accident of the upgrading process. A. On 12/9/07, Dan Waters wrote: > > I just upsized a BE to SQL Server from A2003. The tables, which were all > prefixed with tbl, and now prefixed with dbo_tbl. > > Is there any reason to keep the dbo_ prefix? > > Thanks, > Dan > > > > > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > > _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From accessd at shaw.ca Mon Dec 10 12:01:56 2007 From: accessd at shaw.ca (Jim Lawrence) Date: Mon, 10 Dec 2007 10:01:56 -0800 Subject: [dba-SQLServer] My learning experience In-Reply-To: <01b701c83ad4$eb0748a0$977aa8c0@M90> References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> <29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com> <018301c83a48$eece8f70$977aa8c0@M90> <29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com> <018e01c83aa2$d3cc1760$977aa8c0@M90> <01b701c83ad4$eb0748a0$977aa8c0@M90> Message-ID: Hi John: I am waiting with baited-breathe.... ;-) Jim > OTOH maybe NOBODY except Arthur is interested! ;-) > Next email, the results. It may be a day or so. I have to actually get > this out TONIGHT so I will be doing whatever I have to do to get results. > I will do timings tomorrow sometime. > John W. Colby > Colby Consulting > www.ColbyConsulting.com From jwcolby at colbyconsulting.com Mon Dec 10 12:12:39 2007 From: jwcolby at colbyconsulting.com (jwcolby) Date: Mon, 10 Dec 2007 13:12:39 -0500 Subject: [dba-SQLServer] My learning experience In-Reply-To: References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90><29f585dd0712082116t3c9c629ekbd430499f056fa5@mail.gmail.com><018301c83a48$eece8f70$977aa8c0@M90><29f585dd0712091011n5e99e8acm2afa88e34d1b8f3f@mail.gmail.com><018e01c83aa2$d3cc1760$977aa8c0@M90><01b701c83ad4$eb0748a0$977aa8c0@M90> Message-ID: <01f601c83b58$40224e10$977aa8c0@M90> Join me on the dock tonight, fishing pole in hand... ;-) John W. Colby Colby Consulting www.ColbyConsulting.com -----Original Message----- From: dba-sqlserver-bounces at databaseadvisors.com [mailto:dba-sqlserver-bounces at databaseadvisors.com] On Behalf Of Jim Lawrence Sent: Monday, December 10, 2007 1:02 PM To: dba-sqlserver at databaseadvisors.com Subject: Re: [dba-SQLServer] My learning experience Hi John: I am waiting with baited-breathe.... ;-) Jim > OTOH maybe NOBODY except Arthur is interested! ;-) > Next email, the results. It may be a day or so. I have to actually > get this out TONIGHT so I will be doing whatever I have to do to get results. > I will do timings tomorrow sometime. > John W. Colby > Colby Consulting > www.ColbyConsulting.com _______________________________________________ dba-SQLServer mailing list dba-SQLServer at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/dba-sqlserver http://www.databaseadvisors.com From ssharkins at gmail.com Mon Dec 10 17:26:57 2007 From: ssharkins at gmail.com (Susan Harkins) Date: Mon, 10 Dec 2007 18:26:57 -0500 Subject: [dba-SQLServer] Oh my... Message-ID: <005a01c83b84$2c777fd0$4b3a8343@SusanOne> http://www.guardian.co.uk/technology/2007/jun/21/it.guardianweeklytechnologysection Susan H. From michael at ddisolutions.com.au Tue Dec 18 18:46:30 2007 From: michael at ddisolutions.com.au (Michael Maddison) Date: Wed, 19 Dec 2007 11:46:30 +1100 Subject: [dba-SQLServer] Clustered index References: <017f01c83a0e$ea5f5ad0$977aa8c0@M90> Message-ID: <59A61174B1F5B54B97FD4ADDE71E7D0128A222@ddi-01.DDI.local> Hi John, Came across this discussion on another list. Thought it might be helpful. cheers Michael M I believe Tom is partly correct. But if I build a covering index on LastName, FirstName, Birthdate, it will cover queries that involve LastName only in the WHERE clause and it will cover a query that looks for LastName and FirstName as well as one for LastName, FirstName and Birthdate. It will also cover a query that looks for LastName, Birthdate. It may not cover a query that only looks for FirstName or one that looks for FirstName, BirthDate. Only the query plan would show whether it does or not. The optimizer does keep overall density statistics on the subordinate keys, but not detailed statistics. This is a place where I think column statistics might be useful. Sharon ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of Tom Cooper Sent: Tuesday, December 18, 2007 12:34 PM To: sql2k5 at sswug.org Subject: RE: [SQL2K5] compound index vs singleton? I agree with Adi also. The kindest thing I can say about my answer is it was not clear. I was responding to the previous message in the thread from Drew where he was concerned with covered queries. So I was talking about covered queries only. But I did not specify that. And that was misleading since the typical query does not have a covered index. And if the index does not cover the query than the order of the columns in the index can make a huge difference in the performance since, as Kenneth pointed out, SQL keeps statistics only on the first column, and if the first column not selective enough, SQL will not use the index, but will instead scan the table. But the situation is different for a covered query. AFAIK, SQL will not look at the statistics for the index for a covered query, since the index is covers the query and there is no need to go to the table at all. So if your index is on Country, FName, and LName, in any order, then for the query Select Country, LName, FName >From TestTable Where Country = 'US' And LName = 'Cooper' And FName = 'Tom' And the table has X rows with N rows matching the WHERE condition. Then in the index, those N rows will be together in the index (no matter what the order of the columns in the index), and SQL will seek to the first row in the index that matches the condition, scan thru only those N rows and then stop the scan and return the result. So in that very restricted case, the order of the columns in the index makes no difference (except, of course, if that covered query has an ORDER BY, in which case, the best of all possible worlds would be to have the columns in the index in the same order as the columns in the ORDER BY, so SQL would not have to do a sort before returning the results). I probably should have just said "it depends" Tom ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of Sharon F. Dooley Sent: Tuesday, December 18, 2007 5:16 AM To: sql2k5 at sswug.org Subject: RE: [SQL2K5] compound index vs singleton? I agree with Adi. The order does make a big difference. I once solved a performance problem by moving what seemed to be the most minor of several columns to the beginning of the index. It made a huge difference. Sharon ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of Adi Cohn Sent: Tuesday, December 18, 2007 2:53 AM To: sql2k5 at sswug.org Subject: RE: [SQL2K5] compound index vs singleton? While I agree with most of what you wrote, there is one statement from your message that I don't agree - "If you are referencing all of the columns in the index in your WHERE clause and all of the comparisons are for equality, then AFAIK the order of the columns in the index doesn't make a difference". In my opinion in this case you should order the column in the index according to the columns' selectivity. For example if in the table there are only 2 countries - USA and Canada, then country will be the last column in the index. Adi ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of Tom Cooper Sent: Tuesday, December 18, 2007 3:54 AM To: sql2k5 at sswug.org Subject: RE: [SQL2K5] compound index vs singleton? If you are referencing all of the columns in the index in your WHERE clause and all of the comparisons are for equality, then AFAIK the order of the columns in the index doesn't make a difference. But if you are only referencing some of those column in the WHERE clause or some of the comparisons return a range, the order makes a difference. For example, if you have a table of everyone in the world and your query is SELECT Country, FName, LName FROM WHERE Country = 'US' AND LName = 'Cooper' Then an index on (Country, LName, FName) will be much better for that query than an index on (FName, LName, Country) even though both of those would be covering indexes. With the first index, you would get a seek to the first entry for US and Cooper and then it would only have to read the index from that point until it found the first entry which was not US and Cooper. But with the second index, it would have to scan the entire index. Tom ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of drewg1010101 at aol.com Sent: Monday, December 17, 2007 8:07 PM To: sql2k5 at sswug.org Subject: Re: [SQL2K5] compound index vs singleton? thanks very much, so for the index to be useful as a covering index, is the covering index constructed on the basis of selectivity/dispersion of values through the table OR does order of attributes at creation matter? iow, is it enough that each column referenced in each part of the query (select from where) only be included the create index statement without respect to their order in index creation statement? thanks again drew -----Original Message----- From: Sharon F. Dooley To: sql2k5 at sswug.org Sent: Mon, 17 Dec 2007 7:06 pm Subject: RE: [SQL2K5] compound index vs singleton? No way! While it may be smart and use two single-column indexes, it is not guaranteed that it will do so. If your queries involve multipart where clauses, you will definitely benefit from multi-column indexes. If you create two single-column indexes, you will also have two indexes that have to be maintained and two sets of statistics that will have to be updated. Assuming that these are non-clustered indexes, you will get the biggest bang for the buck with covering indexes. You can use SQL 2005's included columns feature to include non-key columns. Sharon ------------------------------------------------------------------------ -------- From: listbounces at sswug.org [mailto:listbounces at sswug.org] On Behalf Of drewg1010101 at aol.com Sent: Monday, December 17, 2007 4:15 PM To: sql2k5 at sswug.org Subject: [SQL2K5] compound index vs singleton? i seem to remember that SQL Server will make use of two simple indexes if the optimizer determines that is the cheapest way to resolve a query, the reason i care is because if the server will combine simple indexes appropriately, i should NOT be creating multiple column indexes is that correct? thanks drew From jlawrenc1 at shaw.ca Mon Dec 24 11:21:20 2007 From: jlawrenc1 at shaw.ca (Jim Lawrence) Date: Mon, 24 Dec 2007 09:21:20 -0800 Subject: [dba-SQLServer] OT Christmas In-Reply-To: <012701c8390f$978270e0$977aa8c0@M90> References: <012701c8390f$978270e0$977aa8c0@M90> Message-ID: <8E3ADC09A1CE4749B979C41BA0BF0064@creativesystemdesigns.com> Hi All: At the risk of being politically incorrect: Merry Christmas and a Happy New Year to the best list on the Internet... :-) Best Regards Jim