jwcolby
jwcolby at colbyconsulting.com
Sat Jun 12 08:26:02 CDT 2010
I am doing ETL but I am doing very specific, very repetitive ETL. I am not trying to migrate the world in and out. What I am writing is an application to handle Stan's entire world. I do many different jobs for him and they all revolve around list processing. Internal to my database the lists all have a common component (name / address) that is the level that is exported out / in. What I am doing is automating what I used to do manually, using C# and SQL Server stored procedures. I had over time developed the stored procedures and ran them manually. Now I run them from my application. I am sure you would be fascinated to see what I do here. John W. Colby www.ColbyConsulting.com Mark Breen wrote: > Hello John, > > I agree with Michael here, only one persons opinion matters here and that's > yours as only you really now the true effect. > > Is it not possible to do a method A and Method B comparison? > > I am presuming that each time you face the decision the results are > significantly different that it is difficult to identify the true answer. > If that is the case, the again, only you will really know. > > Can I ask, have you read up much on ETL? The works you are doing nowadays > is AFAICS, ETL and I believe that there are plenty of ETL tools. In fact it > is a subject area in itself and my instinct tells me that traditional > approaches to data management is different when you are doing ETL. > > I only saw it for a few months when I worked for a while in an Oracle Shop, > but they had a whole ETL department with their own tools. Those tools did a > lot of the heavy lifting that you are coding. > > HTH somewhat > > Mark > > > On 10 June 2010 01:57, Michael Maddison <michael at ddisolutions.com.au> wrote: > >> Hi John, >> >> I think you will only get opinions, if that. >> You are going to have to test each scenario IMO. >> Another option for testing is to enable/disable the covering indexes. >> You probably need to update statistic after enabling if the data has >> changed. >> If the data hasn't changed then this 'should' be your best option. >> >> Cheers >> >> Michael M >> >> >> I have processes which process huge tables (65 million records are >> common) and update specific >> fields with the results of user defined functions. For example I have a >> function which builds a >> text "code" using four passed in values. These four values come from >> four fields of the record and >> the resulting code is placed back in another field of the same record. >> >> ATM I build a "cover index" just before updating this field, perform the >> update, then delete the >> index when done. >> >> I am wondering whether you guys believe that the time to build the cover >> index will be recovered by >> less time taken to perform this update. IOW does it take sufficiently >> less time to get data from an >> index vs a table scan that it is worth building the index? >> >> -- >> John W. Colby >> www.ColbyConsulting.com >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> >> No virus found in this incoming message. >> Checked by AVG - www.avg.com >> Version: 9.0.829 / Virus Database: 271.1.1/2926 - Release Date: 06/10/10 >> 04:35:00 >> >> _______________________________________________ >> dba-SQLServer mailing list >> dba-SQLServer at databaseadvisors.com >> http://databaseadvisors.com/mailman/listinfo/dba-sqlserver >> http://www.databaseadvisors.com >> >> > _______________________________________________ > dba-SQLServer mailing list > dba-SQLServer at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/dba-sqlserver > http://www.databaseadvisors.com > >