[dba-VS] [AccessD] Visual Studio and source control

[John Colby]

Jim,

Would I use the cloud if I started today?  Maybe but probably not. One 
issue is that every month I have to export 500 million names and 
addresses into csv text files, 500K records at a clip, push them to 
disk, then push those files to VMs where 3rd party software processes 
those csv files.  The results are pushed over the internet to California 
for further processing on a server at Accuzip, and results brought back 
to my VMs, whereupon the reassembled text files, with many extra fields, 
are re-imported into SQL Server.

I have a SQL Server doing what it does best, 4 (and potentially more) 
VMs doing what only they can do, Accuzip's server doing what only it can 
do, a development VM running my custom written (and pretty complex) 
control process which automates this whole process.

There are just some things that don't match the cloud very well. Today 
the cloud wants to charge on some "per something" basis.  Per I/O, Per 
transaction, Per core, per etc.

I have two physical boxes which does this entire thing and, once built 
and programmed, essentially costs me nothing more than hosting fees.

Yep, it cost me a lot of time and effort, but it gained me 11 years of 
significant income (so far).  Understand that I started out doing all 
this stuff "manually", and piece by piece automated it.  Today these 
files export out, process and import back in with perhaps a couple of 
hours of my time.  And yet I am still paid, even more than I was when I 
did it all manually.  When I started I processed a single database, 65 
million records and put in perhaps 50 hours a week doing all of this 
processing manually.  Today I process 500 million records for perhaps 5 
- 10 hours a month.  That is the power of automation and that is what I 
have done over those 11 years.

On 11/30/2015 6:35 PM, Jim Lawrence wrote:
> Hi John:
>
> Well of course you are not downloading the entire core with all the rigs and every version that will run on you smartphone up to a 3D Imax, of the movie just you can rip and view it on your TV but all that detail is saved.
>
> What you have is an excellent application along with the biggest and baddest home computer anyone could assemble. That said, today, few individuals would ever consider that as an option for resolving their big data problems. Most system admins would move their needs off to the web (Cloud). It is basically cheaper, faster and easier...how much would you calculate all your hours times say $65 per hour... Add the price of components, assemble, shipping, self-training, ongoing maintenance and so on. What you database server look like to me is a beautiful, hand-built, high performance sports-car...a real thing of beauty.
>
> If you were starting from scratch today would you follow the same route?  You wouldn't necessarily have to move everything to a public Cloud.
>
> I would bet that with maybe ten beater boxes, not particularly tricked-out, connected in a single network, a single file-system, where all the systems work as a single entity, the same functionality (or far greater) could be obtained along with superior performance, far greater reliability and probably for a lot cheaper...right in your basement.
>
> I have used this video before but it does lend its self towards the new way data is managed. It shows a IBM system but really what we are seeing is a system with a number motherboards, cores and memory all connected into a single entity. With all those cores, memory and access to a group of OSS programs (services) all abstracted and assembled through an application like Juju (https://jujucharms.com/ ...which is actually what Microsoft Azure uses.)
>
> https://www.youtube.com/watch?v=VWBNoIwGEjo
>
> Aside: I have been playing at this for a while with this tech with some success but given a combination of, my life is not my own and laziness, the progress has been slow. ;-)
>
> Jim