Darryl Collins
darryl at whittleconsulting.com.au
Wed Nov 16 17:02:35 CST 2011
No probs Shamil, It is always good to explore these issues. This role I am in now is a little unique compared to other positions I have had. In the past I have usually been involved with building rock solid user solutions that are rolled out over a business. They have to look good, function well and be nice and strong and stable as the user are just users who do what they do. They generally also follow a structured business workflow or plan and are fairly constrained on their input and outputs. If I have had the advantage of working with SQL Server I would usually get access to a big enterprise setup which was managed by dedicated DBA's with all the bells and whistles. There were always rules about what could and couldn't be done etc. In this sort of environment I can see your point about TCO and economies of scale and maintenance etc. Now in this role, I work with a small company in a team of 4 other guys and we manipulate large volumes of data which are project specific. Each project is largely unique so whilst we have a sort of framework we use, there is a lot of 'start from scratch' stuff in the early days when we get the data from the clients. In this role there is no need for nice GUI's for the users, or functionality or anything like. It is mostly pure data and brute force processing power to get the clients data into a useable framework for our processes. In reality we use a whole stack of software tools, From Excel, Access, SQL Server Express to dedicated mining software with GemcomWhittle and Surpac. The plan was not so much to replace SQL Server Express, but replace it for this job and see how it works. If MySQL has the grunt and space to get a result then I will use it this time. It may not be the best tool for next time. We really do use the best tool for the dataset we have. Sometimes that is Excel, although it usually is Access. That is what I love about this role. It really is very different and hands on - without any hassle of having to spend hours and hours building a functional UI or even a functional DB. I mean we do built GUI for us, but they don't have to be rock solid and stable as they guys here are all data nerds like me and generally know there way around this stuff. One of the best and most interesting roles I have ever had. Now, based on this, I am probably not the best person to ask. One day we may have a dedicated and central server for processing data, but right now all of our heavy duty processing machines (about 25 of them) run our propriety software that actually produces the results for the clients. Think of what we do as drive the heavy bulldozers and trucks that mash the data and prepare it for processing. Hope that makes sense. Cheers Darryl -----Original Message----- From: accessd-bounces at databaseadvisors.com [mailto:accessd-bounces at databaseadvisors.com] On Behalf Of Salakhetdinov Shamil Sent: Wednesday, 16 November 2011 7:35 PM To: Access Developers discussion and problem solving Subject: Re: [AccessD] New SQL Server license scheme is RADICALLLY more expensive Hi Darryl -- Thank you for your reply. But you didn't account Total Cost of Ownership (TCO) which includes your project development and support costs, did you? I''m not asking for concrete figures but for evaluation based on your or others past experiences. Once again: TCO - C#/.NET 4.0 (VS2010) vs. mySQL + ...(?) .... Did you try to split your MS SQL Express db into several parts? Would that splitting introduce too much complexity to your business application development? How do you process your large data volumes? .... Instead of "blindly switching to mySQL" wouldn't it be better first try to evaluate long term costs? .... Darryl, I'm not trying to insult you or others with my questions here - these are more questions to myself how inexpensive is that "mySQL free cheese" in long run? Anybody tried to calculate that? Yes, I know many companies do not have that much of free money to pay to MS SQL licences up-front but with proper development plan/road-map there should be a way to make that licences affordable for them in long run - I mean if that companies have a profitable business and your development makes this business even more profitable then MS SQL should be a better choice provided your back-end isn't just a "dumb" relational db but needs rich (T)-SQL and many other features you're getting with MS SQL "out-of-the-box"... Thank you. -- Shamil 16 ноября 2011, 02:39 от Darryl Collins <darryl at whittleconsulting.com.au>: > > Hi Shamil, > > Initially it is just the sheer volume of records. I am working with a 'block model' which is used to map open pit mines and there are many millions of records to process (about 86 million in this case). Each block has a unique X Y Z co-ord that places it in space, these 3 co-ords are basically the key for each record as should be unique. > > However in this case the client said there should be 83M in the model, but when we counted all the combined data sources we had 86 million, so we suspected there were duplicates. Loading 83 million records pushed SQL Express right to the edge of the size restriction limit, it left little room to do any actual processing work. The original data came to us in 8 separate MS Access databases, each one having around 8 million rows of data. > > The free version of MySQL doesn't have the same size limits as SQL Server Express, and with the Free GUI's (I am currently testing Toad and MySQL Workbench) it has been a fairly simple process to get started. > > Today I am hoping to get into the more complicated things - it is very early days, but hey, even If you are using the full grunt paid system on a stand alone server, it is a fraction of the cost of SQL Server these days - I would seriously consider it. I love the full version of SQL server, but I can say that, as I have never PAID for it. It has always been a company expense - someone else's proble. But even so, with this new plan it is looking bloody pricey even for those with serious pockets. > > Now. MySQL - The Process so far: > > Looked at the manual: > << http://dev.mysql.com/doc/refman/5.5/en/index.html>> > > Download the Free community edition: Easy << > http://dev.mysql.com/downloads/mysql/>> > > Install: Fairly easy > > Attempt to do 'stuff' via the text window: Welcome back to 1998 - this was kind of freaky, was expecting something a lot more, ummm, polished. > << http://dev.mysql.com/doc/refman/5.5/en/database-use.html>> > > Decided there had to be a better way: Used Google and downloaded a GUI: Much better - phew! Relief for sure. > << http://dev.mysql.com/downloads/workbench/>> > Or > << > http://www.toadworld.com/Freeware/ToadforMySQLFreeware/tabid/561/Defau > lt.aspx>> > > Download ODBC connector for MySQL: Painless << > http://dev.mysql.com/downloads/connector/odbc/3.51.html>> > > Create MySQL Database: Easy - although I want to look at more > 'options' on this Push or Pull data from MS Access to MySQL - Fairly easy - need to poke around a bit on the parameters. > > Check data types > << http://dev.mysql.com/doc/refman/5.0/en/storage-requirements.html>> > > Today I need to look at some serious data mashing. Will keep you posted. > > Cheers > Darryl. > > -----Original Message----- > From: accessd-bounces at databaseadvisors.com > [mailto:accessd-bounces at databaseadvisors.com] On Behalf Of > Salakhetdinov Shamil > Sent: Tuesday, 15 November 2011 6:41 PM > To: Access Developers discussion and problem solving > Subject: Re: [AccessD] New SQL Server license scheme is RADICALLLY > more expensive > > Hi Darryl -- > > I wonder, what raw (server side views, SPs, UDFs, triggers, ...) data processing/manipulation are you doing with your large dataset, which doesn't fit MS SQL Express? > What data processing/manipulation is done on your client side - just conceptual view of it?... > > Thank you. > > -- Shamil > > 15 ноября 2011, 06:22 от Darryl Collins <darryl at whittleconsulting.com.au>: > > Well, > > > > Been an interesting couple of days. Have a very large dataset, far too large for Access so I have been playing with it in SQL Server Express, but been hitting issues with that as well - the data just fits (just) but it leaves almost no space do perform any actions on it. So now I am trying MySQL. Been a bit of a learning curve this morning, for example, it was awful until I found a decent GUI (that needs to be downloaded separately). But so far so good, I am liking it. Got it hooked into MS Access for import / export and the rest of it seems rather doable as well. Fairly intuitive so far, which I like. > > > > Will be a change as I have used SQL server for a few years now, but Express is getting too limiting and there is no way we can justify paying for a dedicated server. > > > > See how this turns out - early days yet so I am not getting too pleased with things, but an encouraging start none the less. > > > > Cheers > > Darryl. > > > > -----Original Message----- > > From: accessd-bounces at databaseadvisors.com > > [mailto:accessd-bounces at databaseadvisors.com] On Behalf Of > > Salakhetdinov Shamil > > Sent: Tuesday, 15 November 2011 7:52 AM > > To: Access Developers discussion and problem solving > > Subject: Re: [AccessD] New SQL Server license scheme is RADICALLLY > > more expensive > > > > Thank you, Arthur, > > > > I thought you have used more powerful hardware - but if it works well for you, no problem. > > I've even a bit "weaker" system than you do (and just 3GB RAM), and I can have several VS2010 sessions, as well MS SQL, MS Word, Excel and Access etc. running simultaneously. They work rather smoothly but it's becoming as you say "confusing" to have all that stuff opened on one physical box, so I'm considering upgrading my hardware to something more modern (but not as mighty as JWC has) to run development for different customers in different VMs... > > > > -- Shamil > > > > 13 ноября 2011, 22:21 от Arthur Fuller <fuller.artful at gmail.com>: > > > My hardware is pretty modest, Shamil, especially when compared with JWC's. > > > On the OS side, I run Windows 7 Ultimate. On the hardware side, > > > the workstation is a gracefully aging HP with 4GB of RAM and a few > > > hard disks and a dual-layer DVD burner. Oh yeah, there's also a > > > USB hard disk with a mere 250 GB. That's about it. Oh, I forgot > > > the flash-memory sticks: a pair of 16 GB sticks. I'm considering > > > mounting the new version of Solaris on one of them. > > > > > > I should add that I don't run everything all at once; what's the > > > point of running Ubuntu and Mint simultaneously? The one problem > > > that I have with my setup is that I don't have a Windows XP VM. > > > Within the Ubuntu VM and also the base Windows 7 boot, I run > > > Chrome as my browser, but each instance opens pages related to the > > > particular OS. I find this approach leads to less confusion, and > > > since I'm going to become 64 in two days, I have to consider confusion as a major topic LOL. > > > > > > A. > > > > > > On Sun, Nov 13, 2011 at 12:49 PM, Salakhetdinov Shamil <mcp2004 at mail.ru>wrote: > > > > > > > Hi Arthur -- > > > > > > > > > At the moment, I'm running Windows 7 as the native boot, and > > > > > then running Oracle VirtualBox inside that, and inside that, I > > > > > run various VMs, including Ubtuntu, Mint and Windows 8 Preview CTP 3. > > > > > Everything works just ducky. > > > > > ...for now this arrangement suits me to a tee. > > > > > > > > May I know what is your hardware to run all that stuff? > > > > > > > > > > > -- > > > AccessD mailing list > > > AccessD at databaseadvisors.com > > > http://databaseadvisors.com/mailman/listinfo/accessd > > > Website: http://www.databaseadvisors.com > > > > > > > -- > > AccessD mailing list > > AccessD at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/accessd > > Website: http://www.databaseadvisors.com > > > > -- > > AccessD mailing list > > AccessD at databaseadvisors.com > > http://databaseadvisors.com/mailman/listinfo/accessd > > Website: http://www.databaseadvisors.com > > > > -- > AccessD mailing list > AccessD at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/accessd > Website: http://www.databaseadvisors.com > > -- > AccessD mailing list > AccessD at databaseadvisors.com > http://databaseadvisors.com/mailman/listinfo/accessd > Website: http://www.databaseadvisors.com > -- AccessD mailing list AccessD at databaseadvisors.com http://databaseadvisors.com/mailman/listinfo/accessd Website: http://www.databaseadvisors.com