jwcolby
jwcolby at colbyconsulting.com
Sat Oct 15 08:30:06 CDT 2011
Hans, >> To be more accurate, NoSQL is intended to be a solution for companies that are expecting rapid growth and cannot rely on vertical scaling alone in order to keep up with demands on resources. I don't see this anywhere. Point me to anywhere that any company is even thinking about NoSQL to run their *business* side of the house. Show me *anything* where *anyone* is developing book keeping or banking or manufacturing kind of databases using NoSQL. To be more accurate, NOSQL is intended to be a solution for companies expecting rapid growth in *document storage*, and needing to *search documents*. >>It's not nonsense, just because it doesn't apply to you. :) I did not say "it" (NOSQL) was nonsense, I have been saying that it it nonsense to keep trying to fit that square peg in this round hole. It is nonsense to keep telling me I need it when (as you are saying) it doesn't apply to me! I read an article by one of the founders of (I believe) Hadoop. What he said was that NOSQL was *NOT* a replacement for SQL based languages, but a solution for places where SQL databases don't fit. The things I do demand relational data. Relationships are the core of my business. My data is large, but they are not large individual chunks (paragraphs or pages or documents) but lots of records with lots of attributes. I have 600 million records in about 30 table pairs. The tables are pairs, each table related to one other with a pk/fk. One table contains name / address / hash fields and a PK. The other table has attributes about the people in that first table. I have (in 15 tables) 300 million records with first name, last name, addr1, city, state, zip, plus a handful of other fields discussing the validity of the address itself. I have to index on and pull addresses based on specific attributes of those addresses. I have indexes on and pull data about those people records based on attributes (fields) in the attribute table. As an example I have to pull those 300 million addresses out every month and run them through a third party program to track people moving. That software requires the name / address fields and hands me back those same fields plus a bunch more that discuss the validity of those addresses (is the address still valid? Is the address complete?) as well as move information. That third party program requires the data in CSV and uses Foxpro to process it. How in the world does this sound like Hadoop? I think my friend Jim just has some misconceptions about what my data is. Given how much I have discussed my "database from hell" with its 600 attribute fields I am a little puzzled how he could not understand what I do. Or perhaps he (or I) misunderstands what NoSQL is. From what I am reading, NoSQL is not about handling relational data or hundreds of tiny attributes (fields) of an object and selecting records based on those attributes. NoSQL (AFAIU) is about storing documents and allowing you to search those documents. I don't have a single field in all of my data that stores more than about 80 characters. I have tables, related to other tables, each of which may have literally hundreds of fields, each field being anywhere from one character (yes / no) to 60 characters (email address). In fact the email address is the single biggest field in all of my data. I have to select small (a few million) record sets based on "where" clauses examining those fields. I have to join the information in these 15 table pairs to select records based on commonality. How does this sound like NoSQL? Every time my friend Jim comes at me with "you need NoSQL" I spend more time trying to see what it is about NOSQL that fits my situation. I am not blithely ignoring him. I have spent hours now reading stuff about this technology, and every time I keep reading stuff by the very people who design NoSQL saying that *it is not a replacement for SQL*. These people say that NOSQL does not do SQL kind of stuff easily. These people say NOSQL is about spreading the load of searching millions of *documents* across an entire server farm. These people are saying that it tough (requires entire new languages, technology and knowledge base) to get the data split out across that server farm and to reassemble the search results *but* that the results are worth it *when* you are dealing with billions of *documents*. I am a one man show. I don't own a server farm. I don't have billions of documents and I am not going to acquire billions of documents. I am just tired of being told how my situation is going to be helped by a technology specifically and intentionally designed to handle the storage and search of *documents*, when I don't have a single document in my entire database. *THIS IS GETTING OLD!!!* I am thrilled that NoSQL exists and that it helps those that it helps. What I am *not* seeing is a single case study where they are taking an SAP process and doing it on NoSQL. Flattening 10 thousand tables in a massive SQL based data processing system and making it run on NoSQL. What I am *not* seeing is anyone claiming that in 5 (or even 30) years the SQL language will cease to exist because of NoSQL. And what I am not thrilled about is a constant "you need NOSQL" when there is never any explanation about how this very cool but not applicable technology applies to me. IOW, LMTFA!!! John W. Colby Colby Consulting Reality is what refuses to go away when you do not believe in it On 10/15/2011 2:47 AM, Hans-Christian Andersen wrote: > > > To be more accurate, NoSQL is intended to be a solution for companies that are expecting rapid growth and cannot rely on vertical scaling alone in order to keep up with demands on resources. > > Trust me, John, if you are in a situation like this, no amount of memory, cpu or compression can save you. It's not nonsense, just because it doesn't apply to you. :) > > - Hans > > > > On 2011-10-14, at 3:37 PM, jwcolby wrote: > >> LOL. And maybe you will eventually discover that NoSQL is targeted at people with a a million blade servers / million dollars in a data center and will quit haranguing me? ;) >> >> I see NO NOSQL in my future. >> >> I have already solved my issues the same way that these guys did. Jillions of cores and 64 gigabytes of RAM, and compression. My hour long processes are under a minute or two. >> >> It would be interesting to actually show you what I do Jim. You would instantly quit this nonsense. >> >> John W. Colby