[dba-Tech] VoltDB and its treatment of RAM on multiple servers

Tue Feb 20 06:26:45 CST 2018

Jim,

I may be way off base here, but I think that an implementation in the cloud
might defeat the whole purpose of the VoltDB design, which is performance.
>From what I gather, it's designed to handle massive amounts of input, such
as a Bloomberg feed or a worldwide gaming site or tracking the votes in an
election, polling site by polling site. My thought (and I could be wrong
about this, since I know little about cloud technologies) is that there
would be a performance hit due to talking to the cloud. SQL Server is
designed for much smaller loads than VoltDB. The latter can easily handle
50k TPS, and upwards.

The examples given in the VoltDB documentation imply a level of hardware
vastly beyond my feeble resources, such as a dozen servers in a cluster,
each housing 64 GB of RAM.

On the other hand, there is a free community edition which could be run on
a single server. There, the point would be to load the entire database
(smallish, say 50 GB) into RAM, with a few hundred simultaneous users.
Periodically and automatically, the RAM database updates the hard disk(s).

I certainly don't pretend to understand a lot of the VoltDB implementation.
Many of its concepts I've never encountered before. Lots of the concepts
assume a collection of servers, with partitioned tables and even
partitioned indexes, and smallish tables  (i.e. lookups) duplicated on each
server, and a lot more that is beyond my meager experience.

Thanks for the links. As time permits, I will follow up, and further my
education.

Arthur

On Tue, Feb 20, 2018 at 1:29 AM, Jim Lawrence <accessd at shaw.ca> wrote:

> Arthur:
>
> PS I could suggest that you check out the Framework Apache Flink:
>
> https://en.wikipedia.org/wiki/Apache_Flink
>
> It is basically a structure that allow processing (CPU core and RAM use)
> across a network or cluster...ideal for Cloud computing.
>
> Jim
>
> ----- Original Message -----
> From: "accessd" <accessd at shaw.ca>
> To: "Discussion of Hardware and Software issues" <
> dba-tech at databaseadvisors.com>
> Sent: Monday, February 19, 2018 9:37:20 PM
> Subject: Re: [dba-Tech] VoltDB and its treatment of RAM on multiple servers
>
> You have to pick one of then most complex subjects on computers these
> days. There are a few applications that are free and OS for your Linux
> computer. One is MPI...a very old application:
>
> To do this the program(s) accessing the CPU/RAM resources must be
> specifically designed to access said resources. A system set up in this
> manner is called a cluster, and the typical way resources are shared is
> with a protocol called MPI (message passing interface). It is a free
> download and using it with Linux can yield a powerful cluster (possibly
> even a super computer) for minimal cost, but again it is useless unless you
> have programs that were specifically designed to take advantage of MPI.
> There are some good cluster tutorials out there, if you are still
> interested you should check one out.
>
> http://www.mcs.anl.gov/research/projects/mpi/
> ...and...
> http://www.mpich.org/
> ...and...
> http://www.mpich.org/static/downloads/3.2.1/mpich-3.2.1-installguide.pdf
> http://bit.ly/2FaMpA4
>
> Then there is PVM3 or Parallel Virtual Machine, version 3. Here is where
> to download the package:
>
> http://www.tucows.com/preview/39014/PVM
>
> Then there is applications under full development like (Very interesting):
>
> https://ramcloud.atlassian.net/wiki/spaces/RAM/overview
>
> I have tried some of these products that allow the file system to extend
> to all RAM and/or hard drive space but have consistently run into issues
> that I could not find a solution for. It all boiled down to older and
> limited computers being added to a cluster. It became an issue of huge
> backups and restoring which took days... OTOH, I learned how to effectively
> do backups properly and now have twice as much hardware dedicated to
> backing up the network as in the network. Of course if someone wanted to
> give me $50K and let me have a free month... ;-)
>
> If you wanted to run an application that requires a substantial amount of
> RAM, it would be better to use a Cloud based source that has all that
> functionality built right in. I would recommend that you use the services
> of DigitalOcean as there is no fixed amount of resources...you pay for what
> you use and for the amount of time you use it. I know of some companies
> doing testing of their application on DO and only do testing for an hour or
> two. Example: 192GB RAM, 32CPUs and 12TB of SSD (plus high-speed bandwidth)
> is $1.43 an hour...then there is optimized Droplets with compute optimized
> virtual machines with dedicated hyper-threads from best in class Intel CPUs
> for CPU Intensive applications like CI/CD, video encoding, machine
> learning, ad serving, batch processing and active front-end web
> servers...less RAM and HD space but only 0.95 per hour. I know of no other
> Cloud provider that gives that level of graduation.
>
> If VoltDB was written in one of the distributive optimized languages then
> it could use multi-cores across a network:
>
> https://en.wikipedia.org/wiki/List_of_concurrent_and_
> parallel_programming_languages
> http://bit.ly/2EDazSK
>
> The old favourites like Erlang and the modern GO are two such programming
> languages. I do think that Gustav and Shamil might have some more insights
> into this environment as well.
>
> * I do have another possible solution but will send you the information
> off line.
>
> Jim
>
>
> ----- Original Message -----
> From: "Arthur Fuller" gmail.com>
> To: "Discussion of Hardware and Software issues" databaseadvisors.com>
> Sent: Saturday, February 17, 2018 8:48:37 AM
> Subject: [dba-Tech] VoltDB and its treatment of RAM on multiple servers
>
> VoltDB, the (relatively) new database from Michael Stonebraker, intended
> for in-RAM databases needing to perform 50K+ transactions per second, has
> some way of treating a bunch of servers as a logical unit. IOW, 8 servers
> each equipped with 64GB of RAM can be regarded as a single 512GB unit of
> RAM. I understand how this approach can work with hard disks, but I have no
> clue how to make it work with RAM. Do you?
>
> The documentation and even packaging of VoltDB suggests that its ideal
> hardware environment is Linux (since the Windows version is installed and
> run using Docker). I don't know enough about Linux to know whether this
> "union" of servers' RAM is built-in or an innovation provided within
> VoltDB. (Speak up, Jim and other Linux gurus!)
>
> Whatever the answer, I can't see this working with the typical mix of
> Windows boxes most of us, other than the thick wallets, being able to
> cobble this together, especially since those of us with multiple boxes
> typically run different versions on different boxes, so we can support
> clients running various versions of Windows, Access, SQL, Office, etc. Add
> to that the problem that on Windows VoltDB runs only in Docker. So I think
> this is a strictly-Linux solution.
>
> Assuming that I'm correct (Linux-only), I'm still faced with the problem of
> how to implement it. Assume modest hardware, say three boxes of varying
> vintage each running the same version of Linux, and all connected. Further
> assume that as yet, VoltDB is not installed. Is it possible to treat their
> collective RAM as a single logical unit of RAM? If so, how so?
>
> --
> Arthur
> _______________________________________________
> dba-Tech mailing list
> dba-Tech at databaseadvisors.com
> http://databaseadvisors.com/mailman/listinfo/dba-tech
> Website: http://www.databaseadvisors.com
>
> _______________________________________________
> dba-Tech mailing list
> dba-Tech at databaseadvisors.com
> http://databaseadvisors.com/mailman/listinfo/dba-tech
> Website: http://www.databaseadvisors.com
>
> _______________________________________________
> dba-Tech mailing list
> dba-Tech at databaseadvisors.com
> http://databaseadvisors.com/mailman/listinfo/dba-tech
> Website: http://www.databaseadvisors.com
>

-- 
Arthur