Search
RubyConf 2012
Allow me to reintroduce myself, my name is MagLev
Edit
#maglev
#ruby-implementation
#data-persistence
#smalltalk
#object-relational-mapping-orm
#database-transactions
Summarized using AI

Allow me to reintroduce myself, my name is MagLev

Jeese Cooke • November 01, 2012 • Denver, Colorado • Talk

In this presentation, Jesse Cook delves into the capabilities of MagLev, a Ruby implementation that runs on a Smalltalk VM, contrasting it with typical Ruby applications that rely on relational databases. He recounts how Avi Bryant's insights in 2007 regarding Ruby's future paved the way for MagLev and its inherent advantages, particularly its transparent object persistence, which fundamentally simplifies data management in Ruby applications.

Key Points:

  • Introduction to MagLev: MagLev is a Ruby implementation akin to MRI, JRuby, etc., but built on the Smalltalk VM, specifically using the GemStone/S object database.
  • Features of GemStone/S: With nearly three decades in operation, it offers robust object persistence functionalities that ease the burden of data management compared to traditional relational models.
  • Object Persistence Model: MagLev allows seamless creation and management of Ruby objects that withstand lifecycle changes, facilitating persistence without conventional database interactions.
  • Impedance Mismatch Issue: Cook explains the problem of traditional relational models where Ruby objects must be serialized into database rows and vice versa, making data handling cumbersome. MagLev solves this by maintaining object relations and state transparently.
  • Examples and Mechanisms:
    • Persistent Root Functionality: Demonstrates how objects can be persisted within a persistent root, with automatic handling of object states at the end of request cycles, making it unnecessary to manage object storage explicitly.
    • Transaction Management: Transactions in MagLev are simplified, allowing developers to begin, commit, or abort them without heavy procedural overhead.
    • Worker Queue and Leaderboard Functionality: Showcases how to implement features like a worker queue with minimal code, emphasizing the ease of persisting and retrieving objects compared to other tools like Redis, while also illustrating the advantages of utilizing Ruby's built-in data structures.
  • Distributed Nature of MagLev: Cook discusses how MagLev's architecture supports distributed systems, where objects can be accessed across multiple VMs, enhancing scalability and availability.
  • Conclusion and Invitation for Collaboration: The talk concludes with an invitation to explore the distributed functionality on a larger scale and an encouragement for the audience to engage with the MagLev community.

Takeaways:

  • MagLev represents a significant shift in how Ruby applications can manage data, offering a more intuitive, object-oriented approach to persistence.
  • The potential for creativity and efficiency in developing Ruby applications is elevated through the capabilities of MagLev, encouraging wider adoption and experimentation among developers.
  • Understanding the integration of historical practices from Smalltalk can inform better strategies in modern Ruby applications.

Allow me to reintroduce myself, my name is MagLev
Jeese Cooke • Denver, Colorado • Talk

Date: November 01, 2012
Published: March 19, 2013
Announced: unknown

Remember when Avi Bryant spoke at RailsConf '07 about Ruby's future, and how it'd be slick if we could get to where Smalltalk was 30 years ago? Well, we kinda have, in some respects, with the 1.0 release of MagLev, a Ruby implementation running on a Smalltalk VM.

What makes MagLev really special? How about a baked-in object database that sits on top of a VM almost 30 years in the making? There are a lot of cool things you can do with this type of persistence, but it's quite different from what we do in a typical Ruby app.

Let's explore what this great implementation has to offer, and learn how to help it along.

RubyConf 2012

00:00:15.960 um so hello good morning I'm Jesse Cook I'm working with Thunderbolt labs and uh I'm here to talk about
00:00:22.240 magav uh I'm a little nervous so if I throw up like that guy just take
00:00:28.960 pictures and tweet to Ruby friends um that'll make me feel a lot better in point and laugh
00:00:35.239 also uh so there's this guy Obby Bryant and back in 2006 before or sorry 2007
00:00:41.879 before Wales conf he wrote this really cool uh article called Ruby and other gems and in it he talked about this uh
00:00:50.840 Small Talk product called gemstone which I had never heard of um and he likened
00:00:56.239 it to rails what he said was that rails is this framework that does all this kind of cool persistence stuff but there
00:01:03.320 are these other products out there um specifically gemstone that do a lot of the stuff for
00:01:09.280 you um of course my slide is kind of cut off so as he he compared basically the
00:01:15.439 storage engine um the memory cache and the worker processes um and likened how we use you
00:01:22.479 know a relational database to store objects um we might use like memcache back then or redus nowadays to do some
00:01:29.240 type of uh um uh shared memory um caching and then
00:01:34.520 we have our actual app servers and uh he thought that it would
00:01:40.119 be really interesting to try to get this working in Ruby um and so uh he ends his
00:01:47.439 art his little blog post saying so there you have it gemstone it's like rails but faster and easier if only it ran Ruby
00:01:55.360 and this sparked a really interesting conversation uh that ended up with this product called maglev which I'm here to
00:02:01.399 talk about today so what is maglev maglev is a ruby
00:02:06.439 implementation um just like there's MRI rubinius J Ruby uh Mac Ruby but it is
00:02:12.800 built on top of a small talk product called uh gemstone s and uh gemstone the company has been
00:02:20.400 around for 30 some years now gemstone s has been around for 20 25 almost 30
00:02:27.280 years uh it's a very solid product and and it offers some really interesting features that uh make um our lives as
00:02:36.200 rubious a lot easier in terms of uh persisting our objects and there are some other really cool things inside of
00:02:42.400 maglev itself that uh I hope to talk about today uh so gemstone s the product is uh
00:02:50.239 basically a small talk VM that has a built-in object database um and it's a
00:02:57.640 uh used in in Mission critical applications all across the world uh UBS
00:03:03.640 uses it um Jack in the Box uses it which I think is Mission critical
00:03:10.040 um o a shipping container company used it to manage all of their uh shipping
00:03:16.280 containers millions of shipping containers worldwide um but the the reason why I
00:03:23.440 think it's so special uh is it has this thing built in called transparent object persistence and that's the object datab
00:03:29.840 B that's um available via the small talk uh
00:03:35.040 product so there's this great quote um by Abdi Grim he was uh one of the Rogues
00:03:42.120 and he said I've often wish that I could just do something or I just change a bunch of models and at the end of the request cycle something magical happen
00:03:49.560 some something magical goes through and collects up all the models that I have changed and persist them I haven't
00:03:54.840 really gotten to that level yet sometimes I feel like active record is holding me back from that of course he
00:04:00.040 talking about a typical rails app this was on a ruby Rogues episode with David larby talking about domain driven design
00:04:08.079 and uh David chimed in with a punchline well you can have that you just have to
00:04:13.120 switch to Java or.net and I was driving and that made me really sad um because
00:04:19.639 that's not actually true we've um we've have this awesome mag product but it
00:04:25.560 doesn't quite have a lot of knowledge or adoption yet so um the other interesting thing is right
00:04:32.600 after I heard that as I was driving up to Seattle I uh was passed by an O truck
00:04:39.759 um so they are all over the place it's uh pretty neat so gemstones everywhere
00:04:45.479 they they uh it's a a great product so um sorry the p is cut off um
00:04:56.639 the thing that we deal with uh on a day-to-day basis is basically that um persisting objects is
00:05:02.680 hard um I think that uh when we comes to the relational model um there's uh and
00:05:09.840 I'll talk about a little bit impedance mismatch between what we have in our uh
00:05:15.199 Ruby application and then how we store our data and the um you know relational databases do a very good job of this um
00:05:23.240 you know people use the document databases people use redis all sorts of uh um cool products but
00:05:30.639 um to an extent it starts to fall apart um you have to split up your data
00:05:37.039 you have to have parts of your data stored over here and we all we all know this right we all build applications we use postgress or MySQL or whatever um
00:05:44.919 but with magv you don't have to do that because your objects um as you design them in your Ruby application can be
00:05:51.520 persisted and used anywhere um so persistence in O uh going
00:05:58.759 back to small talk talk was supposed to be simple um it basically was something you didn't have to think about you could
00:06:05.400 create objects in your running program and then uh you could close your program
00:06:10.440 and you could open your program back up and your objects would still be there um it seemed like a really nice world to
00:06:16.319 live in um and so you know Small Talk has that and that seems to be fantastic and
00:06:23.280 it's it's uh enabled by this thing called the image and that's basically how I think of this kind of like Fusion
00:06:29.960 plasma ball where you have all your objects floating around um and they can all their relations and everything is
00:06:36.000 just connected so uh currently you know as I
00:06:41.560 said we store objects as rows um in databases uh you know relational uh
00:06:47.639 maybe we store them as documents in something like or react um as
00:06:52.680 they're Marshal to some other data format or even as strings in in reddis
00:06:59.160 um which is kind of a um a little a little bit of a shame because we're ripping apart our objects um and we're
00:07:05.400 ripping apart the relations um and I think from a
00:07:10.639 um holistic point of view that's kind of a mean thing to do to your objects um
00:07:16.080 but we have to for most most of the things we want to get done um but the other thing that does is
00:07:22.319 there's kind of a a a shift in the way we have to think about um retrieving our
00:07:28.120 objects even so you know we have the the typical fine by name finder and rails um
00:07:35.680 or we uh the user creat is a is a bit more simple but you know this is basically uh the API that we use to talk
00:07:42.680 to our database but it gets thrown down into some query language that's
00:07:47.720 transferred across the wire via strings um which is a pretty primitive um way to
00:07:54.479 uh talk about this Rich world that we live in with objects um and the other thing that
00:08:00.360 happens with these other storage systems is that they start leaking into our code
00:08:06.039 and so you can think of SQL as being like a The Leaky abstraction with object-oriented programming where you
00:08:12.159 have finders um that are you know need valid SQL in order to um find the things
00:08:21.000 you that you need to find so uh one of the the terms that really bugs
00:08:28.240 me is poo plain old Ruby object um and that's one of the things that we kind of tend not to deal with too much um uh in
00:08:36.159 in like a typical rail app let's say because um what developers usually think about
00:08:42.719 is the the storage mechanism um and so you know it's not that I have a post and
00:08:48.720 post is just a normal class it's that a post is an active record object uh and
00:08:54.760 in order to um solve a bunch of problem or to um avoid a bunch of problems you have to generally think about um uh the
00:09:03.800 object as an instance of an active record object not as just a plain old Ruby
00:09:10.240 object um so this kind of uh speaks to the object relational impedance mismatch
00:09:16.440 which is this fancy term for um you know you've got your objects over here and
00:09:22.200 then you have the way to store them uh over over there and over there could be who knows where right like it's kind of
00:09:29.120 nice it's a bit of a black box but it's only a black box to an extent there's um you know you still have to to to know
00:09:36.880 enough about it and there's really um there's some great articles you know the the Cunningham and Cunningham the C2
00:09:43.079 Wiki is uh fantastic resource to read more about this
00:09:49.880 um so getting back to AI um back in 2007 he gave the keynote at
00:09:56.600 rails and uh he was kind of talking about this gemstone product a little bit and uh he said you know the future of
00:10:03.399 Ruby is here it's just that we have it over in the small talk world and you guys should come and get it and he was
00:10:08.440 mainly talking about um having a fast VM and having um a nice persistence model
00:10:15.000 that you essentially don't really have to think about um and so we have that with magl
00:10:21.920 um so I'll kind of start with a a simple example to show you um what it's like to
00:10:28.279 work with maglev um so first um let me kill this for a second so maglev is a um a ruby
00:10:35.240 implementation right now it's uh 187 that's targeting 187 um there's a team
00:10:41.519 working um in Germany to get it up to uh 19 193 and then hopefully Ruby 2.0 soon
00:10:49.680 um but it's um it's basically Ruby except it's got what you could think of as an object database built into it so
00:10:58.160 I'll go through some examples to hopefully get you little familiar with um what it's like to work with magl so
00:11:05.240 the first thing is um there's this there's this notion
00:11:12.279 of a persistent route and um make sure I don't get too close so the persistent
00:11:18.680 route is uh where you want to stick anything that you want to persist um and
00:11:24.120 so what I mean by that is let's say I've got a class um and I create
00:11:29.800 an instance of the class uh my C Pier now I go through and I kind of add
00:11:35.760 the precis the Precision route is essentially kind of like a hash um or actually it is a hash um and so when I
00:11:42.880 add Pierre um into the hash and then I run this thing uh commit
00:11:50.160 transaction um my the the instance is just basically magically saved right so there's a bunch of stuff that happens on
00:11:56.360 the back end um the VM uh um stores it in memory but then it also
00:12:02.480 takes a um a representation the the same representation that's in memory and it
00:12:07.519 writes it to disk um and then this is available in any other instance of uh
00:12:13.760 magb that you boot up so let me let me say that again so I can
00:12:19.399 persist uh an object in one running VM and then I can pull it up and have that
00:12:25.079 in the same instance of the object in another VM I don't have to go out to a database um it's just
00:12:33.720 there um and so you can see that here uh basically you know I this little example
00:12:40.440 is me just showing you that uh um pulling up another VM oh and by the way
00:12:45.480 I call it P root because having persistent roote everywhere is kind of long
00:12:51.560 um so it's a very simple mechanism and that's what's really nice about it um it's very easy to understand there's no
00:12:57.959 save um um there's there's not necessarily any validations though there could be um it's just this very simple
00:13:04.360 notion of transactions so uh the magav VM the the
00:13:09.560 gemstone VM basically has two modes it has the default transient mode and um a
00:13:15.760 persistent mode and so the transient mode basically uh being the default
00:13:22.079 means that when you start running it any instance that you um create won't necess
00:13:28.320 it won't be NE necessarily saved um to the stone which is kind of the the
00:13:34.279 database um that everything shares um and so what I mean by that is you can
00:13:40.720 still um have things kind of local to the VM but there's still um it it won't
00:13:46.880 be saved off and then available to other VMS that connect so you can run in a persistent
00:13:53.759 mode and the persistent mode actually um um allow s you to uh essentially save
00:14:02.120 like whatever um it they both start what's like a transaction so maglab is
00:14:07.519 transaction based um but uh the persistent mode basically wraps your
00:14:12.880 whole program in this uh persistent block and uh anything that's um um
00:14:20.360 required any classes that are defined are then uh um you're able to persist
00:14:26.880 them uh so transactions right so uh transactions are are pretty simple to
00:14:33.600 understand you begin a transaction you uh might abort a transaction if you
00:14:39.800 don't want the stuff anymore and then you commit a transaction in kind of a simple way to
00:14:45.880 uh think about persistence um so when you start up a maglev VM there's an
00:14:52.079 implicit uh begin transaction um uh if you want to throw things away
00:14:59.440 and start over you can abort a transaction uh and what that does both begin transaction and abort transactions
00:15:06.800 kind of grab a fresh view which I'll explain uh in a second
00:15:12.440 um so maglev works by having this gemstone VM um kind of running wherever
00:15:20.240 doesn't doesn't necessarily matter where it's running um and then in the gemstone
00:15:26.240 parlance but we'll call it they they call them gem which is a running instance um which is the the little
00:15:34.120 blocks at the bottom um so what happens is when you abort a transaction um the running gemstone or
00:15:42.160 sorry the running uh Ruby program essentially grabs a fresh copy from the stone so anything that has changed in
00:15:49.680 any other VM is then available locally to your VM to your uh gem so you get
00:15:56.519 essentially like a snapshot of the database and uh on the flip side when you commit
00:16:03.199 the transaction um things are basically pushed back up to the stone um so it's a
00:16:09.639 it's a you know there's there's um it's kind of like you know active record save
00:16:15.519 to an extent um but it happens across anything that's changed in your in your
00:16:21.279 program since the last time you aborted so one thing that uh gem has
00:16:29.399 that's really powerful is what's called persistence by reachability and uh as the example kind
00:16:36.600 of shows which I'll explain um is that basically you don't have to stick everything into that persistent route um
00:16:43.759 anything that is inside the persistent roote anything that is attached to something that's inside the persistent
00:16:49.560 roote can be persisted um so the example here is that I have an array um oh and
00:16:56.040 by the way all like the like normal classes like string um arrays hashes those things are all
00:17:02.000 persistable kind of by default you don't have to do anything special with those um so I have you know an array and I put
00:17:09.120 an instance of a cat in it and I'm actually persisting the array I'm not persisting um the the instance of the
00:17:16.959 cat itself and when I start up another VM um
00:17:22.319 I can pull that out and it's the same object so notice I'm actually comparing the instance of the cat that's in the
00:17:28.520 array not the two arrays so that's essentially persistence by
00:17:35.240 reachability and it goes all the way down the graph right so if my cat Pierre
00:17:41.000 has uh an array or whatever of toys um and I append um a ball of yarn to it
00:17:48.919 when I go to get that back out later it is the same object so it's they're the
00:17:54.919 same objects all the way down the graph
00:18:03.679 so what are things we can easily persist today uh with you know active record or any type of om we've got
00:18:10.880 arrays uh lists essentially we have um sets which is
00:18:16.559 basically an array with some type of um constraint on it um hashes either with
00:18:22.640 like a redus or um you know or H store whatever uh and then count ERS
00:18:29.360 which get a little tricky because you you know might have to do some locking if you're using a relational or um you
00:18:35.000 know use something like redus but these are all things that we uh have in Ruby yet we have to translate out into our um
00:18:44.400 uh data store and also sorted sets um which is you know something that I've
00:18:49.679 used in Rea several times and it's a it's a wonderful feature um but the common theme among
00:18:55.840 all of these is uh I have to uh manage serialization
00:19:01.200 somehow um it's more expensive than dealing with just the objects themselves
00:19:07.400 because I have to decompose the object um put it in some type of form that I
00:19:12.880 send across to something else um or get back from something else and then build up the object
00:19:19.280 again um and I may want to uh use some other type of data structure that is
00:19:26.080 more exotic than um you know an array way so something like a KD tree if
00:19:31.240 you're doing some type of mapping program yeah guys KD trees um or a bloom
00:19:39.880 filter um or a Judy AR Ray or something that Google told me was a leftist tree
00:19:49.320 um um so the the point that uh I'm trying to make with this is that going
00:19:54.720 back to kind of the mismatch is that it's not always just about our the objects themselves but it's about the
00:20:01.320 relations um and it's the it's the graph that we build up like that's the that's
00:20:06.559 the fun part of uh uh programming that's the part that you know um gets me going
00:20:12.720 and that you know you kind of design these systems and you you uh design the way that components interact with each
00:20:19.520 other um so closure right closure is kind of like the new hotness and and a
00:20:25.000 lot of those guys tend to not necessarily agree with a lot of the ways that we do things over in the O World um
00:20:31.799 but there was this uh really interesting talk that um I'm blanking on his name and my notes aren't working um that one
00:20:39.039 of the guys at relevance did about uh closure um called the impedance mismatch is our
00:20:44.840 fault um and uh the one the one nice thing he had to say about oo was
00:20:52.080 essentially transactions the transactions are awesome and so if you think about the way you normally do um
00:20:58.799 you know your rails app or your U um you know your your ruis script whatever your
00:21:05.280 Sinatra app um you're not really using transactions like it's a very uh simple
00:21:11.360 way to think about um uh persisting data and so I thought that this was a really
00:21:17.720 interesting quote so the only nice thing I'll say about databases as they exist today or about Joba or other o languages
00:21:24.320 the only nice thing I'll say is about transactions transactions are awesome transactions are composable we can
00:21:31.120 understand how to reason about them which is very important and they're part of what give databases their greatness so I've got no qualm with transactions
00:21:38.640 and I thought this was really interesting because I talked to some of the implementers of maglev um and uh I
00:21:45.279 was talking with them about um datomic the new kind of closure uh database F
00:21:52.080 they use like facts and kind of stores every like the state um and gemstone has
00:21:58.679 basically done that for a really long time like you may not be able to go back and get all of the history um at each
00:22:05.640 commit but it's basically uh something that they you know did 20 30 years ago
00:22:13.120 um that's now making its way out to uh other languages which is which is great
00:22:18.200 um but I would like to see people kind of also use that in Ruby um so uh we'll run through another
00:22:26.520 quick example um there's kind of the typical blog example um in in the rails
00:22:33.440 world and um what was really fun about this it's just a simple Sinatra app but
00:22:38.840 I was just dealing with classes like I didn't have any migrations I didn't have to set up like SQL light or postgress um
00:22:47.120 there were no dependencies um so there were no drivers um as long as like magl installs like
00:22:54.400 you will be able to persist your objects um which can be really fun to um to use
00:23:00.640 so uh yeah so this very simple um Sinatra app you know I basically load up
00:23:06.320 posts and blogs um if I want to get a post I just use an actual detect right like I'm not
00:23:14.760 I'm not doing like a fine by or using find ID or anything like I'm not having to necessarily worry about um um you
00:23:22.799 know the representation out in the other world like I just want to find just like I would in any other like typical Ruby
00:23:31.840 program uh on the create side um it's very simple right like I just create and
00:23:37.720 then I append the post to the the uh list of blog posts um do some redirect
00:23:45.520 and like that's essentially it but what enables this is kind of uh
00:23:50.840 going back to what obdi um had asked for which was something that kind of
00:23:56.000 magically collected up all the changes and and uh persisted them and so if we
00:24:02.240 uh this is something that um uh Peter on the maglev team had kind of whipped up
00:24:07.760 it's a basic rack middleware um and so if you think about the request response life cycle like you
00:24:14.840 can uh abort on the way in and then um you want to figure out if you uh at the
00:24:22.120 end with this kind of commit if cool um method basically on the status like you
00:24:28.039 know we want to figure out how we're going to sorry if we're going to uh
00:24:33.200 commit the changes that have happened in this request and the commit of cool basically
00:24:38.840 says well if the status is like something that's like relatively successful uh then commit otherwise
00:24:45.720 abort and like that's that's it right any changes that have been made are persisted and uh any other VM that
00:24:52.440 connects will see that and uh it's a very simple Paradigm um and then I'll kind of whip
00:24:58.799 through these they're not really that important but basically my um classes were uh just normal Ruby classes right
00:25:06.159 like the um initializing the the blog itself like I'm just dealing with normal arrays
00:25:13.720 um the post you know um I've got like
00:25:18.799 just it's all just normal just normal Ruby um the authors just a normal class
00:25:24.960 um and uh it's really easy to kind of get going with with it right so um if I
00:25:30.480 have like what I call a bootstrap script I can uh load the classes inside
00:25:37.960 a persistent block and um I can set things up and then I basically just
00:25:45.200 commit so there's this kind of dasm commit thing um which is the same thing as saying magb commit at the end of the
00:25:51.159 script and that's it like objects are created and they will stick around until
00:25:56.799 you delete them um and after I gave so I set up I kind of did this example um for
00:26:05.000 Cascadia Ruby and I whipped this thing up like in the morning I thought it it'd be a nice little demo I made the objects
00:26:11.720 I wanted to make sure it worked right like I don't know a couple weeks ago like I was kind of poking around in
00:26:17.960 my in my uh magav looking around the persistent route and like there were my
00:26:24.039 there was my blog and there was like the post that I the example post I put in there um and what what kind of struck me
00:26:32.679 was that they were they were the objects themselves like they were the instances I didn't have to like you know my
00:26:38.320 program contained them I didn't have to go out and get them um so this interesting little um realization that
00:26:46.159 that was a very it was very pleasant it was um kind of a nice way to interact with my
00:26:52.320 program um so another example that I've uh talked about before is uh something
00:26:57.720 simple like a worker queue um so we have these things right we've got sidekick and rescue and and
00:27:03.799 those are like absolutely great libraries um but you can write something
00:27:08.840 very similar in like 35ish lines of code um so if you think about um what these
00:27:16.240 worker cues do is basically you want to put something like a proc in right you want to be able to to um run like a call
00:27:24.039 method or something and the work will get done
00:27:29.159 um so uh an example that you could use in um something like maglev is that
00:27:35.760 basically you can just you know keep on adding jobs right like let's just add jobs forever and then we'll commit and
00:27:42.679 they'll be available to be pulled off this array at some point but now the interesting thing to see here is that
00:27:49.360 I'm persisting a proc so procs aren't per you can't technically you can
00:27:55.799 Marshall procs um in other implement ations but uh they the implementers
00:28:01.200 don't recommend you doing it um so this is an interesting example because in
00:28:06.799 maglev you can persist any object like you can persist a proc if you want uh
00:28:13.080 you can persist threads through continuations um which people do uh to
00:28:18.720 debug things um in small talk and there's some examples in Ruby world but
00:28:24.600 um they use continuations and I don't understand those things
00:28:30.320 um so on the worker side uh you know we've got basically this Loop that
00:28:36.919 aborts a transaction so it grabs a fresh view of um the
00:28:42.200 repository it pops off um some unit of work to do it immediately commits so
00:28:49.840 that uh that instance that worker has that job and nobody else has that job
00:28:55.799 and then it calls it so um one thing I haven't talked about
00:29:02.600 is like a commit failure so what happens if two VMS try to pop off the same
00:29:08.200 thing um so magav has a notion of a a a failed commit
00:29:13.840 exception um but the retry logic is really simple like it's for this is just
00:29:20.440 redo like there's there's no there's there's no other cue to manage um it's
00:29:26.600 you know just a oneline and like you're kind of good to
00:29:31.679 go um so on the worker side you know we may have a a failed transaction
00:29:38.519 because um oh sorry this was what was this one
00:29:44.240 oh on the producer side so maybe two VMS tried to uh lock that Q array at the
00:29:49.840 same time um or sorry append to that Q array at like exactly the same time um
00:29:58.120 and then on the worker side you know same thing with popping it off so the the the code is up there U
00:30:04.760 but the point of this exercise was that it was like 34 line it might be a lineer
00:30:11.679 more or less now um but it was like 34 lines of of Ruby code just Ruby code um
00:30:17.120 no tests of course but it's pretty simple to understand and so I was like okay well like what is that you know I
00:30:23.600 never used slow count before and I was like I just want to know how many lines of code it is and I don't want to have to count them
00:30:28.919 and then he came out with this dollar amount I was like well first of all 56 Grand a year seems a little low um so
00:30:35.600 let's let's assume we get we get paid all right and we double that um so this you know 776 bucks let's say it's like
00:30:41.960 two grand right like two grand to implement a a a distributed worker Q
00:30:47.880 system in 35ish lines of Ruby um that's a it's a very powerful mechanism to have
00:30:54.880 so it shows that we can do very um what would be complex things very simply because we don't have to worry about our
00:31:01.519 persistence model um so uh one of the other examples
00:31:07.720 that I want to share is a leaderboard um it's one of the things that reddis is uh supposedly great great with and um I
00:31:15.639 have some friends that I've uh um I'll be using as examples um but uh um
00:31:22.399 essentially you know let's say the the competition is the um you know we want to get to 100 points
00:31:28.480 points so I've got some users in my system and you know me and Sprout are at
00:31:34.679 99 and Shawn is at 98 points um and the
00:31:40.840 approaches that we can take to persist this on a a you know normal like um in any other Ruby implementation
00:31:48.000 would be to use something like redus um so the the sorted set to come
00:31:53.080 up with a way to use a relational database um or come up with some way use
00:31:58.320 like Ruby's sorted set which um contrary to popular belief as I was flying here
00:32:04.120 um I was watching a presentation he was like yeah sorted sets they're great Ruby doesn't have that it's not true Ruby
00:32:09.720 does have a sorted set you require set you'll get the sorted set like we have these data structures so we don't
00:32:15.480 necessarily have to go out to these um other uh systems to use them so it's
00:32:21.559 like okay for this instance like let's use let's use redus uh this the sorted set seem like a great tool to use for
00:32:28.919 this application uh but the problem I ran into was that um two people could have a
00:32:36.039 score of 100 but that doesn't mean that they both won like I had another um um
00:32:42.279 how do how do you figure out if you know who won only one person can win this competition um how do I know if it was
00:32:49.399 me or how do I know if it was Sprout um and so the idea um in a normal so kind
00:32:56.000 of how you use that with a the spaceship operator um would just be to Simply
00:33:01.159 compare their scores right um and this kind of you know gives us a nice simple
00:33:07.399 way to compare to uh leaderboard items U but a better winner criteria is
00:33:15.799 the first person to 100 points um and I uh um actually contacted Peter that
00:33:23.679 works on redus and I was like Hey like is there a way to um um see who got to a
00:33:30.120 certain number of points first and unfortunately there wasn't um which kind of put a big damper on this new somewhat
00:33:38.480 simple criteria um because in Ruby it's like
00:33:44.200 just you know one more um kind of set of of um comparisons right so first I see
00:33:52.159 like two people are similar uh or sorry first we compare on score and then if
00:33:57.840 the scores are the same then we compare on the time stamp of the
00:34:03.799 score um and so using reddis um you know the the advantages were that it was
00:34:09.599 always sorted on insert which was great um and reddis is super fast um but the
00:34:17.000 the cons were that it only sorted on one score criteria which was essentially
00:34:22.960 like useless for for this application when a new requirement came in
00:34:28.320 um and the reddish driver always returns an array um it'd actually be kind of nice for sort of for sets if it did
00:34:34.800 return an actual set but it doesn't um so you know we could have
00:34:39.839 done this with a relational database um so one nice thing is that it can sort over any field um so we could have
00:34:47.480 gotten the score and the time stamp um but the and and it's relatively fast you
00:34:53.639 know the my ssql and postgress are pretty quick but um the the downsides are that it always
00:35:00.880 sorts at query time um so you are constantly figuring this thing out every
00:35:05.960 time you request this you're you're figuring out uh what the what the the ranking
00:35:13.119 is and again it returns this um I could make it return like a sorted set but the
00:35:19.040 driver itself just returns a list um which is not necessarily what I want to deal with so the nice thing about using the
00:35:26.160 sorted set in Ruby was that it was a richer data structure like it actually made sense to to use this thing we
00:35:32.160 already have um it was composed of Ruby objects like instances of the
00:35:38.119 leaderboard item class which was which was really um nice like you know Ruby's a programming language it has objects
00:35:44.720 like let's use them um it does sort on insertion and um I could sort over any
00:35:51.040 criteria with the spaceship operator um the only weird thing I found is that if things depending on how you set it up if
00:35:57.720 you you might have to sort before you iterate um if you change things but it
00:36:02.839 was kind of a minor thing and um not really that big a deal performance-wise like it it didn't matter um so the moral
00:36:09.920 of the story was that um you know I was able to keep the logic when I when I did
00:36:15.240 this this uh example I was able to keep the logic in the app itself like in my
00:36:20.960 Ruby code um I didn't have to uh figure out what was going on in the database
00:36:26.119 itself and I was able use data structures that we have in Ruby um and
00:36:33.640 if I didn't have exactly what I wanted I could build them myself like it you know Ruby's awesome I'd build whatever I want
00:36:40.560 so if I wanted to build a data structure I could use it um the reason why I included this
00:36:47.599 example is because with maglev it would have been absolutely trivial to do this and persist this across any number of um
00:36:54.680 you know app servers um or any anything that could basically connect to the VM those things would already be there the
00:37:01.280 Sorting would work everything would would just work um so the one thing that I really
00:37:09.240 wanted to do not like a live demo but well kind of a live demo uh and I I
00:37:14.319 heard that's risky um so I didn't do it but the the the Fantastic thing about
00:37:20.200 maglev is that it's distributed um and so uh what I what I mean by that is um
00:37:30.280 so if I have a stone um running on this computer and I start up any number of those Little Gems like you know mag Ruby
00:37:37.839 run some code um it will connect to this local machine and it will have all those
00:37:42.880 objects and that's cool but where it's really powerful is when you distribute
00:37:48.000 it over servers anywhere so these object the stone is really smart about how it
00:37:54.319 um moves these objects around and and um since the transaction approach is very
00:37:59.440 simple any computer that can connect to a stone um can see the object that it's
00:38:05.319 allowed to see because there's all these crazy um security policy things that you can do uh in the VM um and so what I
00:38:12.440 wanted to do and I didn't um was to get you all to help me with this um because
00:38:19.079 it's actually quite simple for you all to see what I have on my
00:38:24.599 stone um you basically you know use breake to kind of interface with magav
00:38:30.040 um so you start up this thing called net ldi and then um on the stone so like
00:38:35.599 this could be my slice up in up in the cloud or whatever um you start magav
00:38:42.839 right and so I have this object repository up there in the cloud and as long as you can connect to it um you
00:38:49.480 would be able to see objects and you would be able to um make changes and write new
00:38:55.640 objects and uh so on my laptop all I would really need to do is uh start net
00:39:01.839 ldi which basically does like starts a demon and then connects or make sure
00:39:07.400 ports are set up and stuff and oh I totally
00:39:15.240 forgot that looks like crap um anyway so I would connect to the stone
00:39:21.000 host um which would basically say which would connect the magb running on my laptop to GL the stone up wherever and I
00:39:30.400 would be able to see um which sorry it looks bad the exact same objects so
00:39:37.640 that's what I mean by like a distributed um object persistence um you don't have
00:39:42.960 to have like active record code over here and active record code over here you don't have to um build up a gem as
00:39:49.800 long as things are um in both sorry as long as the instances are persisted to
00:39:55.480 the stone they're available everywhere
00:40:01.319 um and so you know one would think that the killer feature of this is the transparent object persistence but I
00:40:07.760 maintain that it's the distributed transparent object persistence um and that that's what will
00:40:14.000 allow people when they when you start playing with this to do some really interesting things much quicker much uh
00:40:21.200 easier than you would if you were to use you know um a relational database or some other data store
00:40:30.119 uh so matz talked about motivation and uh I'd like to share um some motivation
00:40:36.560 as to why you might want to play with maglo um so first of all you will only ever deal with objects you won't deal
00:40:42.760 with rows you won't deal with documents you won't deal with like however Rus store stuff um I mean keys right
00:40:50.599 essentially Keys um as or sorry strings as keys but uh um you will only deal
00:40:56.599 with objects that's a really nice world to be in um you get to learn from the
00:41:01.720 past like I'm just now learning how Okay small talk people are of course
00:41:08.200 smart right but like they've done so much and they've like the history is so
00:41:15.359 rich we have um so much to learn from them um in terms of implementing you
00:41:21.440 know VMS which you know rubinius did with its C++ VM um in terms of best
00:41:27.000 practice IES you know all this kind of stuff like it's it's a really great way for us to become um better
00:41:35.000 programmers and it makes your life easier like you don't have to worry about you know like is your database
00:41:41.200 server down because with maglev either uh your program runs or it doesn't and
00:41:46.440 if it doesn't run that means it can't connect to the stone and like that's really simple um you don't have all these dependencies out there everywhere
00:41:53.400 that you have to keep track of um so I think I've only got a couple minutes left I want to say um uh I'd
00:42:01.119 love to actually try out the distributed thing on a larger scale I've really only tried the distributed thing like on my
00:42:07.359 laptop but I'd love to get a bunch of people together and see if we can't get it working um on a on a little bit of a
00:42:14.119 larger scale so if you're interested in that come find me um I'll take questions if anybody has
00:42:20.520 questions for a couple minutes this is going to hurt isn't it Josh
00:42:27.280 yeah how how do you deal with doing queries that uh I mean I think gemstone
00:42:33.480 has some capacity for doing SQL type queries so if you're trying to do something that you want to it's it's
00:42:40.359 like a relational type query that would be better to grab just the right set of
00:42:45.800 data from the stone rather than pulling grabbing all data and running through some Ruby thing yeah I I haven't looked
00:42:51.680 into um so the question was you know does gemstone have like this or it does have this SQL type um query language I
00:42:58.880 actually haven't really looked into that too much um it does have uh built-in indexing um so if you're uh you can if
00:43:08.160 you're I guess worried about um performance to an extent like maybe you don't want to pull everything down um I
00:43:14.880 mean everything pretty much comes down anyway like the VM handles most of that for you um so it's kind of nice you
00:43:20.720 don't really have to think about it but um there's indexing stuff that's built in that helps with some of the performances of queries if you if you
00:43:27.160 need it uh let's go with yeah what can you tell us about um sharding and large data
00:43:34.200 sets and scalability and stuff like that on okay um so sharding there's no
00:43:40.000 sharding um because there's basically the stone um there could be uh some I
00:43:47.000 guess what if you have a St that's so big that it doesn't fit on any piece of metal that exists
00:43:53.000 today thank you um so as long as you can like Network attach
00:43:59.319 storage you can grow the size of your repository so it's basically limited the the actual stone the num the number of
00:44:06.319 objects is essentially unlimited uh you can store whatever you want to store what you may be able to have um like the
00:44:13.359 VM will push stuff off to dis when necessary and it'll pull stuff off of dis when it needs it but uh in terms of
00:44:20.280 like the number of objects you can store it's unlimited because you just need more dis space you basically tell the
00:44:26.760 the Stone to like use another extent and it can grow so as long as you have like
00:44:32.559 network attached storage or some way to add capacity that just sound what you're
00:44:37.680 describing though sounds like for certain types of applications where you want to have a bunch of servers with
00:44:42.920 like data in memory because you just your data set is of such a nature that
00:44:48.440 you can't really uh you can't wait to swap like Google search for example you
00:44:54.880 know Google doesn't want to have to pull some data off of the dis in order to return search results to
00:45:01.880 you yeah so I don't think Google would use would you would I mean that's a I
00:45:07.000 guess that's a different problem for normal applications use something like this for something maybe not exactly
00:45:13.319 Google but I mean something kind of like that sure um so o like they've got a
00:45:19.599 Global Network of offices um people are creating objects all the time um and you
00:45:26.720 know I'm sure there are things deep down in the VM that you can I think I know
00:45:32.119 there are things deep down in the VM that you can kind of tune um but the kind of the whole point is that like you
00:45:39.440 get whatever like you get what you need like the VM kind of gives you and if it can't find it locally it will go find it
00:45:46.800 kind of up the tree like the network tree to wherever the stone
00:45:51.880 is does the uh does the stone persist
00:45:58.040 yes
00:46:04.160 so instances would uh yes other instances
00:46:09.760 so the question was um does magav persist the class definition so classes are objects so it persists them um and
00:46:18.680 uh yes so if you do add a method it will get picked up and there aren't strategies for like migrations um which
00:46:25.280 haven't been thought about a lot in the in like the Ruby version but in small talk they kind of have that figured out
00:46:31.319 there hasn't been a lot of work done quite yet um I don't I don't know if we
00:46:36.599 actually have more time I mean kind of have much time as we want because we're at lunch
00:46:44.920 huh let's take yes what version of Ruby does it run so uh mag right now runs 187
00:46:52.520 it runs most of 187 um and the uh they're working on one nine support
00:46:58.680 right now yes what about uh reporting and
00:47:03.800 business intelligence softw off of the data that she got forist so uh the
00:47:09.760 question was about reporting um or data analysis so um there there are things
00:47:14.960 built into the VM itself that um kind of give you a lot of like the analytics you
00:47:20.119 might want um but you know you have objects and they're just there so you
00:47:25.400 could kind of like build up whatever type of reporting you want and that's something that o o chose gemstone um
00:47:33.280 back in like the 90s because you know they've got offices everywhere and they have each office has different reporting
00:47:40.119 uh requirements and so it allowed them to you know do massive reporting
00:47:48.720 stuff so this another scalability question I guess uh so if you have one of the great things about Redd is it
00:47:54.960 doesn't care what langage you WR with this it seems like oh now I have a JavaScript app or have a closure app and
00:48:01.680 I talk to it and get Ry objects seems like one of your things learning from
00:48:07.200 the past I feel like we learn from so that passing around objects is
00:48:12.520 exactly right so um so the the I guess the statement was
00:48:18.800 that you know reddis is cool because it's so how do you integrate with other oh how do you so you don't I mean so the
00:48:25.359 same thing you would right now if you have a Javascript app but your back end is in Ruby you're going to use Json so
00:48:31.760 the same thing uh in using Mago essentially um but the the power is that
00:48:38.240 if you need to um build up some other component of your application in Ruby
00:48:44.119 you don't have to duplicate um code you don't have to like the objects are just there like I I know I kind of like wave
00:48:50.880 my hand when I say that but I swear to God like that's that's how it works like they're just they're just there like
00:48:56.119 it's great and when you start playing with it you'll see that and hopefully it will blow your mind
00:49:04.559 yes uh so the Q is something I one thing
00:49:11.880 about the Q um I whipped it up in like five minutes like before a talk so it is
00:49:18.440 not um it is not great but it yes
00:49:25.559 so okay okay I think that's it let's go to lunch thanks
JC
Jeese Cooke
1 talk
Explore all talks recorded at RubyConf 2012
+46