Summarized using AI

Dissecting a Ruby Block

Pat Saughnessy • November 01, 2012 • Denver, Colorado • Talk

In the talk titled Dissecting a Ruby Block, Pat Saughnessy explores the intricacies of Ruby blocks and their internal implementation, underscoring their significance in Ruby programming.

The presentation is structured as follows:

  • Introduction to Ruby Blocks: Saughnessy starts with a lighthearted acknowledgement of the audience's commitment. He emphasizes that blocks are a defining feature of Ruby, contributing to its fun and elegance.

  • Importance of Ruby Internals: He outlines three main reasons why understanding Ruby's internals is beneficial for developers:

    • It enhances the depth of knowledge about the language.
    • It provides insights from the core team who crafted Ruby’s internal structures, which can inform better programming practices.
    • It is ultimately enjoyable to explore how a language works under the hood.
  • Defining a Block: Saughnessy defines a block as a snippet of Ruby code encapsulated between do and end, and he explains that blocks are Ruby's implementation of closures. He provides a practical example to illustrate how blocks can reference variables from their surrounding context.

  • Dual Nature of Blocks: He highlights the duality of blocks—they function as individual methods while still being connected to the surrounding environment, which is critical in understanding how Ruby operates.

  • Terminology and Historical Context: Drawing on historical foundations from computer science, Saughnessy explains how the term "closure" was first introduced in a technical paper by Sussman and Steele in the 1970s, framing closures as a combination of a function and its environment.

  • Differences Among Lambdas, Procs, and Blocks: The talk also addresses related structures in Ruby, such as lambdas and procs. Saughnessy explains how they differ yet fundamentally relate back to closures.

  • Meta Programming and Closure: Lastly, Saughnessy introduces the concept of metaprogramming through a detailed example of defining methods dynamically in Ruby using closures. He demonstrates how closures can access variables from their environment in ways that traditional method definitions cannot.

In conclusion, Saughnessy stresses that understanding these concepts can lead to improved Ruby development practices, while also appreciating Ruby's elegant implementation of blocks and closures. He encourages the audience to explore these themes further through his book, Ruby Under a Microscope. This exploration not only improves coding skill but also deepens one's appreciation of the Ruby language.

Dissecting a Ruby Block
Pat Saughnessy • Denver, Colorado • Talk

Date: November 01, 2012
Published: March 19, 2013
Announced: unknown

More than any other feature of the language, in my opinion blocks are what make using Ruby fun. But what is a block, exactly? What would I see if I could cut one open and look inside? During this talk we'll:

Explore Ruby's internal implementation of blocks, lambdas, procs and bindings. Learn how closures and metaprogramming are related in Ruby internals. Discover what metaclasses and singleton classes are and how Ruby uses them. Do you really need to know how Ruby works internally to be a good Ruby developer? Probably not. But taking a peek under the hood can help you better understand the language... and is a lot of fun!

RubyConf 2012

00:00:15.200 hi oh too loud so um i realize i'm the last person between you guys and the bar so i'm
00:00:21.359 really impressed that you're still here um there's been some amazing talks today i've seen a lot of them some amazing
00:00:27.599 talks going on right now so thanks a lot for coming i really appreciate it uh and thanks to ruby central i don't know if
00:00:33.280 any of you guys are in the room um for giving me the chance to talk at rubyconf it's like a once in a lifetime opportunity i'm really excited
00:00:40.160 uh it's it's thrilling for me to be here so thanks a lot my name is pat shaughnessy and i'm going to be talking
00:00:45.520 about blocks so that's right i'm going to talk for 45 minutes about ruby blocks
00:00:51.360 in fact together we're going to be dissecting a ruby block so don't worry there'll be no blood no gore
00:00:57.920 no lab animals will be harmed in this experiment what what i'm going to do is look at how
00:01:03.280 ruby implements blocks internally and see what we can learn about blocks by doing that and i'll be talking about
00:01:10.080 max's ruby implementation today not about j ruby rubinius other versions of ruby
00:01:15.680 before i get into blocks let me just start by saying a little bit about myself you probably haven't heard of me before i work at a company called
00:01:22.080 mckinsey which is a management consulting firm but i'm not a consultant i just i'm just a ruby developer i write
00:01:27.520 code for them um in my spare time i read a blog about ruby development at pat shaughnessy.net
00:01:34.240 and i'm really excited i actually just finished my first book i wrote a book i've been working for the last hey thank
00:01:40.320 you so i just released a book uh this week on monday called ruby under a microscope
00:01:45.520 and it's all about ruby internals so i've taken six months off my my job and i went into my just
00:01:53.040 sat at my house and i wrote about how ruby works and the way i the way i like to think of
00:01:58.399 it is i went on a long road trip it was kind of like visiting japan or a foreign country where you don't really know the
00:02:04.000 language and you're seeing all these amazing exotic sites and you're um you know but
00:02:09.280 you begin to learn a little bit of local language you get to know people there and you can kind of appreciate their customs and their food and that's what
00:02:15.360 it was like for me so the book is sort of a travel journal about ruby mri internals i also touch on j
00:02:22.239 ruby and robinius in the book so if you're into internal stuff and actually the presentation today this
00:02:28.640 is really a synopsis of one of the chapters from the book please check it out
00:02:34.560 so the first question you probably have for me is okay pat why ruby internals who cares about how ruby works all i
00:02:41.040 care about is that ruby does what it's supposed to do you know if i it takes a value if i insert something into a hash
00:02:46.879 or if i save it into array all i need to know is that i'm going to get it back when i give it the same key um i i don't
00:02:53.200 care how ruby works i'm busy i have a day job i have to get stuff done i have to um
00:02:58.239 you know maybe i have an open source project at night i work on in my spare time or on the weekends you know who has time to look into ruby internals and
00:03:04.640 figure it out you know i took six months off and i did it i'm kind of crazy maybe but um here are the reasons why i did it
00:03:10.640 and maybe why you should learn about ruby internals too so i can think of three reasons um the first one is i
00:03:16.080 think if you study ruby internals it can help you become a better ruby developer so i think you can learn
00:03:22.159 about the language more deeply than you may have you can learn it more deeply than you did before
00:03:28.239 you can learn about how ruby was intended to be used how matts and the rest of the core team intended the language to be used not
00:03:34.799 just use it the way that you happen to learn how to use it and i think you can learn a lot and just
00:03:40.000 become a better developer for it maybe a second reason could be there's a lot to learn from the core team you know
00:03:46.159 the koichi and the other team people on team are they're really smart people they've solved difficult problems
00:03:51.440 and maybe you can learn from their solutions their algorithm their code and learn about you know maybe pull some
00:03:57.040 ideas from them that you can try on your own projects so there's a lot of amazing solutions in there that we can all learn
00:04:02.239 from and it's a little inaccessible unless you take the time to learn it and i think the last reason the most
00:04:07.599 important one is just a lot of fun i had a blast i had so much fun learning about ruby internals and i hope you guys get a
00:04:13.680 little taste of that today if you if you like it then check out the book there's a lot of really amazing stuff going on
00:04:19.440 in there it's fascinating to me to learn how a language works internally you know i never knew this i'm not a computer
00:04:25.040 science major or anything and so for me to learn about how language worked internally it was just it was just super
00:04:30.800 fun so that's the most important reason have fun um so today i'm going to talk about blocks you know here's a block and it
00:04:38.160 seems kind of hard to believe that there's really that much interesting to say about blocks you know what can we say about them well we can call them you
00:04:43.919 can call a block you can pass in an argument to a block like this um and you know most of us especially
00:04:50.000 here rubyconf all you guys can tell me how this you know what this code will do in two seconds or less you can tell me
00:04:55.680 what this code is going to do you know what am i possibly going to say what can we say about blocks that's that
00:05:00.960 interesting so hold that thought i'll get to the blocks in a second the real reason however i want to talk about blocks is
00:05:08.560 when i think back to when i first came across ruby and i imagine a lot of you here had the same experience you know
00:05:14.160 you first saw ruby for me was in 2008 maybe and before that i was a php
00:05:19.199 uh java c plus plus developer i did a few other random things too you know so if i wanted to write code
00:05:25.199 that did this in c i would have to say you know int i semicolon uh four
00:05:30.800 parentheses i equals zero semicolon i less than ten semicolon i plus plus
00:05:35.840 parentheses brace blah blah blah it was a much more verbose way of doing the same thing and when i first came across
00:05:41.440 this it might have been 2008 i thought to myself wow this is something different i haven't seen anything like
00:05:47.280 this before you know it was it seemed really elegant and beautiful to me and it got me to
00:05:52.560 think twice about ruby i thought to myself ruby is something special and when i think back about that it it
00:05:58.960 really was the blocks the way that you can use blocks and ruby that got me to fall in love with the language so you
00:06:04.800 know aside from all the technical stuff which we're about to go into for me the real reason i wanted to talk to you guys today about blocks was
00:06:11.440 it's what i like the most about ruby uh and you know think back to your own history where you discovered the
00:06:16.479 language and what it was about ruby that made you fall in love with it enough to come to ruby conf
00:06:22.000 so for me it was blocks so what am i going to say about blocks well the first thing i want to do is define
00:06:27.440 what is a block you know if we ask that question what is a block what kind of answer do we get you know and so the way
00:06:33.440 i want to do that is imagine if i could slice open the block and look inside you know what's inside of a block are there
00:06:39.039 moving parts in there am i going to see blood guts chromosomes you know amoebas if i were to slice a piece of a block
00:06:44.880 add look it under look at it under a microscope you know what would i see and so the first half the presentation
00:06:51.039 i'm going to talk about what blocks are i'm going to answer that question what is a block
00:06:56.880 and we'll find out in a few minutes that blocks are ruby's implementation of closures and we'll define what a closure
00:07:03.440 is along the way uh and then at the second half i'm going to switch gears and talk about closures and meta programming and how those two
00:07:10.000 things are related so two different very different concepts but i think they're actually related to each other and so um
00:07:15.759 we'll talk about that second so let's get into it so if you were to look inside the ruby c source code and
00:07:22.000 you were to try to figure something out about what blocks are the first thing that you would see is that there's a c
00:07:27.360 memory structure called rb underscore block and i'm not showing the contents of it
00:07:32.720 here and the reason why is i want to sort of discover as we go along what's in this structure
00:07:38.319 and but the nice thing about studying ruby internals is if you have a question like this you know what is a block or
00:07:44.479 what is an object what is a module you know you can actually directly answer that all you need to do is go into the c
00:07:49.919 code find the definition of this or that or the other thing and that's your answer you know you
00:07:54.960 don't have to think about it very much you can just look at how it was implemented and so that's maybe another reason for studying ruby internals is
00:08:01.120 you can answer all these sorts of questions that you might have about how the language works or how it was intended to be used
00:08:07.759 so before we go any further i worry about my diagrams uh you know today and also in my ebook i have a lot of
00:08:12.800 pictures it's like an illustrated book um my pictures are not definitive exhaustive definitions of
00:08:19.680 exactly how ruby works i'm not going to show you every single thing inside the block structure
00:08:25.440 and there's the reason why is i want to point out to you the details about ruby internals that i think are important for
00:08:31.440 for you for ruby developers to know there's a lot of other very technical things and a lot of nitty gritty details
00:08:37.279 that maybe aren't as interesting and so i'm going to gloss over ignore some of those details and so my diagrams are
00:08:43.680 over simplified versions of reality so that'll make them hopefully easier for you to understand as well and made it
00:08:49.600 easier for me to draw them too so what we're going to do is uh we're going
00:08:54.800 to do a series of experiments and the reason i call the book ruby under a microscope is i wanted to use sort of a
00:09:00.320 scientific method for discovering and learning things about ruby you know it's one thing to just start talking about
00:09:05.839 how it works and what it does it's another thing to test your hypothesis with an experiment um so i'm not going
00:09:11.279 to do real experiments today i'm just going to do a you know a few simple thought experiments wouldn't have much time but in the book i go into more
00:09:17.200 detail about one thing or another and actually run some code the experiments are written in ruby code so you can download them and
00:09:23.440 run them yourself if you're interested so the first thing i'm going to do is really the most basic thing you can do
00:09:28.959 with a block is call a block so how do we call a block so not rocket science all of you know
00:09:34.240 how to call a block here's an example um here's a block that takes a string value
00:09:39.519 the quick brown fox the rest of that of course is jumps over the lazy dog that didn't fit on the
00:09:44.880 slide but we're going to come back to that string over and over again and you're going to get very bored and tired of hearing me say that
00:09:51.200 and we're going to save it in a string variable call a local variable called str and then we're going to print that
00:09:57.279 out and we'll do that 10 times so not very interesting code but what it does is it allows me to answer the question
00:10:03.360 what is a block and so the first the most obvious answer to that question is well it's right there on the screen it's
00:10:09.600 everything between the do keyword and the end keyword you know the first most obvious definition of a block is it's
00:10:16.000 just a code snippet it's something that um it's code that you type in it's sort of like a function
00:10:22.160 if we were to look inside of ruby you know through a microscope so to speak this is what you would see
00:10:28.240 so that rb block c structure i'm showing that on the right and now i'm showing you one of the
00:10:33.839 fields or one of the values inside that and there are many more but here's one of them it's called iseq and i think and
00:10:40.320 you know help me out here core team i think that stands for instruction sequence and that's a pointer and it points to
00:10:46.800 an instruction sequence that's what i'm showing on the left side um and an instruction sequence is a series
00:10:53.760 of bytecode or virtual machine instructions the first one is put string set dynamic et cetera et cetera don't
00:11:00.160 worry don't try to read that don't try to understand it i'm not going to explain any of that today i have a lot
00:11:05.360 more of that in my book but what i wanted to point out is this is really what a block is it's a snippet
00:11:12.560 of code and internally this is what it looks like um just taking a step back for a moment i will say just one uh you know maybe
00:11:18.560 one minute about how the virtual machine works and how these by code instructions work at a ten
00:11:23.760 thousand foot level when you look at ruby from you know from a distance the way ruby works at least starting with
00:11:28.959 ruby one nine is you give it some your code ruby parses that and compiles your ruby code into
00:11:34.800 these instructions and then executes them with a virtual machine and that's what koichi was talking about earlier
00:11:40.839 today so so anyway this is our first answer to the question what is a block it's a
00:11:47.760 it's a snippet of code or a function now let's do another experiment this time i want to look at how you can
00:11:54.240 reference variables from the parent scope and that sounds kind of complicated but it's really actually obvious so if you were to look at a
00:12:00.240 block like this so now i'm doing the same thing i'm putting the same string out ten times i'm just doing it in a
00:12:05.760 more complicated silly fashion and so what i've done here is i've taken the first half of the string the quick
00:12:12.000 round fox and i've saved that in the same str variable but i'm i'm doing that in the outer scope
00:12:18.320 um then i call the block 10 times and in the block i say str2 so i have a second variable and on that one i'm saving the
00:12:25.200 second half of the string jumps over the lazy dog and then i have a put a statement that
00:12:30.240 says put s you know string and string two so pretty simple code you can all understand this
00:12:36.320 and in fact you know what's so interesting about this it's all kind of obvious all you guys write this this sort of code probably every day this is
00:12:42.399 probably a reflex you probably just type this in without even realizing it um you know why am i pointing this out to you
00:12:48.560 well for me there's something actually kind of profound and interesting going on here it has to do with the question again what is a block so the way i look
00:12:55.839 at it is blocks have kind of this dual personality they're kind of schizophrenic you know they have on the
00:13:01.680 one hand they're a separate function that i'm calling on the other hand they're part of the surrounding function
00:13:07.760 and so that put a statement can access equally well it can access str
00:13:13.360 and it can access str2 it doesn't it doesn't matter and that's kind of obvious we all take that for
00:13:18.639 granted but it's not you know don't take it for granted a lot goes on inside of ruby to make that work so let's take a
00:13:24.399 look inside of ruby and see how that's implemented um so what i'm going to do is i'm going to step through this slowly and i'm
00:13:30.240 going to go through this in great detail and explain how this is implemented so the first thing we do is we take that string the fox string and we save it in
00:13:37.360 a variable str so what does ruby do when you save a value in a variable so if we were to
00:13:43.519 look inside of ruby again you can tell these are super oversimplified diagrams but what i'm
00:13:49.360 trying to get across here are the essential ideas so first of all on the left we have a stack
00:13:55.040 this is something called the virtual machine's internal stack so the way the virtual machine works and if
00:14:01.199 you heard the presentation earlier you know this it's a stack based machine as it runs those instructions that we saw
00:14:06.880 earlier set local get local whatever is it runs those internal instructions those instructions typically use
00:14:13.360 they take values off of this internal stack they do some kind of operation and they push the result back on the stack
00:14:20.240 so this internal stack is actually not something that you ever see as a ruby developer or really need to know much about
00:14:26.160 but you know just remember that the virtual machine inside of ruby is a stack-based machine
00:14:31.199 and so when we save a value in a local variable like str in this example that value gets saved on that stack
00:14:38.480 internally okay on the right is actually a different stack so on the right there's something called the rb control frame
00:14:44.720 structure and i'm showing one member from that structure and quiche asked earlier you know how many members are in
00:14:49.920 this and i think there were 11. so there's a lot more going on here i'm not showing you but what i wanted to point out is that there's one pointer the dfp
00:14:56.880 which i think stands for dynamic frame pointer that indicates that ruby uses to keep track of where the variables are
00:15:03.839 for the current um for the current scope so what i need to explain here is that as your ruby program runs and you call
00:15:10.480 from one method to another to a block to a lambda you have this ruby call stack that you build up and if you ever run
00:15:16.560 the command put s color that will print out that whole call stack and so each level in that call stack is represented
00:15:22.959 by this rb control frame structure and so what you have to imagine here is i don't really have time to explain all
00:15:28.639 this today is that there's a stack or an array of these control frames that represent your ruby call stack so if you
00:15:35.279 think about it inside of ruby there are two stacks there's the ruby stack on the right and there's the
00:15:40.480 internal virtual machine stack on the left and those two things are related and connected anyway let's get back to blocks so how
00:15:47.360 does this work so now we've saved a value in str and now we're going to call the block so what happens
00:15:53.600 well actually we're not calling the block are we what are we doing here what we're doing here is we're taking a number
00:15:59.279 10 and we're considering that to be an object that's the receiver of a message
00:16:04.320 so 10 is an instance of the fixed num or integer class and we're sending a message to it which is the times method
00:16:11.040 so what we're doing is we're calling the times method on that class fixed num
00:16:16.079 and that has nothing to do with the block in fact we're not calling the block at all what we're doing is we're taking that block and we're passing it
00:16:22.000 into the method as an argument so it's um there's more going on here than you might think so let's look at
00:16:28.000 what happens inside of ruby at the moment when you pass a block into a method call as an argument so again
00:16:34.320 we're not calling the calling the block yet we're simply referring to that block for the first time so when you do that
00:16:39.600 when you when you first refer to or use a block ruby says oh i have a new block
00:16:44.800 i need to create a new rb block structure and so what it does and again i'm
00:16:50.000 showing an oversimplified version of the same structure but it creates a second value in there and it copies it the dfp
00:16:57.199 it copies it from the control frame to the block okay so remember that dfp that was a
00:17:02.720 pointer over to the internal stack where the local variables for that method were
00:17:08.079 stored so what's going on here is interesting is that ruby is saving information inside the block about where you first
00:17:15.760 referred to that block from where you decided to pass it into a method so
00:17:21.280 blocks are not as simple as they seem they're not just a code snippet they have this other value inside of them which is a
00:17:27.439 pointer to the environment where you reference them from so interesting
00:17:32.880 let's hold on to that thought for men and see where where we go okay so now we're calling uh the 10 dot
00:17:38.480 times method so we're calling into the times method inside the fixednum class and so whenever you call a method in
00:17:45.200 ruby it creates another stack frame on that internal stack it pushes a bunch of values on there and there's a lot of
00:17:51.120 details i'm not showing but i just want to get across the fact that as you call deeper and deeper into your code it
00:17:56.559 creates more and more of these stack frames and now this fixednum times class this is implemented in c this is not
00:18:02.320 something that you write yourself but it works the same way that you probably would write it loops over you know the number zero one two up to n
00:18:09.200 minus one so in this case we'll go up to nine and finally now it's time to call a
00:18:14.320 block so that will that code inside of the fixednum class will yield to the block yield is just a fancy word to use
00:18:20.400 for calling the blocks function um and uh and it'll also pass in that integer argument if you want to if you
00:18:26.799 want it to okay so now i'm in the block i'm going to create that str2 variable where i save jumps over the lazy dog and
00:18:34.559 then i'll print out the two strings so let's see how that works okay so now i'm getting a little more
00:18:40.080 complicated in my internal stack my yar of internal stack now i have three stack frames in fact there's more than three
00:18:46.480 i'm glassing that over too there's a fourth stack frame in there but just to keep things simple today on the bottom
00:18:52.480 we have the parent scope where the str variable was in the middle we have another scope for
00:18:58.480 the times method and then on the top we have the top of the stack is now being used as we run
00:19:04.240 the code inside the block and so that's where the str2 variable goes
00:19:09.440 and so the put s call when it says put s str str2 the code in put us is able to
00:19:14.960 access both str2 in the current scope and then using that dfp point to go down
00:19:20.960 to the parent scope farther down on the stack and find that str variable and the reason it can do that is that when you
00:19:27.200 yield to a block in ruby when you call a block ruby takes that dfp pointer out of the
00:19:32.559 block and puts it on to the stack so that your all the code inside your
00:19:37.840 block is able to access the environment where you originally again where you first referred to that block so ruby's
00:19:44.640 in that way sort of merging those two things together the original surrounding environment and the
00:19:50.080 current scope of the block itself so that's uh you know back to that dual personality that's the way blocks are
00:19:56.320 able to behave in this way how they're smart enough to do both things um so just taking a step back and
00:20:02.720 reviewing we found uh so far we've discovered two two fields or values inside the block
00:20:08.080 structure we have the iseq pointer this is the instruction sequence that points over to the again a series of bytecode
00:20:16.080 instructions that the virtual machine is going to run so this is the compiled version of my function
00:20:21.120 and then it has that dynamic frame pointer or dfp that points down to the stack and again not to where the stack
00:20:27.679 is you know at the moment we run the code but to the referencing environment where i first referred to that stack to
00:20:34.159 that block so it turns out in computer science there's a fancy name for this combination of two ideas a function and
00:20:42.000 the environment where you refer to that function from and that was invented a long time ago back in 1975 i don't know if that's off
00:20:49.360 the edge of the slide so back in 1975 uh an mit professor named gerald sussman
00:20:54.880 and his colleague guy steele wrote a paper called scheme an extended an interpreter for
00:21:00.240 extended lambda calculus sounds really complicated and it's pretty it's a pretty hardcore academic paper it's
00:21:06.320 actually online you can download this and read it yourself i did the other day it was really fun
00:21:12.240 i love this stuff i'm kind of a language geek and this paper defined and and explained
00:21:18.559 their implementation of a language called scheme scheme is a dialect of lisp and what's interesting about scheme and
00:21:24.960 scheme is still being used today uh is that it was the first time that in computer science there was a formal
00:21:30.080 definition of the word closure so they define inside this um this academic paper what a closure is and here's how
00:21:36.400 they defined it so we'll just read through this for a second in order to solve this problem i don't remember what problem they were solving
00:21:42.400 we introduced the notion of a closure uh which is a data structure containing a lambda expression whatever that is
00:21:49.760 and an environment to be used when that lambda expression is applied to its arguments so kind of hot you know come
00:21:56.240 you know convoluted language here a lambda expression that's a fancy way of saying a function or a method and um applying that lambda expression
00:22:03.760 to its arguments just means to call the function and pass in the arguments but if you think about this for a minute
00:22:09.840 this is actually what we have in a ruby with a block we have a function and an environment to be used when we
00:22:17.120 call a function you know backing up this is what a block is it's the function on the top left and in the environment on
00:22:23.120 the lower left and so uh and so what we've done here is we've shown that blocks and ruby are are they're ruby's
00:22:30.320 implementation of closures so ruby has taken and i don't know if matt did this on purpose or you know indirectly but
00:22:35.760 ruby's taken um a concept from the 1970s and applied it to a language that was
00:22:41.120 invented in the 1990s and that we're still using today so i find that kind of interesting
00:22:46.880 so let's move on and talk about lambda so lambdas are related to blocks and let's see if we could do experiment to figure out you know what lambdas are
00:22:53.520 but before we get to the lambdas themselves you know where is the word lambda come from you know why in the world does ruby have a keyword that is a
00:23:00.320 greek letter lambda is a letter in the greek alphabet what does it have to do with computer science
00:23:06.159 um so it turns out the same thing that ruby borrowed the idea of lambda or the word lambda from lisp this is the first
00:23:12.559 computer that ever ran uh the lisp language this is from the early 1960s list was invented by john mccarthy in
00:23:19.200 the late 1950s actually so this is what is that almost 50 years ago now
00:23:26.240 and in lisp john mccarthy introduced the idea of a lambda which is a way and so
00:23:31.440 here's a list how you would create an anonymous function i'm not a list developer and many of you here might be
00:23:37.280 what i think this is doing is it's defining a function there's a lot of parentheses in lisp but what i think
00:23:42.720 it's doing is defining a function that takes the argument and then divides that argument by two and returns that value
00:23:48.559 so it's uh but it's not doing that immediately it's returning a function as a value and so what lisp introduced and
00:23:55.039 other functional programming languages do this too they allow you to treat code or functions as data values
00:24:01.679 uh and so they you know quote unquote they treat code as a first class citizen so in ruby you can do the same thing
00:24:07.440 ruby borrowed the lambda key word from lisp and um here's uh you know here's my same
00:24:12.480 silly example um now i'm doing in in an even more complicated and confusing manner so what i want to do is
00:24:19.039 demonstrate how you use lambda in ruby so we're going to do the same thing we're going to print out the same string
00:24:25.039 not 10 times just one time today well for this experiment so how does this work
00:24:32.000 a lot of you guys might know exactly how lambdas work but in case some of you don't or haven't used one in a while let me go through it so the way this works
00:24:38.960 is on the bottom i call a method display message calls into the method in the method we
00:24:45.520 save the quick brown fox the first half of the string in a variable str same thing we did earlier and then we have a
00:24:51.200 block that takes um the first half of the string and prints it out concatenated with the second half of the
00:24:56.480 string except the difference here is i don't call the block immediately instead i say lambda and what lambda does is it
00:25:02.720 returns that block as a value it converts it into a data value of some kind and that becomes since it's the
00:25:08.640 last thing in the display message method that becomes the return value for that method and then finally we say dot call
00:25:16.159 at the very bottom so it's not until the bottom here where i say call that it prints out the full string and you can
00:25:21.919 try this on your own computers so let's go through this and see how this is implemented inside of ruby you
00:25:28.240 know what goes on when you when you use lambda so we'll go through it one line at a time slowly the first thing is i
00:25:34.000 take the fox string and i put it in the str variable so same story here um you know no surprise we save the string on
00:25:41.679 the stack and there's a control frame structure um and the bottom empty one is the outside scope and then we have one
00:25:48.000 for the message uh for that in that method that we're calling this should be display message
00:25:54.080 um okay what do we do next so now we call lambda so what happens inside of ruby when you call the lambda keyword so
00:26:00.799 it's going to be a complicated diagram don't panic i'm going to go through this slowly let's see if we can figure this out together
00:26:06.799 and let me take a couple minutes to just explain this picture so before i get to all the boxes and arrows on the right
00:26:12.559 let me talk about the two words on the left stack and heap so what i'm indicating there is there's really two
00:26:17.679 different types of memory in your ruby process or really in any process for that matter there's stack memory that's
00:26:23.360 what we've been talking about so far where you push values onto a stack you pop them off the stack is used for
00:26:29.120 values that are very temporary in nature so they're only valid or they're only needed for a short period of time
00:26:34.559 usually during the context of one method or function on the on the bottom under the dotted
00:26:40.320 line we have the heap and a lot of you may know the heap is for where as for when you want to save things for a long
00:26:45.520 period of time you know when you um want some value to persist around or you know the lie around for a while so you can
00:26:51.120 use it later and the biggest or most obvious example of using the heap in a ruby program is
00:26:56.559 when you create an object it gets saved in the heap it ruby allocates the memory out of the heap and uses it for that
00:27:02.240 object so what's going on here when i call lambda what does it do well the first thing it does let's start
00:27:08.080 from the top left it takes that internal stack where we have that str variable and it copies it from the stack it
00:27:14.640 copies that stack frame down into the heap so it creates a persistent sort of
00:27:19.840 um a new second copy of all the same information but what's special about that is it's now in the heap so it can
00:27:26.240 persist for a longer period of time and the other thing ruby does it creates on the bottom and i don't know if
00:27:32.000 everyone can see this some of you are pretty far back there's a structure called rb underscore env
00:27:37.760 and this is a what's called the environment object i assume and this is a sort of a structure
00:27:43.279 that ruby uses to manage that heap copy of the stack so interesting what else is going on
00:27:49.279 here the other thing that ruby does on the right is it takes the control frame structure and it copies it into
00:27:54.320 something called an rb underscore proc structure and as you might know some of you the return value of a lambda
00:28:01.440 key of the lambda keyword is actually a proc object and so this is ruby's internal representation of a proc
00:28:07.840 and you can see it puts in there the dfp pointer the iseq pointer and a couple other things
00:28:14.240 and one interesting detail to notice here is that inside the proc structure is a block structure
00:28:20.000 uh and so in fact what we what we're seeing here by looking at the way these things are implemented is that procs and
00:28:25.520 lambdas and blocks are all really the same thing they're all closures they're all that combination of the iseq pointer
00:28:32.559 the code or the function and the dfp pointer which is the referencing environment so there's all really
00:28:38.720 different ways of looking at the same thing and so when you call lambda it creates that new object it copies the stack
00:28:44.399 frame or that environment into the heap and then it um and then it returns that proc to you as
00:28:50.799 a return value okay so finally we're ready to print out the string and we're ready to say
00:28:56.360 displaymessage.call and it's going to print out the string it'll work just fine but what i wanted to point out here is something interesting which is
00:29:02.799 if you look at this carefully and think about it how is it possible that when i call
00:29:08.320 dot call at the bottom it prints out the whole string if you look at that first variable str
00:29:13.919 that's a local variable inside a method normally when a method returns all the variables that you create in a method
00:29:19.919 are you know they're free they're released they're gone forever but obviously here that str variable lives
00:29:25.760 on it's got sort of a second lease on life and it's able to do more you know more good for you do more in your program how does that work well that's
00:29:32.880 really back up to this picture that's the whole reason why ruby is doing all this work all this copying all these new
00:29:39.440 structures and pointers all this is just so that you can continue to refer to the environment where you called lambda from
00:29:47.120 so at the moment i call lambda that environment not only what's inside the lambda but the surrounding values are
00:29:52.559 all saved away so that later when i say call it works fine and looking at how that's implemented
00:29:59.120 inside of ruby it's sort of a similar picture except the opposite so what we do now is ruby takes the dfp
00:30:06.159 pointer from the block that's inside that proc and it copies it up into a new stack frame that it sets up to run the code
00:30:13.279 inside the block and remember in the block we're creating str2 that's the dog part of the string
00:30:19.919 so that is pushed onto the stack now on the top left and it saved that dfp pointer there
00:30:25.440 which points now it points out to the heap where the referencing environment or that you know copy of the reference
00:30:31.520 of that environment is saved so it's just like calling a normal block except for that detail the dfp is now pointing
00:30:37.679 out to the heap so that's how lambdas work so let's move on that's enough about
00:30:43.039 blocks and we've seen how blocks and laminas and procs are all the same thing and they're all
00:30:48.080 they're all closures now let's shift gears and talk about closures and meta programming so two different very very
00:30:53.840 different ideas how are they related and you know what does meta-programming have to do with closures
00:30:58.960 so first of all um so let's do an experiment using a closure to define a method so what in the world does that mean
00:31:05.840 so let's take another example let's say i create a now i'm going to wrap my exciting fox dog string and wrap it up
00:31:11.440 in a class so i'm going to create a class called quote and you all know how to create methods what you do is you say
00:31:16.960 class the name of a class quote and you say def this def that def the other thing you just start typing in method
00:31:22.399 definitions and use the def keyword def all very obvious so how do you do meta programming what
00:31:29.279 is meta programming well one way of doing meta programming in ruby is to dynamically create methods and all that
00:31:35.200 means is instead of saying def def i can say define method display message so the
00:31:41.360 only difference between this what you normally do and this the define method meth method
00:31:47.600 is that you pass the name of the meth of the new method as a parameter to it so the name of my new message my new method
00:31:54.240 is display message i should have thought of a different example that wasn't a tongue twister
00:31:59.360 so here i am defining a method in a more confusing convoluted manner that doesn't make as much sense that's what meta
00:32:04.559 programming really is all about isn't it so um but you know in a real example what
00:32:10.240 you would do is you would probably have a loop here you might create 10 or 20 methods all the same time you're going through the you know each name each
00:32:16.720 method's name is maybe an array or maybe the name of the method is from user input or from a database query so there
00:32:22.320 might be some really valid reasons for doing something like this but what i want to point out here is something different i think there's
00:32:28.000 actually some there's another minor detail here you may not have noticed that's different between this
00:32:33.360 and this which is that when you say define method and you pass in the the name then you have to say do
00:32:39.679 end so the the methods body is actually a block you're passing a block into this define
00:32:47.039 method method and that's not what we did over here we just said def the name and then we started typing in the code
00:32:53.519 and remember blocks are closures so because the block is a closure it means that the code inside of the block
00:33:00.240 can access things in its surrounding environment now in this example there really is nothing interesting around
00:33:06.080 that there's no there's no reason to do this but imagine if in your in your ruby application there was an interesting
00:33:11.679 environment there was a place in your code where you had a lot of interesting values you wanted your new method to be
00:33:17.120 aware of and to have access to so i'll make up a contrived example let's say i only remember to put the fox
00:33:24.080 the first half of the string in the in the instance variable at str but then i discovered later oh i have
00:33:30.240 the dog the second half of the string str2 somewhere else in my code you know how can i how can i allow my exciting
00:33:37.360 method to print out the same thing both strings together well since it's a closure it's able to access
00:33:43.760 the instance variable so it's going to become a method of this quote class it's able to access equally well at str
00:33:51.360 on the top and str2 which is in the surrounding scope there
00:33:57.600 one confusing detail here here is i have to say quote dot send define method that's because the define method method
00:34:04.240 is a is private to the module class but other than that it's actually not that complicated so this is how you can
00:34:09.599 define a method using a closure okay let's take another example um let's
00:34:14.720 look at sort of the classic example of meta programming so meta programming really means to write code that writes
00:34:20.639 code and in ruby the simplest way to do that is to say is to use the eval function if i create
00:34:27.040 a string like put s two plus two uh again this is silly but you can imagine
00:34:32.399 a real example where you're dynamically writing code that based on some kind of data you're getting from somewhere
00:34:38.320 you can pass that string into eval and what it will do is it will recursively call that virtual machine again and
00:34:44.960 compile that code into those bytecode instructions and then run them and so in this case it'll print out four
00:34:50.320 so not really interesting but one one detail about eval that maybe you didn't know is that it's also a closure so the
00:34:56.480 code inside of this string can access the environment around the call to eval
00:35:02.240 so what does that mean and how does that work well another way of doing this is to use the binding keyword this allows you to
00:35:08.960 formally specify that environment that you want the eval to work with um so so imagine for a minute i went
00:35:15.839 back to my quote class and i said def get binding so i create a method called get binding and the reason i want to do
00:35:21.839 that you'll see in a minute the reason i want to do that is to be able to access the scope or the environment inside of
00:35:27.920 this class so well so i can access that at str variable so hold that thought let's move on and
00:35:34.880 look at creating a new instance of the quote class so i'll say object equals quote dot new so now i have a new object
00:35:42.720 what i want to do is so let me take a moment and explain just for for 10 seconds or less how ruby implements
00:35:48.880 objects and i have a lot more about this in my book um there's a um a lot more detail whole chapter on
00:35:54.800 objects but in a nutshell whenever you create an object ruby creates a structure called our object at least for
00:36:01.119 custom objects that you write you know for strings and arrays there's something else but one of the other details i didn't
00:36:07.040 show in that control frame structure is that there's another thing called the self pointer that points to the current
00:36:13.599 value of self the object in which you're running at the moment so all of you are probably familiar with self but if not
00:36:19.440 just imagine that as you go through your ruby call stack and you call one method calls another calls another each one of
00:36:25.040 those methods belongs to some object some instance of some class and so the way ruby implements that is in this
00:36:31.760 control frame structure and there's one of these for each level in your ruby call stack there's a self pointer that
00:36:37.119 points to which instance uh which object instance is currently the cell the current value
00:36:43.119 of self um the r object um structure itself has some other things in it but the most
00:36:48.400 important two values are there's a class with a k that's the class pointer that we'll point out to indicate which class
00:36:55.440 this object is instance of and then there's another thing called iv ptr or the instance variable pointer and
00:37:01.359 this points to an array of the instance variables for this object so in this case we're going to have at
00:37:06.720 str in that in that array okay so back to our example now we know something about how objects work
00:37:12.079 internally and i've created an object obj equals quote.new and now i call eval
00:37:17.680 and what i want to have happen and this of course this will work you can try this if you really want to but what i want to do here is allow the compiler
00:37:25.200 when it compiles that string to compile it in the context of the obj object
00:37:31.520 and so the way you can do that in with the eval method is you can pass in a second parameter which is that binding
00:37:38.640 and remember we got that so we're saying obj.getbinding so backing up to my class at the bottom
00:37:45.200 we had that method that we saw earlier get binding and that just calls this special this mysterious keyword binding
00:37:52.800 and what does that do and what happens at the moment i call binding and what is a binding anyway
00:37:58.160 so well we can see what it's intended to do it's going to tell eval when you compile that code do it in the context
00:38:06.480 of this binding and that binding can be anywhere in your application and so how does that work what happens
00:38:12.400 inside of ruby when i say binding well let's take a look
00:38:17.680 so this is this sort of a similar picture to what we had earlier we have on the top we have the stack on the
00:38:22.880 bottom we have the heap and on the top we have the control frame structure now i'm going to point out there's the self pointer there along
00:38:29.200 with the dfp and all the same things will happen with the stack it'll copy the stack out into
00:38:34.720 the heap and it'll set the dfp pointer to the heap copy i'm not showing that in this picture just to keep things simple
00:38:41.599 but when you call binding ruby will also call it'll also create a new structure called rb underscore binding on the
00:38:48.880 lower right and in that structure it'll save the file name and the line number where you call the binding keyword in case you
00:38:55.599 want to you know debug something you can or if you print out the binding it'll tell you but it also creates this thing
00:39:01.200 called env which points to the same environment structure we saw earlier with the lambda keyword
00:39:07.599 except this time what i'm actually showing inside that environment a block structure so it
00:39:12.640 turns out there's another block structure in the environment and i'm also showing an additional value
00:39:18.000 here inside the block called self so when you call binding it copies not only
00:39:23.520 the dfp from that control frame into the new block it copies also that self pointer and the self pointer will point
00:39:30.240 over to the r object the to the current value itself and that's how when i you know compile
00:39:36.720 that code using eval it's able to access all the way down on the lower left the at str variable
00:39:43.520 so what we've seen here is bindings lambdas procs and blocks are all really the same thing they're all
00:39:49.839 closures and they're all different ways of looking at the same thing and i find you know i think this is all
00:39:55.520 fascinating okay so and then just reviewing you know going back to my original question what
00:40:01.280 is a block now we've seen there's uh three things in this block structure rb block structure we have first the iseq
00:40:07.839 the instruction sequence that points to the function we have the dfp the dynamic frame pointer that
00:40:13.520 points to the referencing environment and now we have this third thing which is the cell pointer that points to our
00:40:18.800 object structure which is the current value itself so this is really the best definition of
00:40:24.960 you know what is a block and it's also the way uh the best it's the way that ruby has implemented closures
00:40:31.440 so in ruby the cl closure in ruby is a function combined with its referencing
00:40:36.560 environment and i guess what you could call the object environment so the con the objects context or the context of uh
00:40:43.599 in which you're calling that block so that's it for today i'll let you guys go at the end of the day if you like
00:40:49.760 this stuff and you're into ruby internals you want to learn more you know buy my book ruby under a microscope
00:40:55.359 and then you'll find it online on my website thanks a lot
00:41:31.440 you
Explore all talks recorded at RubyConf 2012
+45