00:00:17.240
this talk is about Nikita ostensibly which is uh something that I named after
00:00:22.760
something we have in rinus called the agent interface so Nikita I thought that went well together but um
00:00:31.519
should have reversed those slides what what I'd really like to talk
00:00:37.480
about is why I have never ever ever heard anyone say this and I have never tired well I have tired I've never
00:00:43.920
stopped hearing about how small talk has these fantastic tools and how Java has these Fantastic Tools so I'm kind of a
00:00:50.399
big big picture person I want to talk a little bit about a forest instead of some trees we have lots of trees that
00:00:56.239
we'll get to in a little bit little bit um but I want to talk about some ideas because I I want I want to know
00:01:03.239
why nobody is quoted as saying that Ruby has really good
00:01:08.680
tools usually what we see and I apologize these slides are totally up I don't I don't know what the show off does but that did not look
00:01:15.479
like that like two minutes ago so bear with me a lot of what we talk about with Ruby
00:01:21.799
is something like this Ruby is a beautiful language it's my favorite language I work really really hard on rubinus and Ruby spec because I love
00:01:27.640
Ruby and I want it to be the language that people can can use we talk a lot about how developers
00:01:33.840
feel when we use Ruby we like Ruby Ruby makes it easier for us it makes us
00:01:39.560
happy and when we look at something like rails rails is a terrific framework whatever you may think about how rails
00:01:46.320
goes in this direction or that direction at at one particular time the fact is if you look on the internet almost every
00:01:52.719
single web framework out there either tries to differentiate itself from rails or it tries to tell you that it's
00:01:58.920
exactly like rails well if it's exactly like rails then why would you use the thing that's a
00:02:04.159
knockoff of the real thing right so rails has really redefined web development and it's defined the um a
00:02:11.520
huge domain that Ruby has used in right Ruby at this point in time is no
00:02:20.599
longer just the language that uh people programing because it makes them happy
00:02:25.640
uh and it's not just uh language for web application development it's actually now pretty serious business and when I
00:02:31.200
say serious I mean like billions of dollars serious right so billions of dollars is a lot of people's salaries
00:02:36.400
it's a lot of families wellbeing it's a lot of businesses um success it's uh
00:02:42.440
it's an entire ecosystem right so who here is paid to write
00:02:49.440
Ruby yeah that's almost every single one of you right four five years ago that
00:02:55.239
would have been much less six seven years ago it would have been a couple people right so huge huge transition
00:03:02.959
companies have invested in Ruby and they have bet on Ruby um I'm very very happy
00:03:08.799
to work for engine Ard engine has very generously sponsored a lot of work for Ruby they sponsor the rubinia team they
00:03:15.360
sponsor the J Ruby team not just two rubies they sponsor still like maintenance of 186 so that's three
00:03:22.080
rubies they sponsored a ton of development on Rails uh they've sponsored development on
00:03:27.280
fog they've really done a lot to Foster um Ruby as a community and the
00:03:34.280
success of Ruby um they just recently opened an office in Portland where I've been for like three and a half years
00:03:39.480
it's really beautiful you're welcome to come and we are hiring however if you look around here
00:03:46.959
and you know and you've been in the Ruby Community for a few years there are people that aren't here today and that's
00:03:52.519
distressing to me there are people that are leaving Ruby and they're leaving Ruby for other languages other systems a
00:04:01.239
lot of people are really excited about node a lot of people are really excited about closure and Scala and these are
00:04:06.319
all very interesting and fantastic systems they offer almost nothing I
00:04:11.519
think in this domain that Ruby doesn't recently this polyglot word
00:04:19.239
right has has sort of become in in my perception sort of the new liberal right it's like you want to be nice to
00:04:25.600
everybody so if you like that language over there and I like this language over here we're all we're all cool we're all friends it's all good and who am I to
00:04:32.199
tell you that your you know religion is not as good as my religion that sort of
00:04:38.479
thing if someone was running for office and they basically made this argument like hey all the candidates are good but
00:04:54.720
person how's that all right so
00:05:00.440
not only uh you know do we have this these these um other languages that are pulling intellectual um sort of Mind
00:05:10.160
share from Ruby but we also have this sort of confusion often that um the way
00:05:16.759
Ruby has been implemented is confounded with the language itself right so MRI is the reference implementation but it's
00:05:23.280
not actually Ruby it's one it's one implementation of Ruby and so when I
00:05:29.280
when I deal with these sort of things I get really frustrated because Ruby has some fantastic stuff and if people are
00:05:34.960
leaving Ruby for reasons that are not at all related to the language but something related to the implementation
00:05:40.560
that's really frustrating um and damn it you can't see this so basically she's saying this was
00:05:47.160
a lot of fun to write but it's really slow right so she's disappointed right
00:05:52.560
it's like she she basically sort of bet on this thing when someone's disappointed then that that emotion they
00:05:59.280
they they need to sort of work that out and and one of the ways that you kind of work that out potentially is by blaming
00:06:04.639
and so you're like yeah Ruby BL right so then we get to this this idea you know I
00:06:09.840
mean business is not easy how many how many startups fail nine out of 10 something like that oops and business is
00:06:17.039
really about customers right if you don't have any customers you don't have a business and businesses are constantly
00:06:24.120
balancing that decision between fear that someone is going to steal all their customers and sort of gried of oh my God
00:06:30.280
if I had all the customers in the world I'd make this much revenue right so a lot of companies probably aren't
00:06:36.800
choosing rails in this sort of middle of the bell curve where it started out Innovative companies were like hey I can
00:06:42.599
deliver faster cool we're going to be Innovative companies in the middle of that are doing it more along the lines
00:06:48.440
of oh those guys are going to get to Market before I can what can we do use rails okay we'll get there right so
00:06:54.319
that's uh that's a that's a very different decision because a fear-based decision is something that can be made very irrational
00:07:00.120
right versus a decision where somebody's like I'm really excited about this let's do it finally developers are not paid to be
00:07:08.479
happy they are paid to make customers happy right and if the business needs
00:07:15.360
right for the customer happiness and the developer happiness Collide
00:07:20.680
um guess who wins I mean just ask some Java developers that struggled through
00:07:25.960
all that so what I want to do actually now is I want to play if if I can in something for you guys that I think
00:07:32.599
is is critically important for us to sort of see and I'm
00:07:37.919
wondering if I'm going to be able to do this do I have to have my mic on the
00:07:45.080
computer for this to pick up sound person we'll find
00:07:58.440
out Services Group of Twitter so if you think of all of Twitter and try to decompose its business logic we have
00:08:05.639
users we have tweets and we have timelines that serve comprises the entire app U my group runs all the
00:08:11.759
scalable infrastructure around those three things so we build all the architecture needed to take a tweet in
00:08:18.080
fan a tweet out we have a couple thousand Tweets in a second come in we need to Fan it out to a couple hundred,
00:08:23.960
people every single time one of those come in those might be triggered by web delivery SMS delivery uh emails might go
00:08:30.680
out we might do Apple push notifications there's a whole lot of things that go on within Twitter and my group runs all
00:08:36.719
that infrastructure um so what I want to talk about is is the story of like how we're
00:08:42.640
moving from rubian rails to the jvm so this is this is jack's first sketch of
00:08:48.880
of Twitter uh somewhere around 2006 we decided to build a website so everyone knows what websites are they're a
00:08:55.360
collection of documents usually served from a directory structure they're either statically generated or
00:09:00.560
dynamically generated um they we have the standard uh rest like commands on
00:09:06.560
top of them you're either to do a get a put or a post so that was Twitter in about 2006 and until recently we started
00:09:13.640
thinking twwiter as Twitter more about service so people are generating events
00:09:19.120
in real time all around the world and we need to percolate those events through our system and act upon them in real
00:09:25.800
time so that means not just a couple thousand requests a second to get on Baseline but handle these big huge
00:09:31.920
spikes that come in the name of our game is real Time That's the hardest thing that Twitter has to do and accomplish
00:09:38.000
it's not an option for us to deliver a tweet a couple of minutes later like we really want to get that tweet out in a
00:09:43.480
couple of seconds if not faster so along with an evolution of our service has
00:09:48.800
come an evolution of our technology so uh we are probably the largest rubian rails um website on the planet right now
00:09:57.880
all our majority of our front and traffic is still served from rails like we have rails hitting a bunch of backend
00:10:03.519
Services uh composing all the responses back and then returning them up up into the cloud but recently we've been
00:10:11.480
thinking a lot more about moving and we have been moving a lot of our technology over to the JPM in fact this started
00:10:17.160
around 2008 so 2008 we moved our queuing system a system we called Starling at
00:10:22.440
the time to kestral so Starling was and Ruby krol and Scala around 2008 we acquired a company called samai which
00:10:29.440
handles search. twitter.com all that infrastructure is on Java and as we move along and as we continue as my team
00:10:36.360
continues to decomposed a main Twitter application we're rebuilding all those pieces of infrastructure on the jvm and
00:10:43.079
I'm going to try to go through a bunch of the reasons why we're doing that so I have a slew of problems and that's
00:10:50.160
interesting formatting I have a slew of problems uh on my team first off we have
00:10:55.639
a huge number of concurrent connections coming into coming into Twitter like on an off day we're handling a hundred
00:11:02.680
couple hundred thousand requests a second to our infrastructure we're pumping out a lot of Io if any of you
00:11:08.720
has ever played with our fire hose again on an off day you're talking about tens of megabits that are coming out of our
00:11:14.360
fire hose at any given time all those being pushed out as quickly as we possibly can and we have very few
00:11:19.880
persistent objects so that's basically the definition of a server workload so
00:11:25.839
the reason why I have a bird on the screen um mainly because every single project that Twitter is codeen named
00:11:31.760
after a bird at some point um is we actually spent a long time trying to get Ruby to do what we needed to do we we
00:11:39.120
patched the MRI we actually built a whole new Java uh garbage collection engine to run within the runtime to make
00:11:45.959
it faster and a lot more efficient for us to use to actually tackle these huge server workloads that we're getting and
00:11:52.560
we're going to continue to do this type of work like we're continuing to push Ruby as far as we possibly can because
00:11:58.440
our entire infrastructure has been built upon it but we're also looking for opportunities to go and join bigger
00:12:04.920
communities find more developers use better tools and just evolve in a
00:12:10.160
direction that the world is going in in a way that we know will work well so for that we're relying on the
00:12:16.880
jvm um so my needs we already talked about server workloads but as I'm going
00:12:22.920
through in decomposing my application we actually also want flexibility in language so we're starting to pull
00:12:28.720
things out of what we call our monolith so our statuses are coming out our timelines are coming out our users are
00:12:34.880
coming out to their own backend services so we want to have flexibility in all
00:12:39.920
the different axes we possibly can and language is one of them um so as a lot
00:12:45.279
of people know we run Scala for the majority of our runtime backend systems so um all our storage systems have a a
00:12:53.399
Scala layer on top of them that sort of handles charting I'll talk about that in a bit um but we also have a lot of java
00:13:01.000
within within Twitter itself in fact Scola and Java are two first class languages at Twitter we support them
00:13:07.279
both equally now a lot of our runtime code critical path code is in Scala
00:13:12.560
however a lot of our relevance engines our search engine our advertising systems and a bunch of other things are
00:13:18.360
being run written in pure Java right now and the new kit on the Block within our
00:13:24.000
infrastructure is run enclosure so we've been experimenting and toying with running closure more and more production
00:13:29.880
the beauty about all this is that my Ops Team gives me no grief because we simply
00:13:34.959
deliver them jar files to deploy and manage as long as they give them similar ways to Monitor and monitor and manage
00:13:42.360
all that code so what I mean by that is similar ways to look at garbage collection logs similar ways to get
00:13:47.519
stats exported from all our runtime Services similar ways to deploy them our op team really they have an opinion but
00:13:53.920
they don't care as much as to what the engineering team may be up to so closure came around with our back type
00:13:59.560
recently and it's something we're going to start investing in more and more so the last thing that I want to
00:14:07.279
touch upon and what's one of the biggest reasons that we're moving to Java is it's a real concurrency model I know
00:14:14.199
that's that's a a kind of crazy thing to say to a bunch of java people in a room but we I'm coming from a world where we
00:14:21.120
have not the best threading model on the on MRI inside the Ruby VM so it means
00:14:26.920
that we can't really experiment in the way ways we want to and some real performance numbers uh right now a
00:14:33.399
single request to a Twitter API takes tens of milliseconds we run in a request per worker model so one request comes in
00:14:40.800
is given to a worker to deal with that artificially gives us a low QPS that we can handle queries per second that we
00:14:46.959
can handle per box um and it and and it just hampers our growth our machine
00:14:52.839
density is not as good as we think it could be so we've been experimenting and we've built a system we call finagle
00:14:59.800
finagle is our generic RPC mechanism we've released open source for anyone to use to handle all the backend traffic
00:15:06.519
our service can handle as well as new incoming front end traffic we've abstracted way how to do asynchronous
00:15:13.639
event driven networking so we can just write beautiful and elegant code in either Scola or Java or basically any
00:15:20.639
other Java jvm language we choose and one of these final servers can pump out
00:15:25.959
tens of thousands of queries a second doing the exact same thing that our Ruby machines can do but and I'll talk about
00:15:32.519
this a bit later we've changed our efficiency our efficiency metrics because we've done something like this
00:15:38.240
so fundamentally Twitter is a real time and event- driven problem we're trying to
00:15:45.199
build a system that'll react to the world and be able to scale as the world scales as people talk about events we
00:15:51.680
want to handle them in real time as one of the people who's working on the core infrastructure and manages a group of
00:15:58.199
around 35 people running writing the core infrastructure we want to fundamentally rethink the way that we
00:16:03.720
work on Twitter so Ruby doesn't provide us the ability to do scatter gather algorithms for example so I want to be
00:16:10.480
able to have the ultimate flexibility in constructing timelines for users so as a request comes in I want to hit hundreds
00:16:17.199
of backend machines compose responses back and return in 50 milliseconds and
00:16:22.800
that's something I can't do in Ruby right now in Ruby like you can see some of our code we literally have to go a
00:16:28.120
hit every single backend server in seral and to figure out what to do at the end with careful timeouts along the way I
00:16:34.160
just want to hit them all one shot and pull the results back so fundamentally the jvm gives us something that Ruby
00:16:41.440
can't so lastly the things I trade in inside my infrastructure is I trade in
00:16:47.720
Ram and CPU I trade in bandwidth and I trade in machine density so RAM and CPU
00:16:52.920
is obvious we're building a bunch of systems that allow us to operate more as our clouds inside Twitter so we can just
00:16:58.519
simply launch processes into the data center and say I need this much RAM this much CPU
00:17:05.720
okay you can watch the entire thing the uh the URL is in my slide
00:17:14.280
um essentially one
00:17:27.840
sec okay so he said a couple of interesting
00:17:33.919
things right I swear I heard him say beautiful and elegant code in Java so I don't know if we want to take the rest
00:17:39.320
of what he said seriously or not however a couple of things he did say is that Ruby and he he kind of distinguished
00:17:45.160
between MRI and Ruby at one point but most of it is Ruby Ruby can't do scatter gather because it has no concurrency
00:17:51.200
Ruby can't do good garbage collection Ruby is not a suitable language for a near real time complex
00:17:57.840
system that has massive server workload I don't know like there's tons of Ruby on
00:18:03.960
desktops but I would guarantee that there's well I would guarantee I would estimate that there are probably several times as many instances of Ruby running
00:18:10.600
on servers around the world than they are on the people's laptops that are sitting in places and doing development
00:18:16.360
so uh two blaring black eyes for Ruby if you read if you see the entire thing he basically says also that he can hire
00:18:23.640
better developers if he um goes to Java so he hits on these these four things
00:18:29.919
right garbage collection concurrency performance and tools right when you deploy this stuff you want to have good
00:18:36.280
tools so that you can actually see what's going on in this VM yes I am worried because while
00:18:43.480
Twitter is one company Twitter is a company that a lot of people looked at when they first started using Ruby on
00:18:48.520
Rails they're a company that other people are looking at seriously and if you um they're visible company he gave
00:18:56.000
that talk as a keynote at oscon oscon is a big and very broad conference there are people from all over the place that
00:19:02.280
attend that um the companies that are making the decisions based on fear are going to be
00:19:08.840
companies that say something like oh my god Ruby can't scale it might get us there but maybe we should use something
00:19:14.200
else because we're going to need to scale and of course they don't need to scale right it's one of those premature optimization things but decisions based
00:19:20.559
on fear are not always rational so yes I'm worried I'm also worried
00:19:26.520
because uh I live in Portland and W Cunningham and alen Worf's Brock two you know like way up here pioneers of the of
00:19:34.440
the previous generation we we sat down with them recently and and Allan pointed out that small talk went from a language
00:19:40.080
that was you know widely used and only like maybe IBM ignoring it to to a language that was near irrelevant within
00:19:46.080
the span of about 18 months right so um
00:19:51.679
complex systems have inflection points you can be going like this and suddenly go like this you don't just go like this
00:19:58.120
that sort of thing so yes I am I am very worried
00:20:06.640
and one one thing that's fairly true is that we're all still kind of kids inside right that's why Ruby makes us excited
00:20:12.960
when we're having fun that sort of thing um I I don't want to see the sad day when people are basically told no you
00:20:18.880
have to use a language that many many people have said this language is not for me I don't want to use it but
00:20:24.799
someone else is making that decision not you because you can't deliver with the language that makes you happy
00:20:32.120
so um slides so if tools are the answer right what kind of questions are
00:20:40.159
we actually asking and I think the number one question is this how do I understand this code I need insight into
00:20:46.440
my code Ruby is a dynamic programming language that means that a program that you write in Ruby is kind of like a
00:20:52.240
class in Ruby when you actually want to use an instance of a class you have to instantiate it when you want to when you
00:20:57.960
want to run an instance of your program there's actually real objects with real types like this might be an array or it
00:21:03.919
might be a string flowing through your program so your your running program is kind of an instance of this dynamically
00:21:09.760
type program that you expressed in Ruby code fear of the unknown is one of the
00:21:16.760
most important motivators I think for people uh sort of running to something
00:21:22.159
like static typing thinking that it's going to give them some greater security however uh the type systems in complex
00:21:30.640
statically typed languages can be T and complete as well which means that you can make the compiler not terminate not
00:21:36.559
just your program right so there there's nothing in static typing that necessarily gets you some greater
00:21:42.240
security but it might suggest some security and so people are willing to sort of take that because they're not
00:21:47.559
examining their their basically Their Fear of the unknown so the key for me
00:21:52.679
for tools is how do we give people insight into their code right
00:21:59.039
thinking about like inside in code I want to look at a couple of things and this is intentionally reversed usually
00:22:05.640
you should have a problem and figure out the solution right what I'm seeing happening is that we have a bunch of
00:22:10.720
solutions and then we go looking for the problems so these are the the four things that I've just sort of picked out
00:22:16.720
that I've have you know um the um topical this like how much they're being
00:22:22.200
discussed have kind of gone up and down but um there's been discussions about
00:22:28.520
optional typing we need to add optional typing to Ruby to make it faster this argument is false however it doesn't
00:22:34.960
stop people from proposing it over and over again so Ruby is about Behavior right it's about the things that your
00:22:41.400
your um it's it's how the objects in your program act right you send them a
00:22:47.039
message so sort of the idea of you are what you do right we call this duct
00:22:52.480
typing which I'm not a fan of that word but it's it communicates the idea but the idea is behavior you are what you do
00:22:57.840
classes on the the other hand organized code they are a tool that we use intellectually to manage complexity
00:23:04.240
right the most important thing that classes do is they provide the encapsulation that's necessary so that
00:23:11.039
an object can manage its state without something else getting in the way so encapsulation is really super important
00:23:17.520
and classes provide that much more than than anything else there's sort of I
00:23:23.600
think sort of taken this from Josh suess's words but basically there's sort of these three primary ideas in in
00:23:29.360
object orientation encapsulation polymorphism which is means that you can name the method the same thing and
00:23:36.159
inheritance but inheritance doesn't have anything to do with classes prototype systems use inheritance just as well as
00:23:42.600
class-based systems so I want to emphasize that classes are more a tool for us to think about
00:23:47.919
code there's something that concerns me even more than optional typing and this is hidden typing I I'm going to call
00:23:54.480
this hidden typing because I don't know what else to call it it's basically this idea where we start imposing typing like
00:24:00.200
requirements on Ruby code but they're not really visible and I I couldn't find exactly the right word to use for
00:24:06.080
concealed but it's essentially that Ruby can't participate so these requirements are
00:24:11.360
not in Ruby and they may be either sort of undefined in the Ruby ISO standard
00:24:16.919
right it doesn't actually say whether it's going to impose this requirement or they're just not documented and because
00:24:22.279
they're hidden from Ruby Ruby can't participate so we have coercion methods like integer capital I integer Bing or
00:24:28.159
like float the sort of thing Ruby can participate in those sort of coercion protocols right they can't participate
00:24:34.320
when we have something like this BL BL BL blah oh if RB object is kind of
00:24:40.080
suddenly anything that wants to participate in something that is sort of float like has to be a subass of
00:24:48.399
numeric hidden in the C code somewhere if you implement 2f which this will
00:24:54.279
accept otherwise you don't get to participate unless you subass
00:25:00.640
numeric if this was Ruby code that we were looking at instead many people would say hey there's a smell in this
00:25:06.200
code I see you're like hardcoding something to numeric here why are you doing that the real big problem is now that
00:25:13.279
now Ruby the language is dictating the structure of our programs instead of
00:25:18.440
just giving us the semantic that we are able to use to describe our computation
00:25:24.399
right dictating that structure um um Jim W probably has some
00:25:30.120
really fancy word for it but uh I'll just call it coupling for one uh brittleness right it's also
00:25:39.039
unnecessary and I'll go one step further and say it's wrong Ruby should not be this way so the big thing about types is
00:25:46.919
types tell you what you cannot do types don't tell you what you can do they tell you what you cannot do the compiler says
00:25:52.000
you can't call that method on that object because it's not the right type the compiler doesn't give you a long
00:25:57.039
list of all the things you can do with that object well by the way if you don't like that method you can do this one over here it doesn't do those sort of
00:26:02.279
things it says you can't do that right and like walls people will either try to defeat them go over them bash their
00:26:09.880
heads into them you know that sort of thing but they will create
00:26:15.080
unnecessary structure that increases the complexity of the system complexity is what is the most
00:26:23.559
important thing that we have to deal with and often complexity will lead to pain so I don't want pain in Ruby and I
00:26:30.480
don't want optional or hidden typing in Ruby refinements are man these damn slides they one of the leading proposals
00:26:38.120
for 2.0 they've been talked about by Ms and other people and you know like talks about what's coming up in Ruby and um I
00:26:45.279
think that they are the worst possible idea that I've seen get a great deal of attention after optional typing they're
00:26:52.320
primary purpose is to hide incompatible changes from you and from your
00:26:57.399
co-workers and every everybody that sees the code base after you and they take something that is so important naming
00:27:04.159
which is one of the things that we in like with our minds our ability to think
00:27:09.360
naming is one of the things that enables us to handle complexity more so than
00:27:14.600
almost anything else and it basically says when you're looking at this code
00:27:20.080
and you see the name of that method it could do something totally different than what you thought it could do because someone doesn't want to go
00:27:25.720
through integration testing with their libraries even if we have refinements we still have to
00:27:31.559
have testing to make sure that the refinement stuff that they did is still compatible with something else that you
00:27:37.399
do over here when you start putting together these libraries so hasn't taken away the need for that we still need robust integration testing and it does
00:27:44.320
not help understand your whole program it instead adds more complexity
00:27:51.320
right more than anything refinements suggest to me what if programming what
00:27:57.480
if someone changes that method and I depend on it okay you know what if they change that method and it breaks your
00:28:02.600
code you either don't use that Library you fix the library or you fix your code because maybe your code is actually
00:28:08.120
broken so what if programming is really based on fear right what is going to happen if something happens uh or
00:28:15.120
arrogance I know better than you about how this should work and I'm going to change 2s to return you know the
00:28:21.200
something else because 2s why wouldn't you know return a string that sort of thing so the other the other thing is that ref
00:28:28.360
refinements I think are an attempt to deal with this problem and this is this is an honest
00:28:33.880
problem the idea that Ruby itself in the standard library has to integrate horizontally right because multiple
00:28:41.399
applications are going to be built vertically on it however any one of those particular vertical applications
00:28:46.720
does not need to integrate you can go do things that have would break completely that application over there because
00:28:52.360
their code is never going to live together however if you're going to make a library right it can't break
00:28:59.200
Ruby so that's the the idea of the horizontal versus vertical integration one more thing DW um I think that it
00:29:07.320
could be named that here's my problem with DW so there's a big Twitter thing you should turn on DW now please do not
00:29:13.640
misquote me and say that I'm encouraging you not to write good code because I am encouraging you to write good code
00:29:19.279
however I don't think DW is the way to good code it's hidden in the parer there are no Ruby objects that you get to mess
00:29:25.399
with it's fairly ad hoc there is no programmatic control over whatsoever you
00:29:30.559
cannot tell it useless use of equal in this thing is okay because I you know
00:29:35.679
wrote rspec and it's actually going to do something when you call that method over there um and it's really doing no
00:29:41.679
semantic analysis it's a couple of edge cases that it basically says oh I think you've got a problem
00:29:47.240
here okay so to reorient right we
00:29:53.720
watched a video by the guy who's in charge of development at Twitter one of the biggest US users of Ruby and and
00:29:59.919
rails in the world and he told us that there are four critical issues that Ruby must address garbage collection
00:30:05.919
concurrency performance and tools I didn't hear him mention refinements Ruby is basically bringing
00:30:13.240
in KN to a gunfight the world of technology is not going to sit around while we're happy about this wonderful
00:30:19.960
language that we get to programming so what are we doing about this here's what
00:30:26.519
reinius is doing we have generational garbage collection true concurrency the
00:30:33.440
2.0 work has been merged into Master whether you're running 18 mode or the one nine work that we're working on
00:30:39.519
right now you have true thread concurrency on your Ruby threads has a Jus in time compiler for
00:30:46.840
performance and we want to build tools we're building the foundations for
00:30:52.559
tools so here's a little status update since Ruby conf last year concurrent threads are in one nine features are
00:30:59.320
coming along we're running 18 and one n on CI you can check Travis and see how it's going we're passing
00:31:05.080
36,000 LW some number of of one n specs I'm going to work on encoding as soon as I'm done with this talk Windows support
00:31:11.440
is coming the VM completely Builds on Windows it doesn't link the C extensions yet that's coming and Evan has been
00:31:17.039
working really hard on just in time compiler improvements which are going to
00:31:22.080
take Ruby performance to the next level the rinus garbage collector real quick
00:31:27.120
it is a precise garbage collector which simply means that every single place where an object exists in the Heap the
00:31:33.799
re the reinus garbage collector knows that that's an object reference it does not scan Stacks looking for stuff it
00:31:39.840
walks up a sort of a parallel variable scope stack so it knows exactly where to find all those objects um I think
00:31:46.919
Precision is something that's a nice feature in garbage collections I mean if we want to think of antonyms for precise
00:31:53.000
right would we want one of those garbage collectors no it's generational the Young Generation copy for compaction and
00:31:58.919
the the the key most important thing is that the garbage is never seen garbage collection is a misnomer uh for copying
00:32:06.880
compaction you basically start with what object you know is alive and all the things that object references you copy
00:32:12.080
over here you never see the garbage so the fewer objects that live the actual
00:32:17.720
shorter the the garbage collection time is and we see one or two millisecond uh
00:32:22.960
garbage collections in the Young Generation if stuff lives long enough it goes into the mature generation which takes more time time but it's a great
00:32:30.399
way to reduce the amount of work that's actually done you're looking at small sections of the Heap where objects excuse me are most likely to die uh the
00:32:37.799
generational garbage collection is based on this thing called the generational hypothesis which is that objects live
00:32:43.559
fast and die young concurrency fully concurrent Ruby threads I can repeat that 700 more times
00:32:51.519
if you think that threading is not the way the world is going you should probably go talk to somebody who's been
00:32:57.880
been writing um or or building hardware for the past decade because they're going to many many cores and they're
00:33:04.519
optimizing those cores for running threads concurrent versus parallel what we' what we've built is
00:33:11.559
the ability for Ruby threads to run concurrently if your Hardware supports it they will run in parallel concurrent
00:33:17.159
is more the idea that at any sort of Step things can make progress but we
00:33:22.279
used to have concurrency on single processor single core uh things where you got your time slice and someone else
00:33:28.159
got there so just to to sort of keep that idea clear what you can now do is
00:33:33.399
run your your Ruby threads in parallel on your multicore things uh if you
00:33:39.799
really don't like threads there are there's a fantastic formalism called actors that have been in rinas for a
00:33:45.480
long ass time uh now people are saying I'm going to Scola because they have actors I'm like what I'm going to
00:33:51.080
closure because they act okay so we have actors uh and then the Justin Time compiler is about making Ruby order of
00:33:59.200
magnitudes faster not like 20% or 10% faster so order of magnitude like 2x
00:34:05.039
faster or 3x faster or 5x faster that's the the point of the JP order of magnitude faster okay
00:34:12.960
tools rubinius implements Ruby differently and so everything I've set up to this point like um is uh is
00:34:21.879
basically to try to give you the reason why when I say this it should make a
00:34:29.879
difference right um reinius uses
00:34:35.040
Ruby and uh that's a confusing thing unfortunately like a long time ago said
00:34:41.240
R Ruby and Ruby that's kind of the goal of rubinius and that's sort of thing it's been very confusing to people so what I want I want to focus on instead
00:34:47.040
is that rubinius uses Ruby to make a consistent system if you uh write a c
00:34:52.520
program that says 1 + one uh C doesn't know anything about adding one plus one
00:34:59.720
actually and neither does Ruby and it never will um C compiles to the CPU
00:35:05.359
instruction for loading one into a register and loading one into a register and executing the ad instruction and
00:35:11.000
that's the exact same thing that Ruby has to do as well uh Ruby though uses a
00:35:16.680
method send for Plus instead right so rubinius is not written in Ruby if you
00:35:23.680
want if you if by in Ruby you mean I can go into the VM directory and read all this Ruby Cod no the VM is in C++ the
00:35:29.560
garbage collector is in C++ the jit compiler is in C++ although we're experimenting with some pure Ruby jit
00:35:35.280
stuff but the point of in Ruby or that rub Bas uses Ruby is that the entire core Library all the things that you are
00:35:41.800
doing in the Ruby semantics comes from Ruby code when you go um and use hash in
00:35:48.760
your program that's rub real Ruby code and when the justing time compiler is looking at what's getting hot it can
00:35:55.040
actually go get that Ruby code put it in your Ruby code code optimize it all together and um and make it faster but
00:36:01.960
it's also the the idea of the consistency so instead of that RB object type whatever has to be numeric right
00:36:09.119
which is um sort of calling this way with a c function and doing some other stuff we are dispatching through Ruby
00:36:16.079
code so you can actually subclass array or hash and see uh your methods called
00:36:21.760
instead of something called out in code where you can't fully participate that's the idea of rubinia so it consists
00:36:27.800
system um something like 1 + one uh complex Expressions it's sort of like
00:36:33.800
Ruby or C or anything else it's sort of a tree way on that leaves there are fundamental primitive things that have
00:36:39.720
to be done right all the way when you get to the end like adding two things together can't be done in Ruby so it's not in
00:36:45.839
Ruby uh how does reinius use Ruby well there's this word it's a pretty cool
00:36:51.720
word uh homo means same and icon means symbol and homo ionic means that I can
00:36:59.720
take some Ruby code and I call 2 A on it and I get back Ruby objects so I can
00:37:05.560
represent so the Ruby code is a string and these objects are also in Ruby uh so
00:37:12.599
I can represent Ruby as either text or as a data structure I can uh do other things like
00:37:19.960
I can look at that a little differently I can look at it as s Expressions if I want um I don't know why you would
00:37:25.280
except they're very compact it's nice but I I wouldn't recommend trying to process them do anything useful uh and
00:37:31.040
you can just call it directly so we support that as well but a 2 a is exactly what the compiler does when you load your Ruby uh
00:37:38.079
program it runs essentially to a so that's available so this is this is the idea of the consistency in Ruby there's
00:37:45.359
this idea of everything being an object well let's look at some Ruby code so everything's an object okay so we're
00:37:52.240
going to define a class we're going to define a module right when the when we Define the class A = 1 is something runs
00:38:00.680
that makes that happen when you define a module something runs that makes you know b equal 2 um when we Define a
00:38:06.960
method okay that's what we're used usually used to thinking about as where you know code execution happens when we
00:38:14.319
uh send a block along with the method send then that block has some execution
00:38:20.359
stuff too um when we write a ruby script we might
00:38:27.040
do something right there right might not be defining a class or anything we just say puts
00:38:32.359
hello if you go into reinius and you and in IRB you do this you get an actual
00:38:38.319
object back from Death right that object is a compiled method if
00:38:44.680
you were to run a script through uh our compile command you get a
00:38:53.079
compiled method so a method body a script body class a module a block all
00:39:00.200
these things are just compiled methods so here we're GNA open a class and we're
00:39:05.839
going to define a method on it touch method right this is just all all ruby
00:39:10.880
objects that we're dealing with here and if we um here's the actual method for the
00:39:18.280
class itself it's going to it's going that's where that's where def m is actually going to get created and we
00:39:24.560
have a compiled method everything in Ruby that can do anything right is this idea of execution
00:39:32.599
and what we represent that at as in rubinius is a compiled method object so the the summary of all this is
00:39:40.800
that Ruby rubinius uses Ruby to build this consistent system everything in Ruby is an object everything that can do
00:39:46.960
anything in rubinius is a compiled method and there are some really cool stuff that um that we're going to be
00:39:52.640
able to do with that so back to this idea right so next year I'd like to say 2012 Ruby has the best tools right here
00:39:59.880
is my idea so I'm calling this Nikita after the rubinius agent it's my idea of a tool right go make your own idea of a
00:40:06.280
tool now I want you to write tools also I don't want this to be like well there's like rails so nothing else is
00:40:12.599
appropriate either use rails if you don't no I want you to write tools using the foundations that we've built in in
00:40:18.160
rubinius but what I'm going to describe is one tool that I've been thinking about for a long time the basic idea is
00:40:23.839
that it's client server if I have reinius VMS running on the server I here
00:40:29.400
want to be able to talk to them right so essentially I need to separate out the
00:40:34.599
the functionality so there's there's going to be a client server mode and it has a completely separate guey right so
00:40:41.760
I don't want to tie rinas to some particular uh language binding I've done
00:40:49.680
desktop applications since the early 90s and there is no end of pain trying to
00:40:54.720
get Ruby to bind to some other uh some other thing or you get these really horrible inconsistencies like if you use
00:41:00.839
like a Java library for like awt I mean on OS 10 we're really really lucky use
00:41:06.040
it on some other platform where Apple didn't spend all the time to make it look good it's really crazy so we're going to use data as The Binding instead
00:41:12.319
we're not going to do any um any language binding stuff and finally the I
00:41:18.319
want it to be very modular but I'm not going to create a whole new system for modularity right gem install so it's going to be like gem install Nikita
00:41:24.800
browser and you're going to have a class browser that sort of thing so trying to keep it really simple
00:41:31.240
um we have all those compiled methods what can we do with them well here's what we can do with them if we put all
00:41:36.520
those compiled methods in a database then we start to blur the
00:41:42.079
division between your Ruby program as source code and your Ruby program as a running program right as your program
00:41:49.240
runs the reinius uh compiler the the justtin time compiler the inline caches
00:41:54.280
if you saw constant sc's talk earlier we have object where we're tracking all the types of things that we see that
00:42:01.040
information is easily extracted and can be annotating these compiled methods as
00:42:06.560
metadata right so you can actually see not just um what your say method call
00:42:13.920
what other named things your method calls but actually the values that sort of come through
00:42:19.880
right uh we can Implement a class browser you've compiled all your code we can just go through this database of
00:42:25.800
compiled methods and we can find out everything as you saw about which methods they Define uh what other
00:42:31.480
classes they're nested in all this stuff is just transparent because the the the bite code the compiled method and the
00:42:38.040
bite code it contains is an exact replica of the source code right so we
00:42:43.160
can create coverage and profiling tools as your program runs once or twice or multiple times there's just very
00:42:48.440
convenient place to store coverage information you could run your app in the test and then you could run your app
00:42:53.760
and and uh and do some real like requests to it and you could like look at that and say wow my tests are
00:42:59.280
covering like 500 times more code than this application is actually using right
00:43:04.800
and then if you talk to Michael feathers he wrote a really interesting thing about the carrying cost of code what if you're worrying about testing all this
00:43:10.640
damn code that no one is ever using right then you might be able to like spend your your time somewhere else you
00:43:16.160
can do bite code instrumentation that bite code is just a stream of things describing something you can actually get in there and uh and change stuff
00:43:22.920
around and this is how easy that would be to do so we have this kind of cool method called uh d Dynamic method and
00:43:28.119
what you can do is you can create from Pure from Pure um bite code a method and what I've done here is I've actually
00:43:34.760
I've actually made this method that'll tell you its name so instead of just
00:43:40.640
printing the name out I could instrument that by code to call some stat or tracking package and say guess what I
00:43:47.079
got called furthermore these are my these are my local variables these are the the arguments that passed in here
00:43:53.319
you can have a look at those as well and it wouldn't change anything else that happened in that method right I've done
00:43:59.079
it without like the tooling just to like represent uh a proof of concept but this
00:44:04.359
is a very very powerful thing and Java for instance has this so I think we should as well uh as transformation
00:44:10.839
sometimes a tree like uh is a really handy data structure and so using a tree once you get the the source code
00:44:18.000
into you can do all kinds of stuff if you've heard of heckle um um I think Dan
00:44:23.119
Cub has sort of already prototyped a Heckle using reinus code or either bik
00:44:28.480
code or um outs out of time I'm almost done
00:44:33.599
another thing we can do is uh better debugging um the one of the challenges
00:44:39.839
with debuggers and I mean if you talk to Tim for instance this guy right here Tim fog working on the magli he has some
00:44:46.680
really cool stuff going on with uh the debuggers right Small Talk has this great idea like you're working along and you like fail and then you like stop
00:44:52.880
there and you do some other stuff and continue right pretty neat idea um you have to be running though right so what
00:44:58.760
I would like to do is explore uh ideas for debugging that that span this blurred space that we would now make
00:45:04.920
with uh between running and and not running using that compiled method database uh finally documentation uh I
00:45:12.480
absolutely 100% Adam detest documentation in my code um I think it's
00:45:19.559
terrible and the reason for that is that one it's very difficult to translate right so you have it in English what if
00:45:25.240
you want other other things well you extract and then you translate all that and you do this sort of thing um it clutters up stuff it's never usually uh
00:45:33.839
the the primary purpose of code is not to document stuff over there so what I really really really really really want
00:45:40.440
is a fantastic system where the documentation lives over here and can be organized and structured right
00:45:46.839
structured for people to read not which is not the way that we necessarily structure the methods in a class or
00:45:53.160
something like that right and very easily connect the two so I I really really am excited about that we can do
00:45:58.920
analysis remember that- WTF flag that we looked at before so we could actually look through this bite code and see
00:46:05.200
stuff like this variable was used before it was defined this is a really really really easy thing to do and we can start
00:46:10.800
adding levels of semantic analysis to it we can look at hey you never even use this variable or I find no method like
00:46:17.839
this okay but here's a problem right Ruby is a dynamically typed language right so what happens if you do module evalve and you define something that
00:46:24.359
doesn't exist syntactically right lexically it's nowhere in the code with
00:46:29.480
a compiled method database you can say you know what I saw a method defined like this at this point in time so this
00:46:35.160
method I don't see in code I see it was defined dynamically however so it's probably cool but you might want to
00:46:41.000
check that out and then you can do all sorts of code quality analysis right uh and furthermore you can build
00:46:48.680
custom tools I I don't have all the ideas about how this stuff should go I just desperately want us to have it so
00:46:54.000
please start building it so the summary is
00:47:00.079
Ruby is at a critical point I could be wrong I could be the uh was it Pi Piper
00:47:05.160
anyway the guy who's like saying oh my God you know crying wolf maybe anyway uh I'm not trying to lead you off the cliff
00:47:11.040
what I'm and I'm not trying to cry wolf on necessarily but I I am very concerned I've been working on ruinas for five
00:47:17.079
years almost and uh and Ruby spec for as long and uh watching things evolve and
00:47:22.720
so I want to give you my best um my best warning because I I absolutely love Ruby
00:47:29.079
I work very very hard to make Ruby better um I want you to note one thing I didn't change anything about Ruby right
00:47:37.839
we started Ruby spec in rubinius to ensure that the compatibility among all
00:47:43.119
the different implementations was as close to 100% as possible so as Ruby developers we didn't have to worry about
00:47:48.880
where we're writing our Ruby and where we're running our Ruby and all kinds of stuff like that right I did not change Ruby I didn't suggest changing Ruby in
00:47:55.440
any way shape or form I simply uh want to add tools and an
00:48:00.839
implementation that answers the demands of Industry the stuff in rubinius is
00:48:06.200
available in 18 or 1 19 mode and while I think it's great that we have Ruby 1 19
00:48:12.119
and I want to see more people use it People BET businesses bet on Ruby years
00:48:17.280
ago and what they had available is 18 they have applications in when and they don't necessarily have time to move them
00:48:23.920
forward we need to respect those businesses they part of the reason that so many of us can raise our hands today
00:48:29.240
and say I am paid to write Ruby finally I don't want to be this person I don't
00:48:34.640
want anybody in this room to to be that little kid inside upset that something
00:48:39.720
that they find fantastically cool and may have changed their whole experience of programming or or whatever may have
00:48:45.880
like taken them from some other field into Ruby we have so many people in Ruby who are not just you know marching out
00:48:52.079
of computer science programs across the world that sort of thing so I don't want to be this person um
00:48:57.480
so here's my plea to you please help Ruby help you this is where you can find reinius I
00:49:05.160
apologize again for these slides man they look so good never will I use show off again uh this is where Nikita stuff
00:49:11.839
will work we will be done um there's several other like directories but you can go there and check that out uh this
00:49:18.359
is where you can find this presentation damn sh um thank you very much and uh I don't
00:49:26.079
think I have any time left but I am very happy to take questions and talk at
00:49:31.280
length about this I leave tomorrow uh just after the conference so um please
00:49:55.640
much
